Skip to content
Knowledge Hub/Article
Article

What Is MLOps? Running Machine Learning Models in Production

We explain what MLOps is, the model life cycle, maturity levels, data drift and retraining, tools such as MLflow and Kubeflow, and governance.

Oğuzhan Gerçek··9 min read
What Is MLOps? Running Machine Learning Models in Production

Short answer: MLOps (machine learning operations) is the set of practices that applies DevOps principles to machine learning, bringing model development and running the model in production into one discipline. What sets it apart from ordinary software is that behavior comes not only from code but also from data: as the data changes, the model goes stale, so it has to be monitored and retrained. The life cycle covers data preparation, training, experiment tracking, a model registry, validation, deployment and monitoring. Google describes maturity in three levels: a manual process, an automated training pipeline, and a pipeline that is itself updated through CI/CD.

What is MLOps?

According to Google Cloud's architecture guide, MLOps is an ML engineering culture and practice that aims at unifying ML system development (Dev) and ML system operation (Ops). AWS defines it as a set of practices that automate and simplify machine learning workflows and deployments, and according to Microsoft it is based on DevOps principles and practices such as continuous integration and continuous delivery. We cover the basics of machine learning in our machine learning article.

What a model does in production is inference. In Google Cloud's definition, AI inference is the process of running a trained model to make predictions on new, unseen data. Alongside real-time inference, which processes each request as it arrives, there is batch inference for large volumes that do not need an immediate response. We cover the hardware difference in our GPU article and the inference bill in our AI FinOps article.

Why is MLOps needed?

According to "Hidden Technical Debt in Machine Learning Systems", the paper D. Sculley and colleagues published at NeurIPS 2015, it is common to incur massive ongoing maintenance costs in real-world ML systems. Some of the paper's findings:

  • Model code is a small part: Because of the glue code written around general-purpose packages, a mature system might end up being at most 5% machine learning code and at least 95% glue code.
  • The CACE principle: "Changing Anything Changes Everything."
  • Data dependencies: Data dependencies cost more than code dependencies.
  • The limits of testing: In a changing world, tests are not sufficient to provide evidence that a system is working as intended.

The MLOps life cycle

Google groups the delivery of a model into eight steps: data extraction, data analysis, data preparation, model training, model evaluation, model validation, model serving and model monitoring. The main MLOps tasks are:

  • Data preparation: Relevant data is selected and integrated from various sources and prepared for the task (see our ETL article).
  • Experiment tracking: Tools such as MLflow Tracking log each run's parameters, code version, metrics and output files.
  • Model registry: A centralized model store such as the MLflow Model Registry keeps model versions, aliases and lineage showing which run produced each model.
  • Validation: According to Google, the model is confirmed to be adequate for deployment, meaning its predictive performance is better than a certain baseline.
  • Model serving: The model is deployed as a microservice with a REST API, as an embedded model on an edge or mobile device, or as part of a batch prediction system (see our API article). A new version usually goes through A/B testing or a canary deployment before it serves all traffic.

Monitoring: data drift, concept drift and retraining

According to Microsoft, unlike traditional software, the behavior of a machine learning system does not only depend on rules specified in code; it is also learned from data. Data distribution changes, training-serving skew and changes in consumer behavior can all cause a model to become stale. IBM's definitions separate two kinds of drift:

  • Data drift (covariate shift): The underlying distribution of the input data changes. Microsoft tracks it by comparing the input distribution with the training data or recent production data.
  • Concept drift: The input variables and the target variable diverge, and the model begins to give incorrect answers because the definitions it learned are no longer valid.

Microsoft recommends starting model monitoring immediately after a model is deployed to production. Google lists four triggers for retraining: a schedule, the availability of new training data, model performance degradation and significant changes in the data distributions. We cover metrics and alerting in our observability article.

MLOps maturity levels

Google describes three levels:

  1. Level 0, manual process: Every step is manual, a new model version is deployed only a couple of times per year, and model predictions are not logged.
  2. Level 1, ML pipeline automation: The training pipeline is automated, and the model is automatically trained in production using fresh data. Data is validated before training, and the model is validated before it is promoted to production.
  3. Level 2, CI/CD pipeline automation: The pipelines themselves are updated rapidly and reliably through a robust, automated CI/CD system, and the trained model is pushed to the model registry.

The concept that arrives with level 1 is continuous training (CT), which Google calls a property unique to ML systems: automatically retraining and serving the models. Microsoft's maturity model splits the same path into five levels, from level 0 (no MLOps) to level 4 (full MLOps automated operations).

MLOps, DevOps and CI/CD

MLOps extends the practices described in our DevOps article. According to Google, CI is no longer only about testing and validating code and components, but also data, data schemas and models; and CD is no longer about a single software package but a system, the ML training pipeline. AWS calls this extension "continuous X": continuous integration, delivery, training and monitoring. Version control also reaches beyond code: according to AWS, the aim is to track changes in machine learning assets so you can reproduce results. We cover pipeline design in our CI/CD article, and DevOps and IaC in our glossary; we handle this layer in our DevOps and platform engineering service.

Tools: MLflow, Kubeflow, Kubernetes and GPUs

The tools below are examples only:

  • MLflow: Its documentation describes it as an open-source AI platform for agents, LLMs and models.
  • Kubeflow: According to its documentation, it is composed of modular, open-source projects that form the Kubernetes-native stack for data and AI workloads. With Kubeflow Pipelines, components and pipelines are authored with the KFP Python SDK and run in containers.
  • KServe: According to the project, a Kubernetes-native platform for serving machine learning models; it supports scale-to-zero for predictive workloads and canary rollouts.

The common ground is Kubernetes. According to the Kubernetes documentation, Pods access GPUs through device plugins, GPUs are only supposed to be specified in the limits section, and node labels steer Pods to the right type of GPU (see our Kubernetes and GPU articles). We cover running container platforms in our cloud-native platforms service, and the design of the GPU and model serving layer in our AI stack design and engineering service.

Operations for large language models: GenAIOps

According to Microsoft, operationalizing generative AI requires extending MLOps investments with generative AI operations (GenAIOps), sometimes known as LLMOps. Traditional machine learning workloads focus on training new models for specific tasks, while many generative AI solutions use existing foundation models that don't require fine-tuning before use. The weight shifts from training to prompts and grounding data, so the knowledge base described in our RAG article becomes part of the pipeline. Microsoft's list of things to monitor includes latency, token usage, 429 errors, content safety and groundedness (see our LLM article).

Governance and traceability

NIST's AI RMF 1.0, released on January 26, 2023, is an AI risk management framework intended for voluntary use; its core is composed of four functions: GOVERN, MAP, MEASURE and MANAGE. Some of its subcategories are squarely MLOps work:

  • MEASURE 2.1: Test sets, metrics and details about the tools used are documented.
  • MEASURE 2.4: The functionality and behavior of the AI system and its components are monitored in production.
  • MANAGE 4.1: Post-deployment monitoring plans are implemented.

According to NIST, maintaining the provenance of training data supports both transparency and accountability. Microsoft's model management records lineage as well: who published a model, why changes were made, and when it was deployed. We cover the management system approach in our ISO/IEC 42001 article and the governance roadmap in our AI governance article.

Frequently asked questions

What does MLOps mean? Machine learning operations: developing a model, putting it into production, monitoring it and retraining it.

What is the difference between MLOps and DevOps? DevOps delivers code; MLOps also versions, tests and monitors the data and the model alongside the code. Because a model goes stale as data changes, MLOps adds continuous training (CT).

What is inference? A trained model producing a prediction or answer on new data. It can run in real time or in batches.

What is the difference between data drift and concept drift? In data drift, the distribution of the input data changes. In concept drift, the relationship between the inputs and the target changes, and the model starts giving wrong answers to the same inputs.

What is LLMOps? Running applications built on large language models; Microsoft calls it GenAIOps. Its focus is on prompts, grounding data, evaluation and content safety.

Sources