Deploying Machine Learning Models at Scale

You have a model that performs well on test data.

Accuracy looks good.

The notebook runs without errors.

The model artifact is ready.

But the most important question is still unanswered:

Can this model run every day, for real users, under real traffic, without becoming a production problem?

That is where the difference between “building a model” and “building a machine learning system” becomes obvious.

Google’s MLOps guidance makes an important point: model code is only one part of a real production ML system. Around it sit data collection and validation, testing, resource management, serving infrastructure, metadata, automation, deployment, and monitoring.

So deploying a model at scale is not simply about putting a model file behind an API.

A production model needs to be versioned, reproducible, observable, rollback-friendly, and scalable.

Start by Defining the Type of Inference You Actually Need

Not every model should be deployed the same way.

Before choosing infrastructure, define when predictions are needed and how quickly they must arrive.

Real-Time Inference

A user or system sends a request and expects a response immediately.

Examples include:

• Fraud detection during payment
• Product recommendations
• Request scoring
• Image classification
• Real-time personalization

In this case, latency matters, and the model usually runs behind an online endpoint.

Batch Inference

Sometimes predictions do not need to happen immediately.

For example, you may want to calculate churn probability for one million customers every night.

In that case, batch inference can be much more practical and cost-effective than keeping an always-on endpoint running.

Asynchronous Inference

Some prediction workloads take longer.

Maybe the payload is large.

Maybe inference takes minutes.

Maybe the request involves heavy processing.

In these cases, it is often better to place the request in a queue and return the result later instead of keeping the client connection open.

AWS distinguishes similar serving patterns in SageMaker: real-time inference for low latency, serverless inference for intermittent traffic, and asynchronous inference for longer-running or larger-payload requests.

So the first scaling question is not:

“How many GPUs do we need?”

It is:

“How is this prediction actually consumed?”

Define the SLA Before You Choose the Infrastructure

“The model should be fast” is not a useful production requirement.

You need measurable targets.

For example:

• P95 latency below 200 ms
• At least 500 requests per second
• 99.9% availability
• Error rate below 0.1%
• Or: Every nightly batch must finish before 6 a.m.

Once those targets are clear, decisions around compute, replicas, autoscaling, batching, and architecture become much easier.

A model can have excellent offline accuracy and still be a poor production choice because it takes two seconds per prediction and requires an expensive GPU.

The best model in the lab is not always the best model in production.

Make the Runtime Reproducible

One of the classic lines in software and ML is:

It works on my machine.

That sentence is not especially useful in production.

A model usually depends on more than the model file itself.

It may depend on:

• Python version
• Libraries
• Runtime
• CUDA
• Drivers
• Preprocessing code
• Feature transformations
• External artifacts

If the training and serving environments differ, the result may range from a simple runtime error to silently different predictions.

A common approach is to package the model and its dependencies inside a container.

MLflow also supports deployment patterns that package model dependencies and runtime requirements so models can be served in consistent environments.

The principle is simple:

The model artifact is not enough. The environment that runs it must be reproducible too.

Do Not Manage Model Versions Like Random Files

Names such as:

model_final.pkl

model_final_v2.pkl

model_really_final.pkl

do not scale well.

In production, you should be able to answer:

• Which model is currently live?
• Which dataset trained it?
• Which code commit produced it?
• Which hyperparameters were used?
• What validation metrics did it achieve?
• Who approved it?
• What was the previous production version?

This is where a Model Registry becomes useful.

MLflow Model Registry stores model versions, metadata, lineage, tags, and aliases, making it possible to distinguish a production model from candidate or experimental versions.

Azure Machine Learning also treats model registration and metadata as part of the MLOps lifecycle, helping maintain traceability across training and deployment.

With a registry, the question changes from:

Which file should we deploy?

to:

Which approved version should receive production traffic?

That is a much healthier production process.

Separate the Model From the Endpoint

A useful serving pattern is to let clients call a stable endpoint instead of connecting directly to a specific model version.

The endpoint can sit in front of multiple deployments.

For example:

Blue → Model v12

Green → Model v13

The client still calls the same endpoint.

Traffic routing decides which model version handles each request.

Azure Machine Learning supports this model directly: a single online endpoint can contain multiple deployments and split traffic between them.

This separation makes model upgrades much easier because client applications do not need to change every time the model changes.

Do Not Send 100% of Traffic to a New Model Immediately

A new model performs better in offline evaluation.

Should you replace the old production model immediately?

Usually not.

Offline metrics do not reveal everything.

Production data may be different.

Latency may be higher.

GPU or memory consumption may increase.

A particular customer segment may behave differently.

There are safer rollout patterns.

Shadow Testing

In shadow deployment, the new model receives a copy of real production traffic, but its response is not returned to users.

The current production model still serves the real response.

Behind the scenes, you can compare:

• Latency
• Errors
• Prediction distributions
• Resource usage
• And, when ground truth is available, model quality

AWS SageMaker and Azure both support shadow or mirrored traffic patterns for evaluating new versions before they affect users.

Canary Deployment

If shadow testing looks good, you can move a small percentage of real traffic to the new model.

For example:

95% → Current model

5% → New model

Then:

90 / 10

50 / 50

And eventually:

0 / 100

This staged rollout limits blast radius.

If the new version behaves badly, only a small portion of users see the issue.

Rollback Should Be Routine, Not a Crisis

If a new model fails, how long does it take to restore the previous one?

If the answer is:

“We first need to find the old model file,”

the MLOps process is not ready.

Versioning, registry, and traffic routing should allow a known-good version to be restored quickly.

Rollback is not failure.

It is a normal part of release engineering.

A good deployment process has a way forward and a way back.

Autoscaling Should Not Depend Only on CPU

When traffic increases, the serving layer needs more capacity.

In container-based environments, Kubernetes Horizontal Pod Autoscaler can scale replicas using CPU, memory, or custom metrics.

But for ML workloads, CPU is not always the right signal.

Depending on the model, better scaling metrics may include:

• Requests per second
• Queue length
• Concurrent requests
• GPU utilization
• GPU memory
• Inference latency
• Tokens per second
• Pending requests

Imagine CPU usage is only 30%, but GPU memory is completely full.

A CPU-based autoscaler may think everything is fine.

So the scaling metric should reflect the actual bottleneck.

Understand Scale Up vs Scale Out

There are two broad ways to increase capacity.

Vertical Scaling

Give one instance more resources.

• More RAM
• More CPU
• A stronger GPU

Horizontal Scaling

Add more serving replicas.

Kubernetes HPA is specifically designed around this horizontal scaling model.

For many real-time systems, horizontal scaling improves both throughput and availability.

But very large models may not fit on a single GPU or even a single node, in which case model parallelism or distributed serving may be required.

So:

“We’ll just add more replicas”

is not a universal answer.

Account for Cold Starts

Autoscaling works well until a new replica takes 40 seconds to become ready.

Startup may include:

• Pulling the model artifact
• Loading it from storage
• Initializing the runtime
• Allocating GPU memory
• Warming caches

If traffic spikes in seconds, the autoscaler may make the correct decision but capacity may arrive too late.

Depending on the use case, you may need:

• Minimum replicas
• Pre-warming
• Predictive scaling
• Smaller container images
• Faster model loading
• Or serverless serving for workloads where cold starts are acceptable

Cold start is not just an infrastructure detail.

It directly affects user experience.

Throughput and Latency Are Often a Trade-Off

Batching can increase predictions per second.

But building a batch may require requests to wait slightly longer.

So higher throughput can increase latency.

For a recommendation system, a few extra milliseconds may be acceptable.

For fraud detection inside a payment flow, they may not be.

Optimization should happen inside the real SLA.

The goal is not simply:

“Maximum QPS.”

The goal is:

The right throughput at the right latency for the business use case.

Monitor Both the System and the Model

A green infrastructure dashboard does not mean the ML system is healthy.

CPU may look fine.

Memory may look fine.

The endpoint may return HTTP 200.

And the predictions may still be getting worse.

Production ML needs at least two layers of monitoring.

Operational Monitoring

Is the serving system healthy?

• Latency
• Throughput
• Error rate
• Timeouts
• CPU
• Memory
• GPU
• Queue length
• Availability

Model Monitoring

Is the ML behavior healthy?

• Prediction distribution
• Feature distribution
• Data quality
• Accuracy or task-specific quality
• Drift
• Bias
• Feature attribution

Azure Machine Learning also treats monitoring of model inputs, operational behavior, and ML-specific issues as part of model operations.

A model returning HTTP 200 is not necessarily a healthy model.

What Is Data Drift?

Imagine a fraud model was trained on last year’s customer behavior.

Then things change.

Customers start buying differently.

A new sales channel is introduced.

New products appear.

Fraud patterns evolve.

The production data distribution no longer looks like the training data.

That change is generally called Data Drift.

AWS Model Monitor describes data quality monitoring by comparing production data statistics with a baseline created from training or reference data.

But drift by itself does not automatically mean the model has failed.

You still need to ask:

• Which feature changed?
• By how much?
• Did model performance change too?
• Is the change expected?

Drift is a signal.

It is not always a verdict.

Ground Truth May Arrive Much Later

For some models, you learn the true outcome quickly.

Spam detection may get fast feedback.

For others, it may take months.

In churn prediction, you need to wait to see whether the customer actually leaves.

In credit risk, the true outcome may take much longer.

So real model quality cannot always be calculated in real time.

In those cases, you can monitor proxies first:

• Input drift
• Prediction distribution
• Confidence
• Missing values
• Business KPIs

Then update real quality metrics when ground truth becomes available.

This is one reason monitoring should be designed around the specific use case, not copied from a generic dashboard.

Do Not Connect Drift Directly to Automatic Retraining Without Guardrails

The idea sounds attractive:

Drift detected → model retrains automatically → new model deploys automatically

But without validation gates, this can simply automate the release of a worse model.

Google’s MLOps guidance places data validation and model validation before delivery.

A retraining pipeline should have gates.

For example, a new model may only become a candidate if:

• Data validation passes
• Evaluation metrics remain above thresholds
• Regression tests pass
• Bias or safety checks pass
• Latency remains acceptable
• And, in sensitive use cases, a human approves the release

Automation should increase speed.

It should not remove judgment.

Watch for Training-Serving Skew

Imagine age is calculated from date of birth during training.

In production, the same feature is calculated using slightly different logic.

The model is identical.

The data looks similar.

But the feature transformation is not the same.

That mismatch can reduce prediction quality.

This is known as training-serving skew.

One of the best protections is to reuse the same feature logic across training and serving whenever possible.

A feature should ideally have one definition.

Not one implementation inside a notebook and another inside the backend.

CI/CD for ML Is More Than Deploying Code

In traditional software, code is usually the main artifact.

In machine learning, more things need versioning:

• Code
• Model
• Data
• Feature definitions
• Training configuration
• Environment
• Pipeline
• Evaluation results
• Infrastructure

That means CI/CD for ML is more than:

docker build && deploy

A mature pipeline may include:

• Source control
• Automated tests
• Training
• Evaluation
• Model registration
• Approval gates
• Infrastructure deployment
• Serving rollout
• Monitoring
• And retraining workflows

Google and Microsoft both treat these elements as core parts of MLOps rather than optional extras around the model.

Production Models Need Lineage

If an important prediction is questioned tomorrow, you should be able to answer:

• Which model version produced it?
• Which dataset trained that model?
• Which training code commit was used?
• Who approved it?
• When was it deployed?
• What were its evaluation metrics?

That traceability is lineage.

Registries such as MLflow help link model versions back to experiments and runs.

Cloud ML platforms also track metadata across training and deployment.

At small scale, this can feel like extra paperwork.

At large scale, with dozens of models and hundreds of versions, it becomes the only reliable way to understand what happened.

Secure the Model Endpoint Like Any Other API

A model endpoint is still an API.

So the same security questions apply:

• Who is allowed to request predictions?
• How is the caller authenticated?
• What permissions does the service account have?
• Does the endpoint need to be public?
• Does the request contain sensitive data?
• What gets written to logs?
• Where are model artifacts stored?
• Are tenants isolated correctly?

Managed ML platforms can also expose private endpoints so inference traffic stays inside controlled network boundaries.

For models that process sensitive data, infrastructure security is part of MLOps.

Not a separate topic.

Measure Cost per Prediction

At scale, cost becomes a first-class metric.

It is not enough to say:

“Our GPU bill was $5,000 last month.”

A better metric might be:

Cost per 1,000 predictions

Or:

• Cost per image
• Cost per recommendation
• Cost per document
• Cost per customer

Now you can compare model versions more intelligently.

Model B may improve accuracy by 1% but cost four times more to serve.

Is that extra 1% worth four times the cost?

That is not only an ML question.

It is a business decision.

Load Test Before Launch

Autoscaling does not replace capacity planning.

Before production, you should know:

• How many requests one replica can handle
• P50 / P95 / P99 latency
• Which batch size works best
• CPU and GPU utilization
• Whether memory grows over time
• Where errors begin
• How long a new replica takes to start

A useful load test does not stop while everything is green.

It should help you understand the breaking point.

Only then can minimum replicas, maximum replicas, and scaling thresholds be based on evidence instead of guesswork.

What Does a Production ML Architecture Usually Include?

There is no single architecture for every project, but a mature ML system often looks something like:

Data → Validation → Training → Evaluation → Registry → Approval → Deployment → Endpoint → Monitoring → Feedback → Retraining

Around that, you usually also need:

• Source control
• CI/CD
• Artifact storage
• Secrets management
• Logging
• Observability
• Infrastructure as Code
• Governance

That is exactly why production ML is much bigger than the model itself.

When Does Kubernetes Make Sense?

Kubernetes is powerful for model serving.

It is also not the default answer for every ML project.

It can make sense when:

• You operate many models
• You need deep infrastructure control
• GPU scheduling matters
• The serving workload is complex
• You already have a platform engineering team
• You need custom autoscaling or deployment behavior

But for one small model with limited traffic, a managed endpoint may be much simpler operationally.

More infrastructure is not automatically better infrastructure.

Good MLOps is not measured by the number of tools in the stack.

A Practical Pre-Production Checklist

Production Is Not the End of the Model Lifecycle

In traditional software, release can feel like a finish line.

In machine learning, release is usually the beginning of a new phase.

The environment changes.

The data changes.

Users change.

Business rules change.

And model performance may change with them.

A model that is the best version today has no guarantee of remaining the best version six months from now.

That is why deploying ML at scale is less about:

“How do we deploy this model?”

and more about:

How do we build a system that can repeatedly deploy, measure, compare, roll back, and replace models safely?

The model matters.

Production is bigger than the model.

If you cannot identify its version, observe its behavior, scale its capacity, or roll it back quickly, it is not truly production-ready yet.

←Back to articles