Personal tools
 

ML Model Deployment

University of New South Wales_022724A
[University of New South Wales]
 

 - Overview

Machine learning (ML) model deployment is the process of integrating a trained, validated ML model into a production environment so it can accept real-world data and return predictions. It transforms a model from an experimental notebook prototype into an accessible, scalable, and reliable software asset that delivers business value.

The end-to-end process typically follows a structured MLOps (Machine Learning Operations) sequence: Train → Validate → Register → Deploy → Serve → Monitor.

(A) Core Components of Deployment: 

A production-ready model deployment relies on three pillars:

  • Deployment Strategy: Setting up the release pipeline (e.g., handling versioning and testing strategies like A/B tests or canary rollouts).
  • Model Serving: Providing the runtime infrastructure and network endpoints so external clients can request predictions.
  • Monitoring & Observability: Tracking inference latency, resource usage, and data/concept drift over time to know when a model needs to be retrained.


(B) Common Strategies for Serving Predictions: 

How you deploy your model depends entirely on your data velocity, latency requirements, and budget constraints.

1. Online (Real-time) Inference: The model processes data instantly via an interactive network request and returns immediate answers.

  • Typical Use Case: Fraud detection, recommendation engines, chatbots.F
  • Tools Used: FastAPI, Triton, TorchServe

2. Batch Inference: Predictions are run on a scheduled interval (e.g., hourly, nightly) over large collections of data.

  • Typical Use Case: Generating weekly marketing emails, bulk bank risk assessments.
  • Tools Used: Apache Spark, Databricks

3. Edge Deployment: The model is compressed and deployed directly onto a client device or web browser.

  • Typical Use Case: Mobile photo filters, autonomous vehicle object detection.
  • Tools Used: TensorFlow Lite, ONNX Runtime.


(C) Standard Technical Deployment Workflow: 

For standard cloud-based online inference, engineering teams typically follow these technical steps:

[Trained Model] ──> [Wrap in API] ──> [Containerize] ──> [Orchestrate/Scale]

  • Step 1: Save & Register the Model Artifact: Export the trained weights (e.g., .pkl, .onnx, .pt) and log them in a central model registry (like MLflow). 
  • Step 2: Wrap in a Web Framework: Build an API endpoint around the model using lightweight frameworks like FastAPI or Flask. This allows applications to send HTTP POST requests with raw data and receive JSON predictions.
  • Step 3: Containerization: Package the code, system configurations, and exact library dependencies into a Docker container. This guarantees the model executes identically in testing, staging, and production environments.
  • Step 4: Scale with Orchestration: Run the container on cloud computing infrastructure. For large-scale traffic, companies use Kubernetes (or specialized tools like KServe) to manage automated horizontal scaling, load balancing, and fault tolerance. 


(D) Production Challenges to Keep in Mind: 

Moving to production introduces software engineering complications that don't exist in a static research environment:

  • Model Drift: Over time, real-world data shifts (e.g., changing consumer behavior), causing a model's accuracy to naturally degrade. Continuous data monitoring is required to alert engineers when it is time to retrain.
  • Compute Constraints: Heavy models (especially deep learning and LLMs) demand significant processing power. Teams use model compression, quantization, and specialized hardware (GPUs/TPUs) to optimize throughput and keep latency down. 

 

 

[More to come ...]


Document Actions