ML Pipeline Types and Use Cases
- [Itsukushima Shrine, Japan]
- Overview
A machine learning (ML) pipeline is a series of interconnected data processing and modeling steps designed to automate, standardize and streamline the process of building, training, evaluating and deploying ML models.
A ML pipeline is a means of automating the machine learning workflow by enabling data to be transformed and correlated into a model that can then be analyzed to achieve outputs. This type of ML pipeline makes the process of inputting data into the ML model fully automated.
A ML pipeline is a crucial component in the development and productionization of ML systems, helping data scientists and data engineers manage the complexity of the end-to-end ML process and helping them to develop accurate and scalable solutions for a wide range of applications.
- Main Machine Learning Pipelines
Pipelines are becoming more and more popular and now they are everywhere in data science, from simple data pipelines to complex machine learning (ML) pipelines. The primary purpose of pipelines is to simplify the process in data analysis and ML.
In ML, the three primary types of pipelines are: feature pipelines which transform raw data into features, training pipelines which use those features to train a model, and inference pipelines which apply the trained model to new data to generate predictions.
Data moves sequentially from the feature pipeline to the training pipeline and finally to the inference pipeline. These pipelines are crucial for implementing MLOps practices, ensuring efficient model development, deployment, and monitoring.
Pipelines can be designed for batch processing (analyzing large datasets at once) or streaming processing (analyzing data as it arrives).
- Feature Pipelines
A feature pipeline is a program that orchestrates the execution of a dataflow graph of feature functions (transformations on input data to create unencoded feature data), where the computed features are written to one or more feature groups.
- Training Pipelines
An ML training pipeline is a series of automated steps within a ML workflow that takes raw data, preprocesses it, extracts features, trains a machine learning model using those features, and evaluates its performance, essentially encompassing the entire process of building and optimizing a model ready for deployment; it aims to streamline the development process by organizing each stage into a structured, repeatable sequence.
- Inference Pipelines
An ML inference pipeline refers to a structured process in ML where new data is fed into a trained model to generate predictions, essentially the operational pathway to apply a model to real-world data, involving steps like data pre-processing, model loading, prediction generation, and output formatting; it can be implemented as a batch process for large data sets or an online service for real-time predictions.
[More to come ...]
