A guide for developers and architects especially those who are not specialized in machine learning to design an MLOps architecture for their organization
Introduction
According to Gartner’s findings, only 53% of machine learning (ML) projects progress from proof of concept (POC) to production. Often there is a misalignment between the strategic objectives of the company and machine learning models built by data scientists. There might be a lack of communication between DevOps, security, legal, IT and the data scientist that causes challenges to pushing the model into production. Finally, the team might find it difficult to maintain the models in production while pushing out new models. It has led to the rise of MLOps which brings the principles of DevOps, such as continuous integration and continuous delivery (CI/CD), automation, and collaboration to the machine learning lifecycle — development, deployment and monitoring.
In this article, I will dive into the following :
- Various steps in the machine learning process
- Different MLOps components and explain why they are necessary without diving too much into the details that only data scientists need to know
- MLOps architecture diagrams based on the size and maturity of the organization
- General suggestions on starting the MLOps journey
Typical Machine-learning process
Let’s start first by understanding the steps involved in the machine learning process.
A machine learning process has the following components:
- Business Problem and Machine learning problem statement: We start the process by identifying the business problem and agreeing that machine learning is the right solution for the problem. The proposed machine-learning solution should produce a measurable business outcome.
- Data Collection, Integration and Cleaning: In this step, data scientists/data engineers collect data, integrate it with different sources, and clean & transform it to make it consumption ready. Data…
