Deployment: Beyond Machine Learning Models and Into the World

Let’s start with the assumption that you know how to build a reliable machine learning model that predicts well and is fault tolerant. Great! But to truly create value, your model will need to extend beyond its theoretical definitions, and be accessible to its intended end users. What’s the point of having a great model if it’s sitting in your Jupyter notebook without reaching an actual human being who will gain value from it?

Take the example of a business user. This person wants to understand the forecast of one of their products for the next three months, and which of their clients are going to churn. They just want an interface from which to easily access this information. What they don’t need is to understand the mechanisms behind how this information is being generated by your model. Knowing how to build this interface, in addition to creating a model, is a key skill.

Deployment is essentially similar to building an end-to-end software product. Machine Learning is just one part of it. If you have good software engineering skills, you’ll find the rest of the pieces quite familiar and relatable: you’ll know how the frontend and backend works, as well as how to build pipelines and deploy.

Most data scientists, however, don’t have this software engineering experience. They usually are analysts skilled in Python, but that’s where their skills end, and the gap is clear. To successfully deploy a model, you’ll need to put together various skills of a software engineer, a data engineer and a machine learning scientist. Having this combination of skills and mindsets will set you apart from people who just build models.

To understand why effective deployment is important, we needn’t look further than self-driving cars, which are already a reality. These cars are equipped with a bunch of cameras that take live videos and photos to identify the objects in front of it, classifying them into a car, human, dog, or just a wayward piece of paper. If the car can’t identify the objects accurately, it won’t be able to maneuver the right way. In addition, the car is also predicting the distance between itself and the objects. In ML terms, this is a classification and regression problem, and is actually a model.

To build this model, you don’t need a car. All you need is a bunch of photos and a laptop. But to move the model into production, you’ll need to install it into a car’s hardware so that it can make maneuvering decisions in real time – within nanoseconds – accurately. Production models are live and used by real people. The theoretical model you built is just one piece of the larger puzzle. For it to be actually used, your model needs to work with other data pipelines and infrastructures.

Indeed, there are several steps between building an amazing theoretical machine learning model and deploying it to users, which form the entire data pipeline. For example, what if instead of the one business user discussed earlier, you had 100,000 users accessing your model concurrently? How would it perform then? Cracking these steps in the pipeline is what will set you apart, and help you realise value from the model you build.

If you’re building great models but are having challenges understanding how to effectively deploy it to users, I’d recommend taking on a product development approach. That is, start by understanding who will be using your model, what they will be using it for, and how they will access this information.

Also think about what will happen if your model’s predictions aren’t accurate. How would you know that your model’s prediction accuracy is consistently off? Accuracy is just one parameter of several other loss functions to consider. What level of deviation is acceptable in this use case? How will you be able to fix any issues in the production model while it is still running? Going back to the example of a self-driving car, imagine if your model is predicting inaccurately while ferrying someone at 70mph speeds. How will you be able to solve these issues in real time?

Return to your model with these questions in mind, to inform how you’ll train it. Work on building a data pipeline that cleans the data on its own, and constantly feeds new data to the model. Building these pipelines will require engineering skills. Plus, keep in mind that this has to happen in real time, and you shouldn’t always expect an analyst to train a production model manually. As you’re probably working on several models at once, identify how you’ll score them and which factors will make you choose one model over another for that particular use case.

Here is an overview of the factors you’ll need to consider:

• Consider pipeline reproducibility (How can others replicate what you did with little context?)
• Monitor incoming data (How different is it from your training data? How would you prevent data drift?)
• Monitor model performance
• Evaluate alternative models (A/B testing)
• Plan for scaling (Auto scaling)
• Test Extensively (To evaluate edge cases and flag when things go wrong)

Work through this checklist and answer the questions outlined, mapping out the skills you currently have as well as those you need to learn. Then, pick one, and start learning.

• Place Boundaries Around What You Learn

One thing to keep in mind when learning these new skills is to not go overboard such that you’re digressing too far away from your core competencies. I’ve been there too, where I got so engrossed in learning about all I could about data engineering. A lot of what I picked up actually didn’t help me solve the problem at hand and ended up being a waste of time. Data engineering as a specialisation is massive. So what level of detail should you go into?

Remember that you’re specialising in pipeline building or model deployment. The only data engineering skills you’ll need to pick up are those that will help you deploy and scale your model. Be disciplined in what you learn and implement only what is relevant. It might help to seek guidance from someone in the field, so you don’t waste time learning things that don’t add value to your model deployment efforts in the short term.

• Dive Deep Into Your Domain

At the end of the day, your model should add tangible value to users. To be able to build for your users, you’ll need to have a strong grasp of your product and the business you’re in. Logistics, for instance, is a very different industry from health care or self-driving cars. Come at the problem with a product-driven mindset. Invest time into conducting in-depth research into your end users, how they behave, and in what settings your model will be used. This will help you build effective production level models.

Another way you can stay focused on your business use case is by having conversations with your internal tech teams. Don’t stay in your silos. Internal teams might be able to share insights, narrow down on what your company’s needs are, or even suggest technologies that you could pick up instead of learning new skills from scratch. You’d end up saving time on research, as well as avoid building models that don’t really add value to your organisation and users.

• Explore Readily Available Solutions

Now there are companies who have found solutions to this deployment problem. Solutions like Sagemaker, DataRobot and H2O.ai are some examples of ML Ops solutions out there. These are SaaS solutions that make it relatively easier for someone with lesser technical skills to use them. And they’ll take you 80% of the way in terms of deploying your model. But what is their adoption? More importantly, do you know how things work under the hood?

My team has a bunch of models in production, and we use Sagemaker. However, if we didn’t understand what’s happening under the hood, we wouldn’t be able to explain any deviations in its predictions. Models don’t explain themselves, they need a human being to make sense of the decisions that are made. Let’s say you have a trading model that lost a few million dollars. You’d obviously need to know why this happened, and what sequence of decisions led to this mishap so you can justify it.

With any readily available solution, you don’t need to be an expert to make it work. You will, however, still need to have the basic skills needed to understand the nuts and bolts of how it works. Knowing what’s happening will ultimately cut down the time needed to deploy your model.

Once you’ve explored readily available solutions, you could check out open source frameworks such as ML Flow, TensorFlow Serving, Cortex. As these are open source frameworks, they’re not going to be as user-friendly, and you might take a little more time figuring out how they work. But with your understanding of how readily available solutions deploy, a relatively tech savvy team could replicate the same using open source frameworks while cutting costs significantly.

• Expose Your Model to Large Audiences

People rarely build models just to show to their boss or teams. Models are built to scale, and are almost always used by hundreds of thousands of people at once. It’s important to expose your model to a large number of people so that you’ll identify how it might fail. Just as a website crashes when it’s hit with high traffic, a model fails when there are lots of people accessing it for predictions. To understand how to prevent this failure, to balance the load, and so on, you’ll first need to test your model. If it hasn’t failed yet, either you’ve built an excellent model or your user base simply isn’t as big as it should be.

With models and deployment, you could always spend lots of money and take all the time in the world to build the best infrastructure. But is that what’s required to solve your business problems at hand? Does it answer your user’s real needs? Remember, you don’t have to do what you don’t need to do. Let this guide you as you navigate through the process.

1 Explore ML Ops Solutions 

If you’re focused on building models today, explore what’s out there. Solutions such as Sagemaker, MLflow, DataRobot, will work for about 80% of your needs, and you’d be ready to deploy in a short period of time. Just make sure you spend time understanding how exactly they work. 

2 Move On to Open Source Frameworks

Once you are comfortable with your basics, experiment with Open Source Frameworks. It will test your core understanding of the concepts but will eventually improve your ability to deploy models using any solution out there. 

3 Test, Test, and Test Again

Think like a Product Manager and evaluate scenarios where your model would fail. Write tests to evaluate for standard and edge cases. Tests should evaluate all parts of your pipeline, and help perform a root cause analysis in an efficient manner.