ML Ops Explained: Q&A With Senior ML Ops Engineer Matthew Theisen

A Kensho MLOps engineer demystifies what it actually means day-to-day to how MLOps fits into a modern AI company.

Creating the path to reliable machine learning

This blog is part of our career stories series, which gives a glimpse of what it’s like to be a part of our team and share the career stories of Kenshins. Stay tuned for more articles on life at Kensho!

Introduce yourself

I am Matthew Theisen, a Senior Machine Learning Operations (ML Ops) Engineer at Kensho Technologies. I’ve worked at Kensho for 4 years. I became an ML Ops engineer through internal mobility. Before ML Ops, I was a machine learning engineer developing models. I previously worked as a data scientist at an internet media company.

What do you do as an ML Ops Engineer?

At a high level, ML Ops is turning machine learning from an art into a science. Companies are increasingly using machine learning to power their products, but this brings a lot of complexity. It is a challenge to manage the code as well as the data that generate machine learning models. In software engineering, practices like unit testing, version control and continuous integration add rigor and reliability. ML Ops is bringing that same polish and professionalism to machine learning. Think of it as DevOps for machine learning.

As ML Ops engineers, we act as a bridge between the infrastructure and the machine learning teams. A lot of our job is identifying or building tools and deploying them in a way that is thoughtful about the underlying technology and the needs of our users, our machine learning engineers.

How did you get started in the role? Was this always what you wanted to do?

I switched from machine learning to ML Ops when the team was created at Kensho. At Kensho, we realized that a dedicated team was needed to manage our machine learning tools. It so happened that my interests aligned with this business need. Since starting in the software industry, I’ve been interested in learning about the different layers of the stack, so this was a way to get closer to the infrastructure.

To be honest, in college I didn’t want to work with computers. Both my older brothers are software engineers and I wanted to do something different. I studied chemical engineering, but sometimes fate has other plans. After a PhD in bioengineering, I heard about machine learning as a new and exciting field. It turned out I enjoyed working with computers, and machine learning was a good fit for me.

How does working remotely affect your job?

I have been working remotely since the beginning of the pandemic. For the most part, it has been seamless. Kensho has been a multi-office company for a while, so the video conferencing and messaging tools were already in place. Kensho is also a relatively small company, so it’s easy to be acquainted with most of the people, even if you’re not seeing them face-to-face. The ability to work remotely is also helpful with the arrival of my daughter, Lucy.

What is the interview process like for someone applying to ML Ops roles?

It’s similar to other software roles. There will probably be technical screens related to coding and system design. Companies will be looking for knowledge of machine learning model lifecycle management, and familiarity with some of the major frameworks.

(Note: Kensho has openings for ML Ops roles)

Imagine you’re back in high school or college. How would you start getting involved in ML Ops?

I would start by majoring in something related to computers! Beyond formal education, there are also more specific resources. ML Ops is an emerging field, but there are already communities convening to share information. One I have enjoyed is mlops.community.

A great thing about software careers is that you probably already have the ultimate software learning and development tool (a computer). You can easily get hands-on with any of the languages and open source frameworks out there. A lot of companies don’t require formal education in software, especially for internships, which is how I got my start in this industry.

What are some of the common challenges you face in your role and how do you handle them?

One of the big challenges we’ve faced is transitioning from older to newer sets of tooling. Building tools for a company that has pre-existing systems is more challenging than starting from scratch. The way we handle this is by planning carefully and making sure both sets of tools are supported for a period of time while we drive adoption of the newer tools. We tend to pull our users over with improved usability and features, rather than push by setting hard deadlines for shutting down old systems.

(Read more about Kensho’s ML Ops philosophy and tools.)

What do you like most about your work?

There is a lot of satisfaction in understanding the different layers of the tech stack and using that knowledge to build tools that make machine learning more robust. That is high leverage work. It makes our machine learning teams more productive, our models more reliable, and our products better. When we see big user successes after rolling out tools, it’s very rewarding. It makes the thoughtful design and rollout processes worth it.

What’s next? How do you see your career evolving over the next 2 years? 5 years?

There will be some centralization around the more successful frameworks. We are starting to see a few of the winners emerging now, and that process will continue. The need for ML Ops will grow in the short run. This is probably a transitional phase as companies roll out new tooling. In the medium term, as we get more standardization around frameworks, it’s possible that some of the ML Ops work will be split between DevOps or infrastructure on the one hand, and machine learning or data engineers on the other.