A platform for the

Complete Machine
Learning Lifecycle
Corey Zumar
March 27th, 2019
Outline
• Overview of ML development challenges

• How MLflow tackles these challenges

• MLflow components

• Demo

• How to get started

2
 Machine Learning
Development is Complex

3
 μ
ML Lifecycle λθ Tunin
g
Scal
Data Prep e

μ
Model λθ Tunin
Delta Raw Data Exchang Training g
e Scal
e
Scal
e

Deploy
Governanc
e Scal
e
4
Custom ML Platforms
Facebook FBLearner, Uber Michelangelo, Google TFX
+ Standardize the data prep / training / deploy loop:
if you work with the platform, you get these!
– Limited to a few algorithms or frameworks
– Tied to one company’s infrastructure

Can we provide similar benefits in an open

manner?
5
Introducing

Open machine learning platform

• Works with any ML library & language
• Runs the same way anywhere (e.g. any cloud)
• Designed to be useful for 1 or 1000+ person orgs

6
MLflow Components

Tracking Project Model

s s
Record and query Packaging format General model
experiments: code, for reproducible format
configs, results, runs on any that supports diverse
…etc platform deployment tools

7
Key Concepts in Tracking
Parameters: key-value inputs to your code
Metrics: numeric values (can update over time)
Artifacts: arbitrary files, including data and models
Source: training code that ran
Version: version of the training code
Tags and Notes: any additional information

8
 MLflow Tracking

Tracking APIs
(REST, Python, Java, R)

UI

Tracking Server API

9
MLflow Tracking
import mlflow

with mlflow.start_run():

Tracking mlflow.log_param("layers", layers)

mlflow.log_param("alpha", alpha)

# train model
Record and query
experiments: code, mlflow.log_metric("mse", model.mse())
mlflow.log_artifact("plot", model.plot(test_df))
configs, results, mlflow.tensorflow.log_model(model)
…etc

10
MLflow backend stores
1. Entity Store
• FileStore (local filesystem)
• SQLStore (via SQLAlchemy)
• REST Store

2. Artifact Repository
• S3 backed store
• Azure Blob storage
• Google Cloud storage
• DBFS artifact repo
11
Demo
Goal: Classify hand-drawn digits

1. Instrument Keras training code with MLflow tracking APIs

2. Run training code as an MLflow Project

3. Deploy an MLflow Model for real-time serving

12
 MLflow Projects Motivation
Diverse set of training
tools Result:
ML code is difficult
Diverse set of
to productionize.
environments

13
 MLflow Projects

Local Execution
Project Spec

Code Config
Remote Execution
Dependencie Data
s

14
 MLflow Projects
Packaging format for reproducible ML runs
• Any code folder or GitHub repository
• Optional MLproject file with project configuration
Defines dependencies for reproducibility
• Conda (+ R, Docker, …) dependencies can be specified in MLproject
• Reproducible in (almost) any environment
Execution API for running projects
• CLI / Python / R / Java
• Supports local and remote execution

15
Example MLflow Project
my_project/ conda_env: conda.yaml
├── MLproject
│ entry_points:
│ main:
│ parameters:
│ training_data: path
│ lambda: {type: float, default: 0.1}
├── conda.yaml command: python main.py {training_data}
├── main.py {lambda}
└── model.py
... $ mlflow run git://<my_project>
16
Demo
Goal: Classify hand-drawn digits

1. Instrument Keras training code with MLflow tracking APIs

2. Run training code as an MLflow Project

3. Deploy an MLflow Model for real-time serving

17
 MLflow Models Motivation

Inference
Code

Batch & Stream

Scoring

Serving Tools
ML
Frameworks
18
 MLflow Models
Inference Code

Model Format

Flavor 1 Flavor 2
Batch & Stream
Scoring

Standard for ML
ML models Serving Tools
Frameworks
19
 MLflow Models
Packaging format for ML Models
• Any directory with MLmodel file
Defines dependencies for reproducibility
• Conda environment can be specified in MLmodel configuration
Model creation utilities
• Save models from any framework in MLflow format
Deployment APIs
• CLI / Python / R / Java

20
 Example MLflow Model
run_id: 769915006efd4c4bbd662461
my_model/ time_created: 2018-06-28T12:34
├── MLmodel flavors:
│ tensorflow:
│ saved_model_dir: estimator
Usable with Tensorflow
│ signature_def_key: predict tools / APIs
│ python_function:
loader_module: Usable with any Python
│
mlflow.tensorflow tool
└ estimator/
├─ saved_model.pb
└─ variables/
mlflow.tensorflow.log_model(...)
...
21
Model Flavors Example
PyTorch mlflow.pytorch.log_model()
Train a model

predict = mlflow.pyfunc.load_pyfunc(…)
Flavor 1:
Pyfunc predict(input_dataframe)

Model Flavor 2:
Format model = mlflow.pytorch.load_model(…)
PyTorch
with torch.no_grad():
model(input_tensor)
22
Model Flavors Example

predict = mlflow.pyfunc.load_pyfunc(…)
predict(input_dataframe)

23
Demo
Goal: Classify hand-drawn digits

1. Instrument Keras training code with MLflow tracking APIs

2. Run training code as an MLflow Project

3. Deploy an MLflow Model for real-time serving

24
Get started with MLflow

pip install mlflow to get started

Find docs & examples at mlflow.org

tinyurl.com/mlflow-slack

2525
0.9.0 Release

MLflow 0.9.0 was released this week! Major

features:

• Tracking server supports SQL via SQLAlchemy

• Pluggable tracking server backends
• Docker environments for Projects
• Custom python models 2626
Ongoing MLflow Roadmap
• UI scalability improvements (1.0)
• X-coordinate logging for metrics & batched logging (1.0)
• Fluent API for Java and Scala (1.0)
• Packaging projects with build steps (1.0+)
• Better environment isolation when loading models (1.0)
• Improved model schemas (1.0)

27
Thank you!

28