ModelRefs / MLflow Experiment Tracking — Tutorial

MLflow Experiment Tracking — Tutorial

Log, compare, and reproduce ML experiments — metrics, artifacts, models, and the model registry. Covers Why experiment tracking matters.

Overview

Log, compare, and reproduce ML experiments — metrics, artifacts, models, and the model registry

Level: Advanced. Estimated reading time: 35 minutes.

Why experiment tracking matters

An ML project accumulates hundreds of experiments: different models, hyperparameters, datasets, and preprocessing steps. Without tracking, you cannot answer: - Which hyperparameters produced the best validation F1? - What was the exact dataset version used for the model in production? - Why did experiment 47 beat experiment 31? - Who ran what, and when?

MLflow is an open-source platform for the full ML lifecycle. Four components:

Tracking: log parameters (learning_rate=0.001), metrics (val_f1=0.87), artifacts (plots, confusion matrices), and tags. Compare runs in the UI.

Models: log any model with a standard interface (mlflow.pytorch.log_model, mlflow.sklearn.log_model). Load with mlflow.pyfunc.load_model — framework-agnostic.

Model Registry: version models, add descriptions and tags, transition between Staging/Production/Archived.

Projects: reproducible experiment runs defined in an MLproject file (dependencies, entry points). Re-run any experiment exactly.

Logging runs and organising experiments

The core API:

mlflow.set_experiment("sentiment-classifier"): create or switch to an experiment.

with mlflow.start_run(run_name="bert-lr-3e5"): create a run. Everything logged inside the context block belongs to this run.

mlflow.log_param("lr", 3e-5): log a hyperparameter.

mlflow.log_metric("val_f1", 0.87, step=10): log a metric (with optional step for time-series).

mlflow.log_artifact("confusion_matrix.png"): upload a file (stored in the artifact store — local filesystem, S3, GCS, Azure Blob).

mlflow.pytorch.log_model(model, "model"): log the model weights + metadata.

Autologging: mlflow.sklearn.autolog() or mlflow.pytorch.autolog() automatically logs all standard metrics and the model without explicit calls. Works with sklearn, XGBoost, LightGBM, Keras, and PyTorch Lightning.

Model registry and CI/CD integration

The MLflow Model Registry manages the model lifecycle:

Transition: client.transition_model_version_stage("SentimentClassifier", 1, "Production") — move to Staging → Production → Archived.

Load: mlflow.pyfunc.load_model("models:/SentimentClassifier/Production") — always loads the current production model.

CI/CD integration: after training, a CI pipeline runs evaluation against a held-out test set. If val_F1 > current_production_F1, promote to Production automatically. Rollback: transition the old version back to Production.

Remote tracking server: mlflow server --backend-store-uri postgresql://... --default-artifact-root s3://... centralises all experiments. Useful for teams — everyone's experiments visible in one UI.

Tags for governance: tag runs with git_commit, data_version, model_type, and team. Filter by tag in the UI or API.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MLflow Experiment Tracking — Tutorial.