CI/CD for ML: Pipelines That Test Data, Models, and Systems

Imagine you trained a model in a notebook. It looks great. AUC 0.91. You are excited.

So the question becomes

If you answer "yes," you will eventually have a bad day. The model was trained on yesterday's data, on your laptop, tested on your metrics. Production is a different place. The data schema changes. A feature goes missing. The exported model file computes slightly different numbers than the Python original. Latency doubles under load. None of that shows up in a notebook.

Normal software teams solved a similar problem decades ago with CI/CD: every code change automatically runs tests, and only changes that pass get deployed. CD4ML (Continuous Delivery for Machine Learning) is the same idea, stretched to cover what makes ML special: models depend on data (which mutates without a git commit), on artifacts (which must survive export/conversion), and on systems (feature fetch → model → downstream consumers).

Sculley et al.'s "Hidden Technical Debt in ML Systems" (NeurIPS 2015) named the root cause: ML systems accumulate feedback loops, direct hidden dependencies, and under-constrained experiment surfaces that classic CI never checks. A code-only pipeline simply cannot see those failure modes.

So the pipeline's job description is one sentence: