The Real ML Stack: From Jupyter Notebook to $10B Infrastructure

Stage 0 of the ML stack is a Jupyter notebook, a CSV, and a model.pkl. Don't laugh. Twitter's early recommendation system was basically this. The graveyard of failed ML projects is full of teams who tried to deploy Kubernetes-orchestrated, feature-store-backed, model-registry-managed systems before they'd shipped a single prediction to a single user. This is the honest guide to what the stack looks like at every stage — and when to graduate from each one.

There is a pattern that kills ML projects at startups so reliably you could set your watch to it. The founding ML engineer joins, sets up a sophisticated infrastructure, spends three months configuring MLflow, Kubeflow, and a feature store. Six months later, they haven't shipped a prediction to a user. They resign, citing "organizational dysfunction." The dysfunction was real but it was downstream of a bad decision: optimising for the stack that Google uses rather than the stack that solves today's actual problem.

The right ML stack is not the most sophisticated one. It's the one that matches your engineering maturity, your data volume, your team size, and the stage of your ML product lifecycle.

Stage 0: Proof of concept (seed stage, 1–3 ML practitioners)

Stack: Python, Jupyter, pandas, scikit-learn. Database: Postgres or even CSV. Serving: a Flask API or FastAPI endpoint reading from a pickle file. Experiment tracking: a spreadsheet. Feature store: a function that transforms a database query into a dict.

This is not embarrassing. This is correct. At Stage 0, the question you're answering is: does ML add value to this product? That question does not require Kubernetes. It requires a model that outperforms a heuristic and a deployment mechanism that puts predictions in front of users.

The failure mode here is premature abstraction: building generic feature pipelines before you know which features matter, building a retraining scheduler before you know how fast your data drifts, building a model registry before you have more than one model.

Ship something. Measure it. Then decide what infrastructure the problem actually requires.

Stage 1: From notebook to production (Series A, 2–6 ML engineers)

The inflection point is when you have 2+ models in production, your data volume exceeds what pandas handles comfortably, and you're manually retraining models because you don't have a pipeline.

Stack additions: MLflow or Weights & Biases for experiment tracking (you're now running more than 10 experiments per week and need to compare them). Docker for reproducibility. Airflow or Prefect for pipeline orchestration. A proper feature computation layer (even if it's just functions in a shared library — not a full feature store yet). Cloud training: SageMaker, Vertex AI, or spot instances.

The critical discipline at this stage: don't skip reproducibility. "Works on my laptop" is expensive at this stage because you now have 3–5 engineers reproducing each other's environments. Containerise early.

Also critical: build a shadow evaluation system. Every model you promote to production should run in shadow mode alongside the incumbent for at least 1–2 weeks. The number of production incidents that shadow mode prevents is enormous. The number of teams that skip it, confident in their offline metrics, is also enormous.

Stage 2: Platform thinking (Series B, 5–15 ML engineers)

You now have enough models, enough engineers, and enough training runs that the overhead of managing them manually exceeds the cost of building infrastructure. This is when platform investment pays off.

Stack: Feature store (Feast, Tecton, or Hopsworks — or a homegrown equivalent). Model registry (MLflow Model Registry or custom). CI/CD for models (training pipelines that trigger on data updates, with automated evaluation gates). Monitoring infrastructure (drift detection, prediction distribution tracking). A/B testing framework integrated with your serving layer.

The common mistake here is building too much custom infrastructure. Every hour your ML engineers spend maintaining a custom feature store is an hour not spent on model quality. Evaluate managed services aggressively. SageMaker Feature Store, Vertex Feature Store, and Databricks Feature Store are not perfect but they're maintained by someone else.

The meta-principle: your ML platform should be opinionated enough to enforce reproducibility and safe deployment, but not so opinionated that it slows down experimentation. The best ML platforms feel like they enable rather than constrain.

Stage 3: Scale infrastructure (Series C to IPO, 15–50+ ML engineers)

At this stage, the abstractions that worked at Stage 2 start breaking. MLflow's tracking server becomes a bottleneck. Your feature store's compute layer can't handle the volume. You start having ML-specific reliability incidents (the retraining pipeline fails silently; a feature value distribution shifts and no one notices for two weeks; the model serving layer has a memory leak that only manifests under sustained load).

Stack investments: Dedicated ML platform team (separate from ML practitioners). Custom training infrastructure on bare metal or reserved instances (spot instances have too much variance for latency-sensitive training jobs). Multi-region serving with failover. Online/offline feature consistency testing (automated regression tests that verify serving features match training features on known inputs). Full lineage tracking (which model version made which prediction for which user — required for debugging and regulatory compliance).

The companies that do this well (Airbnb, Uber, Lyft, Spotify) all reached the same conclusion: the generic tools weren't good enough for their specific constraints, and they built custom solutions. Uber's Michelangelo, Airbnb's Bighead, Lyft's Flyte, LinkedIn's Pro-ML — all of these exist because the off-the-shelf tools had gaps that became expensive at scale.

Stage 4: The hyperscaler stack (FAANG, 100+ ML engineers)

At Google, Meta, Amazon, and Microsoft, ML infrastructure is itself a product. Google's TFX (TensorFlow Extended) pipeline framework is used externally but was built internally. Meta's FBLearner Flow handles thousands of model training runs per day. Amazon SageMaker is a productisation of what Amazon's internal ML teams needed.

The key characteristic of hyperscaler ML stacks is that they optimise for different things than earlier stages: reproducibility at the billion-parameter scale, incremental training on streaming data, hardware utilisation (TPU/GPU allocation is itself a complex scheduling problem), and ML regulatory compliance (model cards, audit trails, explainability tooling).

Most engineers will never work at this stage. That's fine. The engineering challenges are fascinating but the job is more infrastructure than ML. The people building TFX write very little Python and a lot of C++.

The question you should always ask:

Before adding any tool to your ML stack, ask: what specific failure mode does this tool prevent? If you can't name the failure mode, you don't need the tool yet. Feature stores prevent training-serving skew — but only if you've observed training-serving skew causing a problem. Model registries prevent "which version of the model is in production?" — but only if you have enough models that this is actually confusing.

Stack complexity is technical debt that compounds. Build for the problems you have today, with one quarter of headroom for where you'll be in six months. That's it.