GNNs in Production at Scale

PinSage, fraud detection, drug discovery, dynamic graphs, feature engineering, real-time inference

A drug molecule has atoms as nodes and bonds as edges. A protein has residues as nodes and spatial contacts as edges. You want to predict whether the drug binds to the protein — a graph-graph matching problem. Hand-crafted features (molecular fingerprints, protein descriptors) have been used for this task for decades. But the features must be designed by domain experts, they are fixed at design time, and they discard structural information that doesn't fit the feature schema. GNNs learn task-specific representations directly from the molecular graph, capturing the geometric and chemical compatibility between drug and protein that hand-crafted features miss.

This is the general pattern for GNN applications: wherever entities have structure (molecules, proteins, social networks, knowledge graphs, circuit layouts) and tasks depend on that structure, GNNs outperform feature-engineering approaches by learning the relevant structural representation end-to-end.

Moving a GNN from an academic benchmark to production exposes problems that benchmark papers omit: graphs with billions of edges, millisecond latency requirements, continuous updates that invalidate cached embeddings, and predictions that must be explainable to analysts. PinSage (Ying et al., 2018) is the canonical case study — from a 2-layer GraphSAGE prototype to a system serving hundreds of millions of users. Its most important innovations are not architectural: random walk importance sampling, MapReduce offline embedding computation, and ANN serving are the engineering decisions that made billion-scale GNN deployment feasible.

NOT this. "GNNs are only used for node classification." GNNs support node classification (protein function prediction), link prediction (friend recommendation, drug-target interaction), graph classification (molecule property, circuit quality), and graph generation (drug design). The readout function changes — per-node output for node classification, pair scoring for link prediction, global pooling for graph classification — but the message-passing backbone is the same. The drug-target binding task above is a graph-graph matching problem that uses GNN encoders on both graphs plus cross-attention for compatibility scoring.

Key points

• PinSage (Pinterest, 2018): 3B pins, 18B edges on a pin-board bipartite graph. Full-batch GCN impossible — infeasible at any node count near 3B. Solution: random walk-based neighborhood sampling — define each pin's neighborhood via L1-normalized random-walk visit counts (importance sampling), with production using a neighborhood size of T=50 most-visited pins. This captures second-order proximity and is more robust than uniform sampling because popular boards don't dominate the neighborhood — pins visited via multiple distinct short walks get higher weight than pins reachable through a single high-degree hub.
• PinSage scalability stack: offline embedding via MapReduce pipeline (GPU machines compute mini-batch embeddings, written to RocksDB); online serving via FAISS ANN on precomputed embeddings (< 10ms for top-1,000 similar pins); curriculum training starting with easy random negatives and progressively using semantically hard negatives from the embedding space. This infrastructure stack — not the GNN architecture — is what makes 3B-pin scale work. The architecture is 2-layer GraphSAGE.
• Fraud detection on transaction networks: nodes = users + merchants, edges = transactions with features (amount, time, merchant category, device ID). Key challenges: temporal causality (training must never use features from future timestamps), adversarial adaptation (fraudsters change patterns after detection), and ring structure as a fraud signal (fraudsters create A→B→C→A cycles). Structural features like betweenness centrality and cycle count often provide stronger signals than node content features alone.
• Drug discovery: atoms as nodes (atomic number, charge, hybridization), bonds as edges (bond type, aromaticity). Tasks: molecular property prediction, drug-target binding affinity, reaction yield. Datasets are small (1K–100K molecules). Solution: pretrain on large unlabeled molecular databases (ZINC, ChEMBL) then fine-tune — SSL pretraining provides the 10× labeled data reduction that makes molecular GNNs practical. Used in AlphaFold's structural inputs and property-prediction pipelines at major pharmaceutical companies.
• Dynamic graphs in production: most production graphs change continuously — new users sign up, transactions occur every second, friendships form and dissolve. Three approaches: (1) Snapshot-based — retrain or update a static GNN on graph snapshots at regular intervals; simple but misses inter-snapshot dynamics. (2) TGAT (Temporal Graph Attention): embeddings are functions of node features plus temporal encodings of event timestamps, Transformer-style. (3) TGN (Temporal Graph Network): nodes carry memory states updated by each new interaction, capturing long-term user behavior across batches. TGN is the production standard for real-time recommendation.
• Real-time GNN inference latency: a 2-layer GNN for a user with 1,000 connections, each with 1,000 connections, requires 1M feature lookups per inference. Done as sequential single-key lookups at sub-millisecond Redis latency each, that's minutes, not milliseconds — three orders of magnitude too slow for real-time ranking. Production solution: precompute and cache 1-hop aggregations nightly; update cache on new edges via event-driven invalidation. Decouple feature store freshness (updated every minute) from embedding freshness (recomputed every few hours). The two are different concerns with different latency requirements.
• Feature engineering often beats architecture improvements in practice. Structural features beyond node content: degree (in and out separately for directed graphs), clustering coefficient, PageRank or personalized PageRank, Node2Vec topology embeddings, temporal features (average edge age, edge creation rate in last 7/30/90 days). These can be precomputed and added as node features, giving the GNN access to higher-order structural information without adding depth — and without the over-smoothing risk that depth adds.
• Production serving infrastructure pattern: feature store (Redis/RocksDB, sub-millisecond lookup) + graph store (adjacency lists in distributed key-value store) + embedding store (FAISS index for ANN) + batch recompute pipeline (Spark + GPU workers for hourly/daily refresh) + event stream (Kafka for real-time edge additions, triggering embedding refresh for high-priority nodes). The GNN model is often deployed unchanged for months; embedding quality degrades more from stale graph data than from model staleness.

Recap

• GNNs beat feature engineering when structure carries the signal — molecules, proteins, social nets, circuits — learning representations end-to-end.
• PinSage is the canonical case study: 3B pins, 18B edges, <10ms serving. Innovations are engineering, not architecture (2-layer GraphSAGE).
• PinSage stack: random-walk importance sampling + MapReduce offline embeddings + FAISS ANN serving + curriculum hard negatives.
• Fraud detection: temporal causality (no future features), adversarial adaptation, ring structure ($A{\to}B{\to}C{\to}A$) as signal.
• Drug discovery: small labeled data → pretrain on ZINC/ChEMBL (SSL) then fine-tune → ~10× labeled-data reduction.
• Dynamic graphs: snapshots → TGAT (temporal encodings) → TGN (per-node memory) = production standard for real-time recsys.
• Production is a pipeline, not a model: feature store + graph store + embedding store + batch recompute + Kafka event stream; staleness hurts more than model age.

Check your understanding