PySpark · ML Systems Lab

Reading the Spark Execution DAG: The Diagnostic Skill Nobody Teaches

Every Spark performance problem is visible in the UI — if you know what to look for. Most engineers glance at the job page and see "completed." Senior engineers open the stages tab, find the task duration histogram, and know within 60 seconds whether the problem is skew, shuffle, or serialization. Here's the complete reading guide.

The Spark UI at port 4040 (or the History Server) tells you everything you need to debug a slow job. Most people only check if it completed. Here's how to actually read it.

The Jobs tab: your entry point

Each action (collect, write, count) triggers a job. Jobs are composed of stages, and stages are composed of tasks. The jobs tab shows you duration and whether any stages were skipped (cached). Start here — find the job that took longest.

The Stages tab: where the real diagnostic begins

Click into the slow job. You see its stages, each corresponding to a shuffle boundary (any wide transformation: groupBy, join, repartition, distinct). The stage table shows:

Duration: wall-clock time for the stage

Input/Output/Shuffle Read/Shuffle Write: data volume at each phase

Tasks: how many tasks ran, how many failed

The ratio that matters most: if Shuffle Read is 10x the Input size, you have a skewed join or explosion in cardinality. If Shuffle Write is large but Shuffle Read is small in the next stage, data is being generated and discarded — look for unnecessary explode() calls.

```python # Read key Spark UI metrics programmatically from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate() sc = spark.sparkContext

# After a job completes, check stage metrics status = sc.statusTracker() for job_id in status.getActiveJobIds() or status.getJobIdsForGroup(None): info = status.getJobInfo(job_id) for stage_id in info.stageIds: stage = status.getStageInfo(stage_id) if stage: print(f"Stage {stage_id}: tasks={stage.numActiveTasks}, " f"shuffle_read={stage.inputBytes}, " f"shuffle_write={stage.shuffleWriteBytes}")

# The two metrics that tell you the most: # 1. shuffleWriteBytes >> shuffleReadBytes → skewed partition # 2. max(taskDuration) / median(taskDuration) > 5 → data skew ```

The Task Duration Histogram: skew detector

Click into a stage and scroll to the task metrics. The task duration histogram is the single most useful chart in Spark debugging. It shows the distribution of time spent across all tasks.

A healthy histogram: roughly normal or uniform distribution. All tasks finish within 2x of each other.

A skew signature: most tasks finish in 0.2s, one task takes 120s. The job's total duration is dominated by that single slow task. This is data skew — one partition has far more data than the others. The fix: salting the join key or using skew hints (`spark.sql.autoBroadcastJoinThreshold`, `skewJoin` hint in Spark 3.x).

A stragglers signature: a long right tail with 3–5 tasks taking 3x the median. This is usually resource contention (noisy neighbour on the executor) or GC pressure. Look at the GC time column — if GC time > 10% of task duration, you have memory pressure.

The SQL tab: for DataFrame and SQL queries

The SQL tab shows the physical plan for each query. Here you can see:

BroadcastHashJoin vs SortMergeJoin: broadcast joins are fast (table fits in memory and is copied to each executor). SortMergeJoin requires shuffle on both sides — much more expensive. If you see SortMergeJoin on a small table, force a broadcast with `broadcast()` hint.

Exchange (shuffle): every Exchange node in the plan is a shuffle. Count them. Three joins between large tables = potentially 6 shuffles. Can any of them be avoided by co-partitioning? By denormalization?

Filter pushdown: good plans show filters pushed below joins. If your filter appears above the join in the plan, Spark is shuffling unneeded rows before filtering them.

The Storage tab: cache debugging

If you call .cache() or .persist(), the stored RDD/DataFrame appears here with memory and disk usage. Key checks: is the cached dataset fully materialized (fraction cached = 100%)? If it's 40% cached, the rest spills to disk — and your "cached" job is doing partial re-computation on every access.

The Environment tab: configuration audit

Check spark.executor.memory, spark.executor.cores, spark.sql.shuffle.partitions (default 200 — almost always wrong), spark.default.parallelism. If shuffle.partitions = 200 and your dataset has 10TB, each partition is 50GB and you'll OOM. Rule of thumb: target 100–200MB per partition, set shuffle.partitions accordingly.

The two-minute diagnostic

1. Jobs tab → find the slow job 2. Stages tab → find the stage with the highest duration, check Shuffle Read/Write ratio 3. Stage detail → task duration histogram → skewed? stragglers? uniform? 4. SQL tab → count Exchange nodes, check join types, verify filter pushdown 5. Storage tab → are your caches fully materialised?

After running this five times, you'll start catching Spark performance bugs in code review before they ever hit production.

Continue interactively

Read this post inside ML Systems Lab — with Simplify toggle, interview Q&As, inline glossary, and the MLE Path forward pointer.

Open in MSL →