Why Adding Spark Nodes Often Doesn't Fix a Slow Job: Check Partitions and Skew First

A Spark job is slow, so someone doubles the cluster. The job is still slow, and now it costs twice as much. If that sounds familiar, cluster size was probably never the constraint. This guide covers the checks worth running before approving more nodes, and it separates documented Spark behavior from judgment calls.

The deciding question is simple: can the stage that is slow actually spread its work across more cores? If it cannot, extra nodes are a tax rather than a fix.

A stage is as slow as its slowest task

Spark runs one task per partition, and cores only help if there are enough similarly sized tasks to occupy them. The official tuning guide says a cluster will not be fully used unless parallelism is high enough for each operation, and it suggests two to three tasks per CPU core as a starting point.

That guidance is a floor for utilization. It says nothing about how evenly the work is spread across tasks. A stage with 800 tasks can still be dominated by one task holding most of the data, because a stage finishes only when its last task does. More nodes make the other 799 tasks finish sooner and leave the wall-clock time unchanged. This is the mechanism behind many "more nodes didn't help" outcomes.

Before touching the cluster, sort the symptom into one of three cases:

Case	What you see	What helps
Too few partitions	Idle executors during the stage	A higher partition count, not more hardware
Too many tiny partitions	Scheduling overhead dominates; many small output files	Fewer, larger partitions (coalescing)
A few oversized partitions	Straggler tasks, spills or out-of-memory errors on specific tasks	Fixing the distribution (skew handling, key design)

Only the first case benefits from more parallelism, and even there the fix is usually a setting, not machines.

The 200-partition default is a guess

For SQL and DataFrame shuffles, the knob is spark.sql.shuffle.partitions, and the performance tuning documentation lists its default as 200. That number is static: it does not know whether a shuffle moves a few megabytes or a few terabytes, and either extreme goes wrong in its own way. Too many partitions on a small shuffle produce near-empty tasks. Too few on a very large one produce big tasks that spill or run out of memory.

The tuning guide describes the memory side directly: a single reduce task's working set can be too large even when the whole dataset fits. Its stated remedy is to raise parallelism so each task's input is smaller. That is a partitioning change, although it can also mean you need more capacity afterward.

Adaptive Query Execution (AQE) is meant to take most of this off your hands. It has been enabled by default since Spark 3.2.0, and when post-shuffle coalescing is on, Spark merges small shuffle partitions at runtime. The starting partition count for coalescing equals spark.sql.shuffle.partitions unless you set an initial number, which is why the usual advice is to leave the starting count generous and let AQE shrink it. Coalescing behavior also depends on related settings, such as whether it favors parallelism over the advisory partition size, so read the documentation for your version before assuming a target size.

Skew detection has two gates

AQE's skew handling is real, but its trigger is narrower than many people assume. A partition is treated as skewed only if it exceeds a factor (default 5) times the median partition size and also exceeds an absolute byte threshold (default 256 MB). Both conditions must hold. In practice:

• A partition that is 40 times the median but only 200 MB is not flagged, because it fails the absolute gate. It can still be the straggler.
• If most partitions are already large, the median is high, so the relative gate may never trip for a genuinely hot key.
• The documentation describes skew join handling for shuffle joins. Skew in a plain aggregation is a different case, so confirm what your Spark version covers before relying on it.

The defaults are reasonable. The point is that "AQE is on" does not mean "skew is handled", and the only way to know is to look at actual partition sizes.

Measure before changing any setting

Two measurements matter: how rows are distributed across partitions at the slow stage, and how rows are distributed across the join or grouping key. This PySpark snippet does both with standard DataFrame APIs. The names are placeholders.

from pyspark.sql import functions as F

# 1) Row counts per partition, on the DataFrame just before the slow stage
(df
 .groupBy(F.spark_partition_id().alias("pid"))
 .count()
 .orderBy(F.desc("count"))
 .show(10))

# 2) Heaviest keys on the column you join or group by
(df
 .groupBy("join_key")
 .count()
 .orderBy(F.desc("count"))
 .show(10))

The first query shows whether work is evenly spread across partitions. Row counts are only a proxy, since wide rows can make byte sizes diverge, so cross-check against the per-task input size in the Spark UI. On the stage page, the summary metrics for completed tasks show minimum, median and maximum duration, and a large gap between median and maximum is the classic straggler signature.

The second query shows whether a handful of keys carry most of the volume. If the top key is far ahead of the rest, partition tuning will not fix it, because every row with that key lands in one partition. The remedy lives in the join or the data model: isolate the hot key, salt it, or rethink the grouping.

If the distribution is flat and tasks are simply too few or too large, the AQE settings below are where to start. The values are the documented defaults, so treat them as a baseline to measure against, not a recommendation:

spark.sql.adaptive.enabled=true
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.advisoryPartitionSizeInBytes=64m
spark.sql.adaptive.skewJoin.enabled=true
spark.sql.adaptive.skewJoin.skewedPartitionFactor=5
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=256m

One interaction deserves care. Skewed partitions are split toward the advisory partition size, so if you raise that size, review the skew threshold alongside it. This is reasoning from how the settings relate, not a documented rule, so test it on your own workload.

Make it a platform contract, not per-job tuning

When every team sets its own shuffle partition counts, behavior is inconsistent and there is no shared baseline. A better contract looks like this:

1. Pin the Spark version and review these defaults on every upgrade. Defaults and AQE behavior have changed across versions, so documentation for one version can mislead on another.
2. Record per-stage task count, median task time and maximum task time for production jobs. A wide gap between maximum and median is a skew alarm, and it is cheaper to alert on than to rediscover during an incident.
3. Require the two diagnostic queries in the review of any change justified by "needs a bigger cluster".
4. Treat a hot key as a data-model question for the owner of the data, not a tuning problem to hide inside the job.

The third item is a process rule more than a technical one. It costs a few minutes and sometimes removes a recurring bill.

Where to stay cautious

The figures above are documented defaults, not benchmarks, and none come from a specific pipeline. The two-to-three tasks per core guidance is a starting point, and sources differ on ideal partition sizes, so measure your own stages. Partitioning is also only one cause among several: slow storage reads, an expensive UDF or an oversized broadcast can look similar from a distance. Confirm the diagnosis in the Spark UI stage view before committing to a fix.

A sensible order of work

1. Find the slow stage in the Spark UI and compare its maximum task time to its median.
2. Run the partition and key distribution queries.
3. If tasks are even and cores are busy, scaling out is a reasonable bet.
4. If tasks are too few or too large, adjust partitioning and let AQE coalesce.
5. If one task carries the stage, address the hot key in the join or the data model.
6. Record the before and after numbers so the next review starts from data.

If one task is carrying the stage, extra nodes buy capacity that has nothing to do. Find the straggler first, then decide whether hardware is the answer.