11.15 — Parallelism, retries & flakiness
Two hundred browser tests, run one at a time: twenty-seven minutes. The machinery of scale — workers, sharding, retries — cuts that to minutes. And its price is the topic every team fights about: FLAKINESS. Today: the machinery, the honest retry debate, and the clinic — four causes of flakes, each with a cure you’ve already learned.
workers: 4, // 4 tests at once fullyParallel: true, retries: process.env.CI ? 2 : 0, // CI splits across MACHINES: // $ npx playwright test --shard=1/4 // the report’s honesty: // ✓ checkout works // ⚠ cart total updates (flaky: passed on retry) // ✗ profile saves // reproduce a flake on purpose: // $ npx playwright test cart.spec --repeat-each=20
The serial baseline: 200 E2E tests × 8s ≈ 27 minutes. But look INSIDE each 8 seconds: the browser mostly WAITS — for the network (6.7), for rendering (7.8), for auto-waiting polls (11.6). Waiting doesn’t need the whole machine. You’ve seen this shape before…
The deeper story, with the real names for things — this part is what turns “I saw it” into “I can explain it.”
Worker mechanics worth knowing: tests from the SAME file run in one worker by default (fullyParallel: true loosens this); a worker that crashes is replaced and its tests re-queued; and workers: 1 on CI (11.3’s line) exists because shared CI boxes are slower and noisier — determinism beats speed where evidence quality matters.
Worker-scoped fixtures (11.7’s scope option) build once per worker — a database connection, a seeded catalog. With 4 workers you get 4 of them; design them to coexist (unique schemas/accounts per worker — testInfo.workerIndex is the standard salt, echoing 11.8).
Sharding’s honest cost: the report fragments across machines. The fix is merging — playwright merge-reports combines shard blobs into one HTML report before upload (11.16’s artifact step often does exactly this). One suite, one verdict, one report — whatever the machine count.
Job note: “how do you deal with flaky tests?” is the interview question in this domain — asked everywhere, answered badly by most. Your answer now has four moves: prevention (the clinic’s cures, by default), detection (the flaky mark + repeat-each), diagnosis (trace the failing run, name the cause), and policy (quarantine + P1 fix, zero tolerance, because trust is the suite’s only asset). Four moves, no hand-waving — that’s a hire.
⌨️ build the flake detector
Given repeat-run results, classify each test the way the report does: stable, broken, or the dangerous third thing — with the failure rate computed as evidence.
requirements:
- Write
classify(results)for an array of"pass"/"fail"strings: every one passing →"stable"; every one failing →"broken"; a mix →"FLAKY — investigate"(4.10’s every asks both questions). - Write
failRate(results):N of M runs failed, counted with a filter. - Run three histories: ten passes → print its class; ten fails → print its class; and
["pass","fail","pass","pass","fail","pass","pass","pass","pass","pass"]→ print its class AND its fail rate.
when you press RUN, the console must show exactly:
✏️ Quick check 1
A test fails, retries, and passes. What does the report mark it as?
✏️ Quick check 2
Why is it safe for 4 workers to run 4 tests simultaneously? (what guarantees no collisions — one phrase)
✏️ Quick check 3
A test flakes only when run with others, and its trace shows another test’s data in its cart. Which clinic cause?
🗣️ Now teach it back
Give the full scale-and-flakiness briefing: why workers speed things up (the 9.6 connection), what makes parallelism safe, the retry debate honestly (both edges + the mark), the four clinic causes with their cures, and the zero-flake policy’s reasoning.
Write it as if your friend is sitting next to you. Saved to your journal — future-you will use these notes to teach others.