A Flaky Playwright Hover Test Was a Real Layout Shift (Flake Triage Ladder)

A Playwright hover test on this site failed about one run in four. The usual move is a retry, a waitForTimeout(500), or a note in the config that says "flaky on busy machines". I had all three available and used none of them, because the fourth run in a row of the same failure is not noise. It is a sensor.

This is the ladder I climbed to find what it was sensing, in the order that actually produced evidence. The bottom rung took two commands. The top rung was a real layout shift on the production page.

Last verified Aug 29, 2026 against Playwright 1.61, Chromium 149 headless shell, Next.js 16.2. Version-specific details drift — check the vendor docs before relying on an exact flag or limit.

Why retries and sleeps fail as a fix

retries: 2 and waitForTimeout(500) both change the same thing: how often the test observes the bug. Neither changes whether the bug exists.

They also cost you the only evidence you have. A test that passes on retry is logged as "flaky", and flaky is a bucket nobody opens. Six weeks later the same mechanism breaks a different test, and the bucket is bigger.

The test in question checked that hovering an open-source project card lights up a ::after registration mark. Hover, poll for opacity: 1, assert. Simple, and correct: when it failed, the mark really had not lit.

By the end you will have

1. The Flake Triage Ladder: five rungs, each with the command that produces evidence and the decision it lets you make.
2. A reading of Playwright's own actionability log, so "not stable" and "not visible" send you to different places.
3. A probe pattern that replays the test's steps and dumps pointer position versus element position on failure.
4. Two page-level fixes that removed the flake without weakening the assertion.

Each rung costs less than the theory it replaces

Rung 1: whose server is that

Local Playwright configs commonly carry reuseExistingServer: !process.env.CI. That means the suite trusts whatever answers on the port. Twice on this machine that was another project's next dev. The failures looked like real assertions (missing headings, 404 redirects) and were nothing of the kind.

lsof -nP -iTCP:3000 -sTCP:LISTEN

If the process is not yours, run your suite on another port. Do not kill it; it is somebody's live work.

Rung 2: does it fail alone, and does it fail clean

pnpm exec playwright test e2e/public-funnel.spec.ts -g "specimen hover" --workers=1 --retries 0
git stash && pnpm exec playwright test ... ; git stash pop

Both answers were "yes, sometimes". So the failure was not contention between workers and not my change. That eliminated the two most popular theories in one minute.

Rung 3: ask Playwright what it did

DEBUG=pw:api pnpm exec playwright test ... --repeat-each 6 --retries 0 2>&1 \
  | grep -E 'hover|scroll|retrying|stable' | sort | uniq -c

Six runs, three failures. The failing runs all contained the pair element is not stable and retrying hover action; the passing runs did not. Playwright checks that an element's bounding box holds still across two frames before it acts. When it does not, Playwright retries, and the retry scrolls the element into view again with a different alignment.

That is the mechanism half of the answer: a retry that scrolls.

Rung 4: replay the steps and record the geometry

A short script that does exactly what the test does, in a loop, and on failure prints the card's rectangle before and after, the scroll position, and the :hover chain.

run 8: FAIL opacity=0
  rectBefore  y=367  scrollY=3996
  rectAfter   y=658  scrollY=3696
  mouse       y=538
  hovered     [..., a.oss-card(neighbour), div.flex, p.line-clamp-2]

The card did not move in the document. The page scrolled 300px after the pointer was placed, so the pointer, still at y=538, now sat on the card below. 658 = 1000 - 341 is a block: "end" alignment, which is what Playwright's retry uses.

Why did the scroll keep going after the pointer landed? The site set scroll-behavior: smooth on html, unconditionally. The retry's scroll animated.

Rung 5: why was the card unstable in the first place

Because the page under it was moving. The work section sits in a content-visibility: auto chunk whose size estimate was 800px against a rendered 1756px, and the funnel strip below it was estimated at double its real height. Each activation reflowed the document. The full measurement is its own note: content-visibility shifts your page until you measure the estimates.

Magnet: Flake Triage Ladder

You should see: each rung ends with a decision, not a theory. If you reach rung 5 with a pointer-versus-element mismatch, the bug is on the page, and fixing the test alone would hide it.

The fixes were on the page, and one precondition in the test

1. Measured contain-intrinsic-size estimates per chunk, so activation no longer reflows the page. This is the root cause.
2. scroll-behavior: smooth only under prefers-reduced-motion: no-preference. It was an accessibility miss on its own (WCAG 2.3.3), and it was the mechanism that let a retry's scroll outrun the pointer.
3. In the test, wait for window.scrollY to hold across two frames before hovering. The card rides a view() scroll timeline, so it is at rest exactly when the scroll is. That is a precondition the test was violating, not a sleep.

Result: 16 of 16 runs with retries off, across both projects, on the same machine that failed one in four.

Failure modes

The comfort retry. retries: 2 in the config makes the dashboard green and deletes the signal.

The wrong precondition. My first attempt waited for getAnimations() on the card's ancestors to be empty. A scroll-linked animation is reported as running forever, so the poll never passed. Know what kind of animation you are waiting on.

Fixing the estimate on the wrong chunk. My first measurement mislabelled two chunks and I "fixed" the one that was already right. Print the chunk id next to every number.

Trusting one clean run. A clean-tree pass proves nothing about a one-in-four failure. Run it enough times to see the rate.

The test that was right. Under the shift, the assertion opacity === "1" was false for a real reason. Keep it.

When not to climb the ladder

A test that has failed once in the history of the repo does not need the ladder. Re-run it, and if it passes, write down the date. Two failures is a pattern.

Do not do it for tests that assert on third-party timing you do not control (an embed, a font CDN). Mock the dependency or delete the assertion.

Nor is it a reason to skip --workers=1 in CI on a two-core runner. Contention flakes are real; they are just a different rung.

A flaky test is a measurement with a wide error bar, not a broken instrument

Narrow the error bar and the measurement is usually right.

If the thing under test is an agent's output rather than a hover, the same rule holds: retrying the agent until it passes is how self-certified completion happens. The six-gate control model separates producing evidence from judging it for exactly this reason.

Your next action: pick the test your team calls flaky and take it down the Flake Triage Ladder with retries off; stop at the first rung that ends in a decision.