Email race checks before flaky test retries

When an email test flakes, many teams jump straight to retries. I usually do the opposite. I treat the failure like a timing bug first and a framework problem second. That one habit has saved me more time than another retries: 2 ever did.

This comes up in signup, OTP, and reset-password coverage where the browser is fine, but the inbox arrives a little late, arrives twice, or gets matched by the wrong test. In bug reports I also keep seeing odd phrases like temp mailid or temp gamil com, which is a small clue that the team is already debugging from noisy evidence instead of clean test traces.

Why retries hide the real email race

Retries are not useless, but they are very good at masking root cause. A test that passes on the second run still consumed trust from the team. Google noted years ago that flaky tests create meaningful productivity drag across engineering orgs in its post on flaky tests at Google, and honestly that still feels true in everyday QA work.

The pattern I see most often looks like this:

• Test A creates an account and waits for any recent email.
• The worker is a bit slow, or a previous message is still visible.
• The assertion finds the wrong mail, or no mail, and fails.
• A retry runs with cleaner timing and suddenly "fixes" it.

Nothing was really fixed. The suite just got lucky, which is not the same thing at al.

If your team has already started building stable inbox contracts, this is easier to spot because the failure surface is smaller. If not, retries can make weak mailbox logic look acceptable for way too long.

The signals I check before touching retry counts

Before I change any retry setting, I want four signals:

1. The exact recipient or alias used by this scenario.
2. A correlation value tied to the business event.
3. Poll timing with attempt-by-attempt timestamps.
4. Enough artifacts to tell whether the app, worker, or test got out of sync.

That sounds simple, but many test harnesses only preserve the final timeout error. From there, people start guessing. They compare timestamps by hand, reopen CI videos, and debate whether the provider was slow or the selector was wrong. It gets messy fast.

I also like reading adjacent guidance such as review windows for disposable email checks, because the review mindset matters. An inbox assertion is not only a yes/no check. It is a small incident timeline, and the timeline should be inspectable.

A small harness pattern for Playwright and Cypress

The best fix is usually boring: make the mailbox wait narrower and make the evidence richer.

I prefer a helper that takes the scenario id, the recipient, the expected message type, and a timeout budget. The browser test calls that helper, but the helper owns the polling details. That keeps the framework code thin and the inbox logic consistent.

type WaitForMailInput = {
  recipient: string;
  scenarioId: string;
  subjectIncludes: string;
  timeoutMs: number;
};

async function waitForMail(input: WaitForMailInput) {
  const startedAt = Date.now();
  const attempts = [];

  while (Date.now() - startedAt < input.timeoutMs) {
    const messages = await inbox.list(input.recipient);
    const match = messages.find((message) =>
      message.subject.includes(input.subjectIncludes) &&
      message.headers["x-scenario-id"] === input.scenarioId
    );

    attempts.push({ at: new Date().toISOString(), count: messages.length });

    if (match) return { match, attempts };
    await delay(1500);
  }

  throw new Error(`mail timeout for ${input.recipient}`);
}

That same shape works in Playwright or Cypress because the framework is not the important part. The contract is. I want the helper to answer a precise question: "Did the email for this scenario arrive within the budget?" Not "Did some inbox somewhere look active?" Those are very diferent questions.

One subtle improvement helps a lot: log both the matched message and the near misses. If a test expected a verification email but saw two password-reset mails, that is gold for triage. It tells you the system is alive, just not aligned with the scenario.

A QA checklist for race-proof email tests

Here is the short checklist I use when a team says email tests are flaky:

1. Scope each scenario to a dedicated mailbox or alias.
2. Match on business intent, not newest-message order.
3. Record polling attempts and inbox counts through the full timeout.
4. Save a scenario id that both app logs and test logs can reference.
5. Keep retry counts low until the mailbox contract is clean.
6. Review the artifacts after the first fail, not after the fifth rerun.

If I cannot answer those six points from one CI job, I do not increase retries yet. I tighten observability first. That is usualy where the real win is.

Quick Q&A

Should I remove retries entirely?

No. Retries can still reduce random infrastructure noise. I just do not want them to be the first response to an inbox race.

Is this mostly a Playwright issue?

Not really. I see it in Cypress, API suites, and worker-level smoke tests too. Playwright just makes the sequencing errors easier to notice.

What is the first artifact worth adding?

An attempt log with timestamps, recipient, and matched subject lines. It is tiny, cheap, and surprsingly effective during triage.

Reliable email automation usually gets better when the team stops asking for more patience from the test runner and starts asking for better receipts from the mailbox layer. Once that evidence is clean, you can decide whether retries are helping or just hiding the bug for another week.