End-to-end tests that read real email, without a disposable-inbox service

By Jonas Birmé, Eyevinn Technology

Our signup flow mails you a six digit login code. That one design choice makes end-to-end testing awkward, because a browser automation suite cannot sign up for anything unless it can read a real inbox. Playwright will happily fill the email field. The code arrives somewhere else entirely.

The usual answer is a disposable-inbox API. You call an endpoint, you get a throwaway address, you poll for the message. We ran on one for about two years and it worked. But it was a second vendor, a second API key in CI, and a second thing that can be down while your tests go red for reasons that have nothing to do with your code.

Then it occurred to us that we were already paying for a mailbox. Our workspace on Open Source Cloud has one, with IMAP and SMTP, because every paid plan includes one. This post is how we moved the test suite onto it, and the one thing that bit us on the way.

One mailbox, unlimited addresses

The obvious objection is that a test suite needs a fresh address per test, and a workspace mailbox is one address. Two tests sharing an inbox will read each other's mail, and a signup flow usually refuses an address that already has an account.

Sub-addressing solves this. Mail sent to workspace+anything@users.osaas.io is delivered to the workspace@users.osaas.io inbox, and the tag is yours to choose at the moment you need it. So a test generates workspace+e2e-k3f9x2p1@users.osaas.io, the signup flow treats it as a brand new user because it is a string nobody has used before, and the mail lands in a mailbox you already control. There is no API call, provisioning step or per-inbox cost to deal with.

Two settings have to be right on the mail server for this to work, and they are worth knowing by name even if someone else operates it:

The second setting is the easy one to miss. If it is set to yes, mail to +e2e-k3f9x2p1 quietly lands in a folder named after the tag, your IMAP search of INBOX finds nothing, and your test times out with no clue why. The OSC mailbox has both set the way you want.

Check the address scheme before writing any code

Before writing a line of IMAP, confirm the server actually accepts the addresses you plan to invent. You can do that without sending anything: open an SMTP conversation, name a recipient, then hang up before the message body.

import smtplib
s = smtplib.SMTP('mail.osaas.io', 25, timeout=30)
s.ehlo('probe.example.com')
s.mail('you@example.com')
print(s.rcpt('workspace+e2e-probe@users.osaas.io'))  # (250, b'2.1.5 Ok')
print(s.rcpt('nosuchworkspace+x@users.osaas.io'))    # (550, b'... User unknown ...')
s.quit()

A 250 on the first and a 550 on the second shows that tag stripping works and that the server really does reject unknown users. No mail is delivered, because you never send DATA. This took about a minute and saved building on an assumption.

The client

The whole thing is one file. Connection details for an OSC workspace mailbox are IMAP on mail.osaas.io:993 over TLS, and the credentials are your workspace address plus the password you set when you created the mailbox. In CI those go in two secrets.

Allocating an address needs no network call at all:

function randomTag() {
  const alphabet = 'abcdefghijklmnopqrstuvwxyz0123456789';
  let s = '';
  for (let i = 0; i < 8; i += 1) {
    s += alphabet[Math.floor(Math.random() * alphabet.length)];
  }
  return `e2e-${s}`;
}

export function setupMailInbox() {
  const tag = randomTag();
  return { tag, userEmail: `${LOCAL_PART}+${tag}@${DOMAIN}` };
}

Waiting for the mail is an IMAP search on the recipient header, polled until a subject matches:

const uids = await client.search({ to: addressFor(tag), seen: false }, { uid: true });

The seen: false filter matters, as the problem below showed.

The catch: hand out each message exactly once

The disposable-inbox client we replaced had a "wait for latest email" call that marked each message read as it returned it. We did not think about that, because why would you. Our first version searched all mail for the tag and returned the first match.

Every test passed except one: sign up, then sign out, then sign back in. It failed with error=Verification, which is what you get for submitting an expired or already-used login code. The second wait was finding the first code, still sitting in the inbox, and submitting it. The app was right to reject it.

The fix is to search unread mail only, take the newest match rather than the first, and mark it read before returning it. That reproduces the behaviour we had been getting for free and never noticed:

const match = messages
  .filter((m) => subjectRegex.test(m.envelope?.subject || ''))
  .sort((a, b) => b.uid - a.uid)[0];
if (match) {
  await client.messageFlagsAdd([match.uid], ['\\Seen'], { uid: true });
}

If you migrate off a hosted inbox API, go read what its fetch call did to message state. That behaviour is part of the contract even though it is nowhere in your code.

What it looks like now

The full UI suite after the switch: 187 passed, 2 flaky, 0 failed. The run before it, on the old provider, was 178 passed, 8 flaky, 2 failed. The two failures in the earlier run were a service-page spec and a usage-dashboard spec, neither of which touches email, so I would not read much into the difference, except that nothing got worse.

Delivery is quick. The mail server logs the gap between accepting a message and filing it, and across a full run that sits between 0.18 and 1.1 seconds. We had previously measured the old provider end to end at anywhere from 4 to 45.6 seconds, which is not the same measurement, so treat it as a rough comparison rather than a benchmark. The practical effect is real though: our test timeouts were sized for a 45 second worst case that we have not seen since.

Three smaller notes:

Keep one IMAP connection per worker process rather than one per call. Dovecot limits concurrent connections per user and address, commonly to ten, and a suite that opens a connection every time it waits for mail will sit uncomfortably close to that.

Check the Junk folder in your timeout path. Our login mail scores as clean, but a spam verdict would otherwise look exactly like mail that never arrived, and you will spend an afternoon on it.

Prune the mailbox. Teardown deletes the messages for its own tag, so a run that is cancelled or times out leaves its mail behind forever. Nothing collapses, since the mailbox has no quota, but your IMAP searches get slower as the inbox grows.

What changed

The client is about 270 lines, a generous share of them comments, running against a mailbox that already existed. The shape of the dependency is what changed: the login codes our tests read now arrive on infrastructure we already run, rather than through a third party whose uptime we do not control and whose invoice we were paying mainly to receive a few hundred emails a week.

If you are already running services on Open Source Cloud, the mailbox is under your workspace settings.