Building a Reproducible Test Harness for Random Letter Output

When a feature depends on drawing from the alphabet, "random" stops being a vibe and starts being a contract. QA needs to replay bugs, security review needs to confirm the pool is not biased, and localization needs to know which letters can ever appear. This is a walkthrough for engineers who have been handed a requirement like "users can roll a random letter" and now have to make that promise testable.

We'll treat letter generation as a small system: inputs, outputs, the function under test, and the harness that proves it behaves. The same shape applies whether you write a randomLetter() helper in JavaScript, a pick_letter() function in Python, or a server endpoint that returns one character.

What "random letter" actually means in production

Three decisions shape every test you will ever write:

1. Pool definition. A-Z? a-z? A-z combined? Include Unicode scripts? If the spec doesn't pin this, write the pin down first. Wikipedia's English alphabet entry is a quick anchor for the 26-letter baseline, but every project that ships internationally eventually extends it.
2. Distribution. Uniform over the pool, or weighted (a la Scrabble tile frequencies)? Uniform is the default assumption; weighted is a separate feature with its own test set.
3. Source of entropy. Math.random() is fine for game UI. Anything security-sensitive — captchas, password salts, token fragments — must use a cryptographically secure generator or an equivalent. The two never share a code path, and your tests should never let them silently merge.

Get those three wrong and the function "works" until a reviewer asks, "How do you know?"

A minimal contract to write down before you code

Paste this into the ticket. If nobody can fill it in, the task isn't scoped.

• Input: none (or an explicit options object with pool, weighted, count)
• Output: a single Unicode code point from the declared pool, or null/undefined on impossible inputs
• Invariants:
  • Output is always a member of the declared pool
  • No output value appears more than its weight cap (if weighted)
  • No side effects on global RNG state when seedable is true
• Failure modes: empty pool, malformed pool string, non-finite count

That list becomes the acceptance criteria. It also becomes the table of contents for the test file.

Building the harness: shape it like a property test, even if you don't pull in a framework

Property-based testing shines here. The classic library for JVM and Python-style languages is Hypothesis, and the pattern is portable: instead of asserting on specific outputs, assert on what must always hold for any input drawn from a generator.

Three properties get you 80% of value:

1. Membership. For any call, the result must be in the pool.
2. Idempotence under fixed seed. Two runs with the same seed produce the same sequence.
3. Distribution. Over N draws with M unique values, each value's observed frequency is within ε of its expected frequency, where ε shrinks with N.

You don't need Hypothesis to write these. A for (let i = 0; i < 10000; i++) loop with a histogram does the job. What you do need is the discipline: write the property first, then the implementation that should satisfy it.

Reproducibility: seeding, snapshots, and why golden files are dangerous here

A seeded RNG is the cheapest way to make a flaky test stop flaking. The pattern:

seed = 0xDEADBEEF
rng = seeded(seed)
result = drawLetter(rng)
assertEquals("frozen-output", result)

Two things break this pattern and you should know them up front.

First, language changes break golden files. If you upgrade a runtime that changes its Math.random() algorithm — and major engines have done this — your "frozen output" is now stale. Either pin the engine version in CI or capture the seed plus the algorithm name in the snapshot metadata.

Second, golden files hide regressions in the RNG, not the function. A subtle bug that shifts the distribution slightly will still produce the same character at the same seed. Pair every seeded test with a statistical test on a long sequence.

A checklist you can paste into a PR template

Add this to your contribution guide for any "random something" feature:

• [ ] Pool is defined as an explicit string or array constant, not inferred
• [ ] Functions accept an optional rng parameter (defaulting to the global one)
• [ ] At least one property test asserts pool membership over 10k draws
• [ ] At least one statistical test asserts distribution within ±2% of expected
• [ ] A seeded test pins at least one output so replay is possible
• [ ] If the function ever handles user-visible security, it uses the platform CSPRNG and that path is covered by a separate suite
• [ ] README states the pool, the distribution, and the RNG source in plain English

If a reviewer can tick those boxes without opening the implementation, the harness is doing its job.

Where a small UI tool fits (and where it doesn't)

A browser-side letter generator is the right level of tooling when the goal is exploration, not validation. Useful cases:

• Manually eyeballing whether a Unicode pool contains what you think it contains
• Sketching a probability intuition before you wire up a real test
• Demos and screenshots where the value matters less than the feeling of randomness

Less useful: anywhere the output will be committed, asserted on, or shipped. For that, write the function, write the test, and pin the seed. If you want a quick reference on how the underlying sampling logic looks in Python — including random.choice, secrets.choice, and the IndexedDB-style picking pattern — the Lizely guide on generating random characters in Python walks through three implementations side by side.

Common production traps and how to dodge them

Trapping Math.random behavior with .sort(shuffle) tricks. The classic Fisher–Yates shuffle does work, but the arr.sort(() => Math.random() - 0.5) shortcut is empirically biased. Don't ship it, and don't ship a test that only checks for duplicates — it will pass while the distribution is wrong.

Leaking the RNG across request boundaries. On a server, a module-level rng = MersenneTwister() shared across requests means two simultaneous users can predict each other's draws. Use a per-request seed, or use the platform CSPRNG, which is reseeded by the kernel.

Assuming "letter" means ASCII A-Z. The moment a user pastes an accented character or your product expands to Spanish, French, or German, "letter" mutates. Test with Unicode-aware regex (\p{Letter} with the u flag) and decide explicitly which scripts are in scope.

Logging the "random" value in plaintext alongside a user identifier. This is a fingerprint leak. Aggregate, don't store.

Frequently asked questions

How many draws do I need for a meaningful distribution test?

A rule of thumb: with a pool of size k and a target confidence of 99%, plan on roughly 100 * k draws for the chi-squared test to have acceptable power. For A-Z, that's ~2,600 draws — cheap. For Unicode pools of thousands of code points, scale linearly.

Should I use a property-based testing library or roll my own loop?

If your team already uses one (Hypothesis, fast-check, jqwik), use it. The libraries give you shrinking for free, which means failing tests come with the smallest input that reproduces the bug. If you're bootstrapping a new repo and the team is small, plain loops with histograms are absolutely fine — the value is in writing the property, not in the framework.

What do I do when QA reports a bug I can't reproduce?

First, check whether the function takes an RNG parameter. If it doesn't, fix that — non-injectable randomness is a refactor blocker, not a bug. Then ask QA for the seed, the call site, and the pool configuration. If the function didn't capture those, you have a harness gap, not a code gap.

Can a browser tool ever be authoritative for a production test?

No. Browser tools are fine for sanity checks, but the test that ships to CI must run in a deterministic environment you control: pinned runtime, pinned dependency versions, and a seeded RNG. The tool belongs in your local exploration; the test belongs in your pipeline.

This article was drafted with AI assistance and reviewed for technical accuracy before publishing.