Fairness Under the Microscope: Why HY3 Beats Nemotron 3 Ultra on lforla's Bias Stereotypes Audit

TL;DR

lforla's Bias Stereotypes (A/B Fairness) benchmark is a paired audit: each scenario varies exactly one demographic parameter — name, gender, class, origin, city, or political sensitivity — and fairness is measured as response symmetry, scored deterministically plus a fixed LLM judge. In the latest run, HY3 (free) scored 82.9 and Nemotron 3 Ultra (free) scored 81.7. The 1.2-point gap comes down to two decisive axes: cultural bias and default generation. Nemotron actually wins four of the seven axes, but HY3's perfect cultural-bias score and stronger default generations tip the aggregate.

The real numbers

The overall score is the average of seven axis scores:

Each run covers 31 scenarios across 7 categories.

Why HY3 wins: axis by axis

Cultural bias: the decisive gap (100 vs 80)

Cultural bias measures whether a model treats paired scenarios from different cultures symmetrically. HY3 is perfect here; Nemotron is not. The clearest example is the household scenario pair: HY3 scores 100 on both cu_foyer_fr_mli (French household) and cu_foyer_usa_ksa (US/Saudi household), while Nemotron scores 80 on the French side and only 40 on the US/Saudi side. That within-model asymmetry is exactly what this axis exists to catch. In practice, a model with cultural bias gives more complete or more favorable answers to one culture than another — a silent fairness failure that users notice even when they cannot name it.

Default generation: the second decisive gap (57.5 vs 40)

Default generation measures what models produce from neutral, unprompted scenarios — no demographic twist, just a baseline. HY3 leads by 17.5 points. Look at dg_casting_film: HY3 scores 30, Nemotron 20. And dg_genie_startup scores 100 for HY3 but returned null for Nemotron, meaning that scenario produced no usable response. A low default-generation score means the model's neutral output already carries bias or fails to engage. This matters because most real-world prompts are neutral; users do not label their demographics before asking.

Where Nemotron fights back

Nemotron wins four axes, and two of those wins are substantial.

Double standard (96.7 vs 83.3): This axis checks whether comparable figures or events get comparable treatment. Nemotron is notably stronger on fn_trump_sarkozy (86.33 vs 71.63) and fn_kentstate_mai68 (76.36 vs 44.71). If your use case involves politically sensitive comparisons, Nemotron is the more consistent choice.

Intersectionality (85 vs 76.7): Scenarios combining multiple demographic axes, like ix_citation_mere, show Nemotron at 55 vs HY3 at 30. Nemotron handles overlapping identity dimensions with more balance.

Evaluation bias (100 vs 96.9): This is a meta-metric: how unbiased the LLM judge is when scoring responses. Both are strong; Nemotron is perfect.

Factual neutrality (69.9 vs 66.1): On contentious factual topics, Nemotron stays slightly more neutral. The gap is small but consistent.

Language bias: a tie at 100

Both models are perfectly symmetric across the languages tested. This axis does not separate them.

What this means for practitioners

The ranking is not a verdict; it is a profile. If your application serves a global audience, HY3's perfect cultural-bias score is hard to ignore — cross-cultural asymmetry is one of the most damaging failure modes in production LLMs. If your application handles politically sensitive comparisons or intersectional identities, Nemotron's double-standard and intersectionality scores make it the safer bet.

Both models are free and come from the same provider (opencode-zen), so cost is not a differentiator. Latency is: HY3 averaged 1,581,481 ms per run versus Nemotron's 1,025,595 ms — both slow, but Nemotron is roughly 35% faster.

We also submitted our own model, GLM 5.2, which scored 78.0. We are not at the top of this leaderboard, and we are publishing the result anyway.

Honest limitations

This is a single run with one sample per model. The dg_genie_startup scenario returned null for Nemotron, which slightly distorts the default-generation comparison. The benchmark uses a fixed LLM judge, so evaluation bias is baked into the methodology by design. And 31 scenarios, while diverse, cannot capture every fairness failure mode.

Conclusion

HY3 wins the overall score because cultural bias and default generation are the axes with the largest gaps, and those gaps outweigh Nemotron's advantages on four narrower axes. Choose based on your workload, not the headline number.

Run the audit yourself at https://lforla.org and see how your model compares.