For our Q3 2026 AI Model Leaderboard, we asked DeepSeek-V4-Flash, claude-sonnet-5, and gpt-5.6-terra to generate playbooks for the same three tasks, a simple Nginx site, a ten-switch Cisco Catalyst rollout, and a full Ansible 2.4-to-2.14 upgrade, and then ran everything they produced through Steampunk Spotter, our governance and guardrail platform for AI-generated Ansible and Terraform code.
The result: 140 errors, 94 warnings, and 221 hints across nine runs.
Capability didn’t prevent mistakes
The most consistent problem across all three scenarios came from claude-sonnet-5. It kept writing short module names instead of fully qualified collection names (FQCN) — 12 times in the easiest scenario, 17 times in the mid-complexity scenario, and 27 times in the hardest one.
That gap held even in the scenario where the prompt explicitly named Ansible 2.14 as the target version, a version where FQCN is the expected standard. The model still made the mistake. Being a capable model didn’t make it a reliable one on this specific pattern.
One issue no model could avoid
All three models triggered the same warning in the Ansible upgrade scenario: a changed parameter default between Ansible versions. This wasn’t a model-specific failure. None of these models are purpose-built for Ansible, so none of them track every collection, module, and version-specific interface that exists. Argument specs change over time, and a model trained before a given change simply doesn’t know about it.
That’s a limitation worth planning around, not just noting. Teams running AI-assisted version upgrades should expect this class of warning regardless of which model they use.
Why this matters for your pipeline
Every model-and-scenario combination in this test came back with errors, warnings, or hints. That’s a strong case for a deterministic review step between AI-generated Ansible code and anything that touches production, because model output is probabilistic. The same prompt can produce a clean playbook one run and a flawed one the next. A scanner that catches the same error pattern every time closes that gap; a well-worded prompt only reduces the odds.
This report didn’t test against organizational compliance or security standards, and we didn’t rerun any generations to average out variance. Both choices likely made the results look better than what most teams will see in practice. We’re building both into the next quarterly report.
Get the full breakdown
This post covers the headline numbers. The full report has the complete scenario-by-scenario error tables, the exact prompts used for each test, and the appendix breakdown by error code.
Download the full Q3 2026 AI Model Leaderboard report
Frequently Asked Questions
Which AI model produced the most Ansible errors in this report? claude-sonnet-5 had the highest error count in two of the three scenarios, driven mostly by missing fully qualified collection names (FQCN).
What is FQCN and why does it matter in Ansible playbooks? FQCN stands for fully qualified collection name — using the full module path (like ansible.builtin.copy) instead of a short name (like copy). It’s the expected standard in modern Ansible versions, and Spotter flags it as a deterministic, fixable error.
Can better prompting fix AI-generated Ansible errors? Not really. A prompt instruction can reduce how often certain errors occur, but model output is probabilistic — mistakes can still happen at random, even when the instruction is explicit. Scenario 003 showed this directly: the prompt named Ansible 2.14 by name, and the FQCN errors still showed up.
How does Steampunk Spotter compare to relying on model choice alone? Spotter catches errors, warnings, and risky patterns deterministically, every time, regardless of which model generated the code or how the prompt was worded. Model choice reduces risk; it doesn’t eliminate it.
