This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Third entry in the DEV x Sentry Bug Smash. Entry 1 was a crash with a confusing message. Entry 2 was a freeze the timeout could not catch. This one is quieter and sneakier: valid Python that the sandbox rejects with an error pointing at the wrong thing entirely.
When the open issues run out, fuzz
By entry 3 every obvious open smolagents bug was already claimed or had a competing PR. So instead of reading the issue tracker I pointed a small fuzzer at the piece of smolagents that runs the most untrusted code: LocalPythonExecutor, the sandbox that executes model-generated Python.
The method is boring and effective: feed it ordinary, valid Python one snippet at a time, and flag anything that raises InterpreterError. Valid Python that the sandbox refuses to run is, by definition, a bug, because the model writes valid Python and expects it to work.
That surfaced four unreported bugs in one afternoon. This post is about the one I shipped: dict unpacking.
The bug
config = {**{"temperature": 0.7, "max_tokens": 512}, "top_p": 0.9}
Merging dicts with ** is one of the most common things an LLM writes. Under smolagents it fails with:
InterpreterError: NoneType is not supported.
There is no None anywhere in that line. The message sends you looking for a null value that does not exist.
Why it happens
In Python's AST, a dict literal keeps its keys and values in two parallel lists. For a normal entry the key is an AST node. For a **mapping spread entry, the key is literally None, a signal that says "this is a spread, not a key/value pair."
smolagents evaluated every key by walking expression.keys and calling evaluate_ast(key, ...) on each one. When the key is None, that call falls through every isinstance branch to the catch-all raise InterpreterError(f"{type} is not supported"). So the spread marker got evaluated as if it were an expression, and the model got blamed for a None it never wrote.
Why silence is the expensive part
Here is the part the Sentry view made obvious. The error is handled: the agent catches it and feeds it back to the model as "here is what went wrong, try again." But the message names NoneType, and the model's code has no None, so the model cannot act on it. It retries the exact same valid syntax. And again. Every step burns a real LLM call and a slot in the step budget until the run gives up.
One bug, one misleading message, three identical failures:
Three events on a single issue is not noise. It is the agent stuck in a loop, and without Sentry counting the events you would never see the loop, only a run that quietly underperformed. Sentry's Seer read the same event and reached the exact root cause:
smolagents' LocalPythonExecutor doesn't handle dict unpacking (
**) syntax: None keys in ast.Dict cause an unsupported type error. [...]evaluate_ast(None, ...)matches no isinstance branch and falls to the else clause. The interpreter raises InterpreterError: NoneType is not supported, the agent retries with identical code, burning steps in a loop.
The fix
Evaluate the dict pairwise instead of evaluating keys blindly. A None key means "merge this mapping":
result = {}
for key_node, value_node in zip(expression.keys, expression.values):
if key_node is None:
value = evaluate_ast(value_node, *common_params)
if not hasattr(value, "keys"):
raise InterpreterError(f"'{type(value).__name__}' object is not a mapping")
result.update(value)
else:
key = evaluate_ast(key_node, *common_params)
result[key] = evaluate_ast(value_node, *common_params)
return result
This matches CPython exactly: spreads merge in order, later keys win, and unpacking a non-mapping raises 'list' object is not a mapping.
A reviewer caught my fix being too strict
I first gated the spread on isinstance(value, Mapping). Minutes after the PR opened, OpenAI's Codex reviewer flagged it (P2): CPython does not require the Mapping ABC, it only requires an object with a keys() method. Since the sandbox lets users define their own classes, a duck-typed mapping with keys() and __getitem__() would have been wrongly rejected. I switched the check to hasattr(value, "keys") and added a test for exactly that case. AI wrote the code, AI reviewed the code, I kept score.
After
On the patched build the same line just runs:
app: step 1 ok, config = {'temperature': 0.7, 'max_tokens': 512, 'top_p': 0.9}
One step, no loop, no phantom None.
Numbers
- 4 unreported bugs found by fuzzing valid Python through the sandbox; this is the first fix
- Misleading
NoneTypeerror reproduced on currentmainand 1.26.0 - 3 wasted agent steps per occurrence, visible only because Sentry counts events
- 9 new tests: spreads, double spreads, override order both ways, a duck-typed mapping class, empty spread, non-mapping rejection
- 406 passing, ruff clean
Links
- Issue: https://github.com/huggingface/smolagents/issues/2552
- PR: https://github.com/huggingface/smolagents/pull/2553
The pattern across all three entries: the worst agent bugs do not throw a red stack trace at you. They hand the model a plausible-but-wrong message and let it fail politely, on repeat. Count your events.
