This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked 😀:
Shortform video hooks live or die by two things that usually trade off against each other: does the copy obey a hard constraint (word count for onscreen text), and does it still sound like something a real creator would actually say out loud? I benchmarked whether LLMs can hold a strict wordlimit constraint on UGCstyle Reel hooks without the output turning stiff, robotic, or losing the "curiosity gap" that makes a hook work. Each model was tested across two difficulty levels (6word and 10word limits) and four content niches (haircare, skincare, fitness, and personal finance). Every generated hook was scored two ways: a hard rule check (word count ≤ limit) and an LLMjudged rubric (sounds natural, creates curiosity, no metacommentary about the instructions themselves).
Models Tested 🤖:
I tested 9 models spanning flagship and lightweight tiers, plus one reasoning/nonreasoning pair to isolate whether explicit reasoning helps or hurts a short, punchy creativeconstraint task: Claude Haiku 4.5 (small), Claude Opus 5 (flagship), Gemini 3.5 Flash Lite (small), Gemini 3.1 Pro Preview (large), GPT5.4 Nano (small), Grok 4.20 nonreasoning, Grok 4.20 reasoning, Qwen3Next Instruct, and Qwen3Next Thinking. I picked this spread because constraintfollowing and creative naturalness are usually assumed to move together, and reasoning is usually assumed to strictly help — I wanted to see if either assumption holds for something as small and stylistically specific as a 610 word Reel hook.
FINDINGS 🤞:
The two metrics didn't move together at all — several models nailed the rule while completely missing the point of a hook:
Qwen3Next Instruct : hit the word limit on every single hook (1.0 adherence) but scored 0.0 on naturalness — it consistently produced technicallycompliant lines that read like instructions being followed, not something a creator would say.
Claude Haiku 4.5 showed the same pattern at a smaller scale: strong adherence (0.875) paired with the secondlowest naturalness (0.125).
Gemini 3.1 Pro Preview was the only model to score a perfect 1.0 on both metrics — proof the tradeoff isn't inherent to the task, just hard to hit.
Claude Opus 5, interestingly, went the other way: the lowest adherence among flagships (0.75) but strong naturalness (0.75) — it seemed to prioritize a hook sounding right over strictly counting words.
GPT5.4 Nano was the weakest performer on both axes (0.375 / 0.25), the only model where "small and cheap" clearly cost real quality rather than just style.
The reasoning vs. nonreasoning Grok pair was the most interesting isolated result: both hit perfect wordlimit adherence (1.0), but the reasoning variant scored noticeably higher on naturalness (0.625 vs. 0.25) — for this task, reasoning improved how the constraint was satisfied without costing any compliance. That cuts against the assumption that reasoning traces make short creative copy stiffer.
Qwen3Next Thinking failed outright partway through evaluation with no usable output — worth flagging as its own data point: not every "thinking" variant is a dropin upgrade over its instruct counterpart, sometimes it's simply less reliable endtoend.
What surprised me most ✨:
Perfect rulefollowing was actually common (4 of 8 successful models hit 1.0 adherence), but perfect naturalness was rare — meaning the harder problem isn't teaching a model to count words, it's getting it to not sound like it's counting words while doing so.
What I'd measure next 😀:
Pushing the constraint down to 34 words to find where adherence itself starts to break down, and having the judge model score the model's rejected internal drafts (where available) to see whether low naturalness models are aware they're being stiff or just don't track it at all.
My Benchmark 😊:
https://www.kaggle.com/code/shahzeenanwar/hook-constraint-benchmark-kaggle-challenge?scriptVersionId=352741515
