Cisco Talos conducted extensive testing on 66 model and reasoning combinations from Anthropic and OpenAI to determine their effectiveness in Security Operations Center (SOC) and Digital Forensics and Incident Response (DFIR) tasks. The study focused on identifying a repeatable methodology for model selection rather than crowning a single winner, emphasizing that the "best" model depends on a balance of efficacy, analysis time, cost, and consistency.
Key findings revealed that increased reasoning effort does not always correlate with better performance; in some instances, it led to higher costs, slower response times, and even lower accuracy or higher failure rates. The researchers recommend using a Pareto frontier approach to evaluate tradeoffs and suggest that organizations benchmark specific reasoning levels and personas—such as Threat Hunters or Detection Engineers—within their own unique workflows to find the optimal balance for their operational needs.
