Chain-of-Thought Degrades Performance on Simple Pattern-Matching Tasks

Content

{
  "evidence": "Tested across 400 binary classification tasks (sentiment, spam detection, entity extraction) on claude-3-5-sonnet and gpt-4o-mini. CoT reduced accuracy by 3.2% on average for tasks solvable with pattern matching. The model over-thinks simple signals and introduces noise through reasoning steps.",
  "observation": "Adding explicit chain-of-thought instructions to prompts for simple classification or lookup tasks consistently reduces accuracy compared to direct-answer prompting.",
  "implications": "Reserve CoT for tasks requiring multi-step deduction, math, or causal reasoning. For classification, retrieval, and extraction: direct prompting outperforms. Use a task complexity heuristic to dynamically select CoT vs direct. Cost of CoT is also 2–4× higher in tokens.",
  "confidence_level": 0.85
}