Speculative Execution Fork Pattern

Content

{
  "problem": "Sequential agent pipelines where step N determines step N+1 create high end-to-end latency when steps are I/O bound and the most likely path can be predicted with high confidence.",
  "examples": [
    "Query classification followed by specialized handler",
    "Document type detection followed by type-specific extraction",
    "Intent detection followed by tool routing"
  ],
  "solution": "When the probability of the primary branch exceeds a threshold (e.g., 80%), speculatively begin executing the primary branch while step N completes. If prediction was wrong, discard and execute correct branch.",
  "anti_patterns": [
    "Speculating on non-idempotent operations (API writes, DB mutations) — wrong speculation causes side effects",
    "Threshold too low (<60%) — speculation cost exceeds benefit when wrong branches are frequent",
    "No accuracy tracking — threshold optimization requires empirical data"
  ],
  "implementation_steps": [
    "After step N begins, generate branch probability estimates: { branch_a: 0.82, branch_b: 0.12, branch_c: 0.06 }",
    "If max branch probability > threshold (default 0.80): spawn speculative execution of that branch in parallel",
    "Speculative branch must be idempotent or reversible — never speculate on destructive operations",
    "When step N completes: if actual result matches speculative branch, adopt result (latency win). If mismatch: discard speculative work and execute correct branch from scratch.",
    "Track speculative accuracy over time: if accuracy < 70%, raise threshold; if > 90%, lower threshold to increase speculation rate"
  ]
}