Exercise 04: Cost Analysis

Objective

Calculate and compare costs across 3 routing strategies using the NOTIF scenario.

Required Reading

• Expert walkthrough cost summary
• Expert README -- "Tiered model strategy" table
• Custom Agents | Cursor Docs -- How the model field in agent frontmatter determines cost per invocation
• Agent Skills | Cursor Docs -- How the jg-benchmark-ops skill defines cost/performance evaluation workflows

CursorClaude Code

Cost analysis applies equally to Claude Code deployments. Model pricing (input/output per token) and invocation counts are the same regardless of IDE. The tiered routing savings demonstrated here translate directly to model selection strategies in Claude Code's sequential prompting.

Cost analysis applies equally to Claude Code deployments. Model pricing (input/output per token) and invocation counts are the same regardless of IDE. The tiered routing savings demonstrated here translate directly to model selection strategies in Claude Code's sequential prompting.

Cost model differs from walkthrough

The walkthrough cost summary uses higher illustrative per-issue totals to emphasize relative tier differences. This exercise uses a formal per-invocation cost model with lower unit costs. Absolute totals will differ; the relative tier comparisons are consistent.

Cost Model

Tier	Tokens/invocation	Cost/invocation
Fast	~2,000	~$0.002
Standard	~8,000	~$0.03
High	~20,000	~$0.15

Each pipeline run involves: planner + subplanner + worker + tester + reviewer + git = ~6 agent invocations.

Tasks

1. Strategy A -- All Standard: Route every issue (NOTIF-001, -002, -003) to standard tier.
2. Strategy B -- Tiered Routing: Route to appropriate tier (fast/standard/high) as in Exercise 02.
3. Strategy C -- Standard with Rework: Use standard for everything, but NOTIF-003 needs 1.5x retry cycles because standard tier cannot handle the complexity in one pass.

For each strategy, calculate: - Total invocations (planner counts for standard/high only) - Total tokens - Total cost

Write an analysis explaining when tiered routing pays off vs overpaying for simple tasks.

Output

Write to docs/expert/tutorials/outputs/04-cost-analysis.json:

Expected Output

{
  "strategies": {
    "all_standard": {
      "total_invocations": 18,
      "total_tokens": 144000,
      "total_cost": 0.54,
      "notes": "Overpays for NOTIF-001 (trivial) by using standard agents"
    },
    "tiered_routing": {
      "total_invocations": 17,
      "total_tokens": 108000,
      "total_cost": 0.37,
      "notes": "Optimal -- fast for trivial, standard for standard, high for complex"
    },
    "standard_with_rework": {
      "total_invocations": 21,
      "total_tokens": 168000,
      "total_cost": 0.63,
      "notes": "Standard agents cannot handle NOTIF-003 complexity in one pass, requiring rework"
    }
  },
  "recommendation": "Tiered routing saves 31% vs all-standard and 41% vs standard-with-rework. The savings increase with higher task volume. For teams processing 20+ issues/week, tiered routing is clearly cost-effective.",
  "breakeven_analysis": "Tiered routing pays off when at least 40% of tasks are trivial (fast tier) or when complex tasks would require standard-tier rework. With our 33/33/33 mix, the savings are moderate but scale linearly."
}

Validation

python3 docs/expert/tutorials/verify.py --exercise 04

Checks: valid JSON, 3 strategies present, required fields, recommendation >= 20 words, costs differ between strategies.

Answer

Three strategies compared:

• All standard: ~$0.54 for 3 issues. Overpays for trivial, underpowers complex.
• Tiered routing: ~$0.37 for 3 issues. Fast for NOTIF-001 ($0.01), standard for NOTIF-002 ($0.18), high for NOTIF-003 ($0.90).
• Standard with rework: ~$0.63 for 3 issues. Standard agents on NOTIF-003 need 1.5x cycles.

Recommendation: Tiered routing saves 31% vs all-standard and 41% vs standard-with-rework.

Breakeven: Tiered routing breaks even when at least 30% of tasks are trivial. The real cost of NOT using high tier for complex tasks is hidden in rework cycles.