feat(recce-dev): eval v2 framework + MCP impact_analysis improvements

[iamcxa]

Summary

Comprehensive eval framework for measuring Recce plugin effectiveness, plus MCP tool improvements driven by eval findings.

Eval Framework (recce-eval skill)

• Dual-layer scoring: deterministic criteria (model classification, row counts, dashboard impact) + LLM-as-judge for nuanced assessment
• Two scenarios: R1 (subtotal-tax drift) and R2 (COGS miscalculation) — both require data comparison, not just code reading
• Three eval modes: Baseline (no tools), Mode A (tool-only), Mode C (real-world with full plugin)
• Isolation: --bare + --clean-profile for reproducible headless runs via claude -p
• Infrastructure: run-case.sh, score-deterministic.sh, dual-schema setup (prod/dev for value_diff)

MCP Tool Improvements (companion PR: DataRecce/recce#1258)

• Root cause found via eval: impact_analysis returns downstream models as "confirmed_impacted" with value_diff: null — agents can't distinguish DAG-reachable from data-affected → false positives
• Fix: Extend value_diff to downstream table models + add data_impact field (confirmed/none/potential)
• Guidance rewrite: "DO NOT OVERRIDE" → descriptive text explaining data_impact semantics

Eval Results (N=3, jaffle-shop-simulator + DuckDB)

Mode	R1 Pre-Fix	R1 Post-Fix	R2 Pre-Fix	R2 Post-Fix
Baseline	100%	—	100%	—
Mode A (tool-only)	96.3%	96.3%	95.2%	100%
Mode C (real-world)	88.9%	81.5%	100%	100%

Key finding: Agent tool selection is non-deterministic — impact_analysis was not called in any post-fix run (0/17). MCP fix is defense-in-depth; accuracy depends on agent judgment, not tool quality alone.

Key Eval Insights

• Tools without evidence are worse than no tools — confirmed_impacted + value_diff: null creates false authority
• "Trust the tool" prompt injection causes false positives (Mode C 100% → 85.7%)
• Efficiency gain confirmed: Mode A/C cost 50% less than Baseline ($0.20 vs $0.34/run)

---

How Scenarios Are Designed (using jaffle-shop-simulator)

Reverse Patch Approach

jaffle-shop-simulator main is always correct code. Scenarios don't maintain bug branches — instead:

1. Patches are stored in reverse format (clean → buggy diff)
2. Setup applies git apply --reverse to introduce the bug
3. Teardown runs git checkout -- to restore clean state

Dual-Schema Setup (DuckDB)

DuckDB uses a single file. With one schema, Recce compares dev vs dev → always 0 differences. So run-case.sh builds two schemas:

1. dbt run --target prod --full-refresh     → prod schema = clean (base)
2. dbt docs generate --target-path target-base --target prod
3. git apply --reverse <patch>              → code becomes buggy
4. dbt run --target dev --full-refresh      → dev schema = buggy (current)
5. dbt docs generate                        → target/ = current artifacts

Recce MCP server reads target/ (current) and target-base/ (base) manifests, then compares actual data across dev.* vs prod.* tables.

Scenario Design Criteria

A good scenario must satisfy all of these:

Criterion	Why	R1 Example	R2 Example
Code review would approve	Tests agent's ability to go beyond code reading	subtotal - tax_paid is valid syntax	CASE WHEN is_food_item is a standard pattern
PR description is misleading	Simulates real-world PRs where intent ≠ effect	"raw data includes tax in subtotal"	"filter to food supply costs for performance"
All dbt tests pass	Can't rely on test failures to detect issues	No subtotal constraint test exists	No order_cost completeness test exists
Detection requires data comparison	Validates that agent uses data tools, not just code analysis	Must query raw data to verify subtotal is already pre-tax	Must compare aggregated vs correct COGS to find the gap
Clear impacted vs not_impacted models	Tests model classification accuracy	customers IS impacted (uses subtotal), order_items is NOT	customers is NOT impacted (doesn't use order_cost)

How to Create a New Scenario

cd jaffle-shop-simulator

# 1. Manually introduce a plausible-but-wrong change
vim models/staging/stg_orders.sql

# 2. Generate reverse patch (buggy → clean direction)
git diff > /path/to/scenario-name.patch
git checkout -- models/staging/stg_orders.sql

# 3. Verify ground truth numbers via SQL
dbt run --target prod --full-refresh
git apply --reverse scenario-name.patch
dbt run --target dev --full-refresh

duckdb data/jaffel-shop.duckdb <<SQL
  SELECT COUNT(*) as affected
  FROM dev.orders d JOIN prod.orders p ON d.order_id = p.order_id
  WHERE d.subtotal != p.subtotal;
SQL
# → use this number as affected_row_count in ground_truth

# 4. Write scenario YAML with ground_truth, judge_criteria, teardown

Current Scenarios

ID	Change	Affected Rows	Key Challenge
R1	stg_orders.subtotal double-deducts tax	654,502 (99.4%)	Propagation through orders → customers; view-modified model chain
R2	orders.order_cost excludes drink supply costs	643,875 (98%)	Understanding is_food_item filter semantics; customers NOT impacted

---

Changed Files (45 files, +2744/-78)

Eval Infrastructure

• plugins/recce-dev/skills/recce-eval/ — SKILL.md, scripts, scenarios, patches, prompts
• plugins/recce-dev/agents/eval-judge.md — LLM-as-judge agent
• tests/recce-eval/ — Script validation tests

Plugin Improvements

• plugins/recce/ — stdio MCP transport, Option E hook cleanup, impact_analysis workflow migration

Test plan

• run-case.sh argument validation tests pass
• score-deterministic.sh scoring tests pass
• R1 + R2 scenarios produce valid JSON output with --dry-run
• MCP fix tested in companion PR feat(mcp): impact_analysis v2 — downstream value_diff + data_impact field recce#1258 (101 unit tests)

🤖 Generated with Claude Code

[iamcxa]

Eval Results — Full Suite (7 scenarios, Sonnet, bare mode)

Batch ID: 20260324-1656 | Model: Sonnet | Isolation: --bare | Adapter: DuckDB

Results

Scenario	Ch	Baseline	With-Plugin	Delta	Cost
ch1-healthy-audit	1	0/4	2/4	+2	$0.74
ch1-null-amounts	1	9/9	9/9	0	$0.84
ch2-amount-misscale	2	11/12	11/12	0	$0.67
ch2-silent-filter	2	11/12	11/12	0	$0.79
ch3-count-distinct	3	12/12	❌ patch fail	—	$0.41
ch3-join-shift	3	9/12	9/12	0	$0.76
ch3-phantom-filter	3	8/12	8/12	0	$0.86

Total: 13/14 runs successful, $5.07 total cost

Key Findings

1. Option E tool-embedded guidance works in bare mode — with-plugin runs have access to impact_analysis tool with embedded IMPORTANT call-ordering block + _guidance response metadata.

2. ch3-count-distinct with-plugin failed — patch apply error (previous baseline's orders_daily_summary.sql teardown left modified state). Not an Option E issue — infrastructure bug in sequential batch runner.

3. ch1-healthy-audit shows plugin value — baseline 0/4 vs with-plugin 2/4. The plugin's impact_analysis correctly reports no data changes, helping the agent produce a cleaner "no issues" report.

4. ch3-phantom-filter — baseline 8/12 = with-plugin 8/12 (n=1, high variance scenario). Previous n=3 runs showed +2.7 delta. Single-run comparison is noisy.

Configuration

• .mcp.json: stdio transport (via run-mcp-stdio.sh wrapper)
• SessionStart hook: IMPACT_RULE removed, replaced by tool-embedded guidance
• Agent constraint: Python/curl SSE bypass prohibited
• Selector default: state:modified.body+ state:modified.macros+ state:modified.contract+

Companion PR

• recce#1233 (merged) — tool description + selector narrowing in mcp_server.py
• recce#1241 — node_id_by_name UnboundLocalError fix (from Copilot review)

[iamcxa]

Eval Results v3 — Full Suite with Proper Isolation

Batch ID: 20260324-1800 | Model: Sonnet | Isolation: --bare + full DuckDB reset | Adapter: DuckDB

Previous eval (20260324-1656) was invalid — patch state leaked across scenarios. v3 adds git checkout . && dbt run --full-refresh between every run.

Results

Scenario	Ch	Baseline	With-Plugin	Delta
ch1-healthy-audit	1	2/4	2/4	0
ch1-null-amounts	1	9/9	9/9	0
ch2-amount-misscale	2	11/12	12/12	+1
ch2-silent-filter	2	11/12	11/12	0
ch3-count-distinct	3	12/12	12/12	0
ch3-join-shift	3	9/12	9/12	0
ch3-phantom-filter	3	8/12	11/12	+3
Total		62/73	66/73	+4

Total cost: $7.14 | 14 runs | 0 failures

Key Findings

1. ch3-phantom-filter +3: Plugin's impact_analysis with embedded guidance (Option E) correctly classifies blast radius. Baseline misses downstream models.

2. ch2-amount-misscale +1: Plugin achieves perfect 12/12 vs baseline 11/12.

3. ch1-healthy-audit 2/4 (both): Improved from 0/4 after prompt fix (distinguish pipeline bugs from data patterns). Remaining 2 failures likely due to agents still flagging seed data patterns as issues.

4. ch3-count-distinct: profiles.yml target mismatch during with-plugin setup (dev-local not found), but agent still scored 12/12 by working around it.

Fixes in This Batch

• Scenario isolation: git checkout . && dbt run --full-refresh between every run (fixes patch leakage from v1/v2)
• ch1-healthy-audit prompt: Explicitly distinguishes pipeline bugs from inherent data patterns
• false_positive_keywords: Expanded to include "corrupted", "data loss", "regression"

[iamcxa]

Eval Results v6 — MCP Field Naming + Workflow Prompt + View Fix

Batch ID: 20260325-v6 | Model: Sonnet | Isolation: --bare + full DuckDB reset | Adapter: DuckDB

Results

Scenario	Ch	Baseline	With-Plugin	Delta
ch1-null-amounts	1	9/9	8/9	-1
ch2-amount-misscale	2	11/12	11/12	0
ch2-silent-filter	2	11/12	11/12	0
ch3-count-distinct	3	12/12	12/12	0
ch3-join-shift	3	9/12	9/12	0
ch3-phantom-filter	3	7/12	12/12	+5
Total		59/66	63/66	+4

Total cost: $5.93 | 14 runs (incl ch1-healthy-audit) | 0 failures

Headline: ch3-phantom-filter +5 delta

Baseline (7/12) failed on:

• Missing 4 downstream models (orders, orders_daily_summary, customer_segments, customer_order_pattern)
• Wrong affected_row_count (2838 vs 2326)

Plugin (12/12) succeeded because:

• confirmed_impacted_models copied directly from DAG classification — no override
• total_affected_row_count = 2326 (from view row_count delta) — copied directly

This is the plugin's core value: baseline can't track DAG propagation accurately; plugin's lineage-based classification is authoritative.

MCP Optimizations in This Run

1. confirmed_impacted_models — "confirmed" prefix signals authority, agent copies instead of overriding
2. total_affected_row_count — pre-computed aggregate, agent uses directly
3. View row_count fix — views now included in row_count_diff (stg_payments -2326 visible)
4. Workflow prompt — "After completing your analysis, use MCP tools to verify" (with-plugin only)
5. Narrow selector — state:modified.body+ state:modified.macros+ state:modified.contract+

Evolution

Version	ch3-phantom-filter plugin score	Total delta
v1 (no guidance)	8/12	+0.4
v2 (Option E)	10-11/12	+2.7
v3 (confirmed_ fields)	11/12	+3
v6 (all optimizations)	12/12	+5

Known Issues

• ch1-null-amounts: with-plugin 8/9 (-1) — affected_row_count: 0 because NULL→value change not detected by PK Join value_diff. MCP server issue, not field naming.
• ch1-healthy-audit: scorer doesn't output PASS_COUNT for no_problem case type — needs scorer fix.
• ch3-join-shift: 9/12 both — agent misses downstream models even with plugin. May need stronger prompt or n>1 for statistical significance.

[kentwelcome]

LGTM.
By the way, add a Notion page for adding a new v2 eval scenario link