-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 0.0% passed
- (0/3)
-
-
- Avg 24.2/100
-
-
- $0
- total
-
-
- 281135ms
- avg
-
-
- Unknown
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Behavior Summary
- | Behavior | Total | Passed | Changed | Regressions | Healed |
|---|---|---|---|---|---|
| untagged | -3 | -0 | -0 | -3 | -0 | -
-
-
-
-
- Score per Test
-
-
-
-
-
-
-
- Execution Cost per Query
- | Test | Model | Trace Cost | Latency | Score |
|---|---|---|---|---|
| implement | -Unknown | -$0 | - -360030ms | -20.0 | -
| bug-fix | -Unknown | -$0 | - -324936ms | -32.5 | -
| refactor | -Unknown | -$0 | - -158438ms | -20.0 | -
| Total | — | -$0 | -avg $0.000000 per query | -|
- Trace cost comes from the agent execution trace only. Mock or non-metered tools will show
- $0 even when EvalView used a separate judge or local model during evaluation.
- This check also used 6 EvalView judge calls (2489 tokens).
-
-
-
-
-
-
-
-
-
-
- ✗
- implement
-
- ▾
-
-
-
-
-
-
-
- Model: Unknown
-
-
- Baseline: 2026-04-06 13:26
- Baseline model: Not recorded in snapshot
-
-
-
-
-
- Score Breakdown
-
-
- Tools 0.0% × 30%
- Output 0.0/100 × 50%
- Sequence Correct × 20%
- = 20.0/100
- The response does not implement group_by_key(), does not run the __main__ block, and simply repeats the timeout error. It fails to address the user’s request entirely.
-
- QueryImplement the group_by_key() function in stub.py. The docstring and type signature are already there — the body is just pass. Implement it, then run the __main__ block to verify it prints the correctly grouped output.
-
-
-
-
-
- Why it failed
- - Score 20.0 below minimum 70.0
- Output quality: 0.0/100
- Hallucination detected (70% confidence)
- Tool accuracy: 0.0%
sequenceDiagram
- participant User
- participant Agent
- participant T0 as error
- User->>Agent: Implement the group_by_key f...
- Agent-xT0: error
- T0-->Agent: OpenCode timed out after 360...
- Agent-->>User: OpenCode timed out after 360...
-
-
- Conversation Turns
-
-
- 🔮 Hallucination detected · 70% · [Warning] - Tool 'error' failed/returned error, but agent did not acknowledge the failure · openai/gpt-5.4-mini
- 🛡 Safe
-
-
-
- Unsupported claims: [Warning] - Tool 'error' failed/returned error, but agent did not acknowledge the failure
-(Confidence 70% below threshold 98% - not blocking)
-
-
-
-
-
-
- ✗
- bug-fix
-
- ▾
-
-
-
-
-
-
-
- Model: Unknown
-
-
- Baseline: 2026-04-06 13:26
- Baseline model: Not recorded in snapshot
-
-
-
-
-
- Score Breakdown
-
-
- Tools 33.3% × 30%
- Output 5.0/100 × 50%
- Sequence Correct × 20%
- = 32.5/100
- The response is relevant and acknowledges the bug, but it does not actually fix the code or run the file to confirm the output. It only states an intention to inspect the file, so it fails to satisfy the requested task and provides no concrete correction.
-
- QueryThere is a bug in buggy.py. The find_max function uses range(1, len(numbers) - 1) which means it never checks the last element. Fix the bug so the function correctly returns the maximum value including the last element. After fixing, run the file to confirm it prints 9.
-
-
-
-
-
- Why it failed
- - Score 32.5 below minimum 70.0
- Output quality: 5.0/100
- Tool accuracy: 33.3%
sequenceDiagram
- participant User
- participant Agent
- participant T0 as read_file
- User->>Agent: There is a bug in buggy.py.
- Agent->>T0: filePath=/Users/hidaibar-mor...
- T0-->Agent: path/Users/hidaibar-mor/Down...
- Agent-->>User: Ill fix the bug in the find_...
-
-
- Conversation Turns
-
-
- 🔮 No hallucination · No verifiable factual claims found in output. · openai/gpt-5.4-mini
- 🛡 Safe
-
-
-
-
-
-
-
-
-
-
- ✗
- refactor
-
- ▾
-
-
-
-
- Model: Unknown
-
-
- Baseline: 2026-04-06 13:26
- Baseline model: Not recorded in snapshot
-
-
-
-
-
- Score Breakdown
-
-
- Tools 0.0% × 30%
- Output 0.0/100 × 50%
- Sequence Correct × 20%
- = 20.0/100
- The response does not answer the query, provide a refactoring, or confirm execution. It only contains a context-size error message and no code, so it fails completeness and relevance entirely.
-
- QueryRefactor the function p() in messy.py. It has poor naming, deeply nested conditionals, and uses type() instead of isinstance(). Rewrite it to be clean and readable while preserving the exact same behaviour: filter out None values, strip whitespace from non-empty strings, and pass through all other types unchanged. Then run the file to confirm it still prints {'name': 'Alice', 'age': 30}.
-
-
-
-
-
- Why it failed
- - Score 20.0 below minimum 70.0
- Output quality: 0.0/100
- Tool accuracy: 0.0%
◎ Direct response — no tools invoked
-
-
-
- 🔮 No hallucination · No verifiable factual claims found in output. · openai/gpt-5.4-mini
- 🛡 Safe
-
-
-
-
-
-
-
- Response"Context size has been exceeded."
-
-
-
-
-
-
-
-
-
-
-
- ⬇ Regression
-
-
-
- implement
- 96.0 → 20.0
- -76.0
- lexical 2.9%
- semantic 0.4%
- 70% confidence
- ▾
-
-
-
-
-
-
-
-
- Baselineglobread_fileedit_filebash
- Currenterror
-
-
-
- Why This Changed
-
- tool_reordered
-
- Tool sequence changed: 'glob' at step 1 was replaced by 'error'
- Suggested fix: The agent is calling the right tools but in a different order. If the new order is valid, run `evalview snapshot --variant <name>` to accept it as an alternative.
-
-
-
-
-
-
- Baseline Output
Output matches expected behavior:
-
-- `"fruit"` → `[apple, banana]`
-- `"veggie"` → `[carrot]`
-
-The implementation iterates over each item, skips any missing the key, and appends to the appropriate grou...
Current Output
OpenCode timed out after 360s
--- golden
+++ actual
@@ -1,6 +1 @@
-Output matches expected behavior:
-
-
-
-- `"fruit"` → `[apple, banana]`
-
-- `"veggie"` → `[carrot]`
-
-
-
-The implementation iterates over each item, skips any missing the key, and appends to the appropriate group in the result dict.
+OpenCode timed out after 360s
-
-
-
-
- ▾
- Trajectory Comparison
-
-
-
-
-
-
- ⬇ Regression
-
-
-
- bug-fix
- 100.0 → 32.5
- -67.5
- lexical 33.7%
- semantic 35.2%
- 99% confidence
- ▾
-
-
-
-
-
-
-
-
- Baselineread_fileedit_filebash
- Currentread_file
-
-
-
- Why This Changed
-
- tool_removed
-
- Tools 'edit_file', 'bash' were expected but not called
- Suggested fix: Check if your agent's prompt or logic still triggers 'edit_file'. If the tool was intentionally removed, run `evalview snapshot` to update the baseline.
-
-
-
-
-
-
- Baseline Output
Fixed. The bug was `range(1, len(numbers) - 1)` which excluded the last index. Changed to `range(1, len(numbers))` so all elements are checked. Output confirms `9`.
Current Output
I'll fix the bug in the find_max function in buggy.py. Let me first examine the file to understand the current implementation.
--- golden
+++ actual
@@ -1 +1 @@
-Fixed. The bug was `range(1, len(numbers) - 1)` which excluded the last index. Changed to `range(1, len(numbers))` so all elements are checked. Output confirms `9`.
+I'll fix the bug in the find_max function in buggy.py. Let me first examine the file to understand the current implementation.
-
-
-
-
- ▾
- Trajectory Comparison
-
-
-
-
-
-
- ⬇ Regression
-
-
-
- refactor
- 96.5 → 20.0
- -76.5
- lexical 9.9%
- semantic 12.0%
- 58% confidence
- ▾
-
-
-
-
-
-
-
-
- Baselineread_filetodowriteedit_fileedit_filetodowritebashtodowrite
- CurrentNo tools
-
-
-
- Why This Changed
-
- tool_removed
-
- Tools 'read_file', 'todowrite', 'edit_file', 'edit_file', 'todowrite', 'bash', 'todowrite' were expected but not called
- Suggested fix: Check if your agent's prompt or logic still triggers 'read_file'. If the tool was intentionally removed, run `evalview snapshot` to update the baseline.
-
-
-
-
-
-
- Baseline Output
Output confirmed: `{'name': 'Alice', 'age': 30}`
-
-Here's a summary of what changed in `messy.py`:
-
-| Before | After | Reason |
-|--------|-------|--------|
-| `p(d)` | `process_dict(data)` | Descriptive...
Current Output
"Context size has been exceeded."
--- golden
+++ actual
@@ -1,11 +1 @@
-Output confirmed: `{'name': 'Alice', 'age': 30}`
-
-
-
-Here's a summary of what changed in `messy.py`:
-
-
-
-| Before | After | Reason |
-
-|--------|-------|--------|
-
-| `p(d)` | `process_dict(data)` | Descriptive names |
-
-| `for k in d` | `for key, value in data.items()` | Unpack key/value directly |
-
-| `if d[k] != None` | `if value is None: continue` | `is None` is idiomatic; early-continue flattens nesting |
-
-| `type(d[k]) == str` | `isinstance(value, str)` | `isinstance` is correct (handles subclasses) |
-
-| `if len(d[k]) > 0` / `else: pass` | `if stripped:` | Truthiness check on the already-stripped string; `pass` branch eliminated |
+"Context size has been exceeded."
-
-
-
-
- ▾
- Trajectory Comparison
-
-
-
-
-
-
-
-
-
-
-
-
- 2
- steps
-
-
- 281135ms
- avg latency
-
-
- $0
- total cost
-
-
- 3
- tests
-
-
-
-
-
-
- Step Latencies
-
-
-
- Step Cost
-
-