docs: add search-replace plan

[RyanGroch]

Like most of the files in the plans folder, this file is AI-generated. However, I've checked it over and revised it, and to me it appears sound.

Essentially, it is a plan to test the search_replace tool in order to make sure that it's reliable, and to fix it if it's not.

---

[wwwillchen]

@BugBot run

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[cubic-dev-ai]

No issues found across 1 file

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

[wwwillchen]

@BugBot run

[wwwillchen]

@BugBot run

[wwwillchen]

I think overall a pretty good plan with a few suggestions:

1. i'd just use Dyad Engine + Dyad Pro key, this is going to be much easier than managing 3 API keys and it'll be using the same models. (Dyad Engine basically proxies all the models)
2. I'd have more eval cases: like 10-12 cases and include more complex cases such as (refactor giant 700-line react component into 3 smaller components, etc.)
3. Just because a search-replace applies without error doesn't mean it's correct, you'll need to either spot check them or use another model (i'd probably use GPT 5.4) to judge the output - basically, feed in the prompt + original file + output file and say: does the output file look correct given the prompt + original file?

[wwwillchen]

@BugBot run

[github-actions]

🎭 Playwright Test Results

❌ Some tests failed

OS	Passed	Failed	Flaky	Skipped
🍎 macOS	407	4	5	129
🪟 Windows	404	8	2	129

Summary: 811 passed, 12 failed, 7 flaky, 258 skipped

Failed Tests

🍎 macOS

• chat_input.spec.ts > send button disabled during pending proposal
  • Error: expect(locator).toBeVisible() failed
• queued_message.spec.ts > editing queued message restores attachments and selected components
  • Error: expect(locator).toBeVisible() failed
• queued_message.spec.ts > canceling queued message edit clears restored components
  • Error: expect(locator).toBeVisible() failed
• setup_flow.spec.ts > Setup Flow > node.js install flow
  • TimeoutError: locator.click: Timeout 30000ms exceeded.

🪟 Windows

• context_manage.spec.ts > manage context - smart context
  • Error: expect(locator).toMatchAriaSnapshot(expected) failed
• context_manage.spec.ts > manage context - smart context - auto-includes only
  • Error: expect(locator).toMatchAriaSnapshot(expected) failed
• context_manage.spec.ts > manage context - exclude paths
  • Error: expect(locator).toMatchAriaSnapshot(expected) failed
• context_manage.spec.ts > manage context - exclude paths with smart context
  • Error: expect(locator).toMatchAriaSnapshot(expected) failed
• github.spec.ts > create and sync to new repo
  • Error: expect(locator).toHaveClass(expected) failed
• github.spec.ts > create and sync to existing repo
  • Error: expect(locator).toMatchAriaSnapshot(expected) failed
• github.spec.ts > create and sync to existing repo - custom branch
  • Error: expect(locator).toMatchAriaSnapshot(expected) failed
• setup_flow.spec.ts > Setup Flow > node.js install flow
  • TimeoutError: locator.dispatchEvent: Timeout 30000ms exceeded.

📋 Re-run Failing Tests (macOS)

Copy and paste to re-run all failing spec files locally:

npm run e2e \
  e2e-tests/chat_input.spec.ts \
  e2e-tests/queued_message.spec.ts \
  e2e-tests/setup_flow.spec.ts

⚠️ Flaky Tests

🍎 macOS

• approve.spec.ts > write to index, approve, check preview (passed after 1 retry)
• context_limit_banner.spec.ts > context limit banner shows 'running out' when near context limit (passed after 1 retry)
• context_manage.spec.ts > manage context - smart context (passed after 1 retry)
• logs_server.spec.ts > system messages UI shows server logs with correct type (passed after 1 retry)
• setup_flow.spec.ts > Setup Flow > setup banner shows correct state when node.js is installed (passed after 1 retry)

🪟 Windows

• context_manage.spec.ts > manage context - default (passed after 1 retry)
• setup_flow.spec.ts > Setup Flow > setup banner shows correct state when node.js is installed (passed after 1 retry)

---

📊 View full report

[wwwillchen]

@BugBot run

[wwwillchen]

@BugBot run

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1f2d72fd5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[wwwillchen]

@BugBot run

[RyanGroch]

Re-requesting a review because I've made two large changes at this point:

1. I changed the plan to use Dyad Pro instead of individual API keys.
2. There are now two kinds of cases: "exact match" cases (checked by a simple diff) and "judge" cases which use GPT 5.4 to judge the output. There are 7 exact match cases and 5 judge cases. In general, judge cases are the more complex ones (refactor large component, etc.).

I've also started implementing this, so I'll likely open another PR soon. I can still address any other changes that are needed though.