feat: complete removal of Clerk — DB schema, dead code, configs, frontend, OpenAPI, CI

[Copilot]

Description

Completes the full migration from Clerk to Better Auth by removing every remaining Clerk reference across the codebase. Better Auth was already fully operational; this PR removes the dead/legacy Clerk layer.

Changes

Database (highest priority)

• migrations/0009_remove_clerk_fields.sql — New D1 migration that drops clerk_user_id from users and agent_sessions via SQLite table-recreation pattern. Removes the idx_users_clerk_user_id and idx_agent_sessions_clerk_user_id indexes. Migration follows the established pattern from 0004_users_email_nullable.sql: wrapped in PRAGMA foreign_keys = OFF/ON, temp tables use underscore-prefixed names (_users_new, _agent_sessions_new) without IF NOT EXISTS so partial runs fail loudly, and all pre-existing indexes are recreated (including idx_users_tier).
• prisma/schema.prisma — Removed clerkUserId String? @map("clerk_user_id") from the User model along with its legacy comment block.
• prisma/schema.d1.prisma — Removed clerkUserId @unique @map("clerk_user_id") from D1 User model; updated section header from // Users — synced from Clerk webhooks to // Users.
• prisma/generated/ + prisma/generated-d1/ — Regenerated both Prisma clients (deno task db:generate).

Storage schema fix (surfaced by Prisma regen)

The Neon migration 20260413000000_remove_clerk_fields had already dropped the token_hash column from the sessions table, but the generated Prisma types were stale and CreateSessionSchema / HyperdriveStorageAdapter.createSession() still referenced tokenHash. Regenerating the client surfaced this mismatch:

• src/storage/schemas.ts — Removed tokenHash from CreateSessionSchema.
• src/storage/HyperdriveStorageAdapter.ts — Removed tokenHash from prisma.session.create() call.
• src/storage/schemas.test.ts + HyperdriveStorageAdapter.test.ts — Updated tests accordingly.

Dead code deletion

• worker/services/user-service.ts — Deleted entirely. Contained ClerkUserData, upsertUserFromClerk, deleteUserByClerkId, findUserByClerkId, updateUserTier — all Clerk-only, never imported in production routing.
• worker/services/user-service.test.ts — Deleted (tested only the above dead code).

Worker cleanup

• worker/handlers/admin-handlers.ts — Removed clerk_user_id backward-compat fallback from the role-assignments list query param (?user_id= is the only accepted param now).
• Worker service JSDoc comments updated in admin-analytics-events.ts, admin-logger.ts, admin-audit-service.ts, admin-feature-flag-service.ts, admin-config-service.ts, feature-flag-service.ts.

Config

• .dev.vars.example — Removed the entire Clerk section: CLERK_PUBLISHABLE_KEY, CLERK_JWKS_URL, CLERK_SECRET_KEY, CLERK_WEBHOOK_SECRET, DISABLE_CLERK_FALLBACK, DISABLE_CLERK_WEBHOOKS.

Frontend

• admin/api-keys/api-keys.component.ts — Removed clerkUserId from the AdminApiKey interface, displayedColumns, filter logic, table column definition, and detail panel (the field was never returned by the worker API). Updated search label/placeholder from "Search by prefix or owner" → "Search by prefix or name" to accurately reflect the active filter fields (keyPrefix, name).
• admin/api-keys/api-keys.component.spec.ts — Removed clerkUserId from interface, factory, fixtures, and the stale 'filters by clerkUserId substring' test.
• admin/roles/roles.component.ts — Changed <mat-label>Clerk User ID</mat-label> → <mat-label>User ID</mat-label>.
• admin/roles/roles.component.spec.ts — Replaced 'clerk-abc', 'clerk-xyz', 'clerk-del' test fixture strings with neutral 'user-abc', 'user-xyz', 'user-del'.
• Stale comments updated in admin.guard.ts, app.routes.server.ts, api-key.service.ts, agent-rpc.service.ts, agent.models.ts, webhooks.component.ts, app.config.ts, components/api-keys/api-keys.component.spec.ts.

src/ library

• Stale "Clerk JWT bearer token" descriptions updated in IDiagnosticsProvider.ts, CliApp.deno.ts, ArgumentParser.ts, configuration/schemas.ts.

Scripts / Postman

• scripts/diag-full.ts — Removed the /api/clerk-config diagnostic probe (endpoint no longer exists).
• scripts/generate-postman-collection.ts — Renamed clerkToken → bearerToken collection variable.
• postman/environments/adblock-compiler.environment.yaml + adblock-compiler-local.environment.yaml — Renamed clerkJwt → bearerToken; removed clerkPublishableKey entries.

OpenAPI / generated API specs

• docs/api/openapi.yaml — Removed /api/webhooks/clerk path block, ClerkJWT security scheme, updated enum: [local-jwt, clerk] → [local-jwt, better-auth], updated endpoint descriptions.
• docs/api/cloudflare-schema.yaml and docs/postman/postman-collection.json — Regenerated via deno task schema:generate.

CI fix

• .github/workflows/neon-branch-create.yml — Replaced pnpm/action-setup@v6.0.0 (which caused ERR_PNPM_BROKEN_LOCKFILE in the Neon Branch: Create workflow) with the ./.github/actions/setup-pnpm-node composite action — the same SHA-pinned pnpm/action-setup@fc06bc1... # v4 used by all other CI jobs. Also upgraded the Node.js version from 20 → 22 to match the rest of CI.

Testing

• Unit tests added/updated
• Manual testing performed
• CI passes

deno task preflight exits 0 (fmt, lint, type-check, openapi:validate, schema:generate, check:drift all pass). deno task test — 1407 passed, 0 failed. Parallel validation (Code Review + CodeQL) — no issues found.

Zero Trust Architecture Checklist

Required for every PR touching worker/ or frontend/.
Check each item that applies. If an item doesn't apply, check it and note "N/A".

Worker / Backend

• Every handler verifies auth before executing business logic
• CORS origin allowlist enforced (not *) on write/authenticated endpoints
• All secrets accessed via Worker Secret bindings (not [vars]) — Clerk secret bindings removed from .dev.vars.example
• All external inputs Zod-validated before use
• All D1 queries use parameterized .prepare().bind() (no string interpolation)
• Security events emitted to Analytics Engine on auth failures

Frontend / Angular

• Protected routes have functional CanActivateFn auth guards
• Auth tokens managed via Clerk SDK (not localStorage) — N/A; Clerk SDK fully removed
• HTTP interceptor attaches ****** (no manual token passing)
• API responses validated with Zod schemas before consumption

API Shield / Vulnerability Scanner

Required for every PR touching docs/api/openapi.yaml, worker/routes/, or resource endpoint handlers.

• New/changed endpoints have a unique operationId in openapi.yaml
• Resource endpoints (those with /{id} path parameters) include a security: annotation
• Resource queries are scoped to the authenticated user (WHERE user_id = ?) — not just by ID
• Missing/unauthorized resources return 404 (not 403) to avoid leaking resource existence
• cloudflare-schema.yaml regenerated if openapi.yaml changed (deno task schema:cloudflare)

[github-actions]

🗄️ Database Migration Plan

This is a dry-run. No changes have been applied to any database.
Migrations will be applied per-backend (sequentially) when this PR is merged to main.

📂 Changed Migration Files

Main DB (migrations/):

• migrations/0009_remove_clerk_fields.sql

☁️ Cloudflare D1

Main Database (adblock-compiler-d1-database)

�[0m�[32mTask�[0m �[0m�[36mwrangler�[0m deno run -A npm:wrangler@^4.82.1 "d1" "migrations" "list" "adblock-compiler-d1-database" "--remote"

 ⛅️ wrangler 4.82.2
───────────────────
Resource location: remote 

Migrations to be applied:
┌──────────────────────────────┐
│ Name                         │
├──────────────────────────────┤
│ 0009_remove_clerk_fields.sql │
└──────────────────────────────┘

Admin Database (adblock-compiler-admin-d1)

�[0m�[32mTask�[0m �[0m�[36mwrangler�[0m deno run -A npm:wrangler@^4.82.1 "d1" "migrations" "list" "adblock-compiler-admin-d1" "--remote"

 ⛅️ wrangler 4.82.2
───────────────────
Resource location: remote 

✅ No migrations to apply!

🐘 PostgreSQL

Prisma migrate status

Loaded Prisma config from prisma.config.ts.

Prisma schema loaded from prisma/schema.prisma.
Datasource "db": PostgreSQL database "adblock-compiler", schema "public" at "ep-polished-resonance-a8mefek3.eastus2.azure.neon.tech"

6 migrations found in prisma/migrations
Following migration have not yet been applied:
20260413000000_remove_clerk_fields

To apply migrations in development run prisma migrate dev.
To apply migrations in production run prisma migrate deploy.

---

Validation run: #196

[jaypatrick]

@copilot Run pnpm install --frozen-lockfile
Scope: all 3 workspace projects
 ERR_PNPM_BROKEN_LOCKFILE  The lockfile at "/home/runner/work/adblock-compiler/adblock-compiler/pnpm-lock.yaml" is broken: expected a single document in the stream, but found more
Error: Process completed with exit code 1.

[Copilot]

@copilot Run pnpm install --frozen-lockfile
Scope: all 3 workspace projects
 ERR_PNPM_BROKEN_LOCKFILE  The lockfile at "/home/runner/work/adblock-compiler/adblock-compiler/pnpm-lock.yaml" is bro...

Fixed in commit e6f6fbb. The ERR_PNPM_BROKEN_LOCKFILE error was caused by pnpm/action-setup@v6.0.0 in neon-branch-create.yml installing an incompatible pnpm version — the rest of CI uses the SHA-pinned pnpm/action-setup@fc06bc1... # v4 composite action which works correctly with the lockfileVersion: '9.0' lockfile. Replaced the two-step pnpm+Node setup with ./.github/actions/setup-pnpm-node (same composite action all other CI jobs use) and upgraded Node from 20 → 22 to match.

[github-actions]

🐘 Neon Database Branch Created

Property	Value
Branch name	pr-1585
Neon project	twilight-river-73901472
Migrations	✅ Applied successfully

🔗 Connection string (click to expand)

Tip: Use this connection string in your local .env to test
against this PR's isolated database branch.

ℹ️ What is this?

This PR has an isolated Neon database branch that is a copy-on-write
clone of the main branch. All Prisma migrations have been applied.
The branch will be automatically deleted when this PR is closed or merged.

See docs/database-setup/neon-branching.md for details.

[Copilot]

Pull request overview

Removes the remaining legacy Clerk integration across DB schema, worker, frontend, OpenAPI/docs, and CI, completing the migration to Better Auth and cleaning up stale/generated artifacts.

Changes:

• Drops Clerk-specific columns/indexes from D1 (users.clerk_user_id, agent_sessions.clerk_user_id) and updates Prisma schemas + regenerated clients.
• Removes stale session tokenHash usage from storage schemas/adapter and updates tests.
• Cleans up Clerk references across worker/frontend/docs/postman and aligns CI pnpm/node setup.

Reviewed changes

Copilot reviewed 51 out of 58 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
worker/services/user-service.ts	Deletes Clerk-only user CRUD service.
worker/services/user-service.test.ts	Deletes tests for the removed Clerk user service.
worker/services/feature-flag-service.ts	Updates docs to reference Better Auth instead of Clerk.
worker/services/admin-logger.ts	Updates field docs to remove Clerk wording.
worker/services/admin-feature-flag-service.ts	Updates JSDoc createdBy wording to Better Auth.
worker/services/admin-config-service.ts	Updates JSDoc createdBy wording to Better Auth.
worker/services/admin-audit-service.ts	Updates audit docs to remove Clerk wording.
worker/services/admin-analytics-events.ts	Updates analytics event docs to remove Clerk wording.
worker/handlers/admin-handlers.ts	Removes legacy clerk_user_id query param fallback.
src/storage/schemas.ts	Removes tokenHash from CreateSessionSchema.
src/storage/schemas.test.ts	Updates schema tests to reflect removal of tokenHash.
src/storage/HyperdriveStorageAdapter.ts	Stops writing tokenHash on session creation.
src/storage/HyperdriveStorageAdapter.test.ts	Updates adapter tests to remove tokenHash usage.
src/diagnostics/IDiagnosticsProvider.ts	Updates diagnostics user-id docs to Better Auth.
src/configuration/schemas.ts	Updates CLI bearerToken description to Better Auth.
src/cli/CliApp.deno.ts	Updates CLI help text to Better Auth.
src/cli/ArgumentParser.ts	Updates CLI help/examples to Better Auth.
scripts/generate-postman-collection.ts	Renames Postman variable from clerkToken to bearerToken.
scripts/diag-full.ts	Removes /api/clerk-config diagnostic probe.
prisma/schema.prisma	Removes clerkUserId field from User model.
prisma/schema.d1.prisma	Removes D1 clerkUserId field and updates header comment.
prisma/generated/models/User.ts	Regenerates Prisma types removing clerkUserId.
prisma/generated/models/Session.ts	Regenerates Prisma types removing tokenHash.
prisma/generated/models/AgentSession.ts	Regenerates Prisma types removing clerkUserId.
prisma/generated/internal/prismaNamespaceBrowser.ts	Regenerates Prisma internal enums removing Clerk/tokenHash fields.
prisma/generated/internal/prismaNamespace.ts	Regenerates Prisma internal enums removing Clerk/tokenHash fields.
prisma/generated-d1/models/User.ts	Regenerates D1 Prisma types removing clerkUserId.
prisma/generated-d1/models/LocalAuthUser.ts	Removes generated LocalAuthUser model file.
prisma/generated-d1/models.ts	Removes export of LocalAuthUser model.
prisma/generated-d1/internal/prismaNamespaceBrowser.ts	Regenerates D1 Prisma namespace removing LocalAuthUser + Clerk field enums.
prisma/generated-d1/internal/prismaNamespace.ts	Regenerates D1 Prisma namespace removing LocalAuthUser + Clerk field enums.
prisma/generated-d1/internal/class.ts	Regenerates D1 Prisma client internals after schema changes.
prisma/generated-d1/client.ts	Removes exported LocalAuthUser type.
prisma/generated-d1/browser.ts	Removes exported LocalAuthUser type.
postman/environments/adblock-compiler.environment.yaml	Renames Clerk env var to bearerToken and removes Clerk publishable key entry.
postman/environments/adblock-compiler-local.environment.yaml	Renames Clerk env var to bearerToken and removes Clerk publishable key entry.
migrations/0009_remove_clerk_fields.sql	Adds D1 migration to drop Clerk columns/indexes from users and agent_sessions.
frontend/src/assets/endpoint-registry.json	Removes /api/webhooks/clerk endpoint and updates auth tag description.
frontend/src/app/services/api-key.service.ts	Updates service docs to Better Auth.
frontend/src/app/services/agent-rpc.service.ts	Updates auth docs/comments to Better Auth only.
frontend/src/app/models/agent.models.ts	Updates model docs to Better Auth only.
frontend/src/app/guards/admin.guard.ts	Updates SSR comment to Better Auth and removes Clerk mention.
frontend/src/app/components/api-keys/api-keys.component.spec.ts	Renames mock service variable from Clerk to generic auth naming.
frontend/src/app/app.routes.server.ts	Updates auth route SSR/CSR docs to Better Auth.
frontend/src/app/app.config.ts	Removes Clerk mention from browser-only API comment.
frontend/src/app/admin/webhooks/webhooks.component.ts	Removes Clerk-specific phrasing from the webhook viewer UI copy.
frontend/src/app/admin/roles/roles.component.ts	Renames “Clerk User ID” label to “User ID”.
frontend/src/app/admin/roles/roles.component.spec.ts	Updates fixtures from clerk-* to user-* and adjusts URL expectations.
frontend/src/app/admin/api-keys/api-keys.component.ts	Removes clerkUserId column/details/filter from admin API keys UI.
frontend/src/app/admin/api-keys/api-keys.component.spec.ts	Updates tests/fixtures to remove clerkUserId filtering.
docs/postman/postman-environment.json	Renames clerkToken to bearerToken.
docs/postman/postman-collection.json	Renames token variable to bearerToken and updates auth descriptions; removes Clerk webhook request.
docs/api/openapi.yaml	Removes Clerk webhook path/scheme and updates auth descriptions/security references to Better Auth.
docs/api/cloudflare-schema.yaml	Regenerates Cloudflare schema to reflect Better Auth and removed Clerk webhook/scheme.
deno.lock	Updates lockfile entries after Prisma regeneration/tooling changes.
.github/workflows/neon-branch-create.yml	Switches to shared pnpm/node setup action and bumps Node to 22.
.dev.vars.example	Removes deprecated Clerk env var section.

Comments suppressed due to low confidence (2)

docs/api/openapi.yaml:4719

• security requirements reference BearerAuth, but components.securitySchemes does not define a BearerAuth scheme (only Turnstile, AdminKey, UserApiKey). This makes the OpenAPI document invalid for any endpoint that now lists - BearerAuth: []. Add a BearerAuth http bearer security scheme (or change the affected endpoints back to an existing scheme) so validation and downstream generation are consistent.

    securitySchemes:
        Turnstile:
            type: apiKey
            in: header
            name: turnstileToken
            description: Cloudflare Turnstile token (if enabled on the server)
        AdminKey:
            type: apiKey
            in: header
            name: X-Admin-Key
            description: Admin API key for protected admin endpoints
        UserApiKey:
            type: http
            scheme: bearer
            bearerFormat: ApiKey
            description: "User API key with abc_ prefix. Pass the key as a Bearer token in the Authorization header (e.g. Authorization: Bearer abc_...)."

docs/api/cloudflare-schema.yaml:4853

• This generated Cloudflare schema references BearerAuth in multiple security blocks, but components.securitySchemes does not define BearerAuth (only Turnstile, AdminKey, UserApiKey). Add the missing BearerAuth scheme (or update the security blocks to use an existing scheme) to keep the schema valid and consistent with openapi.yaml.

    securitySchemes:
        Turnstile:
            type: apiKey
            in: header
            name: turnstileToken
            description: Cloudflare Turnstile token (if enabled on the server)
        AdminKey:
            type: apiKey
            in: header
            name: X-Admin-Key
            description: Admin API key for protected admin endpoints
        UserApiKey:
            type: http
            scheme: bearer
            bearerFormat: ApiKey
            description: >-
                User API key with abc_ prefix. Pass the key as a Bearer token in the Authorization header (e.g.
                Authorization: Bearer abc_...).
security: []

[jaypatrick]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all four review items in commit 60a1d49:

• Migration safety — wrapped both table rebuilds with PRAGMA foreign_keys = OFF/ON (matching pattern from 0004_users_email_nullable.sql)
• Temp table naming — renamed users_new → _users_new and agent_sessions_new → _agent_sessions_new, removed IF NOT EXISTS so partial runs fail loudly instead of silently reusing a stale temp table
• Missing index — added idx_users_tier ON users(tier) to the post-rebuild index recreation block
• Search label — updated Search by prefix or owner → Search by prefix or name (and placeholder abc_ or user_... → abc_ or key-name...) to match the actual filter fields (keyPrefix, name)