Skip to content

fix: add pagination to GET /api/workflows to prevent OOM on large workspaces #3578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
guoyangzhen wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: add pagination to GET /api/workflows to prevent OOM on large workspaces #3578

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb99aa7
fix: add pagination to GET /api/workflows (#3435)
guoyangzhen Mar 14, 2026
3a453a2
fix: apply cursor to pagination queries (addressing review feedback)
guoyangzhen Mar 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions apps/sim/app/api/workflows/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { db } from '@sim/db'
import { permissions, workflow, workflowFolder } from '@sim/db/schema'
import { createLogger } from '@sim/logger'
import { and, asc, eq, inArray, isNull, min } from 'drizzle-orm'
import { and, asc, eq, gt, inArray, isNull, min } from 'drizzle-orm'
Copy link
Contributor

@greptile-apps greptile-apps bot Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused gt import

gt is imported from drizzle-orm but is never referenced in the file. This will produce a lint/compile warning and should be removed until the cursor WHERE clause is actually implemented.

Suggested change
import { and, asc, eq, gt, inArray, isNull, min } from 'drizzle-orm'
import { and, asc, eq, inArray, isNull, min } from 'drizzle-orm'

import { type NextRequest, NextResponse } from 'next/server'
import { z } from 'zod'
import { AuditAction, AuditResourceType, recordAudit } from '@/lib/audit/log'
Expand All @@ -27,6 +27,9 @@ export async function GET(request: NextRequest) {
const startTime = Date.now()
const url = new URL(request.url)
const workspaceId = url.searchParams.get('workspaceId')
const cursor = url.searchParams.get('cursor')
Copy link
Contributor

@greptile-apps greptile-apps bot Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cursor parsed but never applied to the query

The cursor variable is read from the query params but is never used in either of the two database queries. As a result, passing ?cursor=<id> is silently ignored and the server always returns the first page — making cursor-based pagination completely non-functional for callers who attempt to use it.

The gt operator imported on line 4 was presumably intended for a WHERE clause like .where(and(...existingWhere..., gt(workflow.id, cursor))), but that call was never added. Since gt is also unused, this will trigger a lint/TypeScript error.

To properly implement cursor-based offset, the WHERE clause needs to be extended, e.g.:

// In the workspaceId branch
.where(
  cursor
    ? and(eq(workflow.workspaceId, workspaceId), gt(workflow.id, cursor))
    : eq(workflow.workspaceId, workspaceId)
)

Note also that using gt(workflow.id, cursor) only gives correct pagination when ordering solely by id. With the current compound ordering [asc(sortOrder), asc(createdAt), asc(id)], a pure id > cursor filter will skip or repeat rows where sortOrder or createdAt differs between pages. A correct keyset cursor needs to encode all three sort columns.

const limitParam = url.searchParams.get('limit')
const limit = Math.min(Math.max(parseInt(limitParam || '100', 10) || 100, 1), 500)
Copy link

@cursor cursor bot Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Silent data truncation for callers expecting all workflows

High Severity

The new default limit of 100 silently truncates results for all existing callers, none of which handle nextCursor pagination. fetchWorkflows in workflows.ts only loads the first 100 workflows into the sidebar registry. use-export-workspace.ts only exports the first 100 workflows, producing silently incomplete ZIP backups. fetchDeployedWorkflows in workflow-mcp-servers.ts may miss deployed workflows. These are the exact large-workspace scenarios this PR targets.

Fix in Cursor Fix in Web


try {
const auth = await checkSessionOrInternalAuth(request, { requireWorkflowId: false })
Expand Down Expand Up @@ -66,12 +69,16 @@ export async function GET(request: NextRequest) {

const orderByClause = [asc(workflow.sortOrder), asc(workflow.createdAt), asc(workflow.id)]

// Fetch limit+1 to detect if there are more pages
const fetchLimit = limit + 1

if (workspaceId) {
workflows = await db
.select()
.from(workflow)
.where(eq(workflow.workspaceId, workspaceId))
.orderBy(...orderByClause)
.limit(fetchLimit)
} else {
const workspacePermissionRows = await db
.select({ workspaceId: permissions.entityId })
Expand All @@ -86,9 +93,15 @@ export async function GET(request: NextRequest) {
.from(workflow)
.where(inArray(workflow.workspaceId, workspaceIds))
.orderBy(...orderByClause)
.limit(fetchLimit)
}

return NextResponse.json({ data: workflows }, { status: 200 })
// Determine if there are more results and set cursor
const hasMore = workflows.length > limit
const data = hasMore ? workflows.slice(0, limit) : workflows
const nextCursor = hasMore ? data[data.length - 1]?.id : null
Copy link
Contributor

@greptile-apps greptile-apps bot Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor value does not match sort order — pagination will produce incorrect results

The nextCursor is set to the last item's id:

const nextCursor = hasMore ? data[data.length - 1]?.id : null

But the query is ordered by [asc(sortOrder), asc(createdAt), asc(id)]. When the cursor is eventually applied as gt(workflow.id, cursor), this filter is only semantically correct when ordering by id alone. In the current compound ordering, rows with a smaller sortOrder but a larger id than the cursor will be excluded from page 2 even though they should appear, and vice-versa.

To avoid skipping or duplicating results, the cursor must encode the full sort key (e.g., sortOrder, createdAt, and id combined — typically as a base64-encoded JSON or composite string), and the WHERE clause must replicate the tie-breaking logic of the ORDER BY.


return NextResponse.json({ data, nextCursor }, { status: 200 })
} catch (error: any) {
const elapsed = Date.now() - startTime
logger.error(`[${requestId}] Workflow fetch error after ${elapsed}ms`, error)
Expand Down