Add pagination support for composite-key queries on NoSQL drivers#408
Open
carllerche wants to merge 7 commits intomainfrom
Open
Add pagination support for composite-key queries on NoSQL drivers#408carllerche wants to merge 7 commits intomainfrom
carllerche wants to merge 7 commits intomainfrom
Conversation
Add limit, scan_index_forward, and exclusive_start_key fields to the QueryPk driver operation, enabling DynamoDB-native pagination. The engine planner now extracts order_by/limit/offset from query statements and maps them to these fields for NoSQL drivers, removing the previous panics that blocked ordering and limit on KV paths. The DynamoDB driver passes these parameters through to the AWS SDK Query API. Add integration tests asserting the correct driver ops are dispatched with pagination attributes set when querying partitioned composite keys with limit and order_by. https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL
Resolve merge conflict in DynamoDB query_pk.rs by combining the secondary index branching logic from main with the pagination parameters (limit, scan_index_forward, exclusive_start_key) from this branch. Also fix &Db -> &mut Db API changes in pagination driver-ops tests and add missing pagination fields to the new secondary index QueryPk initialization in plan/statement.rs. https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL
Combine main's refactored try_fan_out closure API with pagination fields (limit, scan_index_forward, exclusive_start_key) from this branch. https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL
- scan_index_forward: Option<bool> → order: Option<stmt::Direction> - exclusive_start_key: Option<stmt::Value> → cursor: Option<stmt::Value> The K/V engine interface should use generic terminology rather than DynamoDB-specific names. `order` uses the existing `stmt::Direction` type (Asc/Desc) instead of a bool, and `cursor` is a standard pagination term replacing `exclusive_start_key`. https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL
Instead of a standalone test file, add QuerySql structure assertions directly to the existing sort_asc, paginate, and limit tests in one_model_sort_limit.rs. This verifies that order_by and limit clauses flow through to the SQL statements without duplicating test coverage. https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL
The existing pagination tests in one_model_sort_limit.rs all require(sql), meaning DynamoDB never exercises pagination, limit, or ordering. Add a new composite_key_pagination test module with four tests that use a composite partition+local key model (no requires gate), so they run on every driver including DynamoDB: - paginate_composite_key: descending pagination with next/prev navigation - paginate_composite_key_asc: ascending pagination walking all pages - limit_composite_key: limit with and without ordering - sort_composite_key: ascending and descending sort verification Each test asserts the correct driver operation type (QuerySql for SQL, QueryPk for KV drivers). https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for pagination, sorting, and limiting on composite-key models (partition + local key) for NoSQL drivers like DynamoDB. Previously,
order_by(),limit(), andpaginate()were only supported on SQL databases. These operations now work across all drivers by passing pagination parameters through the query execution pipeline.Key Changes
New test suite: Added
composite_key_pagination.rswith comprehensive tests for:.paginate(),.next(), and.prev().order_by()in both ascending and descending directions.limit()Query planning: Modified
statement.rsto extract pagination parameters (limit, sort direction, cursor) from query statements and pass them toQueryPkoperations via a newextract_query_pk_pagination()methodMIR and execution layers: Updated
mir/query_pk.rsandexec/query_pk.rsto carry pagination fields (limit,order,cursor) through the compilation pipelineDriver operation: Extended
driver/operation/query_pk.rswith three new optional fields:limit: Maximum number of items to returnorder: Sort direction (Asc/Desc) for composite keyscursor: Serialized key for resuming paginated queriesDynamoDB implementation: Updated
query_pk.rsin the DynamoDB driver to apply pagination parameters:.limit()for result count limiting.scan_index_forward()to control sort direction.set_exclusive_start_key()for cursor-based paginationRemoved assertion: Deleted the assertion in
lower.rsthat prevented ordering and limiting on non-SQL drivers, enabling these features for all backendsEnhanced SQL tests: Updated
one_model_sort_limit.rswith operation verification to ensure SQL queries include proper ORDER BY and LIMIT clausesImplementation Details
The pagination parameters flow through the compilation pipeline: statement parsing → query planning → MIR → execution → driver operation. This allows NoSQL drivers to implement pagination natively using their query APIs while maintaining the same high-level API for users across all database backends.
https://claude.ai/code/session_019tLeKCnD9izwJjJXYQWygL