tapdb: fix postgres sequence desyncs#2047
Conversation
Migration 31 copied universe_leaves rows with explicit ids, leaving the BIGSERIAL sequence behind the actual data. New inserts then hit duplicate-key errors. Add migration 55 to reset the universe_leaves sequence via setval(). Also reset supply_commit_states and supply_commit_update_types (migration 40 inserted explicit ids into these enum tables); these are benign since no code path auto-increments into them, but fixing them keeps sequence state consistent. On SQLite the statements are replaced with no-ops.
Add TestSequenceConsistency, a Postgres-only test that verifies every BIGSERIAL sequence is consistent with its table's max ID after all migrations have run. This catches migrations that recreate tables and copy rows with explicit IDs without advancing the sequence.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses issues where Postgres sequences become desynchronized from their underlying table data due to legacy migrations that inserted rows with explicit IDs. By explicitly resetting these sequences and implementing a robust automated test to monitor sequence consistency in CI, the changes ensure database integrity for future migrations. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds migration 55 to synchronize Postgres sequences that were desynchronized by previous migrations and introduces a new test to verify sequence consistency. Feedback includes a critical fix for the migration script to prevent failures on empty tables, a reminder to update SQLite replacements if the SQL changes, and suggestions to improve the test logic and SQL safety.
| SELECT setval(pg_get_serial_sequence('universe_leaves', 'id'), COALESCE((SELECT MAX(id) FROM universe_leaves), 0)); | ||
| SELECT setval(pg_get_serial_sequence('supply_commit_states', 'id'), COALESCE((SELECT MAX(id) FROM supply_commit_states), 0)); | ||
| SELECT setval(pg_get_serial_sequence('supply_commit_update_types', 'id'), COALESCE((SELECT MAX(id) FROM supply_commit_update_types), 0)); |
There was a problem hiding this comment.
In PostgreSQL, setval will throw an error if the value provided is less than the sequence's MINVALUE (which is 1 by default for BIGSERIAL columns). Using COALESCE(..., 0) will cause this migration to fail on any database where the universe_leaves table is empty (e.g., a fresh installation or a node that hasn't used the Universe feature). To safely handle empty tables and ensure the next value is 1, you should use the three-argument form of setval with a fallback of 1 and an is_called flag based on whether any rows exist.
| SELECT setval(pg_get_serial_sequence('universe_leaves', 'id'), COALESCE((SELECT MAX(id) FROM universe_leaves), 0)); | |
| SELECT setval(pg_get_serial_sequence('supply_commit_states', 'id'), COALESCE((SELECT MAX(id) FROM supply_commit_states), 0)); | |
| SELECT setval(pg_get_serial_sequence('supply_commit_update_types', 'id'), COALESCE((SELECT MAX(id) FROM supply_commit_update_types), 0)); | |
| SELECT setval(pg_get_serial_sequence('universe_leaves', 'id'), COALESCE((SELECT MAX(id) FROM universe_leaves), 1), (SELECT MAX(id) FROM universe_leaves) IS NOT NULL); | |
| SELECT setval(pg_get_serial_sequence('supply_commit_states', 'id'), COALESCE((SELECT MAX(id) FROM supply_commit_states), 1), (SELECT MAX(id) FROM supply_commit_states) IS NOT NULL); | |
| SELECT setval(pg_get_serial_sequence('supply_commit_update_types', 'id'), COALESCE((SELECT MAX(id) FROM supply_commit_update_types), 1), (SELECT MAX(id) FROM supply_commit_update_types) IS NOT NULL); |
| "SELECT setval(pg_get_serial_sequence(" + | ||
| "'universe_leaves', 'id'), COALESCE((" + | ||
| "SELECT MAX(id) FROM universe_leaves" + | ||
| "), 0));": "SELECT 1;", | ||
| "SELECT setval(pg_get_serial_sequence(" + | ||
| "'supply_commit_states', 'id'), " + | ||
| "COALESCE((SELECT MAX(id) FROM " + | ||
| "supply_commit_states), 0));": "SELECT 1;", | ||
| "SELECT setval(pg_get_serial_sequence(" + | ||
| "'supply_commit_update_types', 'id'), " + | ||
| "COALESCE((SELECT MAX(id) FROM " + | ||
| "supply_commit_update_types), 0));": "SELECT 1;", | ||
| } |
There was a problem hiding this comment.
The strings in sqliteSchemaReplacements must match the SQL in the migration files exactly for the replacement to work. If the SQL in 000055_fix_universe_leaves_seq.up.sql is updated to fix the setval(..., 0) bug, these keys must be updated accordingly. Failure to do so will cause the replacement to fail, and SQLite will attempt to execute the Postgres-specific SQL, leading to migration errors on SQLite backends.
| err := db.QueryRowContext(ctx, fmt.Sprintf( | ||
| "SELECT last_value FROM %s", s.seqName, | ||
| )).Scan(&lastValue) | ||
| require.NoError(t, err) | ||
|
|
||
| var maxID int64 | ||
| err = db.QueryRowContext(ctx, fmt.Sprintf( | ||
| "SELECT COALESCE(MAX(%s), 0) FROM %s", | ||
| s.column, s.table, | ||
| )).Scan(&maxID) | ||
| require.NoError(t, err) | ||
|
|
||
| require.GreaterOrEqual( | ||
| t, lastValue, maxID, | ||
| "sequence %s on %s.%s is behind: "+ | ||
| "last_value=%d, max(%s)=%d", | ||
| s.seqName, s.table, s.column, | ||
| lastValue, s.column, maxID, | ||
| ) |
There was a problem hiding this comment.
The check lastValue >= maxID is not sufficient to guarantee that the next auto-incremented ID will not collide with existing data. In PostgreSQL, if last_value is equal to maxID but the is_called flag is false, the next call to nextval() will return maxID, causing a primary key violation. A more robust check would involve verifying that the effective next value (accounting for is_called) is strictly greater than maxID. Additionally, when constructing SQL queries with identifiers dynamically, it is safer to use quoted identifiers (e.g., via %q in fmt.Sprintf) to avoid issues with reserved words or case sensitivity.
1 participant
Fixes Postgres sequence desyncs caused by a couple of legacy migrations. One is material (in
universe_leaves, via migration 31) and the other two are benign (supply commitment-related enum values, via migration 40). Each desync'd sequence is simply set to its respective table's max id.Also adds a Postgres-only unit test that should catch future desyncs caused by migrations like this in CI; it simply checks that every BIGSERIAL sequence is consistent with its table's max id after all migrations have run.