feat(worker): remove dual-write, make ProcessingState DB-only#750
Merged
thomasrockhu-codecov merged 3 commits intotomhu/processing-state-db-allfrom Mar 10, 2026
Merged
Conversation
- Make db_session a required parameter on ProcessingState - Remove all Redis operations (sadd, srem, smove, scard, srandmember) - Remove PROCESSING_STATE_TTL, _redis_key(), get_redis_connection import - mark_uploads_as_processing is now an explicit no-op (uploads already exist with state_id=UPLOADED which get_upload_numbers counts) - Pass db_session through upload.py schedule_task chain - Remove redis_state workaround in processing.py safety-net trigger, reuse the existing DB-backed state instance - Remove all Redis-only and Redis-mock unit tests Made-with: Cursor
This runs in a finally block where the DB transaction may already be in a failed state. Wrap in try/except so it doesn't mask the original error. The upload stays UPLOADED, which is safe. Made-with: Cursor
The safety-net is best-effort: if the DB query or finisher dispatch fails, the chord callback will still trigger the finisher normally. Wrapping in try/except prevents test failures in environments where the repos table is unavailable. Made-with: Cursor
Contributor
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## tomhu/processing-state-db-all #750 +/- ##
=================================================================
- Coverage 92.22% 92.21% -0.02%
=================================================================
Files 1304 1304
Lines 47933 47908 -25
Branches 1628 1628
=================================================================
- Hits 44206 44178 -28
- Misses 3418 3421 +3
Partials 309 309
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
thomasrockhu-codecov
merged commit fe130ab
into
tomhu/processing-state-db-all
33 checks passed
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #749. Removes all Redis usage from
ProcessingState, making the DB the single source of truth for upload processing state.db_sessionis now a required parameter. All Redis operations removed (sadd,srem,smove,scard,srandmember,expire)._redis_key(),PROCESSING_STATE_TTL, andget_redis_connectionimport deletedmark_uploads_as_processingis now an explicit no-op — uploads start asUPLOADED, whichget_upload_numbers()counts as "processing"upload.py:db_sessionthreaded throughschedule_task→_schedule_coverage_processing_taskprocessing.py: removed theredis_stateworkaround, reuses the DB-backedstateinstance for the safety-net triggerclear_in_progress_uploadsand the safety-net finisher trigger are wrapped in try/except — they run in error-recovery paths where the DB transaction may already be abortedTest plan
schedule_tasksignature) passprocess_uploadpasstest_full_upload,test_full_carryforward) pass