Filter partitions on test unique_stg_ga4__events_event_key#365
Merged
mtcarlone merged 3 commits intoVelir:mainfrom Jan 28, 2026
Merged
Filter partitions on test unique_stg_ga4__events_event_key#365mtcarlone merged 3 commits intoVelir:mainfrom
unique_stg_ga4__events_event_key#365mtcarlone merged 3 commits intoVelir:mainfrom
Conversation
…t_key` to the last 7 days (`event_timestamp` is already in the hash so no need to test all partitions)
Collaborator
|
Hi - getting up to speed with the project. Please give me a little bit of time to go through this and get acquainted / review. Thanks in advance! |
Collaborator
|
@portalhacker can you please merge |
Contributor
Author
|
CI passed ✅. Pending merge @mtcarlone |
Collaborator
|
Thanks @portalhacker - this will go out next week as a part of our Jan release! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce compute by limiting the test
unique_stg_ga4__events_event_keyto the last 7 days (event_timestampis already in the hash so no need to test all partitions)Description & motivation
By default the unique_stg_ga4__events_event_key test scans all partitions which can be very expensive, at 2TB in this example:
Scanning all partitions is not required as the event_key already includes the event timestamp.
The same test on the same data is 100 times less expensive at 22GB when adding a partition filter on the last 7 days.
Checklist
dbt testto validate existing testspython -m pytest .to validate existing testsFor a reason I can't explain most integration test are failing with FileNotFoundError despite being in the unit_tests directory, existing files in other directories, and being able to view the file find the file on direct executions like
python -c "from dbt.tests.util import read_file; print(read_file('../macros/default_channel_grouping.sql'))"ERROR test_macro_default_channel_grouping.py::TestDefaultChannelGrouping::test_mock_run_and_check - FileNotFoundError: [Errno 2] No such file or directory: '../macros/default_channel_grouping.sql'
[... 8 others]
========================================================== 4 passed, 141 warnings, 9 errors in 70.85s (0:01:10) ==========================================================