Skip to content

Filter partitions on test unique_stg_ga4__events_event_key#365

Merged
mtcarlone merged 3 commits intoVelir:mainfrom
adviso:main
Jan 28, 2026
Merged

Filter partitions on test unique_stg_ga4__events_event_key#365
mtcarlone merged 3 commits intoVelir:mainfrom
adviso:main

Conversation

@portalhacker
Copy link
Copy Markdown
Contributor

Reduce compute by limiting the test unique_stg_ga4__events_event_key to the last 7 days (event_timestamp is already in the hash so no need to test all partitions)

Description & motivation

By default the unique_stg_ga4__events_event_key test scans all partitions which can be very expensive, at 2TB in this example:

SCR-20250508-lpzr

Scanning all partitions is not required as the event_key already includes the event timestamp.
The same test on the same data is 100 times less expensive at 22GB when adding a partition filter on the last 7 days.

SCR-20250508-lqwc

Checklist

  • I have verified that these changes work locally
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have run dbt test to validate existing tests
  • I have run python -m pytest . to validate existing tests

For a reason I can't explain most integration test are failing with FileNotFoundError despite being in the unit_tests directory, existing files in other directories, and being able to view the file find the file on direct executions like python -c "from dbt.tests.util import read_file; print(read_file('../macros/default_channel_grouping.sql'))"

ERROR test_macro_default_channel_grouping.py::TestDefaultChannelGrouping::test_mock_run_and_check - FileNotFoundError: [Errno 2] No such file or directory: '../macros/default_channel_grouping.sql'
[... 8 others]
========================================================== 4 passed, 141 warnings, 9 errors in 70.85s (0:01:10) ==========================================================

portalhacker and others added 2 commits May 8, 2025 12:49
…t_key` to the last 7 days (`event_timestamp` is already in the hash so no need to test all partitions)
@mtcarlone mtcarlone self-assigned this Nov 14, 2025
@mtcarlone
Copy link
Copy Markdown
Collaborator

Hi - getting up to speed with the project. Please give me a little bit of time to go through this and get acquainted / review. Thanks in advance!

@mtcarlone
Copy link
Copy Markdown
Collaborator

@portalhacker can you please merge Velir:main into this branch to re-trigger CI? Thank you!

Copy link
Copy Markdown
Collaborator

@mtcarlone mtcarlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Pending CI

@portalhacker
Copy link
Copy Markdown
Contributor Author

CI passed ✅. Pending merge @mtcarlone

@mtcarlone
Copy link
Copy Markdown
Collaborator

Thanks @portalhacker - this will go out next week as a part of our Jan release!

@mtcarlone mtcarlone added this to the Jan 2026 Release milestone Jan 28, 2026
@mtcarlone mtcarlone merged commit 61b013e into Velir:main Jan 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants