-
-
Notifications
You must be signed in to change notification settings - Fork 133
Switch from pre-commit to prek; add security pre-commit hooks
#5141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
2899118
2be2d6c
d7e8aba
e639a94
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -34,22 +34,23 @@ jobs: | |
| - name: Run Zenodo DOI updater | ||
| id: update | ||
| run: | | ||
| pixi run update_zenodo_dois ${{ github.event.inputs.datasets }} | ||
| pixi run update_zenodo_dois "${{ github.event.inputs.datasets }}" | ||
|
|
||
| # Check if any changes were made | ||
| if git diff --quiet; then | ||
| echo "changes=false" >> $GITHUB_OUTPUT | ||
| echo "changes=false" >> "$GITHUB_OUTPUT" | ||
| echo "No changes detected" | ||
| else | ||
| echo "changes=true" >> $GITHUB_OUTPUT | ||
| echo "changes=true" >> "$GITHUB_OUTPUT" | ||
| echo "Changes detected!" | ||
| fi | ||
|
|
||
| - name: Format datasets as labels | ||
| id: format-datasets | ||
| run: | | ||
| DATASETS="${{ github.event.inputs.datasets }}" | ||
| LABELS=$(echo "$DATASETS" | sed 's/ \+/, /g') | ||
| # shellcheck disable=SC2001 | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ShellCheck is like "Why you wanna use sed for this?" |
||
| LABELS=$(echo "$DATASETS" | sed -E 's/[[:space:]]+/, /g') | ||
| echo "dataset_labels=$LABELS" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Create Pull Request | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,7 +39,7 @@ jobs: | |
| set -o pipefail | ||
| pixi update --json | pixi exec pixi-diff-to-markdown >> diff.md | ||
| pixi install --locked | ||
| pixi run pre-commit-autoupdate | ||
| pixi run prek-autoupdate | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is the only place we're programmatically using it. |
||
| PUDL_OUTPUT="/home/runner/pudl-work/output/" pixi run dbt-deps-upgrade | ||
| - name: Make a PR to merge updated lockfiles | ||
| # If we are relocking dependencies on a schedule or workflow_dispatch, we need | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -95,6 +95,28 @@ repos: | |
| hooks: | ||
| - id: actionlint | ||
|
|
||
| # Scan for secrets and credentials | ||
| - repo: https://github.com/trufflesecurity/trufflehog | ||
| rev: v3.94.2 | ||
| hooks: | ||
| - id: trufflehog | ||
| name: TruffleHog | ||
| entry: trufflehog git file://. --since-commit HEAD --fail | ||
| language: golang | ||
| pass_filenames: false | ||
|
|
||
| - repo: https://github.com/Yelp/detect-secrets | ||
| rev: v1.5.0 | ||
| hooks: | ||
| - id: detect-secrets | ||
| args: | ||
| - "--baseline" | ||
| - ".secrets.baseline" | ||
| - "--exclude-lines" | ||
| - "workload_identity_provider:" | ||
|
Comment on lines
+115
to
+116
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Exclude the WIF |
||
| - "--exclude-files" | ||
| - "(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)" | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did a full scan with no exclusions first to see where it would find false positives, and created this filter based on those results. |
||
|
|
||
| ##################################################################################### | ||
| # Our own pre-commit hooks, which don't come from the pre-commit project | ||
| ##################################################################################### | ||
|
|
@@ -150,5 +172,5 @@ ci: | |
| autoupdate_branch: main | ||
| autoupdate_commit_msg: "[pre-commit.ci] pre-commit autoupdate" | ||
| autoupdate_schedule: weekly | ||
| skip: [unit-tests, nb-output-clear, shellcheck] | ||
| skip: [unit-tests, nb-output-clear, shellcheck, trufflehog, detect-secrets] | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They need some resources not available on pre-commit.ci so they don't work there. |
||
| submodules: false | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed the (now) false WIF positives. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,220 @@ | ||
| { | ||
| "version": "1.5.0", | ||
| "plugins_used": [ | ||
| { | ||
| "name": "ArtifactoryDetector" | ||
| }, | ||
| { | ||
| "name": "AWSKeyDetector" | ||
| }, | ||
| { | ||
| "name": "AzureStorageKeyDetector" | ||
| }, | ||
| { | ||
| "name": "Base64HighEntropyString", | ||
| "limit": 4.5 | ||
| }, | ||
| { | ||
| "name": "BasicAuthDetector" | ||
| }, | ||
| { | ||
| "name": "CloudantDetector" | ||
| }, | ||
| { | ||
| "name": "DiscordBotTokenDetector" | ||
| }, | ||
| { | ||
| "name": "GitHubTokenDetector" | ||
| }, | ||
| { | ||
| "name": "GitLabTokenDetector" | ||
| }, | ||
| { | ||
| "name": "HexHighEntropyString", | ||
| "limit": 3.0 | ||
| }, | ||
| { | ||
| "name": "IbmCloudIamDetector" | ||
| }, | ||
| { | ||
| "name": "IbmCosHmacDetector" | ||
| }, | ||
| { | ||
| "name": "IPPublicDetector" | ||
| }, | ||
| { | ||
| "name": "JwtTokenDetector" | ||
| }, | ||
| { | ||
| "name": "KeywordDetector", | ||
| "keyword_exclude": "" | ||
| }, | ||
| { | ||
| "name": "MailchimpDetector" | ||
| }, | ||
| { | ||
| "name": "NpmDetector" | ||
| }, | ||
| { | ||
| "name": "OpenAIDetector" | ||
| }, | ||
| { | ||
| "name": "PrivateKeyDetector" | ||
| }, | ||
| { | ||
| "name": "PypiTokenDetector" | ||
| }, | ||
| { | ||
| "name": "SendGridDetector" | ||
| }, | ||
| { | ||
| "name": "SlackDetector" | ||
| }, | ||
| { | ||
| "name": "SoftlayerDetector" | ||
| }, | ||
| { | ||
| "name": "SquareOAuthDetector" | ||
| }, | ||
| { | ||
| "name": "StripeDetector" | ||
| }, | ||
| { | ||
| "name": "TelegramBotTokenDetector" | ||
| }, | ||
| { | ||
| "name": "TwilioKeyDetector" | ||
| } | ||
| ], | ||
| "filters_used": [ | ||
| { | ||
| "path": "detect_secrets.filters.allowlist.is_line_allowlisted" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.common.is_baseline_file", | ||
| "filename": ".secrets.baseline" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies", | ||
| "min_level": 2 | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_indirect_reference" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_likely_id_string" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_lock_file" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_not_alphanumeric_string" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_potential_uuid" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_sequential_string" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_swagger_file" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.heuristic.is_templated_secret" | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.regex.should_exclude_file", | ||
| "pattern": [ | ||
| "(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)" | ||
| ] | ||
| }, | ||
| { | ||
| "path": "detect_secrets.filters.regex.should_exclude_line", | ||
| "pattern": [ | ||
| "workload_identity_provider:" | ||
| ] | ||
| } | ||
| ], | ||
| "results": { | ||
| "docker/dagster.yaml": [ | ||
| { | ||
| "type": "Secret Keyword", | ||
| "filename": "docker/dagster.yaml", | ||
| "hashed_secret": "a8b730d9cb75d147a6ab37ab39036e47f567513b", | ||
| "is_verified": false, | ||
| "line_number": 5, | ||
| "is_secret": false | ||
| } | ||
| ], | ||
| "src/pudl/analysis/timeseries_evaluation.py": [ | ||
| { | ||
| "type": "Base64 High Entropy String", | ||
| "filename": "src/pudl/analysis/timeseries_evaluation.py", | ||
| "hashed_secret": "a6d0b3075cbbd4c9a3141d744954036b00fe91be", | ||
| "is_verified": false, | ||
| "line_number": 70, | ||
| "is_secret": false | ||
| } | ||
| ], | ||
| "terraform/main.tf": [ | ||
| { | ||
| "type": "Secret Keyword", | ||
| "filename": "terraform/main.tf", | ||
| "hashed_secret": "790739d6bad41c70225e325068b8f8fc9a61628b", | ||
| "is_verified": false, | ||
| "line_number": 131, | ||
| "is_secret": false | ||
| } | ||
| ], | ||
| "terraform/pudl-usage-metrics-dashboard.tf": [ | ||
| { | ||
| "type": "Secret Keyword", | ||
| "filename": "terraform/pudl-usage-metrics-dashboard.tf", | ||
| "hashed_secret": "10b642e314d4e2aaab3fd757c06a18971d02a746", | ||
| "is_verified": false, | ||
| "line_number": 72, | ||
| "is_secret": false | ||
| }, | ||
| { | ||
| "type": "Secret Keyword", | ||
| "filename": "terraform/pudl-usage-metrics-dashboard.tf", | ||
| "hashed_secret": "37e891b7e8956f04d734c16d66eeec245078478c", | ||
| "is_verified": false, | ||
| "line_number": 77, | ||
| "is_secret": false | ||
| } | ||
| ], | ||
| "terraform/pudl-viewer.tf": [ | ||
| { | ||
| "type": "Secret Keyword", | ||
| "filename": "terraform/pudl-viewer.tf", | ||
| "hashed_secret": "3e8db137aa8c9ed287c81c4b79d43e817e86af7c", | ||
| "is_verified": false, | ||
| "line_number": 95, | ||
| "is_secret": false | ||
| }, | ||
| { | ||
| "type": "Secret Keyword", | ||
| "filename": "terraform/pudl-viewer.tf", | ||
| "hashed_secret": "f2636c63dbd332b36f892a92d1ddd4c706ecc71b", | ||
| "is_verified": false, | ||
| "line_number": 100, | ||
| "is_secret": false | ||
| } | ||
| ], | ||
| "test/unit/workspace/datastore_test.py": [ | ||
| { | ||
| "type": "Hex High Entropy String", | ||
| "filename": "test/unit/workspace/datastore_test.py", | ||
| "hashed_secret": "05dc54328edc1e335436ea8cc059a52c19d3cf96", | ||
| "is_verified": false, | ||
| "line_number": 182, | ||
| "is_secret": false | ||
| } | ||
| ] | ||
| }, | ||
| "generated_at": "2026-04-01T15:17:31Z" | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed some linting issues that came up when I ran
prek run --all-files