Switch from `pre-commit` to `prek`; add security pre-commit hooks by zaneselvans · Pull Request #5141 · catalyst-cooperative/pudl

zaneselvans · 2026-03-29T18:01:30Z

Overview

Adds two new pre-commit hooks: trufflehog and detect-secrets
These will help prevent us from accidentally committing passwords, API keys, and access tokens to the repository.
PR includes results of a baseline scan with detect-secrets with the identified suspicious strings marked as false-positives after review, so that they are ignored in the future.
Partly this was inspired by setting up my own dotfiles repo to track my local configurations, and partly by just how easy it is to create multi-thousand line PRs with the help of coding agents... and it would be real bad if they snuck authentication tokens into the codebase.
We are also starting to deploy more services to cloud resources, so there's an increasing risk of doing this.

Documentation

Review and update any other aspects of the documentation that might be affected by this PR.

To-do list

Run pixi run prek --all-files and fix the issues it raises
Run pixi run prek-run to run linters and static code analysis checks.
Review the PR yourself and call out any questions or issues you have.

zaneselvans · 2026-03-29T18:05:21Z

pyproject.toml

 # happen to only be available via PyPI.
 "catalystcoop.pudl" = { path = ".", editable = true }
 # Furo is pure python and the conda package is like a year out of date.
+detect-secrets = ">=1.5"


It's useful to have this in our environment for running or re-running baseline scans, but it's not strictly necessary since that's rare and the hook runs in its own isolated environment. It's not available on conda-forge, so we don't get to clean this section after the docs PR merges.

zaneselvans · 2026-03-29T18:05:41Z

.pre-commit-config.yaml

  autoupdate_commit_msg: "[pre-commit.ci] pre-commit autoupdate"
  autoupdate_schedule: weekly
-  skip: [unit-tests, nb-output-clear, shellcheck]
+  skip: [unit-tests, nb-output-clear, shellcheck, trufflehog, detect-secrets]


They need some resources not available on pre-commit.ci so they don't work there.

zaneselvans · 2026-03-29T18:06:32Z

.pre-commit-config.yaml

+          - "--baseline"
+          - ".secrets.baseline"
+          - "--exclude-files"
+          - "(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)"


I did a full scan with no exclusions first to see where it would find false positives, and created this filter based on those results.

jdangerx

Nice! Nothing blocking.

I think probably we can add a workload_identity_provider: rule which means we have fewer special cases to go through - here's the diff if you want to apply that.

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 46e130920..bdbf2184b 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -112,6 +112,8 @@ repos:
         args:
           - "--baseline"
           - ".secrets.baseline"
+          - "--exclude-lines"
+          - "workload_identity_provider:"
           - "--exclude-files"
           - "(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)"
 
diff --git a/.secrets.baseline b/.secrets.baseline
index 587616db2..85262e0a2 100644
--- a/.secrets.baseline
+++ b/.secrets.baseline
@@ -130,39 +130,15 @@
       "pattern": [
         "(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)"
       ]
+    },
+    {
+      "path": "detect_secrets.filters.regex.should_exclude_line",
+      "pattern": [
+        "workload_identity_provider:"
+      ]
     }
   ],
   "results": {
-    ".github/workflows/build-deploy-ferceqr.yml": [
-      {
-        "type": "Base64 High Entropy String",
-        "filename": ".github/workflows/build-deploy-ferceqr.yml",
-        "hashed_secret": "99359e3ddedb1602025485e4ff9017fa7d71cc77",
-        "is_verified": false,
-        "line_number": 63,
-        "is_secret": false
-      }
-    ],
-    ".github/workflows/build-deploy-pudl.yml": [
-      {
-        "type": "Base64 High Entropy String",
-        "filename": ".github/workflows/build-deploy-pudl.yml",
-        "hashed_secret": "99359e3ddedb1602025485e4ff9017fa7d71cc77",
-        "is_verified": false,
-        "line_number": 110,
-        "is_secret": false
-      }
-    ],
-    ".github/workflows/deploy-pudl.yml": [
-      {
-        "type": "Base64 High Entropy String",
-        "filename": ".github/workflows/deploy-pudl.yml",
-        "hashed_secret": "99359e3ddedb1602025485e4ff9017fa7d71cc77",
-        "is_verified": false,
-        "line_number": 68,
-        "is_secret": false
-      }
-    ],
     "docker/dagster.yaml": [
       {
         "type": "Secret Keyword",
@@ -240,5 +216,5 @@
       }
     ]
   },
-  "generated_at": "2026-03-29T17:53:48Z"
+  "generated_at": "2026-04-01T15:17:31Z"
 }

Switch from pre-commit to prek, a parallelized Rust-based drop-in replacement, to speed up hook execution during development. Also fixes several unquoted shell variable warnings (SC2086) in GitHub Actions workflows and updates the dataset label sed script to handle any whitespace. - Replace `pre-commit` dependency with `prek` in pyproject.toml - Rename pixi tasks: pre-commit-{run,install,autoupdate} → prek-{run,install,autoupdate} - Update AGENTS.md and dev_setup.rst to reference prek commands and docs - Update update-lockfiles.yml to call prek-autoupdate - Fix unquoted $GITHUB_OUTPUT and dataset arg in update-dois.yml - Update sed pattern to handle any whitespace: `sed -E 's/[[:space:]]+/, /g'` - Regenerate pixi.lock file to reflect the new dependencies. - Add an exception to security screening pre-commit hook for WIF tokens

review-notebook-app · 2026-04-01T19:29:57Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

zaneselvans · 2026-04-01T19:30:30Z

@jdangerx I've replaced pre-commit with prek if you'd like to take another look.

zaneselvans · 2026-04-01T19:32:03Z

.github/workflows/build-deploy-docs.yml

        run: |
          DEST_DIR=${{ case(github.ref == 'refs/heads/main', 'latest', github.ref) }}
-          echo DEST_DIR=${DEST_DIR#refs/heads/} > $GITHUB_ENV
+          echo DEST_DIR=${DEST_DIR#refs/heads/} > "$GITHUB_ENV"


Fixed some linting issues that came up when I ran prek run --all-files

zaneselvans · 2026-04-01T19:32:29Z

.github/workflows/update-dois.yml

        run: |
          DATASETS="${{ github.event.inputs.datasets }}"
-          LABELS=$(echo "$DATASETS" | sed 's/ \+/, /g')
+          # shellcheck disable=SC2001


ShellCheck is like "Why you wanna use sed for this?"

zaneselvans · 2026-04-01T19:32:52Z

.github/workflows/update-lockfiles.yml

          pixi update --json | pixi exec pixi-diff-to-markdown >> diff.md
          pixi install --locked
-          pixi run pre-commit-autoupdate
+          pixi run prek-autoupdate


I think this is the only place we're programmatically using it.

zaneselvans · 2026-04-01T19:33:23Z

notebooks/work-in-progress/eia191-annual-vs-monthly.ipynb

Not sure how this guy snuck in without clearing the outputs.

zaneselvans · 2026-04-01T19:34:11Z

.pre-commit-config.yaml

+          - "--exclude-lines"
+          - "workload_identity_provider:"


Exclude the WIF

zaneselvans · 2026-04-01T19:34:50Z

.secrets.baseline

Removed the (now) false WIF positives.

zaneselvans · 2026-04-01T19:35:23Z

pyproject.toml

+grpcio = "==1.78.1"
+grpcio-health-checking = "==1.78.1"
+grpcio-status = "==1.78.1"


Sneaking this pin update in so I don't forget.

zaneselvans · 2026-04-01T21:25:30Z

Popped this out and back into the queue because it got hung up installing the pixi environment for some reason

Add trufflehog and detect-secrets to pre-commit hooks

2899118

github-project-automation bot added this to Catalyst Megaproject Mar 29, 2026

github-project-automation bot moved this to New in Catalyst Megaproject Mar 29, 2026

zaneselvans self-assigned this Mar 29, 2026

zaneselvans added developer experience Things that make the developers' lives easier, but don't necessarily directly improve the data. testing Writing tests, creating test data, automating testing, etc. labels Mar 29, 2026

zaneselvans moved this from New to In progress in Catalyst Megaproject Mar 29, 2026

zaneselvans added the cloud Stuff that has to do with adapting PUDL to work in cloud computing context. label Mar 29, 2026

zaneselvans commented Mar 29, 2026

View reviewed changes

jdangerx approved these changes Apr 1, 2026

View reviewed changes

jdangerx moved this from In progress to In review in Catalyst Megaproject Apr 1, 2026

Merge branch 'main' into trufflehog

2be2d6c

zaneselvans moved this from In review to In progress in Catalyst Megaproject Apr 1, 2026

zaneselvans requested a review from jdangerx April 1, 2026 19:30

zaneselvans commented Apr 1, 2026

View reviewed changes

Update PR template to the new prek-run command.

e639a94

zaneselvans changed the title ~~Add trufflehog and detect-secrets to pre-commit hooks~~ Switch from pre-commit to prek; add security pre-commit hooks Apr 1, 2026

zaneselvans moved this from In progress to In review in Catalyst Megaproject Apr 1, 2026

zaneselvans added this pull request to the merge queue Apr 1, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 1, 2026

zaneselvans added this pull request to the merge queue Apr 1, 2026

Merged via the queue into main with commit 884494c Apr 1, 2026
15 checks passed

zaneselvans deleted the trufflehog branch April 1, 2026 22:07

github-project-automation bot moved this from In review to Done in Catalyst Megaproject Apr 1, 2026

Uh oh!

Conversation

zaneselvans commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Documentation

To-do list

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdangerx left a comment

Choose a reason for hiding this comment

Uh oh!

review-notebook-app bot commented Apr 1, 2026

Uh oh!

zaneselvans commented Apr 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zaneselvans commented Apr 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zaneselvans commented Mar 29, 2026 •

edited

Loading