Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-deploy-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
- name: Compute destination dir
run: |
DEST_DIR=${{ case(github.ref == 'refs/heads/main', 'latest', github.ref) }}
echo DEST_DIR=${DEST_DIR#refs/heads/} > $GITHUB_ENV
echo DEST_DIR=${DEST_DIR#refs/heads/} > "$GITHUB_ENV"
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed some linting issues that came up when I ran prek run --all-files


- name: Deploy to GitHub Pages at the appropriate version path
uses: peaceiris/actions-gh-pages@v4
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/deploy-pudl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,7 @@ jobs:
- name: Determine container image to use
id: container-image
run: |
echo "image=docker.io/catalystcoop/pudl-etl@${{ steps.docker_build.outputs.digest }}" >> $GITHUB_OUTPUT

echo "image=docker.io/catalystcoop/pudl-etl@${{ steps.docker_build.outputs.digest }}" >> "$GITHUB_OUTPUT"
- id: "auth"
uses: "google-github-actions/auth@v3"
with:
Expand Down
9 changes: 5 additions & 4 deletions .github/workflows/update-dois.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,22 +34,23 @@ jobs:
- name: Run Zenodo DOI updater
id: update
run: |
pixi run update_zenodo_dois ${{ github.event.inputs.datasets }}
pixi run update_zenodo_dois "${{ github.event.inputs.datasets }}"

# Check if any changes were made
if git diff --quiet; then
echo "changes=false" >> $GITHUB_OUTPUT
echo "changes=false" >> "$GITHUB_OUTPUT"
echo "No changes detected"
else
echo "changes=true" >> $GITHUB_OUTPUT
echo "changes=true" >> "$GITHUB_OUTPUT"
echo "Changes detected!"
fi

- name: Format datasets as labels
id: format-datasets
run: |
DATASETS="${{ github.event.inputs.datasets }}"
LABELS=$(echo "$DATASETS" | sed 's/ \+/, /g')
# shellcheck disable=SC2001
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShellCheck is like "Why you wanna use sed for this?"

LABELS=$(echo "$DATASETS" | sed -E 's/[[:space:]]+/, /g')
echo "dataset_labels=$LABELS" >> "$GITHUB_OUTPUT"

- name: Create Pull Request
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/update-lockfiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
set -o pipefail
pixi update --json | pixi exec pixi-diff-to-markdown >> diff.md
pixi install --locked
pixi run pre-commit-autoupdate
pixi run prek-autoupdate
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the only place we're programmatically using it.

PUDL_OUTPUT="/home/runner/pudl-work/output/" pixi run dbt-deps-upgrade
- name: Make a PR to merge updated lockfiles
# If we are relocking dependencies on a schedule or workflow_dispatch, we need
Expand Down
24 changes: 23 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,28 @@ repos:
hooks:
- id: actionlint

# Scan for secrets and credentials
- repo: https://github.com/trufflesecurity/trufflehog
rev: v3.94.2
hooks:
- id: trufflehog
name: TruffleHog
entry: trufflehog git file://. --since-commit HEAD --fail
language: golang
pass_filenames: false

- repo: https://github.com/Yelp/detect-secrets
rev: v1.5.0
hooks:
- id: detect-secrets
args:
- "--baseline"
- ".secrets.baseline"
- "--exclude-lines"
- "workload_identity_provider:"
Comment on lines +115 to +116
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exclude the WIF

- "--exclude-files"
- "(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)"
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a full scan with no exclusions first to see where it would find false positives, and created this filter based on those results.


#####################################################################################
# Our own pre-commit hooks, which don't come from the pre-commit project
#####################################################################################
Expand Down Expand Up @@ -150,5 +172,5 @@ ci:
autoupdate_branch: main
autoupdate_commit_msg: "[pre-commit.ci] pre-commit autoupdate"
autoupdate_schedule: weekly
skip: [unit-tests, nb-output-clear, shellcheck]
skip: [unit-tests, nb-output-clear, shellcheck, trufflehog, detect-secrets]
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They need some resources not available on pre-commit.ci so they don't work there.

submodules: false
220 changes: 220 additions & 0 deletions .secrets.baseline
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the (now) false WIF positives.

Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
{
"version": "1.5.0",
"plugins_used": [
{
"name": "ArtifactoryDetector"
},
{
"name": "AWSKeyDetector"
},
{
"name": "AzureStorageKeyDetector"
},
{
"name": "Base64HighEntropyString",
"limit": 4.5
},
{
"name": "BasicAuthDetector"
},
{
"name": "CloudantDetector"
},
{
"name": "DiscordBotTokenDetector"
},
{
"name": "GitHubTokenDetector"
},
{
"name": "GitLabTokenDetector"
},
{
"name": "HexHighEntropyString",
"limit": 3.0
},
{
"name": "IbmCloudIamDetector"
},
{
"name": "IbmCosHmacDetector"
},
{
"name": "IPPublicDetector"
},
{
"name": "JwtTokenDetector"
},
{
"name": "KeywordDetector",
"keyword_exclude": ""
},
{
"name": "MailchimpDetector"
},
{
"name": "NpmDetector"
},
{
"name": "OpenAIDetector"
},
{
"name": "PrivateKeyDetector"
},
{
"name": "PypiTokenDetector"
},
{
"name": "SendGridDetector"
},
{
"name": "SlackDetector"
},
{
"name": "SoftlayerDetector"
},
{
"name": "SquareOAuthDetector"
},
{
"name": "StripeDetector"
},
{
"name": "TelegramBotTokenDetector"
},
{
"name": "TwilioKeyDetector"
}
],
"filters_used": [
{
"path": "detect_secrets.filters.allowlist.is_line_allowlisted"
},
{
"path": "detect_secrets.filters.common.is_baseline_file",
"filename": ".secrets.baseline"
},
{
"path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
"min_level": 2
},
{
"path": "detect_secrets.filters.heuristic.is_indirect_reference"
},
{
"path": "detect_secrets.filters.heuristic.is_likely_id_string"
},
{
"path": "detect_secrets.filters.heuristic.is_lock_file"
},
{
"path": "detect_secrets.filters.heuristic.is_not_alphanumeric_string"
},
{
"path": "detect_secrets.filters.heuristic.is_potential_uuid"
},
{
"path": "detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign"
},
{
"path": "detect_secrets.filters.heuristic.is_sequential_string"
},
{
"path": "detect_secrets.filters.heuristic.is_swagger_file"
},
{
"path": "detect_secrets.filters.heuristic.is_templated_secret"
},
{
"path": "detect_secrets.filters.regex.should_exclude_file",
"pattern": [
"(?x)(dbt/package-lock\\.yml|.*\\.ipynb|docs/.*\\.html|migrations/.*|skills-lock\\.json)"
]
},
{
"path": "detect_secrets.filters.regex.should_exclude_line",
"pattern": [
"workload_identity_provider:"
]
}
],
"results": {
"docker/dagster.yaml": [
{
"type": "Secret Keyword",
"filename": "docker/dagster.yaml",
"hashed_secret": "a8b730d9cb75d147a6ab37ab39036e47f567513b",
"is_verified": false,
"line_number": 5,
"is_secret": false
}
],
"src/pudl/analysis/timeseries_evaluation.py": [
{
"type": "Base64 High Entropy String",
"filename": "src/pudl/analysis/timeseries_evaluation.py",
"hashed_secret": "a6d0b3075cbbd4c9a3141d744954036b00fe91be",
"is_verified": false,
"line_number": 70,
"is_secret": false
}
],
"terraform/main.tf": [
{
"type": "Secret Keyword",
"filename": "terraform/main.tf",
"hashed_secret": "790739d6bad41c70225e325068b8f8fc9a61628b",
"is_verified": false,
"line_number": 131,
"is_secret": false
}
],
"terraform/pudl-usage-metrics-dashboard.tf": [
{
"type": "Secret Keyword",
"filename": "terraform/pudl-usage-metrics-dashboard.tf",
"hashed_secret": "10b642e314d4e2aaab3fd757c06a18971d02a746",
"is_verified": false,
"line_number": 72,
"is_secret": false
},
{
"type": "Secret Keyword",
"filename": "terraform/pudl-usage-metrics-dashboard.tf",
"hashed_secret": "37e891b7e8956f04d734c16d66eeec245078478c",
"is_verified": false,
"line_number": 77,
"is_secret": false
}
],
"terraform/pudl-viewer.tf": [
{
"type": "Secret Keyword",
"filename": "terraform/pudl-viewer.tf",
"hashed_secret": "3e8db137aa8c9ed287c81c4b79d43e817e86af7c",
"is_verified": false,
"line_number": 95,
"is_secret": false
},
{
"type": "Secret Keyword",
"filename": "terraform/pudl-viewer.tf",
"hashed_secret": "f2636c63dbd332b36f892a92d1ddd4c706ecc71b",
"is_verified": false,
"line_number": 100,
"is_secret": false
}
],
"test/unit/workspace/datastore_test.py": [
{
"type": "Hex High Entropy String",
"filename": "test/unit/workspace/datastore_test.py",
"hashed_secret": "05dc54328edc1e335436ea8cc059a52c19d3cf96",
"is_verified": false,
"line_number": 182,
"is_secret": false
}
]
},
"generated_at": "2026-04-01T15:17:31Z"
}
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@
- Pixi environments and tasks are defined in `pyproject.toml` under `[tool.pixi]`
sections.
- PUDL uses ruff to lint and automatically format python code. Before staging files for
a commit, always run `pixi run pre-commit run ruff-check --all-files` and
`pixi run pre-commit run ruff-format --all-files`
a commit, always run `pixi run prek run ruff-check --all-files` and
`pixi run prek run ruff-format --all-files`
- A number of pre-commit hooks are defined in .pre-commit-config.yaml.
- We try to use appropriate type annotations in function, class, and method definitions,
but they are not yet checked or enforced. They are primarily to improve readability
Expand Down
8 changes: 6 additions & 2 deletions docs/dev/dev_setup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -206,14 +206,18 @@ To make sure they are run before you commit any code, you need to enable the

.. code-block:: console

$ pixi run pre-commit-install
$ pixi run prek-install

The scripts that run are configured in the ``.pre-commit-config.yaml`` file.
The scripts that run are configured in the ``.pre-commit-config.yaml`` file. We use
`prek <https://prek.j178.dev>`__, a fast, parallelized drop-in replacement for
pre-commit written in Rust.

.. seealso::

* The `pre-commit project <https://pre-commit.com/>`__: A framework for
managing and maintaining multi-language pre-commit hooks.
* `prek <https://prek.j178.dev>`__: A fast Rust-based replacement for
pre-commit with parallel hook execution.
* `Real Python Code Quality Tools and Best Practices <https://realpython.com/python-code-quality/>`__
gives a good overview of available linters and static code analysis tools.

Expand Down
Loading