[DAPS-1800] - Python Client Build Script & CI Job#1811
Open
[DAPS-1800] - Python Client Build Script & CI Job#1811
Conversation
…//github.com/ORNL/DataFed into feat-DAPS-1800-python-client-build-script-ci
Contributor
Reviewer's GuideAdds a Docker-based build pipeline and GitLab CI jobs for the DataFed Python client, including an image for building/publishing the package to PyPI, a local build script, and wiring into existing image-check and deploy stages, plus a small CMake fix for requirements.txt copying and minor whitespace/formatting updates. Sequence diagram for deploy:pypi:manual CI job uploading Python package to PyPIsequenceDiagram
participant Dev as Developer
participant GL as GitLabCI
participant Job as deploy:pypi:manual job
participant Reg as DockerRegistry
participant Img as datafed-python-client-deploy image
participant PyPI as PyPI_Server
Dev->>GL: Click manual job deploy:pypi:manual
GL->>Job: Start job in stage deploy-pypi-package
Job->>Reg: docker login ${REGISTRY} using HARBOR_USER and token
Job->>Job: Compute DOWNSTREAM_SHA from external/DataFedDependencies submodule
Job->>Job: Optionally modify cmake/Version.cmake
Job->>Reg: Pull base dependencies image
Job->>Img: docker build datafed-python-client-deploy:latest
Note over Job,Img: Uses python/datafed_pkg/docker/Dockerfile
Job->>Job: mkdir -p dist
Job->>Img: docker run to copy /dist/* to CI dist/ directory
Job->>Job: Restore Version.cmake if prerelease was enabled
Job->>Job: ls -lh dist/ to verify artifacts
Job->>Img: docker run twine upload dist/*
Img->>PyPI: HTTPS upload of wheels and sdists
PyPI-->>Img: Upload success response
Img-->>Job: Twine exit code 0
Job-->>GL: Job success with dist/ as artifact
GL-->>Dev: Pipeline shows successful PyPI deployment
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey there - I've reviewed your changes - here's some feedback:
- The COMPONENT name for the Python client is inconsistent between jobs (e.g.,
python-clientin.gitlab/build/build_python_client_image.ymlvspython_clientin the image-check and deploy configs), which will likely break shared registry paths and helper scripts that depend on a consistent COMPONENT; consider standardizing on a single identifier. - The Docker build invocation and dependency image resolution logic are duplicated between
scripts/build_python_package.shand.gitlab/stage_deploy_pypi.yml; extracting the common logic into a shared script or template would reduce the chance of the two paths diverging over time.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The COMPONENT name for the Python client is inconsistent between jobs (e.g., `python-client` in `.gitlab/build/build_python_client_image.yml` vs `python_client` in the image-check and deploy configs), which will likely break shared registry paths and helper scripts that depend on a consistent COMPONENT; consider standardizing on a single identifier.
- The Docker build invocation and dependency image resolution logic are duplicated between `scripts/build_python_package.sh` and `.gitlab/stage_deploy_pypi.yml`; extracting the common logic into a shared script or template would reduce the chance of the two paths diverging over time.
## Individual Comments
### Comment 1
<location> `.gitlab-ci.yml:29` </location>
<code_context>
- local: .gitlab/stage_test.yml
- local: .gitlab/stage_unit.yml
- local: .gitlab/end_to_end.yml
+ - local: .gitlab/stage_deploy_pypi.yml
stages:
</code_context>
<issue_to_address>
**issue (bug_risk):** New `deploy-pypi-package` stage is not added to the global `stages:` list, so jobs in that stage may never run
`.deploy_pypi_base` in `.gitlab/stage_deploy_pypi.yml` is configured with `stage: deploy-pypi-package`, but that stage is not declared in the root `.gitlab-ci.yml` `stages:` list. Please add `deploy-pypi-package` there so these jobs run in the correct stage and pipeline order.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
nedvedba
added
Component: Python API
Collaborator
|
@nedvedba, what was the reason for creating a whole new Dockerfile, there is already one in python/docker/Dockerfile for the client. It would probably make a lot of sense to have a single Dockerfile that is building the client possibly with a layer that will spit outs the python package. The dockerfiles in python/docker/ are a bit dated anyways. If you want an example of intermediate image builds take a look at the the web Dockerfile. |
JoshuaSBrown
reviewed
Dec 9, 2025
Comment on lines
+78
to
+83
| # Manual deployment job - can be triggered via button in GitLab UI | ||
| deploy:pypi:manual: | ||
| extends: .deploy_pypi_base | ||
| when: manual | ||
| rules: | ||
| - when: manual |
Collaborator
There was a problem hiding this comment.
I think this is the best choice too.
Collaborator
|
Another point, when making print statements in the scripts. I would recommend prepending: As is appropriate similar to what we are trying to do elsewhere in the repo and logs. |
JoshuaSBrown
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket
#1800
Description
Added python client dockerfile, build script, and CI job.
How Has This Been Tested?
Run through CI job for both datafed pypi repos with successful deployments.
Tasks
Summary by Sourcery
Add Docker-based build and CI/CD pipeline support for the DataFed Python client, including image build, validation, and PyPI deployment.
Enhancements:
Build:
CI: