Conversation
There was a problem hiding this comment.
Pull request overview
Updates the eval-driven-dev skill to improve eval-driven instrumentation guidance and adds a self-check mechanism intended to detect when the skill content is outdated.
Changes:
- Added a
resources/directory containingversion.jsonand acheck_version.pyhelper script. - Extended
skills/eval-driven-dev/SKILL.mdwith a “Version check” step and substantial guidance updates (hard gates, eval-boundary selection, run harness stage). - Updated
docs/README.skills.mdto list the newresourcesfolder for the skill.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| skills/eval-driven-dev/resources/version.json | Adds local version metadata intended for remote version comparison. |
| skills/eval-driven-dev/resources/check_version.py | Adds a script intended to compare local vs remote skill version metadata. |
| skills/eval-driven-dev/SKILL.md | Adds “version check first” guidance and expands the workflow (notably run harness + hard gates). |
| docs/README.skills.md | Updates the skill’s references column to include resources. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
aaronpowell
left a comment
There was a problem hiding this comment.
In the metadata field of the skill frontmatter we can store a version, so a separate "version file" wouldn't be needed.
But more broadly speaking, I'm a little apprehensive on this because it's trying to solve a tooling problem (tool doesn't see updates) with a custom solution.
I'm going to bring this up with the Copilot team on the topic of versioning, as I know it's something that has been discussed but I'm unsure what the status of it is.
Yea I don't like the custom solution as well, it'd be great if copilot can solve this for all skill distribution so I don't have to do this. |
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.Description
Update the SKILL description to carry out instrumentation and dataset generation better, also add a step to check for latest version of SKILL on Github when SKILL is invoked.
Type of Contribution
Additional Notes
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.