Skip to content

Indexes 6: Changes version_filter field in index schema#1344

Open
dcookspi wants to merge 3 commits into
index-5-repo-cmds-config-and-cli-flagsfrom
index-5-zplus-1-versionfilters-not-strings
Open

Indexes 6: Changes version_filter field in index schema#1344
dcookspi wants to merge 3 commits into
index-5-repo-cmds-config-and-cli-flagsfrom
index-5-zplus-1-versionfilters-not-strings

Conversation

@dcookspi
Copy link
Copy Markdown
Collaborator

@dcookspi dcookspi commented Mar 28, 2026 •
edited
Loading

This changes the version_filter field in the index schema to a flatbuffers list and table representation, instead of a string. This allows the version filter rules (VersionRange entries) to be extracted from the index without string parsing.

The resulting index is slightly larger at 122 MB (see Indexes PR 5: #1340). Index generation time is about the same. But the solve times improve for the larger solves (marked with *):

Requests        | Solution size | Num.    | Solve time  |  Indexed solve time |  Solve time now
                | (# packages)  | Retries | (seconds)   |  (seconds)          |  (seconds)  
------------------------------------------------------------------------------------------------
python          |        4      |    1    |     0.17    |   0.03              |  0.04
boost-python    |        8      |    1    |     0.31    |   0.05              |  0.07
python-torch    |       37      |    2    |     0.58    |   0.15              |  0.14
widget toolset  |       60      |    2    |     3.44    |   0.48              |  0.38
katana toolset  |      181      |   10    |    18.32    |   2.97              |  2.14
nuke toolset    |      280      |   12    |    24.32    |   4.55              |  1.70  (*)
houdini toolset |      211      |   19    |    37.24    |   6.58              |  3.78  (*)
maya toolset    |      403      |   20    |    59.50    |   9.52              |  5.30  (*)

This is the 6th of the chained PRs for adding indexes to spk solves:

  1. Indexes 1: Change Package and related traits to not return references to fields #1336
  2. Indexes 2: Add new_unchecked() constructors to spk schema objects #1337
  3. Indexes 3: Adds flatbuffers schema and SolverPackageSpec for indexes to spk #1338
  4. Indexes 4: Adds Indexes for SPK repositories #1339
  5. Indexes 5: Adds spk repo index subcommand for index generation and updates #1340
  6. this PR
  7. Indexes 7: Adds a lock file around index generation and updates #1354
  8. Indexes 8: Fixes the spk build or mkb crash with indexes enabled #1355
  9. Indexes 9: Adds messaging on package events to kafka #1356

@dcookspi dcookspi self-assigned this Mar 28, 2026
@dcookspi dcookspi added enhancement New feature or request SPI AOI Area of interest for SPI pr-chain This PR doesn't target the main branch, don't merge! labels Mar 28, 2026
@dcookspi dcookspi requested review from jrray and rydrman March 28, 2026 01:02
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 28, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 37.56345% with 123 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/spk-schema/src/fb_converter.rs 39.24% 96 Missing ⚠️
...-schema/crates/foundation/src/version_range/mod.rs 30.76% 27 Missing ⚠️

📢 Thoughts on this report? Let us know!

This was referenced Mar 30, 2026
Open
Open
Open
@dcookspi dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch 2 times, most recently from 83c8d12 to 3dcf4e3 Compare April 2, 2026 01:18
@dcookspi dcookspi force-pushed the index-5-zplus-1-versionfilters-not-strings branch 2 times, most recently from 95154e5 to 6e12345 Compare April 2, 2026 16:43
jrray
jrray reviewed Apr 2, 2026
View reviewed changes
Comment thread crates/spk-proto/schema/spk.fbs Outdated
jrray
jrray requested changes Apr 2, 2026
View reviewed changes
Comment thread crates/spk-schema/src/fb_converter.rs Outdated
jrray
jrray reviewed Apr 2, 2026
View reviewed changes
Comment thread crates/spk-schema/src/fb_converter.rs
@dcookspi dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch 3 times, most recently from 39f3a57 to 50c68f1 Compare April 9, 2026 18:17
@dcookspi dcookspi force-pushed the index-5-zplus-1-versionfilters-not-strings branch 2 times, most recently from c3fb021 to 81fb685 Compare April 13, 2026 20:52
@dcookspi dcookspi requested a review from jrray April 14, 2026 00:04
This was referenced Apr 16, 2026
Open
Open
Merged
Open
@dcookspi dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from 5540bc7 to ab04c9f Compare April 17, 2026 19:55
@dcookspi dcookspi force-pushed the index-5-zplus-1-versionfilters-not-strings branch 2 times, most recently from f31a655 to d9f7cef Compare April 17, 2026 22:53
@dcookspi dcookspi mentioned this pull request Apr 20, 2026
Open
@dcookspi dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from 52b5be2 to 7556016 Compare May 1, 2026 17:40
dcookspi added 2 commits May 1, 2026 10:42
@dcookspi
Changes version_filter field in flatbuffer schema to a flatbuffers
a423dba 
list/table representation instead of a string. The field no longer
needs to be parsed when extracted from the index.

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi
Updated log messages with qualified VersionRangeOperator names, and r…
7aa00a9 
…enamed Version access methods to version.

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi dcookspi force-pushed the index-5-zplus-1-versionfilters-not-strings branch from d9f7cef to 7aa00a9 Compare May 1, 2026 18:14
@dcookspi
Removes unused new_from_version_str constructor
424f49a 
Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
@dcookspi dcookspi mentioned this pull request May 8, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jrray jrray jrray approved these changes

@rydrman rydrman Awaiting requested review from rydrman

Assignees

@dcookspi dcookspi

Labels

enhancement New feature or request pr-chain This PR doesn't target the main branch, don't merge! SPI AOI Area of interest for SPI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dcookspi @jrray