Indexes 5: Adds `spk repo index` subcommand for index generation and updates by dcookspi · Pull Request #1340 · spkenv/spk

dcookspi · 2026-03-20T00:49:12Z

This adds a new repo index subcommand to spk for index generation and updates. It adds the --use-indexes and --no-indexes flags for repository index usage. This also updates resolvo solver to get global variables data from an indexed repository. This allows resolvo to solve without needing to restart its solves.

Indexing

The index is designed to help the solvers with solve times. It doesn't contain enough data to help with other spk operations like building and testing a package.

Indexing can be enabled or disabled in the spk config file. If indexing is enabled, you have to generate an index, with spk repo index, prior to trying to use it. They are not generated on the fly (outside of automated tests).

To generate an index (for the origin):

spk repo index --disable-repo local

To update an existing index, e.g. after a new python package was published:

spk repo index --disable-repo local --update python

The flatbuffer index data is stored in a file in the underlying spfs repo in a index/spk/ sub-directory.

If index use is enabled in the config file, it can be disabled with the --no-indexes command line flag. If index use is disabled by default, it can be enbled with the --use-indexes flag. If index use is enabled, but no index has been generated, spk will fallback to using the underlying repo directly (it acts as it would before this change).

Speed Diferences

Generating the index file on our repo (sizes below) takes about 2 minutes. Updating a package in an existing index, such as after a new build is published, takes a few seconds.

Sample solver time improvements using this indexing

The numbers come from this setup:

an origin repo that has 2245 packages, 23540 versions, 82517 builds (11 erroring, about 30% deprecated), and 141 global vars
the index loads in ~0.0003 seconds unverified, or ~0.2 seconds verified, and is about 76 MB on disk with trimmed down deprecated builds (107 MB with full deprecated builds)
these times are from a rough average of 3-4 runs with index verification disabled
a "toolset" below is a set of requests for the named DCC and our typical in-house plugins and tools

Requests        | Solution size | Num.    | Solve time  |  Indexed solve time, no retries
                | (# packages)  | Retries | (seconds)   |  (seconds)
-----------------------------------------------------------------------------------------
python          |        4      |    1    |     0.17    |   0.03 
boost-python    |        8      |    1    |     0.31    |   0.05 
python-torch    |       37      |    2    |     0.58    |   0.15 
widget toolset  |       60      |    2    |     3.44    |   0.48 
katana toolset  |      181      |   10    |    18.32    |   2.97 
nuke toolset    |      280      |   12    |    24.32    |   4.55 (*)
houdini toolset |      211      |   19    |    37.24    |   6.58 (*)
maya toolset    |      403      |   20    |    59.50    |   9.52 (*)

The indexing doesn't have a noticable impact (to users) on smaller solves. But it allows our larger solves to finish in under 10 seconds, or about 1/6th of the time they currently do. The times marked with (*) are improved further by the changes in PR6: (#1344).

This is the final 5 of 5 chained PRs for adding indexes to spk solves:

codecov · 2026-03-20T02:20:27Z

Codecov Report

❌ Patch coverage is 39.33824% with 165 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/spk-cli/cmd-repo/src/cmd_repo.rs	0.00%	55 Missing ⚠️
crates/spk-storage/src/storage/flatbuffer_index.rs	26.38%	53 Missing ⚠️
crates/spk-storage/src/storage/indexed.rs	50.00%	15 Missing ⚠️
crates/spk-config/src/config.rs	0.00%	14 Missing ⚠️
crates/spk-storage/src/storage/handle.rs	25.00%	12 Missing ⚠️
crates/spk-cli/common/src/flags.rs	75.55%	11 Missing ⚠️
crates/spfs/src/storage/handle.rs	28.57%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

jrray · 2026-03-28T00:36:51Z

+    pub use_indexes: bool,
+
+    /// Do not get the package data from the repo index, always use
+    /// the repo instead. This only applies to non-destructive repo
+    /// operations. This option can be configured as the default in
+    /// spk's config file.
+    #[clap(long, conflicts_with = "use_indexes")]
+    pub no_indexes: bool,


Why have both? Can a "global" option to disable index use despite what an individual repo is configured to do exist at a higher level?

It lets a site (or user) enable indexes in the spk config file (so the default for all uses), and disable them for some command line runs, and visa versa - if a site (or user( disables indexes in the spk config file, this lets them be enabled for some command line runs.

We're likely to enable indexes in the config file, and probably use --no-indexes sometimes (if there's an issue as a workaround, or for testing something). But another site might prefer it the other way around for some reason.

I agree with the concept as you describe the usage pattern but still dislike these two options existing here at the same level in the configuration hierarchy.

We already have a configuration pattern of some config property that can be set in a config file but overridden with an env var (or possibly a command-line option). Having these two with opposite meanings creates confusion about which gets set where and which overrules the other.

This could be a case for needing something other than one or two bool options but use an enum instead:

an option that disables indexes globally and overrules any repo-specific setting

an option that delegates to repo-specific settings (default)

an option that enables indexes globally but doesn't overrule any repo-specific setting (we'd likely use this one in our config file)

(maybe) an option that enables indexes globally and overrules any repo-specific setting, but this one feels questionable

I'd be okay with having a flag like no_indexes that acts as an alias / shortcut for picking the option that globally disables indexes, but this wouldn't map to a setting that lives in the config file.

As per our discussion today, I've updated the spk config file structure to remove the global index settings and have them on each spk repository section, replaced the command line options with a single flag with an enum of values, and changed the defaults to use an index if one exists, except for the local repo.

dcookspi · 2026-04-10T00:53:41Z

Updated the spk repo index --update ... option so it can be specified multiple times, and fixed a couple of bugs related to updating specific package/versions, or a deleted package/version, in the index.

dcookspi · 2026-04-14T17:25:42Z

+                if !update.is_empty() {
+                    // Update the existing index for the given package/version
+                    let start = Instant::now();
+                    let idents: Vec<VersionIdent> = update


Todo:

Swap OptVersionIdent in for VersionIdent for the list of things to update.

I've swapped this in and removed the reliance on the default 0.0.0 version.

… flabuffers index and configuration Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Adds --use-indexes and --no-indexes flags to repository. Updates resolvo solver to get global variables data from an indexed repository. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

…ndles. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

…dex use. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

… option Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

times, and fixes bugs when using it to update a specific package version. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

instead of just the ones for the packages being updated. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

jrray · 2026-05-05T19:04:41Z

+
+            // spk repo index ...
+            Self::Index { repo, update } => {
+                // Generate or update an index a repo. The repo must


Suggested change

// Generate or update an index a repo. The repo must

// Generate or update an index in a repo. The repo must

jrray · 2026-05-05T19:07:38Z

+                        Err(err) => {
+                            // There isn't an existing index, so generate one from scratch that
+                            // will also include the update package version.


Can you match on a more specific error before assuming the problem is that the index doesn't exist?

jrray · 2026-05-05T21:12:05Z

+/// The index use command line setting options that can be used to
+/// override index usage set in the spk config file. The default for a


Suggested change

/// The index use command line setting options that can be used to

/// override index usage set in the spk config file. The default for a

/// The options that can be used to

/// override index usage set in the spk config file. The default for a

I had a really hard time parsing that sentence.

jrray · 2026-05-05T21:14:47Z

+/// using the matching environment variables.
+#[derive(Copy, Clone, PartialEq, Eq, PartialOrd, Ord, ValueEnum)]
+pub enum IndexUse {
+    /// Use the index use settings from the repository configurations


Suggested change

/// Use the index use settings from the repository configurations

/// Use the settings from the repository configurations

It's the "Use ... use" I'm having trouble with.

jrray · 2026-05-05T21:20:41Z

+        // Check whether using the indexes for the repos is globally
+        // disabled by the spk command, such as 'spk repo index' or
+        // 'spk info'.
+        let disable_all_index_use = DISABLE_INDEX_USE.load(std::sync::atomic::Ordering::Relaxed);


Using Relaxed here is inconsistent with using Release on line 70, which would imply pairing with Acquire here. However, since this is only for doing an atomic read/write, it is fine to use Relaxed in both places. Either way, this load is not strictly guaranteed to see a prior store. Use Mutex<bool> for that guarantee.

jrray · 2026-05-05T22:42:22Z

+packages at a once, e.g.:
+`spk repo index -r origin --update python --update zlib`
+
+The `--update` option take a package/version as well. This lets the


Suggested change

The `--update` option take a package/version as well. This lets the

The `--update` option takes a package/version as well. This lets the

jrray · 2026-05-05T22:42:48Z

+
+The `--update` option take a package/version as well. This lets the
+update be restricted to a specific version of a package. This can make
+for shorter update times for packages with large numbers of versions,


Suggested change

for shorter update times for packages with large numbers of versions,

for shorter update times for packages with a large number of versions,

jrray · 2026-05-05T22:43:20Z

+Those commands will read in the existing index for the repository and
+update the versions and builds of the named package in the index. It
+is faster than generating an index from scratch. It has to be run once
+per repository to update the given package or packages in that


Suggested change

per repository to update the given package or packages in that

per repository to update the given package or packages in each

jrray · 2026-05-05T22:44:30Z

+See `spk repo index -h` for more details.
+
+
+## Index vs Repository mismatches - updates are important


The heading capitalization in this document is inconsistent (or I'm not seeing the pattern). Please pick one style.

jrray · 2026-05-05T22:46:37Z

+spfs fs repository.
+
+
+### Structure and types in SPK


Another consistency nit, we can't decide if it's Spk or SPK.

dcookspi self-assigned this Mar 20, 2026

dcookspi added enhancement New feature or request SPI AOI Area of interest for SPI pr-chain This PR doesn't target the main branch, don't merge! labels Mar 20, 2026

dcookspi changed the title ~~Indexes 5: Adds 'spk repo index' subcommand for index generation and updates~~ Indexes 5: Adds spk repo index subcommand for index generation and updates Mar 20, 2026

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from 9da2f7c to d88602e Compare March 20, 2026 01:22

dcookspi requested review from jrray and rydrman March 20, 2026 19:15

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from d88602e to 027a156 Compare March 20, 2026 19:27

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 8530adc to 68bb519 Compare March 20, 2026 19:30

dcookspi commented Mar 20, 2026

View reviewed changes

Comment thread crates/spk-cli/cmd-repo/src/cmd_repo.rs Outdated

dcookspi commented Mar 20, 2026

View reviewed changes

Comment thread crates/spk-cli/common/src/flags.rs

dcookspi commented Mar 20, 2026

View reviewed changes

Comment thread crates/spk-cli/group4/src/cmd_view.rs

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 68bb519 to 1e28bf4 Compare March 20, 2026 19:53

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from 027a156 to cdaf8f9 Compare March 20, 2026 19:55

dcookspi commented Mar 20, 2026

View reviewed changes

Comment thread crates/spk-storage/src/storage/flatbuffer_index.rs

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 1e28bf4 to 854446a Compare March 21, 2026 01:07

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from cdaf8f9 to b70cba2 Compare March 21, 2026 01:08

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 854446a to a58142a Compare March 25, 2026 01:13

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from b70cba2 to 4698954 Compare March 25, 2026 01:15

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch 2 times, most recently from b32582c to 1a43f64 Compare March 27, 2026 19:30

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch 2 times, most recently from b8f7428 to 3871cc5 Compare March 27, 2026 23:28

jrray reviewed Mar 28, 2026

View reviewed changes

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch 2 times, most recently from 39f3a57 to 50c68f1 Compare April 9, 2026 18:17

dcookspi requested a review from jrray April 10, 2026 00:54

dcookspi commented Apr 14, 2026

View reviewed changes

This was referenced Apr 16, 2026

Indexes 7: Adds a lock file around index generation and updates #1354

Open

Indexes 8: Fixes the spk build or mkb crash with indexes enabled #1355

Open

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from e5ee8b6 to 07eb6a8 Compare April 17, 2026 19:34

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from 5540bc7 to ab04c9f Compare April 17, 2026 19:55

dcookspi mentioned this pull request Apr 20, 2026

Indexes 9 - Adds messaging on package events to kafka #1356

Open

dcookspi force-pushed the index-4-indexed-repository-and-fbindex branch from 07eb6a8 to a3c15eb Compare April 30, 2026 22:16

dcookspi added 17 commits April 30, 2026 15:39

Adds new Indexed repository type, related RepositoryIndex traits, and…

8d68d35

… flabuffers index and configuration Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Adds 'spk repo index' subcommand for index generation and updates.

4df7970

Adds --use-indexes and --no-indexes flags to repository. Updates resolvo solver to get global variables data from an indexed repository. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Adds the indexing settings to the config file docs

34c2a47

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Adds deprecated builds count to index generation output

60bb597

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Disables index use for spk info command

399f0d6

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates config docs

9044083

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Adds index docs to reference seciton

289b83c

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Moves index location path creation and access into spfs repository ha…

bb66e4f

…ndles. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates docs on indexes

3e592be

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates to spk config file, command line options, and defaults for in…

4f30958

…dex use. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates spk repo index command to just allow -r reponame command line…

45e4956

… option Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates index docs

051989b

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates spk repo index --update option so it can be specified multiple

073a4c0

times, and fixes bugs when using it to update a specific package version. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Fixed bug caused by reading all version lists from the repository,

51bb6ac

instead of just the ones for the packages being updated. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Changes index updates to use OptVersionIdent

d8b735a

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Cleans the index update variables and comments

3b026bb

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

Updates index/repository configuration file docs

7556016

Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>

dcookspi force-pushed the index-5-repo-cmds-config-and-cli-flags branch from 52b5be2 to 7556016 Compare May 1, 2026 17:40

jrray requested changes May 5, 2026

View reviewed changes

	// Generate or update an index a repo. The repo must
	// Generate or update an index in a repo. The repo must

		/// The index use command line setting options that can be used to
		/// override index usage set in the spk config file. The default for a

	/// Use the index use settings from the repository configurations
	/// Use the settings from the repository configurations

	The `--update` option take a package/version as well. This lets the
	The `--update` option takes a package/version as well. This lets the

	for shorter update times for packages with large numbers of versions,
	for shorter update times for packages with a large number of versions,

	per repository to update the given package or packages in that
	per repository to update the given package or packages in each

		See `spk repo index -h` for more details.


		## Index vs Repository mismatches - updates are important

Conversation

dcookspi commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Indexing

Speed Diferences

Sample solver time improvements using this indexing

Uh oh!

codecov Bot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcookspi commented Apr 10, 2026

Uh oh!

dcookspi Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dcookspi commented Mar 20, 2026 •

edited

Loading

codecov Bot commented Mar 20, 2026 •

edited

Loading

dcookspi Apr 14, 2026 •

edited

Loading