cli: add SNP ID block annotations to Pods based on CPU requirements by daniel-weisse · Pull Request #2214 · edgelesssys/contrast

daniel-weisse · 2026-02-26T11:42:56Z

Update reference value generation to create SNP reference values for up to 8 vCPUs
- This can be adjusted at will, but since each CPU variation results in one more entry in the generated manifest, larger numbers will immensely blow up the size of the manifest
Update ID block generation to calculate ID blocks for up to 8 vCPUs
Embed ID block mappings the the CLI and annotate Pods during contrast generate with the ID blocks required for the requested CPU amount

cli/cmd/common.go

cli/cmd/generate.go

packages/by-name/contrast/reference-values/package.nix

packages/by-name/contrast/snp-id-blocks/package.nix

cli/cmd/generate.go

packages/by-name/contrast/reference-values/package.nix

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

msanft

I think this will work, but I'm a little unsure about the current interface we expose to the user.

cli/cmd/common.go

cli/cmd/generate.go

msanft · 2026-03-18T13:55:42Z

cli/cmd/generate.go

+		podLevelCPU := getCPUCount(spec.Resources)
+
+		// Convert milliCPUs to number of CPUs (rounding up), and add 1 for hypervisor overhead
+		totalMilliCPUs := max(regularContainersCPU, initContainersCPU, podLevelCPU)


I wonder if this matches the user's expectations, or what's done by non-Kata Kubernetes here.

What do you think may be unexpected about this formula? I pointed @daniel-weisse to #2272 for where it comes from.

The thing I was wary about is the round-up. With cgroups and CPU slices, this isn't something to worry about. But when a user shifts some YAML that worked in his non-Contrast deployment to Contrast, we may try to use more CPUs than physically available due to this. I don't think this is something that would be a realistic scenario, though. LMK

Understood, thanks. We'll need to document this in https://docs.edgeless.systems/contrast/howto/workload-deployment/deployment-file-preparation#pod-resources before we consider this feature done, yes. I don't see what we could do to not round up, though, since fractional CPUs don't make sense for VMs.

Scheduler considerations might become interesting, though: I don't think there's a way to tell k8s via runtimeClass to round up the limits.

Do you have a concrete idea on how to proceed with this? I don't see what we could do either.

Just document it, recommeding only integral CPU counts. If rounding does not change the number, there are no problems with unexpected counts or scheduling. But if the user decides to go against that recommendation, this code still does the right thing.

packages/by-name/contrast/reference-values/package.nix

msanft · 2026-03-18T14:06:47Z

packages/by-name/kata/runtime/package.nix

          ]
          ++ [
            "panic=1"
-            "nr_cpus=1"


Should we just set this to the maximum value of 8 statically?
As per the docs, this is:

Maximum number of processors that an SMP kernel could support

We could also just omit this, as this number can also be resolved dynamically: https://elixir.bootlin.com/linux/v6.19.8/source/kernel/cpu.c#L3153-L3166

Yes, there's a ticket for removing the param / Markus mentioned this as well yesterday. Thomas at one point suggested setting this to 32.

8 seems a bit low. I'm not super sure if the memory usage increase from removing the limit is a noticeable problem. I'm voting "remove".

From what I read in the docs, this setting is only relevant for VMs that might have CPUs hot-plugged during runtime, which is not the case for us.

No, but apparently setting this to a constant < the Kernel max will save some memory 🤷🏼‍♀️

Yeah, I think this is only relevant for memory pre-reservation. Perhaps also for hot-plugging, which we don't need to support anyway.

nr_cpus= [SMP] Maximum number of processors that an SMP kernel could support. nr_cpus=n : n >= 1 limits the kernel to support 'n' processors. It could be larger than the number of already plugged CPU during bootup, later in runtime you can physically add extra cpu until it reaches n. So during boot up some boot time memory for per-cpu variables need be pre-allocated for later physical cpu hot plugging.

I read this as "the kernel does the right thing for the number of CPUs plugged at boot; if you plan on plugging more later, set this to the number you're aiming for."

Right, that is also how I understood this, with the caveat that not setting this defaults to the kconfig value of CONFIG_NR_CPUS, which in our case is currently 240. Hence the increased memory pre-reservation if the line is removed altogether.

Not arguing against removing this though, as I said above, I'm for it.

I read this as "the kernel does the right thing for the number of CPUs plugged at boot; if you plan on plugging more later, set this to the number you're aiming for."

"n >= 1 limits the kernel to support 'n' processors". If you don't set it, you will be able to hot-plug up to CONFIG_NR_CPUS as Charlotte said. And this requires reservation of some memory.

Since we already limit this to 240 (I didn't know that when I commented 32 in the ticket), it should be okay.

Not specifying `nr_cpus` on the command line costs us marginal amounts of memory while saving complexity in the TDX RTMR pre-calculation. By dropping this from the command line, we make the kernel fall back to the `CONFIG_NR_CPUS=240` kconfig variable.

msanft · 2026-03-24T13:56:43Z

@burgerdev, @charludo; Addressed my own feedback, PTAL.

charludo

fixup changes LGTM; have not looked into the still-open conversation.

daniel-weisse requested a review from burgerdev February 26, 2026 11:42

daniel-weisse added the changelog PRs that should be part of the release notes label Feb 26, 2026

daniel-weisse force-pushed the dw/cli-id-block-generation branch 7 times, most recently from 2287b59 to 6de728d Compare March 3, 2026 14:23

daniel-weisse marked this pull request as ready for review March 4, 2026 08:48

daniel-weisse force-pushed the dw/cli-id-block-generation branch 3 times, most recently from 3acb557 to 6fd606e Compare March 5, 2026 14:21

daniel-weisse mentioned this pull request Mar 5, 2026

kata-runtime: select correct SNP ID block for platform at runtime #2227

Open

daniel-weisse requested a review from charludo March 9, 2026 14:35

daniel-weisse force-pushed the dw/cli-id-block-generation branch 2 times, most recently from 0304e90 to f63b942 Compare March 9, 2026 14:48

burgerdev self-assigned this Mar 10, 2026

burgerdev reviewed Mar 10, 2026

View reviewed changes

charludo reviewed Mar 10, 2026

View reviewed changes

cli/cmd/generate.go Outdated Show resolved Hide resolved

packages/by-name/contrast/reference-values/package.nix Outdated Show resolved Hide resolved

daniel-weisse force-pushed the dw/cli-id-block-generation branch from f63b942 to 59f9a72 Compare March 10, 2026 10:11

daniel-weisse added 4 commits March 16, 2026 15:17

snp: calculate ID blocks for up to 8 vCPUs

e0e1d8d

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

cli: generate and add ID Block annotations to Pods

c90688e

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

cli: embedd SNP reference values for up to 8 vCPUs

2efae70

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

nodeinstaller: allow ID block related annotations in Kata

cf6cec9

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

daniel-weisse force-pushed the dw/cli-id-block-generation branch from 59f9a72 to cf6cec9 Compare March 16, 2026 14:17

msanft reviewed Mar 18, 2026

View reviewed changes

msanft added 2 commits March 24, 2026 14:49

fixup! cli: generate and add ID Block annotations to Pods

31d1f3b

msanft requested review from burgerdev and charludo March 24, 2026 13:57

charludo approved these changes Mar 24, 2026

View reviewed changes

Conversation

daniel-weisse commented Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msanft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charludo Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msanft commented Mar 24, 2026

Uh oh!

charludo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

charludo Mar 20, 2026 •

edited

Loading