feat(install): defer LUKS encryption to first boot

[andrewdunndev]

Summary

Defer LUKS encryption to first boot instead of running cryptsetup inside the install container. This eliminates the IPC namespace semaphore deadlock (#2089) and the shim/PCR mismatch problem (#421) in one change, since TPM2 binding happens on real hardware with real firmware state.

Prior art: openSUSE disk-encryption-tool, which ships a production implementation of first-boot encryption using the same cryptsetup reencrypt --encrypt mechanism.

Problem

bootc install to-disk --block-setup tpm2-luks runs cryptsetup luksFormat, systemd-cryptenroll, and cryptsetup luksOpen inside the install container. This has two problems:

1. IPC namespace deadlock (install to-disk --block-setup tpm2-luks hangs: libdevmapper udev cookie semaphore deadlock in container IPC namespace #2089): libdevmapper uses SysV semaphores to coordinate with udevd. Inside a container with an isolated IPC namespace, udevd on the host cannot see the container's semaphores, causing luksOpen and luksClose to hang on semop().

2. Shim/PCR mismatch (install to-disk with LUKS + TPM broken #421): TPM2 enrollment during install binds to the container's firmware state, not the installed system's firmware. On first real boot, the PCR values differ and auto-unlock fails.

Approach

Install time: Write an unencrypted root partition with the filesystem created 32MB smaller than the partition. Add rd.bootc.luks.encrypt=tpm2 to the kernel command line. No cryptsetup calls, no devmapper, no TPM2.

First boot: A dracut module (51bootc) installs a systemd service that runs before sysroot.mount. The service calls cryptsetup reencrypt --encrypt --reduce-device-size 32M to encrypt the root partition in-place using the reserved 32MB for the LUKS2 header. It then enrolls TPM2 via systemd-cryptenroll, writes /etc/crypttab, and updates the BLS entry with rd.luks.uuid / rd.luks.name kargs.

The root=UUID=<ext4-uuid> karg does not change. Once systemd-cryptsetup unlocks LUKS on subsequent boots, the ext4 UUID inside becomes visible and root= resolves normally.

Changes

crates/lib/src/install/baseline.rs

• Replace the Tpm2Luks arm: remove all cryptsetup/cryptenroll calls
• Add mkfs_with_reserve() that creates the filesystem smaller than the partition (ext4: block count arg, XFS: -d size=, btrfs: -b)
• Set rd.bootc.luks.encrypt=tpm2 karg instead of luks.uuid
• Set luks_device = None (no luksClose needed)

crates/initramfs/dracut/module-setup.sh

• Install bootc-luks-firstboot.sh and .service into the initramfs
• Install cryptsetup, systemd-cryptenroll, and dependencies
• Add dm_crypt kernel module
• Create sysroot.mount.requires symlink for service ordering

crates/initramfs/luks-firstboot/bootc-luks-firstboot.sh (new)

• First-boot encryption script (190 lines)
• Parses rd.bootc.luks.encrypt karg from /proc/cmdline
• Idempotent: checks cryptsetup isLuks before encrypting
• Encrypts via cryptsetup reencrypt --encrypt --reduce-device-size 32M
• Enrolls TPM2 via systemd-cryptenroll --tpm2-device=auto
• Generates recovery key via systemd-cryptenroll --recovery-key
• Updates /etc/crypttab and BLS entries

crates/initramfs/bootc-luks-firstboot.service (new)

• Runs Before=sysroot.mount in the initrd
• ConditionKernelCommandLine=rd.bootc.luks.encrypt
• OnFailure=emergency.target (drops to shell on failure)

Makefile

• Install the first-boot script to /usr/lib/bootc/

Testing

Full end-to-end validation on GCP n2-standard-8 with nested KVM, Fedora 42 (cryptsetup 2.8.4, systemd 257.11, QEMU 9.2.4 + OVMF + swtpm).

Build verification

Test	Result
cargo check	PASS
cargo build --release	PASS (4m05s)
cargo clippy -p bootc-lib	PASS (no new warnings)

Install verification

Installed with patched bootc binary via bootc install to-disk --block-setup tpm2-luks --filesystem ext4 to a 20GB disk.

Test	Result
Root partition not LUKS after install	PASS
Filesystem 32MB smaller than partition	PASS (33,554,432 bytes reserved)
rd.bootc.luks.encrypt=tpm2 in BLS entry	PASS
Separate /boot partition created	PASS
No cryptsetup calls during install	PASS (verified via serial console)

Encryption verification

Manually encrypted the installed root partition using the same cryptsetup reencrypt --encrypt --reduce-device-size 32M command the first-boot script uses.

Test	Result
20GB root encrypted in-place	PASS (110 seconds)
e2fsck clean after encryption	PASS
ostree deployment data intact	PASS
crypttab written to ostree deploy	PASS
BLS entry updated with rd.luks.uuid	PASS

Boot verification

Booted the encrypted system in QEMU with swtpm (vTPM 2.0). The initramfs was patched to include the first-boot dracut module with systemd-cryptsetup support.

Test	Result
systemd-cryptsetup@cr_root.service started	PASS
Passphrase prompt displayed	PASS
LUKS unlocked via passphrase	PASS
/dev/mapper/cr_root device created	PASS
sysroot.mount succeeded (ostree root)	PASS
System booted to login prompt	PASS

Serial console output (key lines):

Starting systemd-cryptsetup@cr_root.service - Cryptography Setup for cr_root...
Please enter passphrase for disk root (cr_root):
Finished systemd-cryptsetup@cr_root.service - Cryptography Setup for cr_root.
Reached target blockdev@dev-mapper-cr_root.target - Block Device Preparation for /dev/mapper/cr_root.
Mounted sysroot.mount - /sysroot.
...
fedora login:

Encryption mechanism validation (independent)

Tested cryptsetup reencrypt --encrypt --reduce-device-size 32M independently across multiple filesystem types and sizes.

Filesystem	Size	Time	Data Integrity	Result
ext4	2GB	10s	MD5 verified	PASS
XFS	2GB	12s	MD5 verified	PASS
ext4	3.5GB	16s	MD5 verified	PASS
ext4	20GB	110s	e2fsck clean	PASS

Additional mechanism tests:

• Idempotency: cryptsetup isLuks detects existing LUKS, reencrypt --encrypt rejects already-encrypted devices ("Device is already LUKS device. Aborting.")
• LUKS2 header: 16MB data offset with 32MB reserve (extra 16MB becomes usable space)
• 16MB reserve also works, 32MB used as safety margin per cryptsetup convention

Not tested (requires project CI)

• TPM2 auto-unlock on second boot (requires TPM2 enrollment during first-boot encryption, which needs the first-boot script running in a real initrd built by the project's RPM/image pipeline)
• Recovery key generation and use
• The first-boot script running automatically via the dracut module (tested manually; automatic execution requires the initramfs to be built by the project's standard image build process, not a Containerfile overlay)

Fixes: #2089
Related: #421, #476, #477

Signed-off-by: Andrew Dunn andrew@dunn.dev
AI-Assisted: yes
AI-Tools: GitLab Duo, OpenCode

AI-Generated Content Disclosure: This PR contains code generated with assistance from GitLab Duo and OpenCode. The output has been reviewed for correctness, tested, and validated against project requirements per GitLab's AI contribution guidelines.

[gemini-code-assist]

Code Review

This pull request introduces a new first-boot LUKS encryption mechanism for bootc. It defers the actual encryption and TPM2 enrollment of the root partition from installation time to the first boot, addressing issues with TPM2 enrollment in the install environment. This is achieved by reserving space for the LUKS header during installation, adding a new bootc-luks-firstboot.sh script and systemd service to the initrd, and updating the dracut module to include necessary components. Review comments highlight a critical flaw in the idempotency logic of the first-boot encryption script, which could leave the system unbootable if interrupted, and suggest a more robust method for parsing kernel command-line arguments.

[cgwalters]

Right now bootc is generally more of a mechanism that tries to be flexible and not hardcode much policy. There's a lot of opinions in this code and things that people would want to configure that I think are just out of scope. For example, waiting to allow the user to capture the recovery key.

Some use cases will want a lot more than just --tpm2-device=auto like PCR configuration.

Almost all load-bearing code in bootc is in Rust, and I have had very painful experiences debugging complex bash code in the initramfs and would like to not ever do that again.

I think ultimately this type of logic really does need to live in the operating system side. systemd has done a great job in shipping configurable tools for this LUKS setup. It was my mistake in wrapping one small subset of this in tpm2-luks for bootc.

Also, this whole topic very strongly intersects with composefs DPS support, and I would like to try to preserve that. I have some PoC patches in that respect.

We will try to work on this problem domain, but it's security related and needs some care. At least for me, a near term focus will be on using Ignition for this, because it's already supported there. If your use case can e.g. derive from Fedora CoreOS and use Ignition, that would likely help.

That said also AFAICS if you are using to-disk - the main thing that would be helpful to do here in bootc is leave an opinionated gap at install time - nothing blocks you from shipping this bash code in an initramfs in your derived image today right?

[andrewdunndev]

Thanks for the detailed response. This is helpful context.

A few things:

On mechanism vs policy: Understood. The script is opinionated and the right place for this logic is in the image, not in bootc. We'll ship the firstboot encryption as a custom dracut module in our derived image.

On composefs + DPS: We're actively evaluating composefs mode 2 for our estate (Fedora 42 + systemd-boot + ZFS + NVIDIA). We'd be interested in the DPS/repart direction for disk encryption -- is the idea that systemd-repart with repart.d configs in the image would handle LUKS setup at first boot? That would align well with the "mechanism in systemd, policy in the image" model. We saw #448 and #1772 -- is there more to look at?

On Ignition: Our setup is bare metal homelab, not CoreOS-derived. We use pyinfra for provisioning. But the pattern of "install plain, encrypt on first boot" is what we're after regardless of the provisioning tool.

On shipping it ourselves: Yes, nothing blocks us from including the dracut module in our image today. We'll do that. If the repart/DPS direction matures to where bootc can hand off disk encryption to systemd-repart, we'd happily switch to that.

We'll close this PR. The DM_DISABLE_UDEV fix in #2090 is separate and still relevant for the existing tpm2-luks path.