diff --git a/README.md b/README.md index 3c9d0ab..6de4086 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ This repository contains a custom Linux system to build Android Open Source Project in a (mostly) ephemeral environment. Our images, based on NixOS, provide a FHS-compatible enviroment that can run upstream Androids toolchain while being flexible and relatively easy to adapt due to the NixOS module system. -We boot into memory while keeping build state that's too big for memory in an ephemeral `/var/lib/build` partition on disk. That partition will be expanded and (re-)encrypted with a ephemeral key on each boot. -While no state is persisted between boots by default, there's an option to use a second disk as "artifact storage" to store build outputs in air-gapped environments. +We boot into memory while keeping build state that's too big for memory in an ephemeral `/var/lib/build` partition on disk. That partition is encrypted with a fresh random key on each boot. +Persistent, TPM2-bound LUKS partitions store keylime agent state and systemd-encrypted credentials across reboots. A second disk can optionally be used as "artifact storage" for build outputs in air-gapped environments. See [user-guide.md](./docs/user-guide.md) for usage guidance and [docs.md](./docs/docs.md) for a more detailed description of design considerations, used components limitations, and further work. diff --git a/docs/docs.md b/docs/docs.md index 55166aa..da52da8 100644 --- a/docs/docs.md +++ b/docs/docs.md @@ -32,8 +32,8 @@ We created a modular proof‑of‑concept based on NixOS that fulfills most of t * **[`nixpkgs`](https://github.com/nixos/nixpkgs)** - the software repository that enables reproducible builds of up‑to‑date open‑source packages. * **[`qemu`](https://qemu.org)** - used to run virtual machines during interactive, as well as automated testing. Both help to decrease testing & verification cycles during development & customization. * **[`systemd`](https://systemd.io)** - orchestrates both upstream and custom components while managing credentials and persistent state. -* **[`systemd-repart`](https://www.freedesktop.org/software/systemd/man/latest/systemd-repart.html)** - prepares signable read‑only disk images for the builder and resizes and re‑encrypts the state partition at each boot. -* **[Linux Unified Key Setup (`LUKS`)](https://gitlab.com/cryptsetup/cryptsetup/blob/master/README.md)** - encrypts the state partition with an ephemerally generated key on each boot. +* **[`systemd-repart`](https://www.freedesktop.org/software/systemd/man/latest/systemd-repart.html)** - prepares signable read‑only disk images for the builder and creates encrypted partitions at boot. +* **[Linux Unified Key Setup (`LUKS`)](https://gitlab.com/cryptsetup/cryptsetup/blob/master/README.md)** - encrypts mutable partitions. The ephemeral build partition uses a random key per boot; persistent partitions for credentials and keylime state are TPM2-bound. * **[`Keylime`](https://keylime.dev/)** - TPM-based remote attestation framework. The Rust agent runs on the builder; a Python registrar and verifier run on the attestation server. * Various **build requirements** for Android, such as Python 3 and OpenJDK. The complete list is in the `packages` section of `android-build-env.nix`. @@ -54,7 +54,7 @@ This approach guarantees that the same inputs always generate the same output, m Users with `nix` installed can clone this repository, download all dependencies and build a signed disk image, ready to flash & boot on the build machine, in a few simple steps outlined in [README.md](../README.md). The resulting disk image boots on generic `x86_64` hardware with `UEFI` as well as Secure Boot, and provides an isolated build environment. -It contains scripts for secure boot enrollment, a verified filesystem, and an ephemeral, encrypted state partition that holds build artifacts that cannot fit into memory. +It contains scripts for secure boot enrollment, a verified filesystem, persistent TPM2-bound encrypted partitions for credentials and keylime agent state, and an ephemeral encrypted partition for build artifacts. [^reproducible]: *Reproducible* in functionality. The final disk images are not yet expected to be *fully* bit-by-bit reproducible. That could be done, but would require a long-tail of removing additional sources of indeterminism, such as as date & time of build. See [reproducible.nixos.org](https://reproducible.nixos.org/) @@ -68,7 +68,7 @@ Under the hood, the image itself is built by `systemd-repart`, using NixOS modul `systemd-repart` is called twice during build-time: 1. While building `system.build.intermediateImage`: - A first image is built, it contains the `store` partition, populated with our NixOS closure as well as minimal `var-lib-build` partition. + A first image is built, it contains the `store` partition, populated with our NixOS closure. `boot` and `store-verity` remain empty during this step. 2. While building `system.build.finalImage`: @@ -77,29 +77,40 @@ Under the hood, the image itself is built by `systemd-repart`, using NixOS modul 3. The image then needs to be signed with a script outside a `nix` build process (to avoid leaking keys into the world-readable `/nix/store`. No `systemd-repart` is involved in this step. Instead we use `mtools` to read the `UKI` from the image, sign it and - together with Secure Boot update bundles, write it back to `boot` inside the image. -4. Finally, `systemd-repart` is called once more during run-time, in early boot at the start of `initrd`: The minimal `var-lib-build` partition, created in the first step above, is resized and encrypted with a new random key on each boot. That -key is generated just before `systemd-repart` in our custom `generate-disk-key.service`. +4. Finally, `systemd-repart` is called once more during run-time, in early boot at the start of `initrd`: All mutable partitions are created from scratch on first boot. The ephemeral build partition is factory-reset and re-encrypted with a new random key on each boot. +The key is generated just before `systemd-repart` in our custom `generate-disk-key.service`. ### Disk Layout +The build-time image contains only immutable partitions: + | Partition | Label | Format | Mountpoint | |---------------------+----------------+------------------+------------| | **00‑esp** | `boot` | `vfat` | `/boot` | -| **10‑store‑verity** | `store-verity` | `dm-verity hash` | `n/a` | +| **10‑store‑verity** | `store-verity` | `dm-verity hash` | `n/a` | | **20‑store** | `store` | `erofs` | `/usr` | -| **30‑var‑lib-build** | `var-lib-build` | `ext4` | `/var/lib/build` | + +At first boot, `systemd-repart` creates additional mutable partitions: + +| Partition | Label | Format | Mountpoint | Lifecycle | +|--------------------------+----------------------+------------------+----------------------+-----------| +| **31‑var‑lib‑credentials** | `var-lib-credentials` | `LUKS+ext4` (TPM2) | `/var/lib/credentials` | Persistent | +| **32‑var‑lib‑keylime** | `var-lib-keylime` | `LUKS+ext4` (TPM2) | `/var/lib/keylime` | Persistent | +| **40‑var‑lib‑build** | `var-lib-build` | `LUKS+ext4` (random key) | `/var/lib/build` | Ephemeral | - **boot** – Holds the signed Unified Kernel Image (`UKI`) as an `EFI` application, as well as Secure Boot update bundles for enrollment. The partition itself is unsigned and mounted read‑only during boot. - **store-verity** – Stores the `dm‑verity` hash for the `/usr` partition. The hash is passed as `usrhash` in the kernel command line, which is signed as part of the `UKI`. -- **store** – Contains the read-only Nix store, bind‑mounted into `/nix/store` in the running system. The integrity of `/usr` is verified at runtime using `dm‑verity`. -- **var-lib-build** – A minimal, ephemeral state partition. See next section below. +- **store** – Contains the read-only Nix store, bind‑mounted into `/nix/store` in the running system. The integrity of `/usr` is verified at runtime using `dm‑verity`. +- **var-lib-credentials** – TPM2-bound LUKS partition for `systemd-creds` encrypted credentials. Created on first boot, persists across reboots. Becomes inaccessible if Secure Boot keys change (PCR 7 binding). +- **var-lib-keylime** – TPM2-bound LUKS partition for keylime agent state (Attestation Key). Created on first boot, persists across reboots. +- **var-lib-build** – Ephemeral build workspace, see next section. Notably, the root filesystem (`/`) is, along with an optional writable overlay of the Nix store, kept entirely in RAM (`tmpfs`) and therefore not present in the image. There's also no boot loader, because the `UKI` acts as an `EFI` application and is directly loaded by the hosts firmware. ### Ephemeral State Partition -The `/var/lib/build` partition is deliberately designed to be temporary and encrypted. Each time the system boots, a fresh key is generated and the partition is resized to match the current disk size. This ensures that sensitive build artifacts never persist beyond a single session, reducing the risk of leaking proprietary information or to introduce impurities between different builds. +The `/var/lib/build` partition is deliberately designed to be temporary and encrypted. Each time the system boots, a fresh key is generated and the partition is factory-reset. This ensures that sensitive build artifacts never persist beyond a single session, reducing the risk of leaking proprietary information or to introduce impurities between different builds. ### Secure Boot Support @@ -213,7 +224,7 @@ The keylime agent (`services.keylime-agent`) is enabled by default on every buil At boot, the agent reads `/boot/attestation-server.json` (written to the ESP by `configure-disk-image set-attestation-server`) to learn the registrar IP, verifier URL, and CA certificate. It then registers with the registrar and begins periodic attestation. -The agent's **Attestation Key** (AK) is persisted to `/boot/keylime/` so that re-registrations after reboot use the same key, avoiding rejection by the registrar. +The agent's **Attestation Key** (AK) is stored in `/var/lib/keylime/`, which is a persistent, TPM2-bound LUKS partition. This ensures re-registrations after reboot use the same key, avoiding rejection by the registrar. ### Server (Registrar & Verifier) @@ -291,11 +302,11 @@ Two tools are included for PCR management: ## Credential Storage {#credential-storage} -The `credential-storage.nix` module provides TPM-backed persistent storage for secrets on the target machine. It uses `systemd-creds` to encrypt credentials with the machine's TPM, bound to PCR 7 (Secure Boot policy), and stores them on the artifact storage disk. +The `credential-storage.nix` module provides TPM-backed persistent storage for secrets on the target machine. It uses `systemd-creds` to encrypt credentials with the machine's TPM, bound to PCR 7 (Secure Boot policy), and stores them on a dedicated LUKS partition that is itself TPM2-bound. A `credential-store` utility for credentials management is included. See the [user guide](user-guide.pdf) for usage. -Encrypted credentials are kept in `/var/lib/artifacts/credentials/`, which is bind-mounted to `/run/credstore.encrypted/`. This is one of the standard directories that systemd searches when a service uses `LoadCredentialEncrypted=`, so stored credentials can be consumed by systemd services without additional configuration. +Encrypted credentials are kept in `/var/lib/credentials/`, a persistent TPM2-bound LUKS partition. This directory is bind-mounted to `/run/credstore.encrypted/`, one of the standard directories that systemd searches when a service uses `LoadCredentialEncrypted=`, so stored credentials can be consumed by systemd services without additional configuration. \pagebreak @@ -354,12 +365,12 @@ Main components are: - **(3)** First run of `systemd-repart` (`system.build.intermediateImage`): - Starts from a blank disk image. - Store paths from the NixOS closure are copied into the newly `store` partition. - - `esp`, `store-verity` and `var-lib-build` are created but stay empty for the moment. + - `esp` and `store-verity` are created but stay empty for the moment. - **(4)** With a filled store partition, `dm-verity` hashes can be calculated. So we build a new `UKI`, taking kernel & initrd from the NixOS closure and add the root hash of the `dm-verity` merkle tree to the kernels command line as `usrhash`. - **(5)** Second run of `systemd-repart` (`system.build.finalImage`): - Starts from the intermediate image from step **(3)**. - - The `store` and `var-lib-build` partitions are copied as-is. + - The `store` partition is copied as-is. - `dm-verity` hashes are written to the `store-verity` partition. - The unsigned `UKI` from step **(4)** is copied into the `esp` partition. - With that being done, the image is built and contains our entire NixOS closure, including the `fhsenv`, in a `dm-verity`-checked store partition, as well as the `UKI` including `usrhash`. @@ -397,7 +408,7 @@ flowchart TB halt["Display error & halt"] generate-disk-key["(3) Generate ephemeral encryption key"] - systemd-repart["(4) Resize, Format and Encrypt state partition"] + systemd-repart["(4) Create/reset partitions (TPM2 + ephemeral)"] mount["(5) Mount read-only & state partitions"] build-android["(7) `fetch-android` & `build-android` are executed"] android-tools["Android Build Tools (`repo`, `lunch`, `ninja`, etc.)"] @@ -434,16 +445,16 @@ flowchart TB - If it is **active** and our image is booting succesfully, we trust the firmware here and continue to boot normally. - If it is in **setup** mode, we enroll certificates stored on our ESP. Setting the platform key disables setup mode automatically and reboot the machine right after. - If it is **disabled** or in any unknown mode, we halt the machine but don't power it off to keep the error message readable. -3. Before encrypting the disks, we run `generate-disk-key.service`. A simple script that reads 64 bytes from `/dev/urandom` without ever storing it on disk. All state is encrypted with - that key, so that if the host shuts down for whatever reason - including sudden power loss - the encrypted data +3. Before encrypting the disks, we run `generate-disk-key.service`. A simple script that reads 64 bytes from `/dev/urandom` without ever storing it on disk. The ephemeral build partition is encrypted with + that key, so that if the host shuts down for whatever reason - including sudden power loss - the build data ends up unusable. -4. `systemd-repart` searches for the small, empty state partition on its boot media and resizes it before using `LUKS` to - encrypt it with the ephemeral key from **(2)**. +4. `systemd-repart` creates and manages mutable partitions. On first boot, it creates all three: the TPM2-bound credentials and keylime partitions (persistent) and the ephemeral build partition (encrypted with the key from **(3)**). On subsequent boots, the persistent partitions are left untouched while the build partition is factory-reset and re-encrypted with a fresh key. 5. We proceed to mount required file systems: * A read-only `/usr` partition, containing our `/nix/store` and all software in the image, checked by `dm-verity`. * Bind-mounts for `/bin` and `/lib` to simulate a conventional, `FHS`-based Linux for the build. * An ephemeral `/` file system (`tmpfs`) - * `/var/lib/build` from the encrypted partition created in **(3)**. + * `/var/lib/build` from the ephemeral encrypted partition. + * `/var/lib/credentials` and `/var/lib/keylime` from the TPM2-bound persistent partitions. 6. With all mounts in place, we are ready to finish the boot process by switching into Stage 2 of NixOS. 7. With the system fully booted, we can start the build in various ways. In unattended mode (`nixosAndroidBuilder.unattended.enable`), a configurable sequence of steps is executed automatically. In interactive mode, the following scripts are available: * `select-branch` presents a dialog to choose from configured branches (auto-selects if only one is configured). diff --git a/docs/user-guide.md b/docs/user-guide.md index 37c56c9..ab92b98 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -481,7 +481,7 @@ $ credential-store add api-token ~/token.txt Credential names must start with a letter or digit and may only contain letters, digits, dots, hyphens, and underscores. -Stored credentials are encrypted with the machine's TPM, bound to PCR 7 (Secure Boot policy). They are persisted on the artifact storage disk at `/var/lib/artifacts/credentials/` and bind-mounted to `/run/credstore.encrypted/`, which is automatically searched by systemd when services use `LoadCredentialEncrypted=`. +Stored credentials are encrypted with the machine's TPM, bound to PCR 7 (Secure Boot policy). They are kept on a dedicated TPM2-bound LUKS partition at `/var/lib/credentials/` and bind-mounted to `/run/credstore.encrypted/`, which is automatically searched by systemd when services use `LoadCredentialEncrypted=`. The encryption parameters can be customized via `nixosAndroidBuilder.credentialStorage.encryptionFlags` - for example, to also bind to PCR 11 (the specific UKI): diff --git a/modules/credential-storage.nix b/modules/credential-storage.nix index 3e9cfa7..7db5556 100644 --- a/modules/credential-storage.nix +++ b/modules/credential-storage.nix @@ -1,8 +1,8 @@ -# Store systemd credentials persistently on the artifacts disk +# Store systemd credentials on a persistent, TPM2-bound LUKS partition # -# This module creates a `credentials` subdirectory on the artifacts partition -# and bind-mounts it to /etc/credstore.encrypted/ so systemd services can -# load encrypted credentials from persistent storage. +# This module bind-mounts the credential directory to +# /run/credstore.encrypted/ so systemd services can load encrypted +# credentials from persistent storage. # # Credentials should be encrypted with `systemd-creds encrypt` using the # machine's TPM. @@ -14,11 +14,10 @@ }: let cfg = config.nixosAndroidBuilder.credentialStorage; - artifactCfg = config.nixosAndroidBuilder.artifactStorage; in { options.nixosAndroidBuilder.credentialStorage = { - enable = lib.mkEnableOption "persistent credential storage on the artifacts disk"; + enable = lib.mkEnableOption "persistent credential storage on a TPM2-bound LUKS partition"; encryptionFlags = lib.mkOption { description = "Flags to pass to systemd-creds encrypt. See man (1) systemd-creds"; @@ -31,11 +30,12 @@ in credentialDir = lib.mkOption { description = '' - Directory where credentials are stored on the artifacts disk. - This will be bind-mounted to /run/credstore.encrypted/ + Directory where credentials are stored. + This is the mount point of the dedicated credentials partition + and will be bind-mounted to /run/credstore.encrypted/ ''; type = lib.types.path; - default = "${artifactCfg.artifactDir}/credentials"; + default = "/var/lib/credentials"; }; mountPoint = lib.mkOption { @@ -48,7 +48,7 @@ in }; }; - config = lib.mkIf (cfg.enable && artifactCfg.enable) { + config = lib.mkIf cfg.enable { systemd.tmpfiles.rules = [ "d ${cfg.credentialDir} 0700 root root - -" ]; @@ -56,10 +56,7 @@ in fileSystems."${cfg.mountPoint}" = { device = cfg.credentialDir; fsType = "none"; - options = [ - "bind" - "x-systemd.requires=${artifactCfg.artifactDir}.mount" - ]; + options = [ "bind" ]; }; security.polkit.extraConfig = '' diff --git a/modules/image.nix b/modules/image.nix index 4d8e3a5..8f476f3 100644 --- a/modules/image.nix +++ b/modules/image.nix @@ -40,9 +40,19 @@ "mode=0755" ]; }; + "/var/lib/credentials" = { + device = "/dev/mapper/var_lib_credentials_crypt"; + fsType = "ext4"; + neededForBoot = true; + }; + "/var/lib/keylime" = { + device = "/dev/mapper/var_lib_keylime_crypt"; + fsType = "ext4"; + neededForBoot = true; + }; "/var/lib/build" = { device = "/dev/mapper/var_lib_crypt"; - fsType = config.systemd.repart.partitions."30-var-lib-build".Format; + fsType = config.systemd.repart.partitions."40-var-lib-build".Format; neededForBoot = true; }; "/boot" = { @@ -129,8 +139,6 @@ "-Efragments,ztailpacking" ]; - mkfsOptions.ext4 = [ "-Eroot_owner=1000:1000" ]; - # OVMF does not work with the default repart sector size of 4096 sectorSize = 512; @@ -177,31 +185,58 @@ Minimize = "best"; }; }; - "30-var-lib-build".repartConfig = { - Type = "var"; - Label = "var-lib-build"; - # We want to start out with a very small partition in the image, and add - # the real minimum size to to systemd.repart.partitions below instead, - # in order to resize it during boot. - SizeMinBytes = "10M"; - }; + # NOTE: 31-var-lib-credentials, 32-var-lib-keylime and 40-var-lib-build + # are NOT defined here. They are created at first boot by + # systemd-repart (see runtime config below). This is intentional: + # repart only formats and LUKS-encrypts partitions during creation. + # Build-time placeholders would be left unformatted, and adjacent + # partitions block in-place growth. }; }; }; ## Run-time configuration of systemd-repart on first boot. - # Reuse settings of the repart-generated image file on first boot - systemd.repart.partitions."30-var-lib-build" = - config.image.repart.partitions."30-var-lib-build".repartConfig - // { - Format = "ext4"; - Encrypt = "key-file"; - SizeMinBytes = "250G"; - # Tell systemd-repart to re-format and re-encrypt this partition on each boot - # if run with --factory-reset, which we do by default. - FactoryReset = true; - }; + # Persistent, TPM2-bound partitions for credentials and keylime agent state. + # These don't exist in the build-time image — systemd-repart creates them + # as new partitions on first boot (which triggers formatting + TPM2 LUKS + # enrollment). On subsequent boots, repart matches them by type+label and + # leaves them untouched (no FactoryReset). + systemd.repart.partitions."31-var-lib-credentials" = { + Type = "linux-generic"; + Label = "var-lib-credentials"; + Format = "ext4"; + Encrypt = "tpm2"; + SizeMinBytes = "64M"; + }; + systemd.repart.partitions."32-var-lib-keylime" = { + Type = "linux-generic"; + Label = "var-lib-keylime"; + Format = "ext4"; + Encrypt = "tpm2"; + SizeMinBytes = "64M"; + }; + + # Ephemeral build partition — factory-reset on every boot. + systemd.repart.partitions."40-var-lib-build" = { + Type = "var"; + Label = "var-lib-build"; + Format = "ext4"; + Encrypt = "key-file"; + SizeMinBytes = "250G"; + # Tell systemd-repart to re-format and re-encrypt this partition on each boot + # if run with --factory-reset, which we do by default. + FactoryReset = true; + }; + + boot.initrd.luks.devices."var_lib_credentials_crypt" = { + device = "/dev/disk/by-partlabel/var-lib-credentials"; + crypttabExtraOpts = [ "tpm2-device=auto" ]; + }; + boot.initrd.luks.devices."var_lib_keylime_crypt" = { + device = "/dev/disk/by-partlabel/var-lib-keylime"; + crypttabExtraOpts = [ "tpm2-device=auto" ]; + }; boot.initrd.luks.devices."var_lib_crypt" = { keyFile = "/etc/disk.key"; device = "/dev/disk/by-partlabel/var-lib-build"; @@ -209,6 +244,25 @@ boot.initrd.systemd = let + # Find the disk we booted from by following the dm-verity device + # tree. The verity backing device is cryptographically bound to + # this UKI's usrhash — so it uniquely identifies our image's disk, + # even if other disks have partitions with the same labels. + findBootDisk = pkgs.writeScript "find-boot-disk" '' + #!/bin/sh + set -e + # dm-verity "usr" is set up by systemd-veritysetup-generator from + # the usrhash= kernel cmdline. Find the dm block device for "usr", + # then walk sysfs from its slave (our store partition) up to the + # parent disk. + dm_dev=$(dmsetup info -c --noheadings -o blkdevname usr) + for slave in /sys/block/"$dm_dev"/slaves/*; do + dev_name=$(basename "$slave") + disk=$(basename "$(readlink -f "/sys/class/block/$dev_name/..")") + ln -sf "/dev/$disk" /run/systemd/volatile-root + break + done + ''; waitForDisk = pkgs.writeScript "wait-for-disk" '' #!/bin/sh set -e @@ -231,6 +285,7 @@ }; # We need to list our scripts here, otherwise store paths won't be in initrd storePaths = [ + findBootDisk waitForDisk generateDiskKey ]; @@ -260,6 +315,8 @@ services = { systemd-repart = { before = [ + "systemd-cryptsetup@var_lib_credentials_crypt.service" + "systemd-cryptsetup@var_lib_keylime_crypt.service" "systemd-cryptsetup@var_lib_crypt.service" ]; after = [ "systemd-udev-settle.service" ]; @@ -282,13 +339,16 @@ }; }; - # Link the read-only nix store to /run/systemd/volatile-root before - # systemd-repart runs. systemd-repart normally looks for the block device - # backing "/", or this path. So this enables systemd-repart to find the - # right device at boot. + # Tell systemd-repart which disk to operate on. Since "/" is + # tmpfs, repart can't discover it on its own. We follow the + # dm-verity backing device (cryptographically bound to this + # image's usrhash) to find the correct disk — even if other + # disks have partitions with the same labels. link-volatile-root = { description = "Create volatile-root to tell systemd-repart which disk to use"; wantedBy = [ "initrd.target" ]; + after = [ "veritysetup.target" ]; + requires = [ "veritysetup.target" ]; before = [ "systemd-repart.service" ]; requiredBy = [ "systemd-repart.service" ]; unitConfig = { @@ -297,9 +357,7 @@ serviceConfig = { Type = "oneshot"; RemainAfterExit = true; - ExecStart = "/bin/ln -sf /dev/disk/by-partlabel/${ - config.image.repart.partitions."30-var-lib-build".repartConfig.Label - } /run/systemd/volatile-root"; + ExecStart = findBootDisk; }; }; }; diff --git a/modules/keylime-agent.nix b/modules/keylime-agent.nix index fb71b03..10a78e7 100644 --- a/modules/keylime-agent.nix +++ b/modules/keylime-agent.nix @@ -32,9 +32,6 @@ let runtimeConf = "/run/keylime/agent.conf"; runtimeCaCert = "/run/keylime/ca-cert.pem"; - # Agent state (AK) persisted here across reboots. - persistDir = "/boot/keylime"; - # Defaults for the push model agent (keylime_push_model_agent). # Only settings read by the push model are included. # registrar_ip, verifier_url, and CA cert paths are placeholders — @@ -97,7 +94,6 @@ let import pwd import grp import re - import shutil import sys SRC = "/boot/attestation-server.json" @@ -154,38 +150,8 @@ let os.chmod(RUNTIME_CONF, 0o440) print(f"Configured: registrar={ip}:{port} verifier=https://{ip}:{vport}") - - # Restore persisted AK from /boot. /var/lib is ephemeral, so without - # this the agent creates a new AK on every boot and the registrar - # rejects re-registration. - PERSIST = "${persistDir}" - WORK = "/var/lib/keylime" - if os.path.isdir(PERSIST): - for name in os.listdir(PERSIST): - src = os.path.join(PERSIST, name) - dst = os.path.join(WORK, name) - if os.path.isfile(src): - shutil.copy2(src, dst) - os.chown(dst, uid, gid) - print(f"Restored agent state from {PERSIST}") - else: - print(f"No persisted state at {PERSIST}") ''; - # Persist agent AK to /boot after the file appears so it survives reboots. - # Triggered by keylime-agent-data.path (PathExists watch). - persistAgent = pkgs.writeShellScript "keylime-persist-agent" '' - set -euo pipefail - src=/var/lib/keylime/agent_data.json - dst=${persistDir} - [ -f "$src" ] || { echo "$src not found, skipping"; exit 0; } - ${pkgs.util-linux}/bin/mount -o remount,rw /boot - mkdir -p "$dst" - cp "$src" "$dst"/ - ${pkgs.util-linux}/bin/mount -o remount,ro /boot - echo "Persisted agent AK to $dst" - ''; - in { options.services.keylime-agent = { @@ -236,7 +202,8 @@ in users.groups.keylime = { }; systemd.tmpfiles.rules = [ - "d /var/lib/keylime 0750 keylime keylime -" + # /var/lib/keylime is a persistent LUKS partition; fix ownership after mount + "z /var/lib/keylime 0750 keylime keylime - -" "d /run/keylime 0750 keylime keylime -" ]; @@ -288,18 +255,6 @@ in description = "Keylime agent data available"; }; - # Persist agent AK to /boot so the UUID survives reboots. - systemd.services.keylime-persist-agent = { - description = "Persist Keylime agent AK to /boot"; - after = [ "boot.mount" ]; - wantedBy = [ "keylime-agent-data.target" ]; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - ExecStart = "${persistAgent}"; - }; - }; - # Report TPM PCR values to the auto-enrollment server so the # daemon can enroll this agent with the full TPM policy. # Reads the agent UUID from agent_data.json (ek_hash field), diff --git a/packages/disk-installer/installer-vm.nix b/packages/disk-installer/installer-vm.nix index c9c776a..ccdf680 100644 --- a/packages/disk-installer/installer-vm.nix +++ b/packages/disk-installer/installer-vm.nix @@ -42,6 +42,7 @@ in useEFIBoot = true; mountHostNixStore = false; efi.keepVariables = false; + tpm.enable = true; # NixOS overrides filesystems for VMs by default fileSystems = lib.mkForce { }; diff --git a/tests/credential-storage.nix b/tests/credential-storage.nix index 9e0fe7b..fc1cb0b 100644 --- a/tests/credential-storage.nix +++ b/tests/credential-storage.nix @@ -7,29 +7,20 @@ { imports = [ ../modules/credential-storage.nix - ../modules/artifact-storage.nix ]; - # artifact-storage references build.sourceDir; provide it without - # pulling in the full android-build-env module. - options.nixosAndroidBuilder.build.sourceDir = lib.mkOption { - type = lib.types.path; - default = "/var/lib/build/source"; - }; - config = { virtualisation.tpm.enable = true; - nixosAndroidBuilder.artifactStorage.enable = true; nixosAndroidBuilder.credentialStorage.enable = true; - # Use a tmpfs instead of a real second disk - fileSystems."/var/lib/artifacts" = lib.mkForce { + # Use a tmpfs to simulate the dedicated credentials partition + fileSystems."/var/lib/credentials" = lib.mkForce { device = "none"; fsType = "tmpfs"; options = [ "size=64m" - "mode=0755" + "mode=0700" ]; }; }; @@ -39,8 +30,6 @@ machine.start() machine.wait_for_unit("multi-user.target") - machine.succeed("mkdir -p /var/lib/artifacts/credentials") - with subtest("credential-store is available"): machine.succeed("credential-store list || true") @@ -71,7 +60,7 @@ machine.fail("echo 'bad' | credential-store add '-dash'") with subtest("encrypted file is not plaintext"): - content = machine.succeed("cat /var/lib/artifacts/credentials/file-token") + content = machine.succeed("cat /var/lib/credentials/file-token") assert "file-secret" not in content, "Credential stored in plaintext!" ''; } diff --git a/tests/integration.nix b/tests/integration.nix index 9b6f0a1..a691324 100644 --- a/tests/integration.nix +++ b/tests/integration.nix @@ -9,7 +9,7 @@ nixosAndroidBuilder.unattended.enable = lib.mkForce false; # Decrease resource usage for VM tests a bit as long as we are not actually # building android as part of the test suite. - systemd.repart.partitions."30-var-lib-build".SizeMinBytes = lib.mkVMOverride "10G"; + systemd.repart.partitions."40-var-lib-build".SizeMinBytes = lib.mkVMOverride "10G"; virtualisation = lib.mkVMOverride { diskSize = 30 * 1024; memorySize = 8 * 1024; @@ -78,6 +78,33 @@ "Secure Boot: enabled (user)", stdout, "Secure Boot is NOT active") ''; + + testPersistence = '' + with subtest("Partition persistence"): + with subtest("Write sentinel files before reboot"): + machine.succeed("echo 'build-data' > /var/lib/build/sentinel") + machine.succeed("echo 'keylime-data' > /var/lib/keylime/sentinel") + machine.succeed("echo 'cred-data' > /var/lib/credentials/sentinel") + + # Verify all three are readable + machine.succeed("cat /var/lib/build/sentinel") + machine.succeed("cat /var/lib/keylime/sentinel") + machine.succeed("cat /var/lib/credentials/sentinel") + + machine.reboot() + machine.wait_for_unit("default.target") + + with subtest("/var/lib/build is ephemeral (wiped on reboot)"): + machine.fail("test -f /var/lib/build/sentinel") + + with subtest("/var/lib/keylime persists across reboot"): + output = machine.succeed("cat /var/lib/keylime/sentinel").strip() + assert output == "keylime-data", f"Expected 'keylime-data', got '{output}'" + + with subtest("/var/lib/credentials persists across reboot"): + output = machine.succeed("cat /var/lib/credentials/sentinel").strip() + assert output == "cred-data", f"Expected 'cred-data', got '{output}'" + ''; in '' import os @@ -96,6 +123,7 @@ ${testSecureBoot} ${testVerity} ${testFHSEnv} + ${testPersistence} machine.shutdown() ''; } diff --git a/tests/keylime-auto-enroll.nix b/tests/keylime-auto-enroll.nix index 10e40ae..9538d9d 100644 --- a/tests/keylime-auto-enroll.nix +++ b/tests/keylime-auto-enroll.nix @@ -103,7 +103,7 @@ in memorySize = 2 * 1024; cores = 2; }; - systemd.repart.partitions."30-var-lib-build".SizeMinBytes = lib.mkVMOverride "1G"; + systemd.repart.partitions."40-var-lib-build".SizeMinBytes = lib.mkVMOverride "1G"; nixosAndroidBuilder.unattended.enable = lib.mkForce false; diff --git a/tests/keylime.nix b/tests/keylime.nix index 05071ec..85b6c4f 100644 --- a/tests/keylime.nix +++ b/tests/keylime.nix @@ -89,7 +89,7 @@ in memorySize = 2 * 1024; cores = 2; }; - systemd.repart.partitions."30-var-lib-build".SizeMinBytes = lib.mkVMOverride "1G"; + systemd.repart.partitions."40-var-lib-build".SizeMinBytes = lib.mkVMOverride "1G"; nixosAndroidBuilder.unattended.enable = lib.mkForce false;