merge main into trunk/l2 by 0x416e746f6e · Pull Request #107 · flashbots/flashbots-images

0x416e746f6e · 2026-03-10T15:01:39Z

subj.

lighthouse 8.0.0 needs it

BoB L1: new tdx-init, LH 8.0.0, fix artifact naming

Switch from Fluent Bit to custom Rust solution

Simplify reproducible lighthouse v8 build process

…g.d, and reverted other buildernet changes

…precation

0x416e746f6e · 2026-03-10T17:15:20Z

mkosi.profiles/gcp/mkosi.extra/etc/hosts

question:

@alexhulbert do you know why is this removed?

functionality isn't affected, see https://www.notion.so/flashbots/Standardized-Mkosi-Networking-Config-3206b4a0d87680479d0be394fe8c84f7?source=copy_link

functionality isn't affected

not entirely:

metadata.google.internal works, yes:

# dig metadata.google.internal ; <<>> DiG 9.20.18-1~deb13u1-Debian <<>> metadata.google.internal ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6621 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;metadata.google.internal. IN A ;; ANSWER SECTION: metadata.google.internal. 3600 IN A 169.254.169.254 ;; Query time: 1 msec ;; SERVER: 169.254.169.254#53(169.254.169.254) (UDP) ;; WHEN: Thu Mar 12 10:10:35 UTC 2026 ;; MSG SIZE rcvd: 69

but just metadata is broken:

# dig metadata ; <<>> DiG 9.20.11-4-Debian <<>> metadata ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40267 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;metadata. IN A ;; Query time: 3747 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Thu Mar 12 10:15:55 UTC 2026 ;; MSG SIZE rcvd: 37

0x416e746f6e · 2026-03-10T17:15:52Z

mkosi.profiles/gcp/mkosi.extra/etc/resolv.conf

question:

@alexhulbert do you know why is this removed?

functionality isn't affected, see https://www.notion.so/flashbots/Standardized-Mkosi-Networking-Config-3206b4a0d87680479d0be394fe8c84f7?source=copy_link

split-horizon DNS is broken b/c of this.

with current config (1.1.1.1 over DoT):

# dig rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net ; <<>> DiG 9.20.11-4-Debian <<>> rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 26087 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net. IN A ;; AUTHORITY SECTION: l2b.flashbots.net. 300 IN SOA ns-cloud-c1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 ;; Query time: 46 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Thu Mar 12 10:19:20 UTC 2026 ;; MSG SIZE rcvd: 172 ^C

same via GCP's metadata:

# dig rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net @metadata.google.internal ; <<>> DiG 9.20.11-4-Debian <<>> rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net @metadata.google.internal ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24134 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net. IN A ;; ANSWER SECTION: rpc.builder.opt-uni-testnet-1301.l2b.flashbots.net. 3600 IN A 10.86.0.15 ;; Query time: 6 msec ;; SERVER: 169.254.169.254#53(metadata.google.internal) (UDP) ;; WHEN: Thu Mar 12 10:19:47 UTC 2026 ;; MSG SIZE rcvd: 95

non splid-horizon DNS resolution doesn't work either, b.t.w.

compare this:

# dig prometheus.l2-meva-uni.gcp.internal ; <<>> DiG 9.20.11-4-Debian <<>> prometheus.l2-meva-uni.gcp.internal ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 44604 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;prometheus.l2-meva-uni.gcp.internal. IN A ;; AUTHORITY SECTION: . 7047 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2026031200 1800 900 604800 86400 ;; Query time: 0 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Thu Mar 12 16:14:15 UTC 2026 ;; MSG SIZE rcvd: 139

with this:

# dig prometheus.l2-meva-uni.gcp.internal @metadata.google.internal ; <<>> DiG 9.20.11-4-Debian <<>> prometheus.l2-meva-uni.gcp.internal @metadata.google.internal ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45668 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;prometheus.l2-meva-uni.gcp.internal. IN A ;; ANSWER SECTION: prometheus.l2-meva-uni.gcp.internal. 3600 IN A 10.86.0.30 ;; Query time: 5 msec ;; SERVER: 169.254.169.254#53(metadata.google.internal) (UDP) ;; WHEN: Thu Mar 12 16:15:02 UTC 2026 ;; MSG SIZE rcvd: 80

ahh, i see.

adding Domains=~internal to base/mkosi.extra/etc/systemd/network/10-ethernet.network should fix the issue. and then for your split-horizon, you can add additional domains via a module-specific
mkosi.extra/etc/systemd/network/10-ethernet.network.d/extra.conf like so:

[Network] Domains=~l2b.flashbots.net

0x416e746f6e · 2026-03-10T17:19:44Z

mkosi.profiles/gcp/mkosi.extra/usr/bin/set-hostname-gcp

question:

@alexhulbert what's the point of this file? (it doesn't seem to be referenced anywhere)

forgot to remove it. hostname stuff shouldn't be in the global GCP profile since buildernet handles hostnames differently (they're set from builderhub there)

0x416e746f6e · 2026-03-10T17:21:48Z

mkosi.profiles/gcp/repart.d/00-uki.conf

+SizeMinBytes=524288000
+SizeMaxBytes=524288000


question:

can we bump this to 1Gb @alexhulbert? (I had corner cases where I needed that much)

I can make it dynamic

dynamic will do too.

0x416e746f6e · 2026-03-10T17:29:09Z

shared/mkosi.extra/etc/chrony/chrony.conf

+
+# Refine trusted sources using cloud hypervisor when available
+server 169.254.169.254 iburst minpoll 4 maxpoll 4
+server 169.254.169.123 iburst minpoll 4 maxpoll 4


question:

@alexhulbert these two look like blanket references to NTP services in different clouds. why not have them in dedicated mkosi.profiles?

these two together form the standards for all cloud providers and these lines do nothing on a non-cloud image, so I added them here for simplicity. see https://www.notion.so/flashbots/Standardized-Mkosi-Networking-Config-3206b4a0d87680479d0be394fe8c84f7?source=copy_link for details

0x416e746f6e · 2026-03-10T17:33:34Z

shared/mkosi.extra/etc/systemd/resolved.conf

@@ -0,0 +1,3 @@
+[Resolve]
+DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com


question:

@alexhulbert is it ok to override this in particular images?

e.g. I use split-horizon DNS in some places => for those to work the resolver must be of the relevant cloud, not external.

the new image actually supports that out of the box via DHCP-peovided domains, no need to override anything. see https://www.notion.so/flashbots/Standardized-Mkosi-Networking-Config-3206b4a0d87680479d0be394fe8c84f7?source=copy_link for details

it might support DHCP-provided domains, but I am talking about split-horizon DNS (so, not DHCP)

im also talking about split horizon DNS. shortcuts like "metadata" or fully qualified names like "metadata.google.internal" correctly resolve using the google hypervisor DNS server, but then regular sites like flashbots.net resolve thru cloudflare over DoT. DHCP doesn't just provide an IP, it also provides a list of LAN asdresses with corresponding hostnames that systemd-resolved can add to its list of hosts.

if you're doing something even more complicated, you can always add a drop in file, though.

pls see comment above. neither split-dns, not just plain internal dns are reliably working.

0x416e746f6e · 2026-03-10T17:36:34Z

shared/mkosi.sync.d/10-setup-apt.sh

question:

shouldn't this only happen in dev images @alexhulbert?

this file provides access to a timestamp-fixed snapshot of the Debian backports repo. This allows us to install newer versions of packages before they're officially released in debian trixie. It's a no-op to your actual package list unless you add "/trixie-backports" to one of your installed packages in your mkosi.conf

I mean, on prod images that we don't have access to we wouldn't even be able to install any packages, neither any kinds of automations are supposed to do that either.

I think that any debian repos should be present only dev images, right?

0x416e746f6e · 2026-03-10T17:39:10Z

kernel/snippets/ubuntu.config

question:

how was this update produced @alexhulbert?

Fryd's recently finished PR produces an equivalent kernel but in a much simpler and more transparent way. Removing this file is the final step in the standardization/simplification process and should be merged into main this week.

I don't understand. my question was about this present file - namely how was it produced. not about parallel work Fryd has done.

or do you mean to say that this file is here temporarily and will be replaced by Fryd's work?

fix: set default file to latest.efi

Bump lighthouse to 8.1.2

feat: add support to Debian kernel source builds and kernel patches

Reorganize Repo Directory Structure

ilyaluk and others added 30 commits October 25, 2025 14:54

wip: gcp: echo gcloud commands for image creation

8ea8fa3

wip: use tdx-init dev branch, fix disk mount and support luks resize

d441a5f

bob-l2: initial commit

92e4bc8

bob-l2: implement fetch-metadata service

095efe5

bob-l2: wip on firewall

78a0309

bob-l2: metadata -> vault

ad0b911

fix p2p port

7b4606e

wip: other tx-init branch

a0e711e

bob-common: bump tdx-init version

2f22972

bob-l1: include trixie-backports for rust 1.89

14fcc2e

lighthouse 8.0.0 needs it

bob-l1: bump lighthouse to 8.0.0

b21e753

bob-l1: move readme, clarify image measurements

a6dd9b5

Drop empty fluentbit filter events

ef5dd27

update commit

2cf8e72

Allow installing packages from debian backports

9a5ef12

Remove old static backports config and update archive timestamp

db80a99

set image ids and versions

1ac8145

add unpack_image.sh script

9780ccf

Switch from fluent-build to custom log-delay code

b9b388d

Simplify reproducible lighthouse v8 build process

89c4333

Change commit hash length in filenames to 6 chars

d8b6fe0

Merge pull request #43 from flashbots/ilya/bob-l1-updates

d98c32c

BoB L1: new tdx-init, LH 8.0.0, fix artifact naming

Support multiple copies of the repo on one machine

2b44885

Use /usr/bin/env bash rather than /bin/bash

a66b144

Add shell.nix file installing lima

8422731

Fix unbound variable in build_rust_package

ae708f1

Fix delay-pipe commit

c9d9bd8

Merge pull request #55 from flashbots/bob-custom-log-delay

a1a8c35

Switch from Fluent Bit to custom Rust solution

Merge pull request #56 from flashbots/lighthouse-8-simplify

f4cfb2b

Simplify reproducible lighthouse v8 build process

Use default mirror uri, not empty file

3c06fc0

fkondej and others added 20 commits March 9, 2026 12:25

chore: remove unused ubuntu.config kernel config snippet

302a9a6

chore: address several PR review comments

e62f51d

re-add python3

4ac5c79

feat: upgrade kernel to 6.18 LTS, move Mirror to per-image configs

24eb27d

fix: pick non dbg .deb

a0f78df

feat: use correct DEB_BUILD_PROFILES for make bindeb-pkg

832bb7e

feat: fail on duplicate kernel config fragment filename

8e563aa

feat: user FLAVOR var instead of hardcoded cloud

bd02658

feat: read Mirror and Release from mkosi config

51ec522

chore: move buildernet kernel config fragments to shared kernel/confi…

66750c5

…g.d, and reverted other buildernet changes

use snapshot in kernel cache hash

71e5928

feat: move kernel build script to mkosi.build.d dir

38830c2

Bump lighthouse version to 8.1.2

41da963

Adjust cargo build profile after lighthouse "reproducible" profile de…

cdabff6

…precation

Using filling SOURCE_DATE_EPOCH to avoid potential non-determinism

e45c4ec

feat: set BuildScripts to glob

6baee6c

feat: build kernel in kernel-build/ subdir to isolate artifacts

427f4f3

feat: internalize kernel FLAVOR and LOCALVERSION env vars

0db7c99

feat: simplify kernel cashe hash logic

555bf10

feat: use nullglob in kernel build script

f50737a

0x416e746f6e self-assigned this Mar 10, 2026

0x416e746f6e commented Mar 10, 2026

View reviewed changes

0x416e746f6e and others added 7 commits March 11, 2026 11:38

fix: set default file to latest.efi

6c876e3

Update GCP measurements export command in Makefile

e2579e0

Merge pull request #109 from flashbots/fix/use-default-file

1849000

fix: set default file to latest.efi

Merge pull request #106 from flashbots/bump_lighthouse_to_8_1_2

e16af8c

Bump lighthouse to 8.1.2

Merge pull request #97 from flashbots/fryd/kernel-debian-source

8e0c26b

feat: add support to Debian kernel source builds and kernel patches

Directory Restructure

1396afd

Merge pull request #113 from flashbots/ah/restructure

f0219c6

Reorganize Repo Directory Structure

0x416e746f6e merged commit 6bf3faf into trunk/l2-merge-main Mar 16, 2026
3 checks passed

		@@ -0,0 +1,3 @@
		[Resolve]
		DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com

		SizeMinBytes=524288000
		SizeMaxBytes=524288000

Conversation

0x416e746f6e commented Mar 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexhulbert Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

alexhulbert Mar 11, 2026 •

edited

Loading