[rpk k8s-multicluster] operator debug bundle

[hidalgopl]

Summary

Adds rpk-k8s multicluster bundle, a diagnostics tool for the multicluster operator analogous to rpk debug bundle for Redpanda. Given a single kubeconfig pointing at one of the operator's host clusters, it discovers the remaining peers via labeled cache Secrets, then collects per-cluster artifacts (operator pod manifest, deployment, logs, metrics, raft status, TLS material) and runs the existing multicluster checks pipeline as both per-cluster and cross-cluster checks. Output is a single zip archive.

This is the operator-side complement to rpk debug remote-bundle — useful for triage of stretch clusters where you need a snapshot of all three operator instances in one shot.

What's in the bundle

stretch-bundle/
├── manifest.json                          # schema, generation timestamp, namespace, redaction + sampling flags
├── status.txt                             # human-readable summary (peers, errors, sizes)
├── errors.txt                             # non-fatal collection errors (omitted when none)
├── cross-cluster/
│   └── checks.json                        # output of cross-cluster checks (e.g. raft quorum)
└── clusters/
    ├── peer-0/
    │   ├── checks.json                    # per-cluster checks (pod, deployment, raft, TLS, ...)
    │   ├── deploy-args.txt                # operator container args (flags as configured)
    │   ├── deployment.yaml                # the multicluster operator Deployment
    │   ├── pod.yaml                       # the active operator Pod
    │   ├── raft-status.json               # raft node status (term, leader, progress)
    │   ├── logs/
    │   │   └── manager.log                # current container log; previous log appended on restart
    │   ├── metrics/
    │   │   ├── t0_metrics.txt             # first /metrics scrape
    │   │   └── t1_metrics.txt             # second /metrics scrape (default --metrics-samples=2 at 10s interval)
    │   └── tls/
    │       ├── ca.crt
    │       ├── tls.crt
    │       ├── tls-key-match.txt          # whether tls.key on disk matches tls.crt
    │       └── tls-secret.yaml            # full Secret manifest (tls.key redacted by default)
    ├── peer-1/
    │   └── ...                            # same layout as peer-0
    └── peer-2/
        └── ...                            # same layout as peer-0

(Sub-directory names under clusters/ are the kube context names from the discovered kubeconfigs — peer-0 / peer-1 / peer-2 shown above for readability.)

managedFields are stripped from every Kubernetes object. tls.key and kubeconfig.yaml Secret payloads are redacted unless --include-private-keys is passed. Per-artifact failures are written to errors.txt rather than aborting the bundle.

Discovery

The user passes a single kubeconfig (--kubeconfig) pointing at any one peer. The tool lists Secrets in the operator namespace matching app.kubernetes.io/component=multicluster-kubeconfig-cache (a new label added in this PR), parses each cached peer kubeconfig, and proceeds against all peers. No pre-merging required.

In environments where the cached peer kubeconfigs have server URLs that aren't reachable from the operator (e.g. local stretch dev envs running vClusters inside a k3d host), pass all contexts explicitly with repeated --context — the bundle command sees len(starting) > 1, skips discovery, and uses only what was given.

Metrics scraping

/metrics is gated by controller-runtime's filters.WithAuthenticationAndAuthorization, so apiserver pod-proxy doesn't authenticate. The bundle instead port-forwards into the operator pod and presents a Bearer token minted via TokenRequest for the operator ServiceAccount. The chart's existing *-metrics-reader ClusterRole (nonResourceURLs: ["/metrics"]) is now bound to that SA so the scrape returns 200 instead of 403 — applies to both operator/chart (single-cluster) and operator/multicluster (multicluster).

By default, two samples are taken 10s apart per cluster (matches rpk debug bundle's default). Multiple samples let an investigator compute counter rate-of-change post-hoc without needing a live Prometheus — particularly useful for rate-style raft signals (leader changes, send errors, drops, RTT). Samples land at metrics/t0_metrics.txt, t1_metrics.txt, … and the manifest records metricsSamples + metricsIntervalSeconds so a bundle reader can interpret the layout.

TLSCheck behavior

TLSCheck is a snapshot read (Get) rather than WaitFor-style polling. A debug bundle should never block on cert-manager finishing — if the TLS secret is missing or mid-rotation, that is itself a diagnostic and surfaces in checks.json. The check now falls straight from deployment-volume matching to scan-by-suffix listing, dropping the brittle synthesis of <SecretPrefix>-multicluster-certificates (which produced invalid Secret names when SecretPrefix was a kube context name like vcluster_..._k3d-harpoon).

Flags

• --kubeconfig / --context / --namespace / --service-name (shared connection flags; --service-name is the app.kubernetes.io/name label value, default operator)
• -o, --output — output path (defaults to ./operator-bundle-<unix-ts>.zip)
• --include-private-keys — keep tls.key and kubeconfig.yaml payloads
• --skip-logs / --logs-size-limit (default 5M, supports MiB / GiB) / --logs-tail-lines (default 5000)
• --skip-metrics / --metrics-samples (default 2, must be ≥ 2) / --metrics-interval (default 10s)
• -v, --verbose — stderr progress trace with timestamps for diagnosing hangs