charts/redpanda: support schema_registry_client SASL credentials via secretRef

[david-yu]

Summary

Re-creation of #1503 (by @AldoFusterTurpin) onto current main, with all original commits preserved and a missing changelog entry + lifecycle golden file update added on top.

The V2 Helm chart had no way to configure SASL credentials for the schema registry's internal Kafka client (schema_registry_client) without storing plaintext passwords in the ConfigMap. This is a regression versus the V1 operator, which already supports this via operator/pkg/resources/configuration.go:488-563.

This PR adds a config.schema_registry_client.saslSecretRef field that references a Kubernetes Secret containing the SASL username and password. Credentials are injected at pod start using the existing redpanda.yaml.fixups mechanism (the same pattern the V1 operator uses), so they never appear in plaintext in the ConfigMap or Helm release history.

How it works

1. User sets config.schema_registry_client.saslSecretRef.name=<secret-name> where the Secret has keys username and password.
2. Helm writes a redpanda.yaml.fixups entry into the ConfigMap telling the configurator init container to patch schema_registry_client.scram_username, schema_registry_client.scram_password, and schema_registry_client.sasl_mechanism.
3. The configurator init container sources the values from env vars backed by secretKeyRef, applies them to redpanda.yaml before writing it to the shared volume, and the Redpanda container starts with credentials in place.

The field names written to redpanda.yaml (scram_username, scram_password, sasl_mechanism) are the documented Redpanda broker properties: https://docs.redpanda.com/current/reference/properties/broker-properties/#schema-registry-client

Step-by-step example (Operator + User CRD)

This walkthrough assumes the Redpanda Operator is installed and that you manage clusters via the Redpanda CR and SCRAM users via the User CR (cluster.redpanda.com/v1alpha2). All commands are namespace-scoped to redpanda — adjust as needed.

The headline benefit of the operator + User CR path is that the SCRAM credential and the configurator-injected env var are populated from the same Kubernetes Secret. The operator watches the Secret, so rotation is a one-step Secret update — no rpk security user update, no race window.

1. Install the operator

helm repo add redpanda https://charts.redpanda.com
helm repo update

helm -n redpanda-system upgrade --install redpanda-operator \
  redpanda/operator --create-namespace --wait

Confirm the CRDs landed:

kubectl get crd redpandas.cluster.redpanda.com users.cluster.redpanda.com

2. Create the namespace and bootstrap superuser secret

kubectl create namespace redpanda

kubectl -n redpanda create secret generic redpanda-bootstrap-user \
  --from-literal=password='REPLACE_WITH_STRONG_PASSWORD'

3. Create the schema registry's SASL user secret

This Secret is the single source of truth for the schema-registry-client SCRAM credential. The new chart field reads it, and the User CR in step 5 reads it. The keys must be exactly username and password (matching corev1.BasicAuthUsernameKey / corev1.BasicAuthPasswordKey):

kubectl -n redpanda create secret generic schema-registry-sasl \
  --from-literal=username='schema-registry-client' \
  --from-literal=password='REPLACE_WITH_SR_STRONG_PASSWORD'

The SASL mechanism is fixed at SCRAM-SHA-512 (see DefaultSASLMechanism at charts/redpanda/secrets.go:24).

4. Apply the Redpanda CR

The operator takes spec.clusterSpec and feeds it into the chart — same field shape as a Helm values.yaml, including the new config.schema_registry_client.saslSecretRef.

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda
  namespace: redpanda
spec:
  clusterSpec:
    auth:
      sasl:
        enabled: true
        # Chart-managed Secret name. users[] is empty because the
        # schema-registry-client user is created via the User CR (step 5),
        # which reads the password from the existing Secret rather than
        # baking it into values.yaml.
        secretRef: users
        bootstrapUser:
          secretKeyRef:
            name: redpanda-bootstrap-user
            key: password
        users: []
    config:
      schema_registry_client:
        # ↓ The new field from this PR. The configurator init container
        # mounts schema-registry-sasl as SCHEMA_REGISTRY_CLIENT_USERNAME /
        # SCHEMA_REGISTRY_CLIENT_PASSWORD env vars and patches
        # redpanda.yaml at pod start via the existing fixups mechanism.
        saslSecretRef:
          name: schema-registry-sasl

kubectl apply -f redpanda.yaml
kubectl -n redpanda wait redpanda/redpanda \
  --for=condition=Ready --timeout=10m

No password appears in spec.clusterSpec. The credential exists only in the schema-registry-sasl Secret. Both the configurator (via this PR's saslSecretRef) and the operator's user reconciler (via the User CR below) read from that one Secret.

5. Apply the User CR

This is what actually registers the schema-registry-client username/password as a SCRAM credential inside the cluster — the operator reconciles it by reading the password out of the same schema-registry-sasl Secret the chart's saslSecretRef points at.

apiVersion: cluster.redpanda.com/v1alpha2
kind: User
metadata:
  name: schema-registry-client
  namespace: redpanda
spec:
  cluster:
    clusterRef:
      name: redpanda
  authentication:
    type: scram-sha-512
    password:
      valueFrom:
        secretKeyRef:
          name: schema-registry-sasl
          key: password
  authorization:
    acls:
      # Schema Registry's internal Kafka client needs read/write access to
      # the _schemas topic (Redpanda creates and reads it for SR state) and
      # to the consumer group it uses to track its offsets.
      - type: allow
        resource:
          type: topic
          name: _schemas
          patternType: literal
        operations:
          - All
      - type: allow
        resource:
          type: group
          name: schema-registry
          patternType: prefixed
        operations:
          - All

kubectl apply -f user-schema-registry.yaml
kubectl -n redpanda wait user/schema-registry-client \
  --for=condition=Ready --timeout=2m

There is no chicken-and-egg: the schema registry retries SASL handshakes against its local Kafka API until the operator finishes reconciling the user. If the cluster comes up before the User is Ready, the schema registry just retries until it can authenticate.

6. Verify nothing is in plaintext

The ConfigMap should contain only CEL fixup expressions — never the actual password:

kubectl -n redpanda get configmap redpanda \
  -o jsonpath='{.data.redpanda\.yaml\.fixups}' | jq

Expected output:

[
  {"field":"schema_registry_client.scram_username","cel":"envString(\"SCHEMA_REGISTRY_CLIENT_USERNAME\")"},
  {"field":"schema_registry_client.scram_password","cel":"envString(\"SCHEMA_REGISTRY_CLIENT_PASSWORD\")"},
  {"field":"schema_registry_client.sasl_mechanism","cel":"\"SCRAM-SHA-512\""}
]

And the configurator init container should source the values from secretKeyRef — not as inline literals:

kubectl -n redpanda get sts redpanda -o yaml \
  | yq '.spec.template.spec.initContainers[]
        | select(.name=="redpanda-configurator").env[]
        | select(.name | test("SCHEMA_REGISTRY_CLIENT"))'

Expected output:

name: SCHEMA_REGISTRY_CLIENT_USERNAME
valueFrom:
  secretKeyRef:
    name: schema-registry-sasl
    key: username
---
name: SCHEMA_REGISTRY_CLIENT_PASSWORD
valueFrom:
  secretKeyRef:
    name: schema-registry-sasl
    key: password

7. Create the demo topic with a Topic CR

Since the operator is already running, we use the Topic CR instead of rpk topic create. This keeps every cluster-state object (the cluster, the SCRAM user, and now the topic) declared the same way and reconciled by the operator.

apiVersion: cluster.redpanda.com/v1alpha2
kind: Topic
metadata:
  name: demo
  namespace: redpanda
spec:
  cluster:
    clusterRef:
      name: redpanda
  partitions: 3
  replicationFactor: 3

kubectl apply -f topic-demo.yaml
kubectl -n redpanda wait topic/demo \
  --for=condition=Ready --timeout=2m

The CR's name (demo) becomes the topic name. To use a different topic name from the CR name, set spec.overwriteTopicName. Extra topic-level configs (e.g. cleanup.policy=compact, tiered-storage flags) go under spec.additionalConfig.

8. End-to-end smoke test with rpk and the Schema Registry

The Redpanda image bundles rpk, and the pods already have RPK_USER / RPK_PASS / RPK_SASL_MECHANISM env vars wired to the bootstrap user — so we can exec into a pod and exercise both Kafka SASL and the Schema Registry without copying any password to our workstation.

Test 1 — Kafka SASL (proves the bootstrap user can produce/consume against the operator-managed topic):

# Produce a message to the topic created by the Topic CR.
kubectl -n redpanda exec -it redpanda-0 -c redpanda -- \
  bash -c 'echo "hello from sasl" | rpk topic produce demo'

# Consume it back.
kubectl -n redpanda exec -it redpanda-0 -c redpanda -- \
  rpk topic consume demo -n 1

If SASL is wired correctly you'll see the message echoed back. A SASL_AUTHENTICATION_FAILED here means the bootstrap user secret is wrong — not related to this PR.

Test 2 — Schema Registry against the cluster (proves the new saslSecretRef works):

This is the actual feature under test. The schema registry's internal Kafka client must authenticate as schema-registry-client to read/write its _schemas topic. Every command below succeeds only if the credentials from schema-registry-sasl were correctly injected at pod start.

kubectl -n redpanda port-forward svc/redpanda 8081:8081 &

# List subjects on a fresh cluster.
curl -sk http://localhost:8081/subjects
# Expected: []

# Register a simple Avro schema for the demo topic's value.
curl -sk -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Demo\",\"fields\":[{\"name\":\"msg\",\"type\":\"string\"}]}"}' \
  http://localhost:8081/subjects/demo-value/versions
# Expected: {"id":1}

# Confirm the subject and version landed.
curl -sk http://localhost:8081/subjects
# Expected: ["demo-value"]

curl -sk http://localhost:8081/subjects/demo-value/versions
# Expected: [1]

curl -sk http://localhost:8081/subjects/demo-value/versions/1
# Expected: full schema JSON

Test 3 — Produce a schema-bound record with rpk:

# Produce a record encoded against schema ID 1.
kubectl -n redpanda exec -it redpanda-0 -c redpanda -- \
  bash -c 'echo "{\"msg\":\"hello schema\"}" | rpk topic produce demo --schema-id 1'

# Consume and have rpk decode using the registered schema.
kubectl -n redpanda exec -it redpanda-0 -c redpanda -- \
  rpk topic consume demo --use-schema-registry value -n 1

If you see SASL_AUTHENTICATION_FAILED or unable to fetch schema from rpk, double-check that the User CR has reconciled:

kubectl -n redpanda get user schema-registry-client -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

It should report "status":"True". If not, inspect the operator's logs:

kubectl -n redpanda-system logs deploy/redpanda-operator | grep schema-registry-client

9. Rotation

The User CR watches the same Secret the chart's saslSecretRef points at, so password rotation collapses to two non-racy steps:

NEW_PASSWORD='NEW_STRONG_PASSWORD'

# 1. Update the Kubernetes Secret in place. The operator watches it and
# reconciles the cluster's SCRAM credential automatically.
kubectl -n redpanda create secret generic schema-registry-sasl \
  --from-literal=username='schema-registry-client' \
  --from-literal=password="$NEW_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

# 2. Bounce the StatefulSet so the configurator init container re-reads
# the Secret and patches redpanda.yaml with the new password.
kubectl -n redpanda rollout restart statefulset/redpanda

The operator typically completes the SCRAM credential update inside a second or two; the pod rollout takes longer than that, so by the time the first restarted pod tries to authenticate, the cluster already has the new credential. No rpk security user update, no race window.

How the previous Helm chart fell short

• The config.schema_registry_client block (in values.yaml and the SchemaRegistryClient Go struct at charts/redpanda/values.go:1195) only exposed tuning knobs — retries, batch sizes, consumer timeouts. There was no field for scram_username / scram_password / sasl_mechanism, and no slot for a secretRef.
• The only escape hatch was config.node (free-form key/value), which gets serialized verbatim into the ConfigMap's redpanda.yaml. Setting credentials that way embeds the cleartext password into the ConfigMap, the Helm release history (Secret of kind helm.sh/release.v1), and any helm get manifest output.
• The V1 operator already solved this — operator/pkg/resources/configuration.go:488 references the schema-registry SASL user secret and emits AddFixup calls plus EnsureInitEnv env vars on the configurator init container, the same pattern this PR brings to the V2 chart. Users moving from the V1 operator to the V2 chart hit a real regression: clusters with auth.sasl.enabled=true saw the schema registry fail to authenticate to its own Kafka API unless they were willing to bake the password into version control.

Authorship

Five commits authored by @AldoFusterTurpin are preserved (cherry-picked from #1503). A single follow-up commit by me adds the missing charts/redpanda changelog entry and regenerates the lifecycle TestV2ResourceClient golden file (every rendered ConfigMap now carries an empty redpanda.yaml.fixups: '[]' entry because the configurator looks for that file unconditionally).

Test plan

• TestSASLFixups (unit): asserts fixup field names and CEL expressions
• TestSASLEnvVars (unit): asserts env var names and secretKeyRef shape
• TestTemplate/sasl-schema-registry-client-secret-ref: verifies env vars in the configurator init container and fixup entries in the ConfigMap when SASL is enabled
• TestTemplate/sasl-disabled-secret-ref-ignored: verifies nothing is injected when auth.sasl.enabled=false
• TestV2ResourceClient lifecycle golden tests pass with the regenerated golden file
• helm lint --strict passes with the feature enabled
• Full go test ./charts/redpanda/ and go test ./operator/internal/lifecycle/ pass locally

🤖 Generated with Claude Code

[AldoFusterTurpin]

@david-yu FYI I just rebased my feature branch onto main and pushed.

[david-yu]

@AldoFusterTurpin Thanks I will pull in your changes.

[david-yu]

@AldoFusterTurpin Are you using the Operator or only the Helm chart? If only Helm chart I assume you are simply creating users using rpk?

[AldoFusterTurpin]

@AldoFusterTurpin Are you using the Operator or only the Helm chart? If only Helm chart I assume you are simply creating users using rpk?

(sorry for the delay, timezones difference 🙂)

We are using the redpanda-operator, and the topics were created using the topic CR and the users using the user CR. I opened an issue with some problems I faced (before I opened this PR), in case you want to check the exact config.

Thank you!

[david-yu]

End-to-end on local kind — bug found and fixed (commit cb8c8c0)

After bumping fs.inotify limits in the Docker VM to get kind past systemd boot, I ran the chart from this PR through a 1-node kind cluster (cert-manager + helm install of charts/redpanda/chart with auth.sasl.enabled: true and config.schema_registry_client.saslSecretRef.name: schema-registry-sasl).

The chart-rendered artifacts looked right at render-time:

• ConfigMap data["redpanda.yaml.fixups"] had the three CEL entries
• StatefulSet's redpanda-configurator init container had SCHEMA_REGISTRY_CLIENT_USERNAME/PASSWORD env vars from secretKeyRef

But the runtime /etc/redpanda/redpanda.yaml inside the pod showed:

schema_registry_client:
    brokers:
        - address: rp-0.rp.redpanda.svc.cluster.local.
          port: 9093
    broker_tls:
        truststore_file: /etc/tls/certs/default/ca.crt
        enabled: true

No scram_username / scram_password / sasl_mechanism. The fixups never reached disk.

Root cause

The chart has two init containers:

Init container	Image	Behavior
bootstrap-yaml-envsubst	redpanda-operator	Runs the Go binary /redpanda-operator bootstrap. Reads bootstrap.yaml.fixups, applies CEL, writes .bootstrap.yaml.
redpanda-configurator	redpanda	Runs a bash script generated by SecretConfigurator() in charts/redpanda/secrets.go:299-358. Uses rpk redpanda config set for specific advertised-listener fields. Never reads redpanda.yaml.fixups.

The fixups in redpanda.yaml.fixups are designed for the Go-based configurator in operator/cmd/configurator/configurator.go (applyFixups()), which the V1 operator's Cluster CR flow invokes. The chart's helm-install path uses the bash configurator instead, so the fixups sit unused.

Why the SR test on a single-broker cluster still appeared to "work": Redpanda runs SR in-process with the broker, so SR's writes to _schemas go through internal Seastar calls — they never exercise the schema_registry_client Kafka client config. In a multi-broker cluster where SR routes to a non-local leader, the missing scram_username would cause SASL_AUTHENTICATION_FAILED.

Why the PR's existing tests missed it: all four (TestSASLFixups, TestSASLEnvVars, TestTemplate/sasl-schema-registry-client-secret-ref, TestTemplate/sasl-disabled-secret-ref-ignored) assert on rendered chart output, not on the bash configurator's behavior. The render-time assertions pass even though the runtime path is broken.

Fix (commit cb8c8c0c)

SecretConfigurator() now emits three additional rpk redpanda config set lines when auth.sasl.enabled && config.schema_registry_client.saslSecretRef != nil:

# Inject schema_registry_client SCRAM credentials from saslSecretRef
# (env vars are projected from the named Secret in the StatefulSet).
set +x
rpk --config "$CONFIG" redpanda config set schema_registry_client.scram_username "${SCHEMA_REGISTRY_CLIENT_USERNAME}"
rpk --config "$CONFIG" redpanda config set schema_registry_client.scram_password "${SCHEMA_REGISTRY_CLIENT_PASSWORD}"
rpk --config "$CONFIG" redpanda config set schema_registry_client.sasl_mechanism "SCRAM-SHA-512"
set -x

Notes:

• set +x around the patches keeps the password out of the init container's stdout (the rest of the script runs under set -xe).
• The lines reference env vars already injected by statefulset.go (the SASLEnvVars() you added) — no new wiring needed.
• The CEL fixups in redpanda.yaml.fixups are unchanged; they continue to work on the V1 operator's Cluster CR path. The new bash lines mirror their effect on the chart's standalone install (and on the V2 operator's Redpanda CR path, which also uses the helm chart).

TestTemplate/sasl-schema-registry-client-secret-ref now also asserts the rendered redpanda-configurator Secret's configurator.sh contains the three field paths and both env var references. The negative test asserts they're absent when SASL is off.

Alternatives considered

• Replace the bash configurator with the Go binary. Cleanest long-term path (single fixup pipeline, no duplication), but a much larger change requiring the chart to depend on the redpanda-operator image for an additional init container. Out of scope for this PR.
• Add a new init container that runs /redpanda-operator apply-fixups. Would need a new subcommand on the operator binary (the existing bootstrap subcommand only handles bootstrap.yaml, not redpanda.yaml). Also a wider change.
• Render the SCRAM fields directly into redpanda.yaml at template time as ${VAR} placeholders + envsubst. Doesn't keep them out of the ConfigMap (defeats the whole point of this PR), so a non-starter.

The bash-rpk-set fix is small, doesn't change the public API of the new feature, and keeps both code paths producing identical on-disk redpanda.yaml.

Verification after the fix

Re-ran the PR's tests and the broader chart suite:

PASS  TestSASLFixups/schema_registry_client
PASS  TestSASLEnvVars/schema_registry_client
PASS  TestTemplate/sasl-schema-registry-client-secret-ref       (now also asserts configurator.sh has the patches)
PASS  TestTemplate/sasl-disabled-secret-ref-ignored              (now also asserts configurator.sh does NOT)
ok    github.com/redpanda-data/redpanda-operator/charts/redpanda/v25                44.165s
ok    github.com/redpanda-data/redpanda-operator/operator/internal/lifecycle        25.204s

🤖 Generated with Claude Code

[david-yu]

Will mark as ready for review and close the original PR.

[RafalKorepta]

I feel like RedpandaYamlFixupFile obsolete from the whole change. The bash script with rpk commands is doing the process of settings schema_registry_client.scram_username and schema_registry_client.scram_password.

I personally would love to see init container that not only expands bootstrap file but redpanda too. We don't have that feature in operator subcommand, but as comment suggested operator V1 have this.

[AldoFusterTurpin]

@david-yu Thank you for testing it and applying the bug fix for one of the paths.

This is my first contribution so I was not aware that there were 2 ways of applying that configuration, that's why I missed the "The chart's helm-install path uses the bash configurator instead, so the fixups sit unused.", sorry about that.

Thanks!

[AldoFusterTurpin]

@david-yu @RafalKorepta Could you please check if this comment makes sense ? I am facing some unexpected problems in the redpanda cluster and I am not sure if the operator config could be related. I would really appreciate that.

[AldoFusterTurpin]

@david-yu I have just opened another PR that could help us with the connections churn we are seeing in our cluster:

feat(operator): make TopicReconciler synchronization interval configurable

Add a --topic-sync-interval CLI flag (default 3s, matching prior behaviour) that sets the reconcile interval for all Topic CRs when spec.synchronizationInterval is not set on the individual resource.

I would really appreciate if you can take a look when you can. Thank you!