Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughThis pull request adds comprehensive documentation for migrating Redpanda clusters between different Iceberg catalog backends. It introduces a new how-to guide ( Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
modules/manage/pages/iceberg/migrate-iceberg-catalog.adoc (1)
82-82: ⚡ Quick winUse auto-title xref here to avoid link-text drift.
Prefer
xref:...[]instead of hard-coded link text so the target page title stays in sync automatically.♻️ Proposed fix
-For full property guidance for each catalog type, see xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[Connect to a REST catalog]. +For full property guidance for each catalog type, see xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[].Based on learnings: “AsciiDoc linking: prefer using xref links with empty brackets … avoid hard-coding link text.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@modules/manage/pages/iceberg/migrate-iceberg-catalog.adoc` at line 82, Replace the hard-coded link xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[Connect to a REST catalog] with an auto-title xref so the link text stays in sync; use xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[] (or xref:manage:iceberg/use-iceberg-catalogs.adoc[] if you want the page title instead of a section) to preserve the target/anchor while removing hard-coded link text.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@modules/manage/pages/iceberg/migrate-iceberg-catalog.adoc`:
- Line 82: Replace the hard-coded link
xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[Connect to a REST catalog]
with an auto-title xref so the link text stays in sync; use
xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[] (or
xref:manage:iceberg/use-iceberg-catalogs.adoc[] if you want the page title
instead of a section) to preserve the target/anchor while removing hard-coded
link text.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ba35c644-bd33-4da0-ad84-3771a53d15e8
📒 Files selected for processing (3)
modules/ROOT/nav.adocmodules/manage/pages/iceberg/migrate-iceberg-catalog.adocmodules/manage/pages/iceberg/use-iceberg-catalogs.adoc
| + | ||
| NOTE: Do not change config_ref:iceberg_enabled,true,properties/cluster-properties[`iceberg_enabled`] at the cluster level. The Iceberg integration must remain enabled at the cluster level so that pending commits can drain to the old catalog. | ||
|
|
||
| . Wait for pending commits to drain. Monitor the `redpanda_iceberg_pending_commit_lag` Prometheus metric on every broker until it reaches `0` for every Iceberg topic-partition. |
There was a problem hiding this comment.
@wdberkeley We don't currently have redpanda_iceberg_pending_commit_lag documented for Cloud or Self-managed. We do currently have vectorized_cluster_partition_iceberg_offsets_pending_commit and vectorized_cluster_partition_iceberg_offsets_pending_translation documented for Self-managed, under Internal Metrics.
Is redpanda_iceberg_pending_commit_lag a new metric that should be added to the Public Metrics page? (Claude also found redpanda_iceberg_pending_translation_lag).
There was a problem hiding this comment.
Hm good question. In the broker code the metrics are redpanda_iceberg_pending_translation_lag and redpanda_iceberg_pending_commit_lag. Those other metrics sound like they could be synonyms or aggregates that we expose but I'm not sure. I think that's a question for Cloud. If users don't have access to the pending commit metric, then they can't safely do the migration without support looking at the metric for them.
There was a problem hiding this comment.
I found redpanda-data/redpanda@adccab8a84 and I do believe these metrics are also exposed in Cloud -- I'll file a separate doc ticket to look closer into why our metrics doc generation process didn't pull those in.
| ifndef::env-cloud[] | ||
| For instructions, see xref:manage:cluster-maintenance/rolling-restart.adoc[]. | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| Coordinate the restart with Redpanda Support. The restart must occur after `redpanda_iceberg_pending_commit_lag` has reached `0` (step 3) and before you resume translation in the next step. | ||
| endif::[] |
There was a problem hiding this comment.
@wdberkeley could you please confirm if the rolling restart is the correct guidance for Self-managed, and that this restart isn't self-serve for Cloud users?
There was a problem hiding this comment.
Yes, a rolling restart of the whole cluster started after commit lag has dropped to zero for all topic-partitions and before re-enabling translation is the right move. The restart is required on all brokers so that they initialize with the correct catalog and the correct catalog authentication.
I'm not sure about how Cloud users do restarts... is it really not self-serve for them? I think #help-cloud could confirm quickly.
There was a problem hiding this comment.
@wzzzrd86 Could you please confirm that we're good to include this restart guidance for Cloud users in this doc?
| branches: [v/*, shared, site-search,'!v-end-of-life/*'] | ||
| - url: https://github.com/redpanda-data/cloud-docs | ||
| branches: main | ||
| branches: 'DOC-2128-migrate-iceberg-catalog' |
There was a problem hiding this comment.
| branches: 'DOC-2128-migrate-iceberg-catalog' | |
| branches: main |
commit before merge
| :learning-objective-2: Pause Iceberg translation and drain pending commits without losing untranslated data | ||
| :learning-objective-3: Apply new catalog configuration and resume translation safely | ||
|
|
||
| Switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data. Use this procedure when moving from the filesystem-based `object_storage` catalog to a managed REST catalog, or when changing between REST catalogs. |
There was a problem hiding this comment.
This first line seems abrupt, like it came out of some other context. The second sentence seems like a better opening sentence.
Use the procedures in this topic when moving from the filesystem-based object_storage catalog to a managed REST catalog, or when changing between REST catalogs. By doing so, you can switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data.
|
|
||
| Switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data. Use this procedure when moving from the filesystem-based `object_storage` catalog to a managed REST catalog, or when changing between REST catalogs. | ||
|
|
||
| The procedure pauses Iceberg translation per topic, lets pending commits drain to the old catalog, applies the new catalog configuration, and restarts the cluster. While translation is paused, retention is temporarily set to infinite to prevent untranslated data from being deleted. |
There was a problem hiding this comment.
| The procedure pauses Iceberg translation per topic, lets pending commits drain to the old catalog, applies the new catalog configuration, and restarts the cluster. While translation is paused, retention is temporarily set to infinite to prevent untranslated data from being deleted. | |
| This procedure guides you through pausing Iceberg translation per topic, draining pending commits to the old catalog, applying the new catalog configuration, and restarting the cluster. |
| rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=<previous-mode> | ||
| ---- | ||
|
|
||
| . Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved in step 1. |
There was a problem hiding this comment.
Referencing a step by number breaks if the order changes. The style guide discourages numbered step references in body text for the same reason it discourages step numbers in headings.
| rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=<previous-mode> | ||
| ---- | ||
|
|
||
| . Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved in step 1. |
There was a problem hiding this comment.
| . Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved in step 1. | |
| . Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved before starting the migration. |
| In either case, you use the catalog to load, query, or refresh the Iceberg table as you produce to the Redpanda topic. See the documentation for your query engine or Iceberg-compatible tool for specific guidance on adding the Iceberg tables to your data warehouse or lakehouse using the catalog. | ||
|
|
||
| After you have selected a catalog type at the cluster level and xref:{about-iceberg-doc}#enable-iceberg-integration[enabled the Iceberg integration] for a topic, you cannot switch to another catalog type. | ||
| To switch to a different catalog type after you have xref:{about-iceberg-doc}#enable-iceberg-integration[enabled the Iceberg integration] for a topic, see xref:manage:iceberg/migrate-iceberg-catalog.adoc[]. |
There was a problem hiding this comment.
| To switch to a different catalog type after you have xref:{about-iceberg-doc}#enable-iceberg-integration[enabled the Iceberg integration] for a topic, see xref:manage:iceberg/migrate-iceberg-catalog.adoc[]. | |
| After you enable Iceberg integration for a topic, you can switch to a different catalog type. See xref:manage:iceberg/migrate-iceberg-catalog.adoc[]. |
|
|
||
| . Wait for pending commits to drain. Monitor the `redpanda_iceberg_pending_commit_lag` metric until it reaches `0` for every Iceberg topic-partition. | ||
| + | ||
| This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. The translation lag metric, `redpanda_iceberg_pending_translation_lag`, can remain non-zero. That value reflects new records the cluster has not yet translated, which is expected while translation is paused. |
There was a problem hiding this comment.
"Which is expected" has an ambiguous antecedent — it could refer to the metric being non-zero, the untranslated records, or the whole situation
|
|
||
| . Wait for pending commits to drain. Monitor the `redpanda_iceberg_pending_commit_lag` metric until it reaches `0` for every Iceberg topic-partition. | ||
| + | ||
| This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. The translation lag metric, `redpanda_iceberg_pending_translation_lag`, can remain non-zero. That value reflects new records the cluster has not yet translated, which is expected while translation is paused. |
There was a problem hiding this comment.
| This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. The translation lag metric, `redpanda_iceberg_pending_translation_lag`, can remain non-zero. That value reflects new records the cluster has not yet translated, which is expected while translation is paused. | |
| This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. A non-zero value here is expected while translation is paused, and reflects new records the cluster has not yet translated. |
| For instructions, see xref:manage:cluster-maintenance/rolling-restart.adoc[]. | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| Coordinate the restart with Redpanda Support. The restart must occur after `redpanda_iceberg_pending_commit_lag` has reached `0` (step 3) and before you resume translation in the next step. |
There was a problem hiding this comment.
Can Cloud customers not initiate a rolling restart themselves? If they can, this guidance may be overly restrictive. If they can't, is "Coordinate with Redpanda Support" the right instruction, or is there a specific support process to reference? I see you included @wzzzrd86 here--good--as we should always check with support before placing guidance in the docs to contact support.
3 participants
Description
Related: redpanda-data/cloud-docs#577
This pull request updates the Iceberg documentation to clarify and improve guidance around migrating between Iceberg catalog backends in Redpanda. The main changes include adding a new migration guide, updating navigation structure, and improving cross-references for users seeking to switch catalog types.
Documentation improvements for Iceberg catalog migration:
migrate-iceberg-catalog.adoc, detailing the step-by-step procedure to safely migrate an existing Redpanda cluster from one Iceberg catalog backend to another without losing untranslated topic data. This includes prerequisites, catalog compatibility verification, migration steps, and troubleshooting.nav.adocto add the new "Migrate to Iceberg Topics" and "Migrate Iceberg Catalogs" pages, and adjusted their placement for easier discovery. [1] [2]use-iceberg-catalogs.adocto direct users to the new migration procedure when switching catalog types, replacing the previous statement that switching was not possible.Resolves https://redpandadata.atlassian.net/browse/
Review deadline:
Page previews
Cloud: https://deploy-preview-1692--redpanda-docs-preview.netlify.app/redpanda-cloud/manage/iceberg/migrate-iceberg-catalog/
Self-managed: https://deploy-preview-1692--redpanda-docs-preview.netlify.app/current/manage/iceberg/migrate-iceberg-catalog/
Checks