From c6e599e4827c7aa6ae3d5f6b6ff9265f5303006b Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Sat, 9 May 2026 06:50:33 +0000 Subject: [PATCH 1/4] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- doc/developer/generated/adapter/catalog/_module.md | 2 +- doc/developer/generated/adapter/catalog/state.md | 3 ++- doc/developer/generated/adapter/catalog/transact.md | 2 +- doc/developer/generated/adapter/coord/command_handler.md | 4 ++-- doc/developer/generated/adapter/coord/group_sync.md | 6 +++++- doc/developer/generated/adapter/error.md | 4 ++-- doc/developer/generated/adapter/notice.md | 3 ++- doc/developer/generated/authenticator/oidc.md | 2 +- doc/developer/generated/catalog/builtin/_module.md | 5 +++-- doc/developer/generated/catalog/builtin/ontology.md | 4 ++-- doc/developer/generated/compute-client/logging.md | 2 +- doc/developer/generated/oidc-mock/_crate.md | 2 +- doc/developer/generated/orchestrator-process/_crate.md | 2 +- 13 files changed, 24 insertions(+), 17 deletions(-) diff --git a/doc/developer/generated/adapter/catalog/_module.md b/doc/developer/generated/adapter/catalog/_module.md index af615fba87aaa..628fde0c7e9a8 100644 --- a/doc/developer/generated/adapter/catalog/_module.md +++ b/doc/developer/generated/adapter/catalog/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog.rs -revision: 7f632d2b4a +revision: fda8736965 --- # adapter::catalog diff --git a/doc/developer/generated/adapter/catalog/state.md b/doc/developer/generated/adapter/catalog/state.md index c48206f7483ef..29bb0dba25839 100644 --- a/doc/developer/generated/adapter/catalog/state.md +++ b/doc/developer/generated/adapter/catalog/state.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/state.rs -revision: 9d0a7c3c6f +revision: fda8736965 --- # adapter::catalog::state @@ -9,4 +9,5 @@ Defines `CatalogState`, the in-memory representation of the entire Materialize c `CatalogState` holds `imbl::OrdMap` collections for databases, schemas, roles, role auth (`role_auth_by_id`), clusters, network policies, the item namespace (keyed by both `CatalogItemId` and `GlobalId`), source references, temporary schemas per connection, comments, default privileges, system privileges, system configuration, and a `notices_by_dep_id` index mapping each `GlobalId` to the optimizer notices that depend on it; it also implements `SessionCatalog` for SQL name resolution. The file also defines `LocalExpressionCache`, a helper used during catalog open and catalog transactions to track which optimizer expressions were found in (or are missing from) the persistent expression cache, avoiding redundant re-optimization. When `deserialize_item` re-parses an existing item (e.g. during a RENAME), it preserves the `optimized_plan`, `physical_plan`, and `dataflow_metainfo` fields from the previous `CatalogItem` incarnation via `plan_fields_mut`; these fields are not reconstructable from `create_sql` alone and must be carried over to avoid silently dropping plans for materialized views and indexes. +Two case-insensitive role lookup helpers support JWT group-to-role sync: `try_get_role_by_name_case_insensitive` finds a single role by lowercased name, and `roles_by_lowercase_name` builds a `BTreeMap` keyed by lowercased name for efficient bulk lookups across multiple group claims. This is the authoritative read-side view of the catalog; mutations go through `catalog::transact` and are applied by `catalog::apply`. diff --git a/doc/developer/generated/adapter/catalog/transact.md b/doc/developer/generated/adapter/catalog/transact.md index 72644ed1fa3c3..56ed2c3b80255 100644 --- a/doc/developer/generated/adapter/catalog/transact.md +++ b/doc/developer/generated/adapter/catalog/transact.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/transact.rs -revision: 05bac54142 +revision: 33ea0b3182 --- # adapter::catalog::transact diff --git a/doc/developer/generated/adapter/coord/command_handler.md b/doc/developer/generated/adapter/coord/command_handler.md index 0a58fbcf3860b..875f67ae4f5c4 100644 --- a/doc/developer/generated/adapter/coord/command_handler.md +++ b/doc/developer/generated/adapter/coord/command_handler.md @@ -1,12 +1,12 @@ --- source: src/adapter/src/coord/command_handler.rs -revision: 122dfd0789 +revision: fda8736965 --- # adapter::coord::command_handler Implements the coordinator's `handle_command` method, which dispatches each `Command` variant to the appropriate handler. Covers session startup and teardown (`Startup`, `Terminate`), authentication, SQL execution (`Execute`, `Commit`), COPY FROM STDIN setup (`StartCopyFromStdin`), webhook lookup, system variable get/set, consistency checks, and frontend-offload variants (`ExecuteSlowPathPeek`, `ExecuteSubscribe`, `ExecuteCopyTo`, `CopyToPreflight`, `ExecuteSideEffectingFunc`, `RegisterFrontendPeek`, `UnregisterFrontendPeek`, `ExplainTimestamp`, `FrontendStatementLogging`) that reduce coordinator bottleneck. -`handle_startup` performs privilege checks, sets session defaults, and emits the initial builtin-table writes for session tracking. +`handle_startup` performs privilege checks, triggers JWT group-to-role membership sync via `maybe_sync_jwt_groups`, sets session defaults, and emits the initial builtin-table writes for session tracking. `CheckRoleCanLogin` (`handle_role_can_login`) handles pre-authentication role existence and login-attribute checks; `InjectAuditEvents` inserts manually injected audit log entries via a catalog transaction. A module-private `RoleLoginStatus` enum (`NotFound`, `CanLogin`, `NonLogin`) and `role_login_status` helper are shared among authentication handlers (`handle_authenticate_password`, `handle_generate_sasl_challenge`, `handle_authenticate_verify_sasl_proof`). diff --git a/doc/developer/generated/adapter/coord/group_sync.md b/doc/developer/generated/adapter/coord/group_sync.md index 24279008e59c3..9051a2097a6a6 100644 --- a/doc/developer/generated/adapter/coord/group_sync.md +++ b/doc/developer/generated/adapter/coord/group_sync.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/group_sync.rs -revision: 0a20c581ea +revision: fda8736965 --- # adapter::coord::group_sync @@ -13,3 +13,7 @@ Manually-granted roles are never touched: the function neither revokes them nor Roles to grant are those in the target set that are absent from both sync-managed and manual sets; roles to revoke are those in the sync-managed set that are absent from the target set. `GroupSyncDiff` bundles the resulting `grants` and `revokes` operation vectors for the caller to apply as a single catalog transaction. + +`Coordinator::maybe_sync_jwt_groups` is the top-level entry point called during connection startup: it checks whether sync is enabled (via `OIDC_GROUP_ROLE_SYNC_ENABLED`), short-circuits when no group claim is present (`groups == None`), and delegates to `sync_jwt_groups`. Errors are either propagated as `AdapterError::OidcGroupSyncFailed` (strict mode, `OIDC_GROUP_ROLE_SYNC_STRICT`) or suppressed with a warning notice (fail-open default). Notices about unmatched groups, reserved-name groups, and sync errors are sent to the client via `notice_tx`. + +`Coordinator::sync_jwt_groups` resolves each group name to a catalog role via case-insensitive lookup (`roles_by_lowercase_name`), filtering out reserved names (`mz_`/`pg_` prefixes) and self-references (where the group resolves to the user's own role). It then calls `compute_group_sync_diff` and executes the resulting grant/revoke ops via `catalog_transact`. diff --git a/doc/developer/generated/adapter/error.md b/doc/developer/generated/adapter/error.md index 44c5292667887..0a11fae3575a9 100644 --- a/doc/developer/generated/adapter/error.md +++ b/doc/developer/generated/adapter/error.md @@ -1,12 +1,12 @@ --- source: src/adapter/src/error.rs -revision: 4002e6039c +revision: fda8736965 --- # adapter::error Defines `AdapterError`, the central error type for the coordinator, with variants covering planning errors, catalog errors, storage/compute controller errors, RBAC violations, timestamp errors, authentication errors, and miscellaneous unstructured errors. -Notable variants include `ConcurrentDependencyDrop { dependency_kind, dependency_id }` for mid-flight dependency drops, `AuthenticationError` (with an inner `AuthenticationError` enum covering `InvalidCredentials`, `NonLogin`, `RoleNotFound`, `PasswordRequired`), `ReplacementSchemaMismatch(RelationDescDiff)` for schema-incompatible replacements, `ReplaceMaterializedViewSealed`, `CollectionUnreadable`, `AlterClusterWhilePendingReplicas`, and `ImpossibleTimestampConstraints`. +Notable variants include `ConcurrentDependencyDrop { dependency_kind, dependency_id }` for mid-flight dependency drops, `AuthenticationError` (with an inner `AuthenticationError` enum covering `InvalidCredentials`, `NonLogin`, `RoleNotFound`, `PasswordRequired`), `ReplacementSchemaMismatch(RelationDescDiff)` for schema-incompatible replacements, `ReplaceMaterializedViewSealed`, `CollectionUnreadable`, `AlterClusterWhilePendingReplicas`, `ImpossibleTimestampConstraints`, and `OidcGroupSyncFailed(String)` for OIDC group-to-role sync failures in strict mode (maps to `SqlState::INTERNAL_ERROR`). Implements `From` conversions from many downstream error types (`PlanError`, `StorageError`, `OptimizerError`, `VarError`, etc.) and maps each variant to a PostgreSQL `SqlState` error code via `code()`; error details and hints are provided through `detail()` and `hint()`. Several constructor helpers on `AdapterError` (e.g. `concurrent_dependency_drop_from_instance_missing`, `concurrent_dependency_drop_from_peek_error`) perform explicit error-kind conversions that are intentionally not automatic `From` impls. The `ShouldTerminateGracefully` trait (private) identifies errors — such as `FenceError::DeployGeneration` — that should cause a clean process exit rather than a panic. diff --git a/doc/developer/generated/adapter/notice.md b/doc/developer/generated/adapter/notice.md index 63d1df1131486..ccce2278af0e1 100644 --- a/doc/developer/generated/adapter/notice.md +++ b/doc/developer/generated/adapter/notice.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/notice.rs -revision: bce428d203 +revision: fda8736965 --- # adapter::notice @@ -8,3 +8,4 @@ revision: bce428d203 Defines `AdapterNotice`, the enum of all informational or warning-level diagnostic messages the adapter emits to clients during query execution. Each variant carries context-specific fields (e.g. object names, timestamps, trace IDs) and maps to a PostgreSQL `SqlState` code, a `Severity`, and optional `detail`/`hint` strings via `into_response`. `DroppedInUseIndex` is a companion struct carrying the index name and the dependent objects preventing its immediate removal. +Three variants cover OIDC group-to-role sync events: `OidcGroupSyncUnmatchedGroup { group }` (Notice severity) for groups with no matching catalog role, `OidcGroupSyncReservedRole { group }` (Warning) for groups that map to reserved role names, and `OidcGroupSyncError { message }` (Warning) for sync failures in fail-open mode. diff --git a/doc/developer/generated/authenticator/oidc.md b/doc/developer/generated/authenticator/oidc.md index 501e4443c7574..422a4f04c9910 100644 --- a/doc/developer/generated/authenticator/oidc.md +++ b/doc/developer/generated/authenticator/oidc.md @@ -1,6 +1,6 @@ --- source: src/authenticator/src/oidc.rs -revision: 0a20c581ea +revision: fda8736965 --- # mz-authenticator::oidc diff --git a/doc/developer/generated/catalog/builtin/_module.md b/doc/developer/generated/catalog/builtin/_module.md index afdfe8313ea22..4fbd2b04b7328 100644 --- a/doc/developer/generated/catalog/builtin/_module.md +++ b/doc/developer/generated/catalog/builtin/_module.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin.rs -revision: 4d59d67c50 +revision: 36df04b4d9 --- # catalog::builtin @@ -11,8 +11,9 @@ Key types include `Builtin` (a generic enum with variants Log, Table, View, M `MZ_CONNECTIONS` and `MZ_SECRETS` are `BuiltinMaterializedView` objects backed by queries over `mz_internal.mz_catalog_raw`, and are registered in `BUILTINS_STATIC` as `Builtin::MaterializedView`. Each builtin struct carries an optional `ontology` field of type `Option` that marks the object as a catalog ontology entity and provides entity-level metadata. `Ontology` captures an `entity_name`, a one-line `description`, a list of `OntologyLink` relationships, and per-column `column_semantic_types` annotations (`&'static [(&'static str, SemanticType)]`). Semantic types are stored here rather than in `RelationDesc` to avoid persist schema mismatches during zero-downtime upgrades. -`OntologyLink` pairs a link `name` with a `LinkProperties` variant: `ForeignKey` (column-to-column reference with `Cardinality`, optional `source_id_type`, `requires_mapping`, `nullable`, and `note`), `Union` (superset view with optional `discriminator_column`/`discriminator_value`), `MapsTo` (ID-type conversion with optional `via` table and `from_type`/`to_type`), `DependsOn` (direct dependency), and `Measures` (metric measurement). `LinkProperties` serializes to JSONB via serde with `#[serde(tag = "kind")]`. +`OntologyLink` pairs a link `name` with a `LinkProperties` variant: `ForeignKey` (column-to-column reference with `Cardinality`, optional `source_id_type`, `requires_mapping`, `nullable`, and `note`), `Union` (superset view with optional `discriminator_column`/`discriminator_value`), `MapsTo` (ID-type conversion with required `source_column`/`target_column` and optional `via` table and `from_type`/`to_type`), `DependsOn` (dependency via a graph-edge table with required `source_column`/`target_column` and optional `extra_key_columns` for composite join keys such as `(id, worker_id)`), and `Measures` (metric measurement). `LinkProperties` serializes to JSONB via serde with `#[serde(tag = "kind")]`. `Cardinality` is `OneToOne` or `ManyToOne`. +Builtin object definitions are split across per-schema submodules: `pg_catalog`, `mz_catalog`, `mz_internal`, `mz_introspection`, and `information_schema`; each re-exports its contents into the parent namespace. The `builtin` submodule generates the `mz_builtin_materialized_views` view, which exposes metadata about all builtin materialized views. The `notice` submodule contains the optimizer-notice tables. The `ontology` submodule contains the static ontology entity definitions used to populate `mz_internal.mz_ontology_*` views. diff --git a/doc/developer/generated/catalog/builtin/ontology.md b/doc/developer/generated/catalog/builtin/ontology.md index 69910805cf9cc..ecb19b16759e2 100644 --- a/doc/developer/generated/catalog/builtin/ontology.md +++ b/doc/developer/generated/catalog/builtin/ontology.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/ontology.rs -revision: ea77b8b38b +revision: ac5e71acee --- # catalog::builtin::ontology @@ -8,7 +8,7 @@ revision: ea77b8b38b Generates the four `mz_internal.mz_ontology_*` views derived from `Ontology` annotations on builtin catalog objects. `generate_views` is the public entry point. -It iterates over all builtins, selects those with an `ontology: Some(...)` field (tables, views, materialized views, and sources), and produces four `Builtin::View` values that are leaked to `'static` lifetime via `leak` — one per view, bounded to startup. +It iterates over all builtins, selects those with an `ontology: Some(...)` field (tables, views, materialized views, sources, and logs), and produces four `Builtin::View` values that are leaked to `'static` lifetime via `leak` — one per view, bounded to startup. The four generated views are: diff --git a/doc/developer/generated/compute-client/logging.md b/doc/developer/generated/compute-client/logging.md index a601d24db34c9..79b8b98c83e47 100644 --- a/doc/developer/generated/compute-client/logging.md +++ b/doc/developer/generated/compute-client/logging.md @@ -1,6 +1,6 @@ --- source: src/compute-client/src/logging.rs -revision: c7616144a8 +revision: 36df04b4d9 --- # mz-compute-client::logging diff --git a/doc/developer/generated/oidc-mock/_crate.md b/doc/developer/generated/oidc-mock/_crate.md index a293be02f5a3b..29e3e7fc4d6ce 100644 --- a/doc/developer/generated/oidc-mock/_crate.md +++ b/doc/developer/generated/oidc-mock/_crate.md @@ -1,6 +1,6 @@ --- source: src/oidc-mock/src/lib.rs -revision: 03db92b55b +revision: fda8736965 --- # mz-oidc-mock diff --git a/doc/developer/generated/orchestrator-process/_crate.md b/doc/developer/generated/orchestrator-process/_crate.md index 4b2de5456e80c..49c330d03abb3 100644 --- a/doc/developer/generated/orchestrator-process/_crate.md +++ b/doc/developer/generated/orchestrator-process/_crate.md @@ -1,6 +1,6 @@ --- source: src/orchestrator-process/src/lib.rs -revision: f6f7d70cc4 +revision: 9c2fb6e98b --- # mz-orchestrator-process From a458e13ff5d8a98cd7cbf6e23d937298cbfd9e02 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Sun, 10 May 2026 06:59:18 +0000 Subject: [PATCH 2/4] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../catalog/builtin/information_schema.md | 27 +++++++++++++++++++ .../generated/catalog/builtin/mz_catalog.md | 22 +++++++++++++++ .../generated/catalog/builtin/mz_internal.md | 24 +++++++++++++++++ .../catalog/builtin/mz_introspection.md | 22 +++++++++++++++ .../generated/catalog/builtin/ontology.md | 2 +- .../generated/catalog/builtin/pg_catalog.md | 18 +++++++++++++ 6 files changed, 114 insertions(+), 1 deletion(-) create mode 100644 doc/developer/generated/catalog/builtin/information_schema.md create mode 100644 doc/developer/generated/catalog/builtin/mz_catalog.md create mode 100644 doc/developer/generated/catalog/builtin/mz_internal.md create mode 100644 doc/developer/generated/catalog/builtin/mz_introspection.md create mode 100644 doc/developer/generated/catalog/builtin/pg_catalog.md diff --git a/doc/developer/generated/catalog/builtin/information_schema.md b/doc/developer/generated/catalog/builtin/information_schema.md new file mode 100644 index 0000000000000..8c8ccf89c5e11 --- /dev/null +++ b/doc/developer/generated/catalog/builtin/information_schema.md @@ -0,0 +1,27 @@ +--- +source: src/catalog/src/builtin/information_schema.rs +revision: 36df04b4d9 +--- + +# catalog::builtin::information_schema + +Defines all built-in catalog objects for the `information_schema` SQL schema. + +This module contains 14 `BuiltinView` definitions exported as `pub static LazyLock` values. Each view implements a standard SQL information schema relation using queries over `mz_catalog` system tables. The views exposed include: + +- `applicable_roles` — roles applicable to the current session user, joining `mz_role_members` and `mz_roles`. +- `columns` — column metadata for all user-visible relations. +- `enabled_roles` — roles currently enabled for the session. +- `key_column_usage` — columns that are part of table constraints. +- `referential_constraints` — foreign key constraint metadata. +- `role_table_grants` — table privileges granted to roles. +- `routines` — stub view for function/procedure metadata. +- `schemata` — all schemas visible to the current role. +- `table_constraints` — primary key and unique constraints. +- `table_privileges` — privilege grants on tables and views. +- `tables` — all tables and views visible to the current role. +- `triggers` — stub view for trigger metadata (always empty). +- `views` — views and materialized views visible to the current role. +- `character_sets` — character set metadata (returns a single UTF-8 row). + +All views grant `PUBLIC_SELECT` access. diff --git a/doc/developer/generated/catalog/builtin/mz_catalog.md b/doc/developer/generated/catalog/builtin/mz_catalog.md new file mode 100644 index 0000000000000..c9d8213379729 --- /dev/null +++ b/doc/developer/generated/catalog/builtin/mz_catalog.md @@ -0,0 +1,22 @@ +--- +source: src/catalog/src/builtin/mz_catalog.rs +revision: 36df04b4d9 +--- + +# catalog::builtin::mz_catalog + +Defines all built-in catalog objects for the `mz_catalog` SQL schema. + +This module exports 78 public items: types, tables, materialized views, views, sources, and indexes. + +**Types** — Materialize-specific pseudo-types and their array forms: `TYPE_LIST`, `TYPE_MAP`, `TYPE_ANYCOMPATIBLELIST`, `TYPE_ANYCOMPATIBLEMAP`; unsigned integer types `TYPE_UINT2/4/8` and their arrays; `TYPE_MZ_TIMESTAMP`, `TYPE_MZ_TIMESTAMP_ARRAY`; `TYPE_MZ_ACL_ITEM`, `TYPE_MZ_ACL_ITEM_ARRAY`. + +**Tables** — `BuiltinTable` statics covering connector-specific metadata: `MZ_ICEBERG_SINKS`, `MZ_KAFKA_SINKS`, `MZ_KAFKA_CONNECTIONS`, `MZ_KAFKA_SOURCES`, and more. + +**Materialized views** — Core catalog entities backed by queries over `mz_internal.mz_catalog_raw`: `MZ_DATABASES`, `MZ_SCHEMAS`, `MZ_CONNECTIONS`, `MZ_SECRETS`, `MZ_TABLES`, `MZ_COLUMNS`, `MZ_INDEXES`, `MZ_VIEWS`, `MZ_SOURCES`, `MZ_SINKS`, `MZ_CLUSTER_REPLICAS`, `MZ_ROLES`, `MZ_OBJECTS`, `MZ_ALL_OBJECTS`, and others. These carry `Ontology` annotations with entity names, descriptions, and `OntologyLink` relationships for the catalog graph. + +**Views** — Additional SQL views for derived catalog information such as `MZ_TIMEZONE_ABBREVIATIONS` and `MZ_TIMEZONE_NAMES`. + +**Sources** — `BuiltinSource` items backed by `DataSourceDesc::Catalog` or `IntrospectionType`. + +**Indexes** — `BuiltinIndex` constants for accelerating catalog queries: `MZ_DATABASES_IND`, `MZ_SCHEMAS_IND`, `MZ_CONNECTIONS_IND`, `MZ_TABLES_IND`, `MZ_TYPES_IND`, `MZ_OBJECTS_IND`, `MZ_COLUMNS_IND`, `MZ_SECRETS_IND`, `MZ_VIEWS_IND`, `MZ_CLUSTERS_IND`, `MZ_INDEXES_IND`, `MZ_ROLES_IND`, `MZ_SOURCES_IND`, `MZ_SINKS_IND`, `MZ_MATERIALIZED_VIEWS_IND`, `MZ_CLUSTER_REPLICAS_IND`, `MZ_CLUSTER_REPLICA_SIZES_IND`, `MZ_KAFKA_SOURCES_IND`, and more. diff --git a/doc/developer/generated/catalog/builtin/mz_internal.md b/doc/developer/generated/catalog/builtin/mz_internal.md new file mode 100644 index 0000000000000..9e4e205403cf6 --- /dev/null +++ b/doc/developer/generated/catalog/builtin/mz_internal.md @@ -0,0 +1,24 @@ +--- +source: src/catalog/src/builtin/mz_internal.rs +revision: 36df04b4d9 +--- + +# catalog::builtin::mz_internal + +Defines all built-in catalog objects for the `mz_internal` SQL schema. + +This is the largest builtin submodule, exporting 186 public items: sources, tables, materialized views, views, indexes, and connections. + +**Sources** (`BuiltinSource`) — Backed by `DataSourceDesc::Catalog` or `IntrospectionType`. Key items include `MZ_CATALOG_RAW` (the raw persist-backed catalog source, system-only access), and storage statistics/status history sources such as `MZ_SOURCE_STATISTICS_RAW`, `MZ_SINK_STATISTICS_RAW`, `MZ_SOURCE_STATUS_HISTORY`, `MZ_SINK_STATUS_HISTORY`, `MZ_STATEMENT_EXECUTION_HISTORY`, `MZ_SESSION_HISTORY`, `MZ_SQL_TEXT`, `MZ_PREPARED_STATEMENT_HISTORY`, and replica metrics/status history sources. + +**Tables** (`BuiltinTable`) — Connector-specific and internal metadata tables such as `MZ_POSTGRES_SOURCES`, `MZ_MYSQL_SOURCES`, and others tracking sink/source details. + +**Materialized views** (`BuiltinMaterializedView`) — Derived catalog views backed by queries over `mz_catalog_raw` and other sources: aggregated statistics, lag histograms, and other derived metrics. + +**Views** (`BuiltinView`) — 161 entries covering the full `mz_internal` SQL surface: cluster introspection, compute operator metrics, peek durations, arrangement sizes, source/sink status, wall-clock lag, statement execution history, privilege management, dependency graph views, and more. Views carry inline SQL queries over `mz_catalog`, `mz_introspection`, and `mz_internal` tables. Access levels vary: most grant `PUBLIC_SELECT`; sensitive views use `MONITOR_SELECT`, `MONITOR_REDACTED_SELECT`, `SUPPORT_SELECT`, or `ANALYTICS_SELECT`. + +**Connections** (`BuiltinConnection`) — System-level connection definitions. + +**Indexes** (`BuiltinIndex`) — 111 index constants accelerating queries on frequently accessed `mz_internal` views and tables. + +Many items carry `Ontology` annotations with `OntologyLink` relationships (foreign keys, union views, dependency edges) for the catalog ontology graph. diff --git a/doc/developer/generated/catalog/builtin/mz_introspection.md b/doc/developer/generated/catalog/builtin/mz_introspection.md new file mode 100644 index 0000000000000..77b04c139cc07 --- /dev/null +++ b/doc/developer/generated/catalog/builtin/mz_introspection.md @@ -0,0 +1,22 @@ +--- +source: src/catalog/src/builtin/mz_introspection.rs +revision: 36df04b4d9 +--- + +# catalog::builtin::mz_introspection + +Defines all built-in catalog objects for the `mz_introspection` SQL schema. + +This module exports 74 public items: `BuiltinLog` statics for per-worker Timely/Differential/Compute log sources, and `BuiltinView` statics that aggregate or reshape those logs. + +**Logs** (`BuiltinLog`) — Each log wraps a `LogVariant` (from `mz_compute_client::logging`) and exposes raw per-worker data from the Timely and Differential dataflow runtimes. Key logs include: + +- Timely scheduling and operator logs: `MZ_DATAFLOW_OPERATORS_PER_WORKER`, `MZ_DATAFLOW_ADDRESSES_PER_WORKER`, `MZ_DATAFLOW_CHANNELS_PER_WORKER`, `MZ_SCHEDULING_ELAPSED_RAW`, `MZ_COMPUTE_OPERATOR_DURATIONS_HISTOGRAM_RAW`, `MZ_SCHEDULING_PARKS_HISTOGRAM_RAW`. +- Arrangement metrics: `MZ_ARRANGEMENT_RECORDS_RAW`, `MZ_ARRANGEMENT_BATCHES_RAW`, `MZ_ARRANGEMENT_SHARING_RAW`, `MZ_ARRANGEMENT_BATCHER_*`, `MZ_ARRANGEMENT_HEAP_*`. +- Compute-level logs: `MZ_COMPUTE_EXPORTS_PER_WORKER`, `MZ_COMPUTE_DATAFLOW_GLOBAL_IDS_PER_WORKER`, `MZ_COMPUTE_FRONTIERS_PER_WORKER`, `MZ_COMPUTE_IMPORT_FRONTIERS_PER_WORKER`, `MZ_COMPUTE_ERROR_COUNTS_RAW`, `MZ_COMPUTE_HYDRATION_TIMES_PER_WORKER`, `MZ_COMPUTE_OPERATOR_HYDRATION_STATUSES_PER_WORKER`, `MZ_COMPUTE_LIR_MAPPING_PER_WORKER`. +- Peek and message logs: `MZ_ACTIVE_PEEKS_PER_WORKER`, `MZ_PEEK_DURATIONS_HISTOGRAM_RAW`, `MZ_MESSAGE_COUNTS_*`, `MZ_MESSAGE_BATCH_COUNTS_*`. +- Prometheus metrics: `MZ_CLUSTER_PROMETHEUS_METRICS`. + +Many logs carry `Ontology` annotations with `OntologyLink` entries expressing foreign-key and dependency relationships between per-worker dataflow entities, used to populate the `mz_internal.mz_ontology_*` views. + +**Views** (`BuiltinView`) — Aggregate the per-worker logs into user-facing introspection views by summing or grouping across workers: e.g., `mz_dataflow_operators`, `mz_dataflow_channels`, `mz_scheduling_elapsed`, `mz_compute_frontiers`, `mz_arrangement_sizes`. diff --git a/doc/developer/generated/catalog/builtin/ontology.md b/doc/developer/generated/catalog/builtin/ontology.md index ecb19b16759e2..56a66abc715a2 100644 --- a/doc/developer/generated/catalog/builtin/ontology.md +++ b/doc/developer/generated/catalog/builtin/ontology.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/ontology.rs -revision: ac5e71acee +revision: fbb2bcd0d8 --- # catalog::builtin::ontology diff --git a/doc/developer/generated/catalog/builtin/pg_catalog.md b/doc/developer/generated/catalog/builtin/pg_catalog.md new file mode 100644 index 0000000000000..1cb1807c4f3d4 --- /dev/null +++ b/doc/developer/generated/catalog/builtin/pg_catalog.md @@ -0,0 +1,18 @@ +--- +source: src/catalog/src/builtin/pg_catalog.rs +revision: 36df04b4d9 +--- + +# catalog::builtin::pg_catalog + +Defines all built-in catalog objects for the `pg_catalog` SQL schema. + +This module contains 114 public items: PostgreSQL-compatible type definitions (`BuiltinType`) and compatibility views (`BuiltinView`). + +**Types** — `BuiltinType` constants cover the full set of PostgreSQL-compatible base types and their array counterparts. Materialize-specific extensions include `TYPE_UINT2`, `TYPE_UINT4`, `TYPE_UINT8`, `TYPE_MZ_TIMESTAMP`, `TYPE_MZ_ACL_ITEM`, and their array forms. All types use static OIDs matching the PostgreSQL type catalog to ensure wire-protocol compatibility. + +**Views** — `BuiltinView` statics implement the `pg_catalog` compatibility layer: `pg_attribute`, `pg_class`, `pg_database`, `pg_depend`, `pg_description`, `pg_enum`, `pg_index`, `pg_indexes`, `pg_locks`, `pg_namespace`, `pg_proc`, `pg_range`, `pg_roles`, `pg_settings`, `pg_tables`, `pg_type`, `pg_views`, and others. These views query `mz_catalog` and `mz_internal` system tables and present the results in the column layout expected by PostgreSQL clients. + +Each `BuiltinView` carries its SQL definition inline, an OID from `mz_pgrepr::oid`, an access list (typically `PUBLIC_SELECT`), and a `RelationDesc` describing its column types. + +The file includes a comment warning that builtin definitions must not be deleted and that column removals or type changes in existing builtins break backwards compatibility with persisted user objects. From c7cad29793c7640b4b5a5fe6fdb47d80a044d8b5 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Mon, 11 May 2026 07:12:10 +0000 Subject: [PATCH 3/4] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- doc/developer/generated/adapter/coord/_module.md | 2 +- doc/developer/generated/adapter/coord/sequencer/inner.md | 2 +- .../generated/adapter/coord/sequencer/inner/_module.md | 2 +- .../generated/adapter/coord/sequencer/inner/cluster.md | 2 +- doc/developer/generated/sql/plan/error.md | 2 +- doc/developer/generated/sql/plan/statement/ddl.md | 2 +- doc/developer/generated/sql/plan/statement/ddl/_module.md | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/developer/generated/adapter/coord/_module.md b/doc/developer/generated/adapter/coord/_module.md index 9c1b53d9d4187..85777ada5f82b 100644 --- a/doc/developer/generated/adapter/coord/_module.md +++ b/doc/developer/generated/adapter/coord/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord.rs -revision: 4d59d67c50 +revision: 433902e0fa --- # adapter::coord diff --git a/doc/developer/generated/adapter/coord/sequencer/inner.md b/doc/developer/generated/adapter/coord/sequencer/inner.md index 7f23d8acc330a..b573873948e37 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner.rs -revision: 07858c9fd7 +revision: 69b338b023 --- # adapter::coord::sequencer::inner diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/_module.md b/doc/developer/generated/adapter/coord/sequencer/inner/_module.md index e55aa7122f42a..aaf35839f3d13 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/_module.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner.rs -revision: 07858c9fd7 +revision: 69b338b023 --- # adapter::coord::sequencer::inner diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md index fd60a35dcd522..31a5fda964b71 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/cluster.rs -revision: eb105c0ce8 +revision: 433902e0fa --- # adapter::coord::sequencer::inner::cluster diff --git a/doc/developer/generated/sql/plan/error.md b/doc/developer/generated/sql/plan/error.md index fb881d62c2ae1..5c7ebb12fa6a7 100644 --- a/doc/developer/generated/sql/plan/error.md +++ b/doc/developer/generated/sql/plan/error.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/error.rs -revision: 721951ce66 +revision: 69b338b023 --- # mz-sql::plan::error diff --git a/doc/developer/generated/sql/plan/statement/ddl.md b/doc/developer/generated/sql/plan/statement/ddl.md index 713f2dd2d8fbb..e37019fee18ed 100644 --- a/doc/developer/generated/sql/plan/statement/ddl.md +++ b/doc/developer/generated/sql/plan/statement/ddl.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl.rs -revision: 16d611fb45 +revision: 69b338b023 --- # mz-sql::plan::statement::ddl diff --git a/doc/developer/generated/sql/plan/statement/ddl/_module.md b/doc/developer/generated/sql/plan/statement/ddl/_module.md index 58e8dc6339d30..3742fe5ccbf25 100644 --- a/doc/developer/generated/sql/plan/statement/ddl/_module.md +++ b/doc/developer/generated/sql/plan/statement/ddl/_module.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl.rs -revision: 16d611fb45 +revision: 69b338b023 --- # mz-sql::plan::statement::ddl From 3b1ebf86843bf8b31bb0aa6b22a90f4f75b0b6cf Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Tue, 12 May 2026 07:02:02 +0000 Subject: [PATCH 4/4] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- doc/developer/generated/adapter/catalog/migrate.md | 2 +- .../generated/adapter/coord/command_handler.md | 2 +- doc/developer/generated/adapter/explain/insights.md | 2 +- .../generated/sql-parser/ast/defs/statement.md | 10 ++++++++-- doc/developer/generated/sql-parser/parser.md | 3 ++- doc/developer/generated/sql/ast/transform.md | 2 +- doc/developer/generated/sql/plan.md | 2 +- doc/developer/generated/sql/plan/_module.md | 2 +- doc/developer/generated/sql/plan/statement.md | 2 +- doc/developer/generated/sql/plan/statement/_module.md | 2 +- doc/developer/generated/sql/plan/statement/ddl.md | 2 +- .../generated/sql/plan/statement/ddl/_module.md | 2 +- .../generated/timely-util/columnar/consolidate.md | 4 ++-- 13 files changed, 22 insertions(+), 15 deletions(-) diff --git a/doc/developer/generated/adapter/catalog/migrate.md b/doc/developer/generated/adapter/catalog/migrate.md index c29b730e17699..bb2aa7d761b8f 100644 --- a/doc/developer/generated/adapter/catalog/migrate.md +++ b/doc/developer/generated/adapter/catalog/migrate.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/migrate.rs -revision: af9155582e +revision: a9a6e4db0f --- # adapter::catalog::migrate diff --git a/doc/developer/generated/adapter/coord/command_handler.md b/doc/developer/generated/adapter/coord/command_handler.md index 875f67ae4f5c4..e443f8dfeaf92 100644 --- a/doc/developer/generated/adapter/coord/command_handler.md +++ b/doc/developer/generated/adapter/coord/command_handler.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/command_handler.rs -revision: fda8736965 +revision: dcf61051e7 --- # adapter::coord::command_handler diff --git a/doc/developer/generated/adapter/explain/insights.md b/doc/developer/generated/adapter/explain/insights.md index dc8e49a6ee1b3..1cba10904804c 100644 --- a/doc/developer/generated/adapter/explain/insights.md +++ b/doc/developer/generated/adapter/explain/insights.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/explain/insights.rs -revision: bce428d203 +revision: f392e046dc --- # adapter::explain::insights diff --git a/doc/developer/generated/sql-parser/ast/defs/statement.md b/doc/developer/generated/sql-parser/ast/defs/statement.md index 9de1e9955b799..f08213500e3cb 100644 --- a/doc/developer/generated/sql-parser/ast/defs/statement.md +++ b/doc/developer/generated/sql-parser/ast/defs/statement.md @@ -1,13 +1,19 @@ --- source: src/sql-parser/src/ast/defs/statement.rs -revision: 9d0a7c3c6f +revision: dcf61051e7 --- # mz-sql-parser::ast::defs::statement -Defines `Statement`, the top-level enum of all SQL statement types supported by Materialize, including DML (SELECT, INSERT, UPDATE, DELETE, COPY), DDL (CREATE/ALTER/DROP for connections, databases, schemas, sources, sinks, tables, views, materialized views, indexes, secrets, clusters, roles, types, functions), and control statements (SUBSCRIBE, EXPLAIN, SHOW, SET, RESET, BEGIN, COMMIT, ROLLBACK, PREPARE, EXECUTE, DECLARE, FETCH, CLOSE, INSPECT, RAISE). +Defines `Statement`, the top-level enum of all SQL statement types supported by Materialize, including DML (SELECT, INSERT, UPDATE, DELETE, COPY), DDL (CREATE/ALTER/DROP for connections, databases, schemas, sources, sinks, tables, views, materialized views, indexes, secrets, clusters, roles, types, functions), and control statements (SUBSCRIBE, EXPLAIN, SHOW, SET, RESET, BEGIN, COMMIT, ROLLBACK, PREPARE, EXECUTE, EXECUTE UNIT TEST, DECLARE, FETCH, CLOSE, INSPECT, RAISE). Each statement variant has a corresponding struct with its specific fields. +`ExecuteUnitTestStatement` represents an `EXECUTE UNIT TEST FOR [AT TIME ] [MOCK , ...] EXPECTED ` statement. Fields: `name: Ident`, `target: T::ItemName`, `at_time: Option>`, `mocks: Vec>`, `expected: ExpectedResultDef`. + +`MockViewDef` represents a single `MOCK () AS ()` clause within an `EXECUTE UNIT TEST` statement. Fields: `name: T::ItemName`, `columns: Vec>`, `query: Query`. + +`ExpectedResultDef` represents the `EXPECTED () AS ()` clause of an `EXECUTE UNIT TEST` statement. Fields: `columns: Vec>`, `query: Query`. + `ConnectionRulePattern` represents a parsed broker-matching pattern with optional leading and trailing `*` wildcards. Fields: `prefix_wildcard: bool`, `literal_match: String`, `suffix_wildcard: bool`. Its `AstDisplay` impl prints the pattern as a single-quoted string with `*` where wildcards are enabled. Derives `Serialize`/`Deserialize`. `KafkaMatchingBrokerRule` represents a `MATCHING` rule inside a `BROKERS (...)` clause that routes brokers matching a pattern through an AWS PrivateLink tunnel. Fields: `pattern: ConnectionRulePattern` and `tunnel: KafkaBrokerAwsPrivatelink`. Its `AstDisplay` impl prints `MATCHING {pattern} {tunnel}`. diff --git a/doc/developer/generated/sql-parser/parser.md b/doc/developer/generated/sql-parser/parser.md index 7e54ac58f23c3..09e5bc492f415 100644 --- a/doc/developer/generated/sql-parser/parser.md +++ b/doc/developer/generated/sql-parser/parser.md @@ -1,6 +1,6 @@ --- source: src/sql-parser/src/parser.rs -revision: 9d0a7c3c6f +revision: dcf61051e7 --- # mz-sql-parser::parser @@ -10,4 +10,5 @@ Implements Materialize's recursive-descent SQL parser. Also exposes `parse_expr` for parsing a single expression, `parse_datatype` for type expressions, `parse_item_name` for parsing a qualified item name (e.g. `"db"."schema"."table"`), and `ParserError` with position information for error reporting. The parser enforces a recursion limit to guard against stack overflow on deeply nested queries. Iceberg sink mode parsing accepts `UPSERT` or `APPEND` as valid values. +`EXECUTE UNIT TEST FOR [AT TIME ] [MOCK , ...] EXPECTED ` is parsed by `parse_execute_unit_test`; individual mock clauses are parsed by `parse_mock_view_def`. Both methods are called from `parse_execute` after the leading `EXECUTE UNIT TEST` tokens are consumed. The private method `parse_list_value` optionally consumes `=`, then parses a comma-separated list enclosed in parentheses or brackets using a provided closure, returning `Vec`. diff --git a/doc/developer/generated/sql/ast/transform.md b/doc/developer/generated/sql/ast/transform.md index f591d4008bb26..25e465592e39b 100644 --- a/doc/developer/generated/sql/ast/transform.md +++ b/doc/developer/generated/sql/ast/transform.md @@ -1,6 +1,6 @@ --- source: src/sql/src/ast/transform.rs -revision: 8f136c6f83 +revision: 3976078ade --- # mz-sql::ast::transform diff --git a/doc/developer/generated/sql/plan.md b/doc/developer/generated/sql/plan.md index c8860ee00941f..ab85d73649882 100644 --- a/doc/developer/generated/sql/plan.md +++ b/doc/developer/generated/sql/plan.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan.rs -revision: 9d0a7c3c6f +revision: dcf61051e7 --- # mz-sql::plan diff --git a/doc/developer/generated/sql/plan/_module.md b/doc/developer/generated/sql/plan/_module.md index 09b1f05ac5bde..5ff1dc093f989 100644 --- a/doc/developer/generated/sql/plan/_module.md +++ b/doc/developer/generated/sql/plan/_module.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan.rs -revision: 9d0a7c3c6f +revision: dcf61051e7 --- # mz-sql::plan diff --git a/doc/developer/generated/sql/plan/statement.md b/doc/developer/generated/sql/plan/statement.md index 4527a3b2cd1a4..13e1b4436bdcc 100644 --- a/doc/developer/generated/sql/plan/statement.md +++ b/doc/developer/generated/sql/plan/statement.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement.rs -revision: 9d0a7c3c6f +revision: dcf61051e7 --- # mz-sql::plan::statement diff --git a/doc/developer/generated/sql/plan/statement/_module.md b/doc/developer/generated/sql/plan/statement/_module.md index 2c6ede5b04dc9..eb5b7e8c264f7 100644 --- a/doc/developer/generated/sql/plan/statement/_module.md +++ b/doc/developer/generated/sql/plan/statement/_module.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement.rs -revision: 9d0a7c3c6f +revision: dcf61051e7 --- # mz-sql::plan::statement diff --git a/doc/developer/generated/sql/plan/statement/ddl.md b/doc/developer/generated/sql/plan/statement/ddl.md index e37019fee18ed..8be02cc30b6cc 100644 --- a/doc/developer/generated/sql/plan/statement/ddl.md +++ b/doc/developer/generated/sql/plan/statement/ddl.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl.rs -revision: 69b338b023 +revision: 7477c3d0a6 --- # mz-sql::plan::statement::ddl diff --git a/doc/developer/generated/sql/plan/statement/ddl/_module.md b/doc/developer/generated/sql/plan/statement/ddl/_module.md index 3742fe5ccbf25..b8bac47c83f76 100644 --- a/doc/developer/generated/sql/plan/statement/ddl/_module.md +++ b/doc/developer/generated/sql/plan/statement/ddl/_module.md @@ -1,6 +1,6 @@ --- source: src/sql/src/plan/statement/ddl.rs -revision: 69b338b023 +revision: 7477c3d0a6 --- # mz-sql::plan::statement::ddl diff --git a/doc/developer/generated/timely-util/columnar/consolidate.md b/doc/developer/generated/timely-util/columnar/consolidate.md index fe99282e71535..820532e11e882 100644 --- a/doc/developer/generated/timely-util/columnar/consolidate.md +++ b/doc/developer/generated/timely-util/columnar/consolidate.md @@ -1,6 +1,6 @@ --- source: src/timely-util/src/columnar/consolidate.rs -revision: b5e66703dc +revision: a766b48485 --- # timely-util::columnar::consolidate @@ -9,7 +9,7 @@ Provides `ConsolidatingColumnBuilder`, a `ContainerBuilder` that consol The builder uses two-level buffering: -1. **AoS staging buffer** (`Vec<(D, T, R)>` with capacity `STAGING_CAP_ITEMS` = 16 384): incoming items are pushed here. When the buffer fills, `consolidate_and_drain` sorts and consolidates in-place using `consolidate_updates`, then drains a multiple-of-`DRAIN_GRAIN` (= 8 192) prefix so that the last in-progress consolidated key remains in staging for potential merging with the next batch. +1. **AoS staging buffer** (`Vec<(D, T, R)>` with capacity computed by `default_staging_cap()` as `(2 * 8 KiB / size_of::<(D, T, R)>()).max(2)`, stored in `staging_cap`): incoming items are pushed here. When the buffer fills, `consolidate_and_drain` sorts and consolidates in-place using `consolidate_updates`, then drains a `staging_cap / 2` prefix so that the last in-progress consolidated key remains in staging for potential merging with the next batch. 2. **SoA accumulator** (three separate columnar sub-containers, one per column): consolidated rows are drained from staging in chunks of `DRAIN_CHUNK_ROWS` = 16 rows, with three sequential per-column passes per chunk to enable autovectorization. After each chunk the serialized size is checked via `indexed::length_in_words`; once it reaches `FLUSH_THRESHOLD_WORDS` (90% of `OUTPUT_TARGET_WORDS` = 2 MiB), `flush_aligned` serializes the accumulator into an aligned `Vec` via `indexed::encode` and enqueues it as `Column::Align`.