Skip to content

feat: Add matillion data cloud support #26773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ulixius9 wants to merge 8 commits into
base: main
Choose a base branch
from
+532 −12
Open

feat: Add matillion data cloud support #26773

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
856a7f4
feat: Add matillion data cloud support
ulixius9 Mar 25, 2026
90f8aef
Update generated TypeScript types
github-actions[bot] Mar 25, 2026
b0ae96a
Merge branch 'main' into matillion_dpc
ulixius9 Mar 26, 2026
b7db483
fix tests
ulixius9 Mar 26, 2026
d070ca4
Merge branch 'matillion_dpc' of https://github.com/open-metadata/Open…
ulixius9 Mar 26, 2026
bd5c69f
fix tests
ulixius9 Mar 26, 2026
f432726
Merge branch 'main' into matillion_dpc
ulixius9 Mar 26, 2026
08e5590
Merge branch 'main' into matillion_dpc
ulixius9 Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ...in/java/org/openmetadata/service/secrets/converter/MatillionConnectionClassConverter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

import java.util.List;
import org.openmetadata.schema.services.connections.pipeline.MatillionConnection;
import org.openmetadata.schema.services.connections.pipeline.matillion.MatillionDPCAuth;
import org.openmetadata.schema.services.connections.pipeline.matillion.MatillionETLAuth;
import org.openmetadata.schema.utils.JsonUtils;

Expand All @@ -30,7 +31,9 @@ public Object convert(Object object) {
MatillionConnection matillionConnection =
(MatillionConnection) JsonUtils.convertValue(object, this.clazz);

tryToConvertOrFail(matillionConnection.getConnection(), List.of(MatillionETLAuth.class))
tryToConvertOrFail(
matillionConnection.getConnection(),
List.of(MatillionETLAuth.class, MatillionDPCAuth.class))
Comment on lines +35 to +36
Copy link

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The indentation in this multi-line tryToConvertOrFail(...) call doesn’t match the project’s standard Java formatting (and is likely to be rewritten by Spotless / google-java-format). Please reformat this block (e.g., by running mvn spotless:apply) so the continuation indentation is consistent and CI formatting checks don’t fail.

Suggested change
matillionConnection.getConnection(),
List.of(MatillionETLAuth.class, MatillionDPCAuth.class))
matillionConnection.getConnection(),
List.of(MatillionETLAuth.class, MatillionDPCAuth.class))

Copilot uses AI. Check for mistakes.
.ifPresent(matillionConnection::setConnection);

return matillionConnection;
Expand Down
46 changes: 46 additions & 0 deletions ...in/resources/json/schema/entity/services/connections/pipeline/matillion/matillionDPC.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
{
"$id": "https://open-metadata.org/schema/entity/services/connections/pipeline/matillion/matillionDPC.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Matillion DPC Auth Config",
"description": "Matillion Data Productivity Cloud Auth Config.",
"javaType": "org.openmetadata.schema.services.connections.pipeline.matillion.MatillionDPCAuth",
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": [
"MatillionDPC"
],
"default": "MatillionDPC"
},
"clientId": {
"title": "Client ID",
"description": "OAuth2 Client ID for Matillion DPC authentication.",
"type": "string"
},
"clientSecret": {
"title": "Client Secret",
"description": "OAuth2 Client Secret for Matillion DPC authentication.",
"type": "string",
"format": "password"
},
"region": {
"title": "Region",
"description": "Matillion DPC region. Determines the API base URL.",
"type": "string",
"enum": [
"us1",
"eu1"
],
"default": "us1"
},
"personalAccessToken": {
"title": "Personal Access Token",
"description": "Personal Access Token for Matillion DPC. Alternative to OAuth2 Client Credentials.",
"type": "string",
"format": "password"
}
},
"required": [],
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MatillionDPC auth config currently has an empty required list, which means an empty object (or one with only region) can pass schema validation. This will allow invalid connections to be saved and then fail later at runtime. Consider enforcing that either personalAccessToken is provided, or both clientId and clientSecret are provided (and ideally disallow providing both auth methods at once).

Suggested change
"required": [],
"required": [],
"anyOf": [
{
"required": [
"personalAccessToken"
]
},
{
"required": [
"clientId",
"clientSecret"
]
}
],

Copilot uses AI. Check for mistakes.
"additionalProperties": false
}
13 changes: 12 additions & 1 deletion .../main/resources/json/schema/entity/services/connections/pipeline/matillionConnection.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"oneOf": [
{
"$ref": "matillion/matillionETL.json"
},
{
"$ref": "matillion/matillionDPC.json"
}
]
},
Expand All @@ -36,10 +39,18 @@
"$ref": "../../../../type/filterPattern.json#/definitions/filterPattern",
"title": "Default Pipeline Filter Pattern"
},
"lineageLookbackDays": {
"title": "Lineage Lookback Days",
"description": "Number of days to look back when fetching lineage events from Matillion DPC OpenLineage API.",
"type": "integer",
"default": 30,
"minimum": 1,
"maximum": 365
},
Comment on lines 28 to +49
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New schema fields (MatillionDPC auth option and lineageLookbackDays) aren’t covered by the existing configuration parsing tests for Matillion. Adding a unit test that parses a valid DPC config (PAT and/or client credentials) and asserts validation failures for missing credentials / out-of-range lineageLookbackDays would prevent regressions in schema-to-model generation and validation behavior.

Copilot uses AI. Check for mistakes.
Comment on lines +42 to +49
Copy link

@gitar-bot gitar-bot bot Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Quality: lineageLookbackDays is DPC-specific but on shared connection

The lineageLookbackDays property is placed at the top-level matillionConnection.json schema, meaning it will be visible for both ETL and DPC connection types. However, its description explicitly says it's for the "Matillion DPC OpenLineage API." If ETL connections don't use lineage lookback, this could confuse users configuring an ETL connection. Consider either:

  1. Moving it inside the matillionDPC.json schema, or
  2. Updating the description to be generic if it applies to both types.

Suggested fix:

Either move lineageLookbackDays into matillionDPC.json, or update the description to not mention DPC specifically if it applies to both connection types.

Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion

"supportsMetadataExtraction": {
"title": "Supports Metadata Extraction",
"$ref": "../connectionBasicType.json#/definitions/supportsMetadataExtraction"
}
},
"additionalProperties": false
}
}
32 changes: 32 additions & 0 deletions openmetadata-ui/src/main/resources/ui/src/generated/api/automations/createWorkflow.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1954,6 +1954,11 @@ export interface ConfigObject {
* List of IDs of your DBT cloud projects seperated by comma `,`
*/
projectIds?: string[];
/**
* Number of days to look back when fetching lineage events from Matillion DPC OpenLineage
* API.
*/
lineageLookbackDays?: number;
/**
* The name of your azure data factory.
*/
Expand Down Expand Up @@ -3624,6 +3629,8 @@ export interface GCPImpersonateServiceAccountValues {
*
* Matillion ETL Auth Config.
*
* Matillion Data Productivity Cloud Auth Config.
*
* Choose between mysql and postgres connection for alation database
*/
export interface ConfigConnection {
Expand Down Expand Up @@ -3774,6 +3781,22 @@ export interface ConfigConnection {
*/
databaseMode?: string;
supportsViewLineageExtraction?: boolean;
/**
* OAuth2 Client ID for Matillion DPC authentication.
*/
clientId?: string;
/**
* OAuth2 Client Secret for Matillion DPC authentication.
*/
clientSecret?: string;
/**
* Personal Access Token for Matillion DPC. Alternative to OAuth2 Client Credentials.
*/
personalAccessToken?: string;
/**
* Matillion DPC region. Determines the API base URL.
*/
region?: Region;
}

/**
Expand Down Expand Up @@ -3804,6 +3827,14 @@ export enum Provider {
LDAP = "ldap",
}

/**
* Matillion DPC region. Determines the API base URL.
*/
export enum Region {
Eu1 = "eu1",
Us1 = "us1",
}

/**
* Storage config to store sample data
*/
Expand Down Expand Up @@ -3964,6 +3995,7 @@ export enum SSLMode {
*/
export enum ConnectionType {
Backend = "Backend",
MatillionDPC = "MatillionDPC",
MatillionETL = "MatillionETL",
Mysql = "Mysql",
Postgres = "Postgres",
Expand Down
36 changes: 34 additions & 2 deletions openmetadata-ui/src/main/resources/ui/src/generated/api/services/createPipelineService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ export interface ConfigObject {
*
* Matillion Auth Configuration
*/
connection?: MetadataDatabaseConnection;
connection?: ConnectionClass;
/**
* Pipeline Service Management/UI URI.
*
Expand Down Expand Up @@ -313,6 +313,11 @@ export interface ConfigObject {
* List of IDs of your DBT cloud projects seperated by comma `,`
*/
projectIds?: string[];
/**
* Number of days to look back when fetching lineage events from Matillion DPC OpenLineage
* API.
*/
lineageLookbackDays?: number;
/**
* Available sources to fetch metadata.
*/
Expand Down Expand Up @@ -704,8 +709,10 @@ export interface AzureCredentials {
* Matillion Auth Configuration
*
* Matillion ETL Auth Config.
*
* Matillion Data Productivity Cloud Auth Config.
*/
export interface MetadataDatabaseConnection {
export interface ConnectionClass {
/**
* Regex exclude pipelines.
*/
Expand Down Expand Up @@ -825,6 +832,22 @@ export interface MetadataDatabaseConnection {
*/
password?: string;
supportsViewLineageExtraction?: boolean;
/**
* OAuth2 Client ID for Matillion DPC authentication.
*/
clientId?: string;
/**
* OAuth2 Client Secret for Matillion DPC authentication.
*/
clientSecret?: string;
/**
* Personal Access Token for Matillion DPC. Alternative to OAuth2 Client Credentials.
*/
personalAccessToken?: string;
/**
* Matillion DPC region. Determines the API base URL.
*/
region?: Region;
}

/**
Expand Down Expand Up @@ -875,6 +898,14 @@ export interface FilterPattern {
includes?: string[];
}

/**
* Matillion DPC region. Determines the API base URL.
*/
export enum Region {
Eu1 = "eu1",
Us1 = "us1",
}

/**
* Storage config to store sample data
*/
Expand Down Expand Up @@ -990,6 +1021,7 @@ export enum SSLMode {
*/
export enum Type {
Backend = "Backend",
MatillionDPC = "MatillionDPC",
MatillionETL = "MatillionETL",
Mysql = "Mysql",
Postgres = "Postgres",
Expand Down
32 changes: 32 additions & 0 deletions ...ain/resources/ui/src/generated/api/services/ingestionPipelines/createIngestionPipeline.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4711,6 +4711,11 @@ export interface ConfigObject {
* List of IDs of your DBT cloud projects seperated by comma `,`
*/
projectIds?: string[];
/**
* Number of days to look back when fetching lineage events from Matillion DPC OpenLineage
* API.
*/
lineageLookbackDays?: number;
/**
* The name of your azure data factory.
*/
Expand Down Expand Up @@ -5914,6 +5919,8 @@ export interface ConfigSourceConnection {
* Matillion Auth Configuration
*
* Matillion ETL Auth Config.
*
* Matillion Data Productivity Cloud Auth Config.
*/
export interface ConfigConnection {
/**
Expand Down Expand Up @@ -6063,6 +6070,22 @@ export interface ConfigConnection {
*/
databaseMode?: string;
supportsViewLineageExtraction?: boolean;
/**
* OAuth2 Client ID for Matillion DPC authentication.
*/
clientId?: string;
/**
* OAuth2 Client Secret for Matillion DPC authentication.
*/
clientSecret?: string;
/**
* Personal Access Token for Matillion DPC. Alternative to OAuth2 Client Credentials.
*/
personalAccessToken?: string;
/**
* Matillion DPC region. Determines the API base URL.
*/
region?: Region;
}

/**
Expand Down Expand Up @@ -6093,6 +6116,14 @@ export enum Provider {
LDAP = "ldap",
}

/**
* Matillion DPC region. Determines the API base URL.
*/
export enum Region {
Eu1 = "eu1",
Us1 = "us1",
}

/**
* Storage config to store sample data
*/
Expand Down Expand Up @@ -6253,6 +6284,7 @@ export enum SSLMode {
*/
export enum ConnectionType {
Backend = "Backend",
MatillionDPC = "MatillionDPC",
MatillionETL = "MatillionETL",
Mysql = "Mysql",
Postgres = "Postgres",
Expand Down
32 changes: 32 additions & 0 deletions ...tadata-ui/src/main/resources/ui/src/generated/entity/automations/testServiceConnection.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1836,6 +1836,11 @@ export interface ConfigObject {
* List of IDs of your DBT cloud projects seperated by comma `,`
*/
projectIds?: string[];
/**
* Number of days to look back when fetching lineage events from Matillion DPC OpenLineage
* API.
*/
lineageLookbackDays?: number;
/**
* The name of your azure data factory.
*/
Expand Down Expand Up @@ -3506,6 +3511,8 @@ export interface GCPImpersonateServiceAccountValues {
*
* Matillion ETL Auth Config.
*
* Matillion Data Productivity Cloud Auth Config.
*
* Choose between mysql and postgres connection for alation database
*/
export interface ConfigConnection {
Expand Down Expand Up @@ -3656,6 +3663,22 @@ export interface ConfigConnection {
*/
databaseMode?: string;
supportsViewLineageExtraction?: boolean;
/**
* OAuth2 Client ID for Matillion DPC authentication.
*/
clientId?: string;
/**
* OAuth2 Client Secret for Matillion DPC authentication.
*/
clientSecret?: string;
/**
* Personal Access Token for Matillion DPC. Alternative to OAuth2 Client Credentials.
*/
personalAccessToken?: string;
/**
* Matillion DPC region. Determines the API base URL.
*/
region?: Region;
}

/**
Expand Down Expand Up @@ -3686,6 +3709,14 @@ export enum Provider {
LDAP = "ldap",
}

/**
* Matillion DPC region. Determines the API base URL.
*/
export enum Region {
Eu1 = "eu1",
Us1 = "us1",
}

/**
* Storage config to store sample data
*/
Expand Down Expand Up @@ -3846,6 +3877,7 @@ export enum SSLMode {
*/
export enum ConnectionType {
Backend = "Backend",
MatillionDPC = "MatillionDPC",
MatillionETL = "MatillionETL",
Mysql = "Mysql",
Postgres = "Postgres",
Expand Down
Loading
Loading