Skip to content

Execution.create() fails with 503 due to credentials annotation typo #6610

@javabrett

Description

@javabrett

Bug

google.cloud.aiplatform.metadata.execution.Execution.create() fails with a 503 error when called without explicitly passing credentials=:

503 Getting metadata from plugin failed with error: before_request

Root cause

Two issues in google/cloud/aiplatform/metadata/execution.py:

1. credentials parameter annotation typo (high severity)

Both create() and _create() use = instead of : for the credentials type annotation:

# Current (broken) -- execution.py lines 103, 181:
credentials=Optional[auth_credentials.Credentials],

# Should be (matches artifact.py, context.py):
credentials: Optional[auth_credentials.Credentials] = None,

Because = is used instead of :, the default value is the typing.Optional[google.auth.credentials.Credentials] type object itself (a _UnionGenericAlias), not None. This non-None type object is passed down to the gRPC auth stack, which attempts to call .before_request() on it, producing the 503.

Introduced in PR #1410 (June 2022). artifact.py and context.py both have the correct : ... = None syntax.

2. Missing ensure_default_metadata_store_exists() call (lower severity)

Artifact.create() calls metadata_store._MetadataStore.ensure_default_metadata_store_exists() before delegating to _create(). Execution.create() skips this call entirely. This means standalone Execution.create() will also fail if the default metadata store hasn't been implicitly created by a prior Artifact.create() or aiplatform.init(experiment=...) call. Context.create() has the same gap but is not addressed here.

Reproduction

from google.cloud import aiplatform
from google.cloud.aiplatform.metadata import execution as metadata_execution
from google.cloud.aiplatform.compat.types import execution as gca_execution

aiplatform.init(project="my-project", location="us-central1")

# This fails with 503:
exec_obj = metadata_execution.Execution.create(
    schema_title="system.CustomJob",
    resource_id="test-execution",
    display_name="test",
    state=gca_execution.Execution.State.COMPLETE,
    metadata={"component_type": "custom_job"},
)

Workaround

Call ensure_default_metadata_store_exists() manually, which also initializes credentials properly:

from google.cloud.aiplatform.metadata import metadata_store
metadata_store._MetadataStore.ensure_default_metadata_store_exists(
    project="my-project", location="us-central1",
)
# Now Execution.create() works

Fix

Two changes to execution.py:

  1. Fix credentials=Optional[auth_credentials.Credentials] to credentials: Optional[auth_credentials.Credentials] = None on both create() and _create()
  2. Add the ensure_default_metadata_store_exists() call in create() (matching Artifact.create())

Environment

  • google-cloud-aiplatform 1.140.0
  • Python 3.13.2
  • macOS / Vertex AI training containers (both affected)

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: vertex-aiIssues related to the googleapis/python-aiplatform API.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions