Bug
google.cloud.aiplatform.metadata.execution.Execution.create() fails with a 503 error when called without explicitly passing credentials=:
503 Getting metadata from plugin failed with error: before_request
Root cause
Two issues in google/cloud/aiplatform/metadata/execution.py:
1. credentials parameter annotation typo (high severity)
Both create() and _create() use = instead of : for the credentials type annotation:
# Current (broken) -- execution.py lines 103, 181:
credentials=Optional[auth_credentials.Credentials],
# Should be (matches artifact.py, context.py):
credentials: Optional[auth_credentials.Credentials] = None,
Because = is used instead of :, the default value is the typing.Optional[google.auth.credentials.Credentials] type object itself (a _UnionGenericAlias), not None. This non-None type object is passed down to the gRPC auth stack, which attempts to call .before_request() on it, producing the 503.
Introduced in PR #1410 (June 2022). artifact.py and context.py both have the correct : ... = None syntax.
2. Missing ensure_default_metadata_store_exists() call (lower severity)
Artifact.create() calls metadata_store._MetadataStore.ensure_default_metadata_store_exists() before delegating to _create(). Execution.create() skips this call entirely. This means standalone Execution.create() will also fail if the default metadata store hasn't been implicitly created by a prior Artifact.create() or aiplatform.init(experiment=...) call. Context.create() has the same gap but is not addressed here.
Reproduction
from google.cloud import aiplatform
from google.cloud.aiplatform.metadata import execution as metadata_execution
from google.cloud.aiplatform.compat.types import execution as gca_execution
aiplatform.init(project="my-project", location="us-central1")
# This fails with 503:
exec_obj = metadata_execution.Execution.create(
schema_title="system.CustomJob",
resource_id="test-execution",
display_name="test",
state=gca_execution.Execution.State.COMPLETE,
metadata={"component_type": "custom_job"},
)
Workaround
Call ensure_default_metadata_store_exists() manually, which also initializes credentials properly:
from google.cloud.aiplatform.metadata import metadata_store
metadata_store._MetadataStore.ensure_default_metadata_store_exists(
project="my-project", location="us-central1",
)
# Now Execution.create() works
Fix
Two changes to execution.py:
- Fix
credentials=Optional[auth_credentials.Credentials] to credentials: Optional[auth_credentials.Credentials] = None on both create() and _create()
- Add the
ensure_default_metadata_store_exists() call in create() (matching Artifact.create())
Environment
- google-cloud-aiplatform 1.140.0
- Python 3.13.2
- macOS / Vertex AI training containers (both affected)
Bug
google.cloud.aiplatform.metadata.execution.Execution.create()fails with a 503 error when called without explicitly passingcredentials=:Root cause
Two issues in
google/cloud/aiplatform/metadata/execution.py:1.
credentialsparameter annotation typo (high severity)Both
create()and_create()use=instead of:for the credentials type annotation:Because
=is used instead of:, the default value is thetyping.Optional[google.auth.credentials.Credentials]type object itself (a_UnionGenericAlias), notNone. This non-None type object is passed down to the gRPC auth stack, which attempts to call.before_request()on it, producing the 503.Introduced in PR #1410 (June 2022).
artifact.pyandcontext.pyboth have the correct: ... = Nonesyntax.2. Missing
ensure_default_metadata_store_exists()call (lower severity)Artifact.create()callsmetadata_store._MetadataStore.ensure_default_metadata_store_exists()before delegating to_create().Execution.create()skips this call entirely. This means standaloneExecution.create()will also fail if the default metadata store hasn't been implicitly created by a priorArtifact.create()oraiplatform.init(experiment=...)call.Context.create()has the same gap but is not addressed here.Reproduction
Workaround
Call
ensure_default_metadata_store_exists()manually, which also initializes credentials properly:Fix
Two changes to
execution.py:credentials=Optional[auth_credentials.Credentials]tocredentials: Optional[auth_credentials.Credentials] = Noneon bothcreate()and_create()ensure_default_metadata_store_exists()call increate()(matchingArtifact.create())Environment