Metadata

zenml.metadata

Initialization of ZenML metadata.

ZenML metadata is any additional, dynamic information that is associated with your pipeline runs and artifacts at runtime.

Modules

lazy_load

Run Metadata Lazy Loader definition.

Classes

LazyRunMetadataResponse

Bases: BaseModel

Lazy run metadata response.

Used if the run metadata is accessed from the model in a pipeline context available only during pipeline compilation.

RunMetadataLazyGetter(_lazy_load_model_name: str, _lazy_load_model_version: Optional[str], _lazy_load_artifact_name: Optional[str] = None, _lazy_load_artifact_version: Optional[str] = None)

Run Metadata Lazy Getter helper class.

It serves the purpose to feed back to the user the metadata lazy loader wrapper for any given key, if called inside a pipeline design time context.

Initialize a RunMetadataLazyGetter.

Parameters:

Name	Type	Description	Default
_lazy_load_model_name	str	The model name.	required
_lazy_load_model_version	Optional[str]	The model version.	required
_lazy_load_artifact_name	Optional[str]	The artifact name.	None
_lazy_load_artifact_version	Optional[str]	The artifact version.	None

Source code in src/zenml/metadata/lazy_load.py

def __init__(
    self,
    _lazy_load_model_name: str,
    _lazy_load_model_version: Optional[str],
    _lazy_load_artifact_name: Optional[str] = None,
    _lazy_load_artifact_version: Optional[str] = None,
):
    """Initialize a RunMetadataLazyGetter.

    Args:
        _lazy_load_model_name: The model name.
        _lazy_load_model_version: The model version.
        _lazy_load_artifact_name: The artifact name.
        _lazy_load_artifact_version: The artifact version.
    """
    self._lazy_load_model_name = _lazy_load_model_name
    self._lazy_load_model_version = _lazy_load_model_version
    self._lazy_load_artifact_name = _lazy_load_artifact_name
    self._lazy_load_artifact_version = _lazy_load_artifact_version

Functions

metadata_types

Custom types that can be used as metadata of ZenML artifacts.

Classes

DType

Bases: str

Special string class to indicate a data type.

MetadataTypeEnum

Bases: StrEnum

String Enum of all possible types that metadata can have.

Path

Bases: str

Special string class to indicate a path.

StorageSize

Bases: int

Special int class to indicate the storage size in number of bytes.

Uri

Bases: str

Special string class to indicate a URI.

Functions

cast_to_metadata_type(value: object, type_: MetadataTypeEnum) -> MetadataType

Cast an object to a metadata type.

Parameters:

Name	Type	Description	Default
value	object	The object to cast.	required
type_	MetadataTypeEnum	The metadata type to cast to.	required

Returns:

Type	Description
MetadataType	The value cast to the given metadata type.

Source code in src/zenml/metadata/metadata_types.py

def cast_to_metadata_type(
    value: object,
    type_: MetadataTypeEnum,
) -> MetadataType:
    """Cast an object to a metadata type.

    Args:
        value: The object to cast.
        type_: The metadata type to cast to.

    Returns:
        The value cast to the given metadata type.
    """
    metadata_type = metadata_enum_to_type_mapping[type_]
    typed_value = metadata_type(value)
    return typed_value  # type: ignore[no-any-return]

get_metadata_type(object_: object) -> MetadataTypeEnum

Get the metadata type enum for a given object.

Parameters:

Name	Type	Description	Default
object_	object	The object to get the metadata type for.	required

Returns:

Type	Description
MetadataTypeEnum	The corresponding metadata type enum.

Raises:

Type	Description
ValueError	If the metadata type is not supported.

Source code in src/zenml/metadata/metadata_types.py

def get_metadata_type(
    object_: object,
) -> MetadataTypeEnum:
    """Get the metadata type enum for a given object.

    Args:
        object_: The object to get the metadata type for.

    Returns:
        The corresponding metadata type enum.

    Raises:
        ValueError: If the metadata type is not supported.
    """
    metadata_type = type(object_)
    if metadata_type in metadata_type_to_enum_mapping:
        return metadata_type_to_enum_mapping[metadata_type]
    raise ValueError(f"Metadata type {metadata_type} is not supported.")

validate_metadata(metadata: Dict[str, MetadataType]) -> Dict[str, MetadataType]

Validate metadata.

This function excludes and warns about metadata values that are too long or of an unsupported type.

Parameters:

Name	Type	Description	Default
metadata	Dict[str, MetadataType]	The metadata to validate.	required

Returns:

Type	Description
Dict[str, MetadataType]	The validated metadata.

Source code in src/zenml/metadata/metadata_types.py

def validate_metadata(
    metadata: Dict[str, MetadataType],
) -> Dict[str, MetadataType]:
    """Validate metadata.

    This function excludes and warns about metadata values that are too long
    or of an unsupported type.

    Args:
        metadata: The metadata to validate.

    Returns:
        The validated metadata.
    """
    validated_metadata = {}

    for key, value in metadata.items():
        if len(key) > STR_FIELD_MAX_LENGTH:
            logger.warning(
                f"Metadata key '{key}' is too large to be "
                "stored in the database. Skipping."
            )
            continue

        if len(json.dumps(value)) > TEXT_FIELD_MAX_LENGTH:
            logger.warning(
                f"Metadata value for key '{key}' is too large to be "
                "stored in the database. Skipping."
            )
            continue

        try:
            get_metadata_type(value)
        except ValueError as e:
            logger.warning(
                f"Metadata value for key '{key}' is not of a supported "
                f"type. Skipping. Full error: {e}"
            )

        validated_metadata[key] = value

    return validated_metadata