Skip to content

Artifact Stores

zenml.artifact_stores

ZenML's artifact-store stores artifacts in a file system.

In ZenML, the inputs and outputs which go through any step is treated as an artifact and as its name suggests, an ArtifactStore is a place where these artifacts get stored.

Out of the box, ZenML comes with the BaseArtifactStore and LocalArtifactStore implementations. While the BaseArtifactStore establishes an interface for people who want to extend it to their needs, the LocalArtifactStore is a simple implementation for a local setup.

Moreover, additional artifact stores can be found in specific integrations modules, such as the GCPArtifactStore in the gcp integration and the AzureArtifactStore in the azure integration.

Attributes

__all__ = ['BaseArtifactStore', 'BaseArtifactStoreConfig', 'BaseArtifactStoreFlavor', 'LocalArtifactStore', 'LocalArtifactStoreConfig', 'LocalArtifactStoreFlavor'] module-attribute

Classes

BaseArtifactStore(*args: Any, **kwargs: Any)

Bases: StackComponent

Base class for all ZenML artifact stores.

Initiate the Pydantic object and register the corresponding filesystem.

Parameters:

Name Type Description Default
*args Any

The positional arguments to pass to the Pydantic object.

 ()
**kwargs Any

The keyword arguments to pass to the Pydantic object.

 {}
Source code in src/zenml/artifact_stores/base_artifact_store.py 
457
458
459
460
461
462
463
464
465
466
467
468
469
470
def __init__(self, *args: Any, **kwargs: Any) -> None:
    """Initiate the Pydantic object and register the corresponding filesystem.

    Args:
        *args: The positional arguments to pass to the Pydantic object.
        **kwargs: The keyword arguments to pass to the Pydantic object.
    """
    super(BaseArtifactStore, self).__init__(*args, **kwargs)
    self._add_path_sanitization()

    # If running in a ZenML server environment, we don't register
    # the filesystems. We always use the artifact stores directly.
    if ENV_ZENML_SERVER not in os.environ:
        self._register()
Attributes
config: BaseArtifactStoreConfig property

Returns the BaseArtifactStoreConfig config.

Returns:

Type Description
BaseArtifactStoreConfig

The configuration.
custom_cache_key: Optional[bytes] property

Custom cache key.

Any artifact store can override this property in case they need additional control over the caching behavior.

Returns:

Type Description
Optional[bytes]

Custom cache key.
path: str property

The path to the artifact store.

Returns:

Type Description
str

The path.
Functions
copyfile(src: PathType, dst: PathType, overwrite: bool = False) -> None abstractmethod

Copy a file from the source to the destination.

Parameters:

Name Type Description Default
src PathType

The source path.

 required
dst PathType

The destination path.

 required
overwrite bool

Whether to overwrite the destination file if it exists.

 False
Source code in src/zenml/artifact_stores/base_artifact_store.py 
308
309
310
311
312
313
314
315
316
317
318
@abstractmethod
def copyfile(
    self, src: PathType, dst: PathType, overwrite: bool = False
) -> None:
    """Copy a file from the source to the destination.

    Args:
        src: The source path.
        dst: The destination path.
        overwrite: Whether to overwrite the destination file if it exists.
    """
exists(path: PathType) -> bool abstractmethod

Checks if a path exists.

Parameters:

Name Type Description Default
path PathType

The path to check.

 required

Returns:

Type Description
bool

True if the path exists.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
320
321
322
323
324
325
326
327
328
329
@abstractmethod
def exists(self, path: PathType) -> bool:
    """Checks if a path exists.

    Args:
        path: The path to check.

    Returns:
        `True` if the path exists.
    """
glob(pattern: PathType) -> List[PathType] abstractmethod

Gets the paths that match a glob pattern.

Parameters:

Name Type Description Default
pattern PathType

The glob pattern.

 required

Returns:

Type Description
List[PathType]

The list of paths that match the pattern.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
331
332
333
334
335
336
337
338
339
340
@abstractmethod
def glob(self, pattern: PathType) -> List[PathType]:
    """Gets the paths that match a glob pattern.

    Args:
        pattern: The glob pattern.

    Returns:
        The list of paths that match the pattern.
    """
isdir(path: PathType) -> bool abstractmethod

Returns whether the given path points to a directory.

Parameters:

Name Type Description Default
path PathType

The path to check.

 required

Returns:

Type Description
bool

True if the path points to a directory.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
342
343
344
345
346
347
348
349
350
351
@abstractmethod
def isdir(self, path: PathType) -> bool:
    """Returns whether the given path points to a directory.

    Args:
        path: The path to check.

    Returns:
        `True` if the path points to a directory.
    """
listdir(path: PathType) -> List[PathType] abstractmethod

Returns a list of files under a given directory in the filesystem.

Parameters:

Name Type Description Default
path PathType

The path to list.

 required

Returns:

Type Description
List[PathType]

The list of files under the given path.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
353
354
355
356
357
358
359
360
361
362
@abstractmethod
def listdir(self, path: PathType) -> List[PathType]:
    """Returns a list of files under a given directory in the filesystem.

    Args:
        path: The path to list.

    Returns:
        The list of files under the given path.
    """
makedirs(path: PathType) -> None abstractmethod

Make a directory at the given path, recursively creating parents.

Parameters:

Name Type Description Default
path PathType

The path to create.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
364
365
366
367
368
369
370
@abstractmethod
def makedirs(self, path: PathType) -> None:
    """Make a directory at the given path, recursively creating parents.

    Args:
        path: The path to create.
    """
mkdir(path: PathType) -> None abstractmethod

Make a directory at the given path; parent directory must exist.

Parameters:

Name Type Description Default
path PathType

The path to create.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
372
373
374
375
376
377
378
@abstractmethod
def mkdir(self, path: PathType) -> None:
    """Make a directory at the given path; parent directory must exist.

    Args:
        path: The path to create.
    """
open(path: PathType, mode: str = 'r') -> Any abstractmethod

Open a file at the given path.

Parameters:

Name Type Description Default
path PathType

The path of the file to open.

 required
mode str

The mode to open the file.

 'r'

Returns:

Type Description
Any

The file object.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
296
297
298
299
300
301
302
303
304
305
306
@abstractmethod
def open(self, path: PathType, mode: str = "r") -> Any:
    """Open a file at the given path.

    Args:
        path: The path of the file to open.
        mode: The mode to open the file.

    Returns:
        The file object.
    """
remove(path: PathType) -> None abstractmethod

Remove the file at the given path. Dangerous operation.

Parameters:

Name Type Description Default
path PathType

The path to remove.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
380
381
382
383
384
385
386
@abstractmethod
def remove(self, path: PathType) -> None:
    """Remove the file at the given path. Dangerous operation.

    Args:
        path: The path to remove.
    """
rename(src: PathType, dst: PathType, overwrite: bool = False) -> None abstractmethod

Rename source file to destination file.

Parameters:

Name Type Description Default
src PathType

The source path.

 required
dst PathType

The destination path.

 required
overwrite bool

Whether to overwrite the destination file if it exists.

 False
Source code in src/zenml/artifact_stores/base_artifact_store.py 
388
389
390
391
392
393
394
395
396
397
398
@abstractmethod
def rename(
    self, src: PathType, dst: PathType, overwrite: bool = False
) -> None:
    """Rename source file to destination file.

    Args:
        src: The source path.
        dst: The destination path.
        overwrite: Whether to overwrite the destination file if it exists.
    """
rmtree(path: PathType) -> None abstractmethod

Deletes dir recursively. Dangerous operation.

Parameters:

Name Type Description Default
path PathType

The path to delete.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
400
401
402
403
404
405
406
@abstractmethod
def rmtree(self, path: PathType) -> None:
    """Deletes dir recursively. Dangerous operation.

    Args:
        path: The path to delete.
    """
size(path: PathType) -> Optional[int] abstractmethod

Get the size of a file in bytes.

Parameters:

Name Type Description Default
path PathType

The path to the file.

 required

Returns:

Type Description
Optional[int]

The size of the file in bytes or None if the artifact store
Optional[int]

does not implement the size method.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
@abstractmethod
def size(self, path: PathType) -> Optional[int]:
    """Get the size of a file in bytes.

    Args:
        path: The path to the file.

    Returns:
        The size of the file in bytes or `None` if the artifact store
        does not implement the `size` method.
    """
    logger.warning(
        "Cannot get size of file '%s' since the artifact store %s does not "
        "implement the `size` method.",
        path,
        self.__class__.__name__,
    )
    return None
stat(path: PathType) -> Any abstractmethod

Return the stat descriptor for a given file path.

Parameters:

Name Type Description Default
path PathType

The path to check.

 required

Returns:

Type Description
Any

The stat descriptor.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
408
409
410
411
412
413
414
415
416
417
@abstractmethod
def stat(self, path: PathType) -> Any:
    """Return the stat descriptor for a given file path.

    Args:
        path: The path to check.

    Returns:
        The stat descriptor.
    """
walk(top: PathType, topdown: bool = True, onerror: Optional[Callable[..., None]] = None) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]] abstractmethod

Return an iterator that walks the contents of the given directory.

Parameters:

Name Type Description Default
top PathType

The path to walk.

 required
topdown bool

Whether to walk the top-down or bottom-up.

 True
onerror Optional[Callable[..., None]]

The error handler.

 None

Returns:

Type Description
Iterable[Tuple[PathType, List[PathType], List[PathType]]]

The iterator that walks the contents of the given directory.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
@abstractmethod
def walk(
    self,
    top: PathType,
    topdown: bool = True,
    onerror: Optional[Callable[..., None]] = None,
) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]]:
    """Return an iterator that walks the contents of the given directory.

    Args:
        top: The path to walk.
        topdown: Whether to walk the top-down or bottom-up.
        onerror: The error handler.

    Returns:
        The iterator that walks the contents of the given directory.
    """

BaseArtifactStoreConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)

Bases: StackComponentConfig

Config class for BaseArtifactStore.

Base configuration for artifact storage backends. Field descriptions are defined inline using Field() descriptors.

Source code in src/zenml/stack/stack_component.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def __init__(
    self, warn_about_plain_text_secrets: bool = False, **kwargs: Any
) -> None:
    """Ensures that secret references don't clash with pydantic validation.

    StackComponents allow the specification of all their string attributes
    using secret references of the form `{{secret_name.key}}`. This however
    is only possible when the stack component does not perform any explicit
    validation of this attribute using pydantic validators. If this were
    the case, the validation would run on the secret reference and would
    fail or in the worst case, modify the secret reference and lead to
    unexpected behavior. This method ensures that no attributes that require
    custom pydantic validation are set as secret references.

    Args:
        warn_about_plain_text_secrets: If true, then warns about using
            plain-text secrets.
        **kwargs: Arguments to initialize this stack component.

    Raises:
        ValueError: If an attribute that requires custom pydantic validation
            is passed as a secret reference, or if the `name` attribute
            was passed as a secret reference.
    """
    for key, value in kwargs.items():
        try:
            field = self.__class__.model_fields[key]
        except KeyError:
            # Value for a private attribute or non-existing field, this
            # will fail during the upcoming pydantic validation
            continue

        if value is None:
            continue

        if not secret_utils.is_secret_reference(value):
            if (
                secret_utils.is_secret_field(field)
                and warn_about_plain_text_secrets
            ):
                logger.warning(
                    "You specified a plain-text value for the sensitive "
                    f"attribute `{key}` for a `{self.__class__.__name__}` "
                    "stack component. This is currently only a warning, "
                    "but future versions of ZenML will require you to pass "
                    "in sensitive information as secrets. Check out the "
                    "documentation on how to configure your stack "
                    "components with secrets here: "
                    "https://docs.zenml.io/deploying-zenml/deploying-zenml/secret-management"
                )
            continue

        if pydantic_utils.has_validators(
            pydantic_class=self.__class__, field_name=key
        ):
            raise ValueError(
                f"Passing the stack component attribute `{key}` as a "
                "secret reference is not allowed as additional validation "
                "is required for this attribute."
            )

    super().__init__(**kwargs)

BaseArtifactStoreFlavor

Bases: Flavor

Base class for artifact store flavors.

Attributes
config_class: Type[StackComponentConfig] property

Config class for this flavor.

Returns:

Type Description
Type[StackComponentConfig]

The config class.
implementation_class: Type[BaseArtifactStore] abstractmethod property

Implementation class.

Returns:

Type Description
Type[BaseArtifactStore]

The implementation class.
type: StackComponentType property

Returns the flavor type.

Returns:

Type Description
StackComponentType

The flavor type.

LocalArtifactStore(*args: Any, **kwargs: Any)

Bases: LocalFilesystem, BaseArtifactStore

Artifact Store for local artifacts.

All methods are inherited from the default LocalFilesystem.

Source code in src/zenml/artifact_stores/base_artifact_store.py 
457
458
459
460
461
462
463
464
465
466
467
468
469
470
def __init__(self, *args: Any, **kwargs: Any) -> None:
    """Initiate the Pydantic object and register the corresponding filesystem.

    Args:
        *args: The positional arguments to pass to the Pydantic object.
        **kwargs: The keyword arguments to pass to the Pydantic object.
    """
    super(BaseArtifactStore, self).__init__(*args, **kwargs)
    self._add_path_sanitization()

    # If running in a ZenML server environment, we don't register
    # the filesystems. We always use the artifact stores directly.
    if ENV_ZENML_SERVER not in os.environ:
        self._register()
Attributes
custom_cache_key: Optional[bytes] property

Custom cache key.

The client ID is returned here to invalidate caching when using the same local artifact store on multiple client machines.

Returns:

Type Description
Optional[bytes]

Custom cache key.
local_path: Optional[str] property

Returns the local path of the artifact store.

Returns:

Type Description
Optional[str]

The local path of the artifact store.
path: str property

Returns the path to the local artifact store.

If the user has not defined a path in the config, this will create a sub-folder in the global config directory.

Returns:

Type Description
str

The path to the local artifact store.
Functions
get_default_local_path(id_: UUID) -> str staticmethod

Returns the default local path for a local artifact store.

Parameters:

Name Type Description Default
id_ UUID

The id of the local artifact store.

 required

Returns:

Name Type Description
str str

The default local path.
Source code in src/zenml/artifact_stores/local_artifact_store.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
@staticmethod
def get_default_local_path(id_: "UUID") -> str:
    """Returns the default local path for a local artifact store.

    Args:
        id_: The id of the local artifact store.

    Returns:
        str: The default local path.
    """
    return os.path.join(
        GlobalConfiguration().local_stores_path,
        str(id_),
    )

LocalArtifactStoreConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)

Bases: BaseArtifactStoreConfig

Config class for the local artifact store.

Attributes:

Name Type Description
path str

The path to the local artifact store.
Source code in src/zenml/stack/stack_component.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def __init__(
    self, warn_about_plain_text_secrets: bool = False, **kwargs: Any
) -> None:
    """Ensures that secret references don't clash with pydantic validation.

    StackComponents allow the specification of all their string attributes
    using secret references of the form `{{secret_name.key}}`. This however
    is only possible when the stack component does not perform any explicit
    validation of this attribute using pydantic validators. If this were
    the case, the validation would run on the secret reference and would
    fail or in the worst case, modify the secret reference and lead to
    unexpected behavior. This method ensures that no attributes that require
    custom pydantic validation are set as secret references.

    Args:
        warn_about_plain_text_secrets: If true, then warns about using
            plain-text secrets.
        **kwargs: Arguments to initialize this stack component.

    Raises:
        ValueError: If an attribute that requires custom pydantic validation
            is passed as a secret reference, or if the `name` attribute
            was passed as a secret reference.
    """
    for key, value in kwargs.items():
        try:
            field = self.__class__.model_fields[key]
        except KeyError:
            # Value for a private attribute or non-existing field, this
            # will fail during the upcoming pydantic validation
            continue

        if value is None:
            continue

        if not secret_utils.is_secret_reference(value):
            if (
                secret_utils.is_secret_field(field)
                and warn_about_plain_text_secrets
            ):
                logger.warning(
                    "You specified a plain-text value for the sensitive "
                    f"attribute `{key}` for a `{self.__class__.__name__}` "
                    "stack component. This is currently only a warning, "
                    "but future versions of ZenML will require you to pass "
                    "in sensitive information as secrets. Check out the "
                    "documentation on how to configure your stack "
                    "components with secrets here: "
                    "https://docs.zenml.io/deploying-zenml/deploying-zenml/secret-management"
                )
            continue

        if pydantic_utils.has_validators(
            pydantic_class=self.__class__, field_name=key
        ):
            raise ValueError(
                f"Passing the stack component attribute `{key}` as a "
                "secret reference is not allowed as additional validation "
                "is required for this attribute."
            )

    super().__init__(**kwargs)
Attributes
is_local: bool property

Checks if this stack component is running locally.

Returns:

Type Description
bool

True if this config is for a local component, False otherwise.
Functions
ensure_path_local(path: str) -> str classmethod

Pydantic validator which ensures that the given path is a local path.

Parameters:

Name Type Description Default
path str

The path to validate.

 required

Returns:

Name Type Description
str str

The validated (local) path.

Raises:

Type Description
ArtifactStoreInterfaceError

If the given path is not a local path.
Source code in src/zenml/artifact_stores/local_artifact_store.py 
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
@field_validator("path")
@classmethod
def ensure_path_local(cls, path: str) -> str:
    """Pydantic validator which ensures that the given path is a local path.

    Args:
        path: The path to validate.

    Returns:
        str: The validated (local) path.

    Raises:
        ArtifactStoreInterfaceError: If the given path is not a local path.
    """
    remote_prefixes = ["gs://", "hdfs://", "s3://", "az://", "abfs://"]
    if any(path.startswith(prefix) for prefix in remote_prefixes):
        raise ArtifactStoreInterfaceError(
            f"The path '{path}' you defined for your local artifact store "
            f"starts with a remote prefix."
        )
    return path

LocalArtifactStoreFlavor

Bases: BaseArtifactStoreFlavor

Class for the LocalArtifactStoreFlavor.

Attributes
config_class: Type[LocalArtifactStoreConfig] property

Config class for this flavor.

Returns:

Type Description
Type[LocalArtifactStoreConfig]

The config class.
docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type Description
Optional[str]

A flavor docs url.
implementation_class: Type[LocalArtifactStore] property

Implementation class.

Returns:

Type Description
Type[LocalArtifactStore]

The implementation class.
logo_url: str property

A url to represent the flavor in the dashboard.

Returns:

Type Description
str

The flavor logo.
name: str property

Returns the name of the artifact store flavor.

Returns:

Name Type Description
str str

The name of the artifact store flavor.
sdk_docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type Description
Optional[str]

A flavor docs url.

Modules

base_artifact_store

The base interface to extend the ZenML artifact store.

Classes
BaseArtifactStore(*args: Any, **kwargs: Any)

Bases: StackComponent

Base class for all ZenML artifact stores.

Initiate the Pydantic object and register the corresponding filesystem.

Parameters:

Name Type Description Default
*args Any

The positional arguments to pass to the Pydantic object.

 ()
**kwargs Any

The keyword arguments to pass to the Pydantic object.

 {}
Source code in src/zenml/artifact_stores/base_artifact_store.py 
457
458
459
460
461
462
463
464
465
466
467
468
469
470
def __init__(self, *args: Any, **kwargs: Any) -> None:
    """Initiate the Pydantic object and register the corresponding filesystem.

    Args:
        *args: The positional arguments to pass to the Pydantic object.
        **kwargs: The keyword arguments to pass to the Pydantic object.
    """
    super(BaseArtifactStore, self).__init__(*args, **kwargs)
    self._add_path_sanitization()

    # If running in a ZenML server environment, we don't register
    # the filesystems. We always use the artifact stores directly.
    if ENV_ZENML_SERVER not in os.environ:
        self._register()
Attributes
config: BaseArtifactStoreConfig property

Returns the BaseArtifactStoreConfig config.

Returns:

Type Description
BaseArtifactStoreConfig

The configuration.
custom_cache_key: Optional[bytes] property

Custom cache key.

Any artifact store can override this property in case they need additional control over the caching behavior.

Returns:

Type Description
Optional[bytes]

Custom cache key.
path: str property

The path to the artifact store.

Returns:

Type Description
str

The path.
Functions
copyfile(src: PathType, dst: PathType, overwrite: bool = False) -> None abstractmethod

Copy a file from the source to the destination.

Parameters:

Name Type Description Default
src PathType

The source path.

 required
dst PathType

The destination path.

 required
overwrite bool

Whether to overwrite the destination file if it exists.

 False
Source code in src/zenml/artifact_stores/base_artifact_store.py 
308
309
310
311
312
313
314
315
316
317
318
@abstractmethod
def copyfile(
    self, src: PathType, dst: PathType, overwrite: bool = False
) -> None:
    """Copy a file from the source to the destination.

    Args:
        src: The source path.
        dst: The destination path.
        overwrite: Whether to overwrite the destination file if it exists.
    """
exists(path: PathType) -> bool abstractmethod

Checks if a path exists.

Parameters:

Name Type Description Default
path PathType

The path to check.

 required

Returns:

Type Description
bool

True if the path exists.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
320
321
322
323
324
325
326
327
328
329
@abstractmethod
def exists(self, path: PathType) -> bool:
    """Checks if a path exists.

    Args:
        path: The path to check.

    Returns:
        `True` if the path exists.
    """
glob(pattern: PathType) -> List[PathType] abstractmethod

Gets the paths that match a glob pattern.

Parameters:

Name Type Description Default
pattern PathType

The glob pattern.

 required

Returns:

Type Description
List[PathType]

The list of paths that match the pattern.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
331
332
333
334
335
336
337
338
339
340
@abstractmethod
def glob(self, pattern: PathType) -> List[PathType]:
    """Gets the paths that match a glob pattern.

    Args:
        pattern: The glob pattern.

    Returns:
        The list of paths that match the pattern.
    """
isdir(path: PathType) -> bool abstractmethod

Returns whether the given path points to a directory.

Parameters:

Name Type Description Default
path PathType

The path to check.

 required

Returns:

Type Description
bool

True if the path points to a directory.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
342
343
344
345
346
347
348
349
350
351
@abstractmethod
def isdir(self, path: PathType) -> bool:
    """Returns whether the given path points to a directory.

    Args:
        path: The path to check.

    Returns:
        `True` if the path points to a directory.
    """
listdir(path: PathType) -> List[PathType] abstractmethod

Returns a list of files under a given directory in the filesystem.

Parameters:

Name Type Description Default
path PathType

The path to list.

 required

Returns:

Type Description
List[PathType]

The list of files under the given path.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
353
354
355
356
357
358
359
360
361
362
@abstractmethod
def listdir(self, path: PathType) -> List[PathType]:
    """Returns a list of files under a given directory in the filesystem.

    Args:
        path: The path to list.

    Returns:
        The list of files under the given path.
    """
makedirs(path: PathType) -> None abstractmethod

Make a directory at the given path, recursively creating parents.

Parameters:

Name Type Description Default
path PathType

The path to create.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
364
365
366
367
368
369
370
@abstractmethod
def makedirs(self, path: PathType) -> None:
    """Make a directory at the given path, recursively creating parents.

    Args:
        path: The path to create.
    """
mkdir(path: PathType) -> None abstractmethod

Make a directory at the given path; parent directory must exist.

Parameters:

Name Type Description Default
path PathType

The path to create.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
372
373
374
375
376
377
378
@abstractmethod
def mkdir(self, path: PathType) -> None:
    """Make a directory at the given path; parent directory must exist.

    Args:
        path: The path to create.
    """
open(path: PathType, mode: str = 'r') -> Any abstractmethod

Open a file at the given path.

Parameters:

Name Type Description Default
path PathType

The path of the file to open.

 required
mode str

The mode to open the file.

 'r'

Returns:

Type Description
Any

The file object.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
296
297
298
299
300
301
302
303
304
305
306
@abstractmethod
def open(self, path: PathType, mode: str = "r") -> Any:
    """Open a file at the given path.

    Args:
        path: The path of the file to open.
        mode: The mode to open the file.

    Returns:
        The file object.
    """
remove(path: PathType) -> None abstractmethod

Remove the file at the given path. Dangerous operation.

Parameters:

Name Type Description Default
path PathType

The path to remove.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
380
381
382
383
384
385
386
@abstractmethod
def remove(self, path: PathType) -> None:
    """Remove the file at the given path. Dangerous operation.

    Args:
        path: The path to remove.
    """
rename(src: PathType, dst: PathType, overwrite: bool = False) -> None abstractmethod

Rename source file to destination file.

Parameters:

Name Type Description Default
src PathType

The source path.

 required
dst PathType

The destination path.

 required
overwrite bool

Whether to overwrite the destination file if it exists.

 False
Source code in src/zenml/artifact_stores/base_artifact_store.py 
388
389
390
391
392
393
394
395
396
397
398
@abstractmethod
def rename(
    self, src: PathType, dst: PathType, overwrite: bool = False
) -> None:
    """Rename source file to destination file.

    Args:
        src: The source path.
        dst: The destination path.
        overwrite: Whether to overwrite the destination file if it exists.
    """
rmtree(path: PathType) -> None abstractmethod

Deletes dir recursively. Dangerous operation.

Parameters:

Name Type Description Default
path PathType

The path to delete.

 required
Source code in src/zenml/artifact_stores/base_artifact_store.py 
400
401
402
403
404
405
406
@abstractmethod
def rmtree(self, path: PathType) -> None:
    """Deletes dir recursively. Dangerous operation.

    Args:
        path: The path to delete.
    """
size(path: PathType) -> Optional[int] abstractmethod

Get the size of a file in bytes.

Parameters:

Name Type Description Default
path PathType

The path to the file.

 required

Returns:

Type Description
Optional[int]

The size of the file in bytes or None if the artifact store
Optional[int]

does not implement the size method.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
@abstractmethod
def size(self, path: PathType) -> Optional[int]:
    """Get the size of a file in bytes.

    Args:
        path: The path to the file.

    Returns:
        The size of the file in bytes or `None` if the artifact store
        does not implement the `size` method.
    """
    logger.warning(
        "Cannot get size of file '%s' since the artifact store %s does not "
        "implement the `size` method.",
        path,
        self.__class__.__name__,
    )
    return None
stat(path: PathType) -> Any abstractmethod

Return the stat descriptor for a given file path.

Parameters:

Name Type Description Default
path PathType

The path to check.

 required

Returns:

Type Description
Any

The stat descriptor.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
408
409
410
411
412
413
414
415
416
417
@abstractmethod
def stat(self, path: PathType) -> Any:
    """Return the stat descriptor for a given file path.

    Args:
        path: The path to check.

    Returns:
        The stat descriptor.
    """
walk(top: PathType, topdown: bool = True, onerror: Optional[Callable[..., None]] = None) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]] abstractmethod

Return an iterator that walks the contents of the given directory.

Parameters:

Name Type Description Default
top PathType

The path to walk.

 required
topdown bool

Whether to walk the top-down or bottom-up.

 True
onerror Optional[Callable[..., None]]

The error handler.

 None

Returns:

Type Description
Iterable[Tuple[PathType, List[PathType], List[PathType]]]

The iterator that walks the contents of the given directory.
Source code in src/zenml/artifact_stores/base_artifact_store.py 
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
@abstractmethod
def walk(
    self,
    top: PathType,
    topdown: bool = True,
    onerror: Optional[Callable[..., None]] = None,
) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]]:
    """Return an iterator that walks the contents of the given directory.

    Args:
        top: The path to walk.
        topdown: Whether to walk the top-down or bottom-up.
        onerror: The error handler.

    Returns:
        The iterator that walks the contents of the given directory.
    """
BaseArtifactStoreConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)

Bases: StackComponentConfig

Config class for BaseArtifactStore.

Base configuration for artifact storage backends. Field descriptions are defined inline using Field() descriptors.

Source code in src/zenml/stack/stack_component.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def __init__(
    self, warn_about_plain_text_secrets: bool = False, **kwargs: Any
) -> None:
    """Ensures that secret references don't clash with pydantic validation.

    StackComponents allow the specification of all their string attributes
    using secret references of the form `{{secret_name.key}}`. This however
    is only possible when the stack component does not perform any explicit
    validation of this attribute using pydantic validators. If this were
    the case, the validation would run on the secret reference and would
    fail or in the worst case, modify the secret reference and lead to
    unexpected behavior. This method ensures that no attributes that require
    custom pydantic validation are set as secret references.

    Args:
        warn_about_plain_text_secrets: If true, then warns about using
            plain-text secrets.
        **kwargs: Arguments to initialize this stack component.

    Raises:
        ValueError: If an attribute that requires custom pydantic validation
            is passed as a secret reference, or if the `name` attribute
            was passed as a secret reference.
    """
    for key, value in kwargs.items():
        try:
            field = self.__class__.model_fields[key]
        except KeyError:
            # Value for a private attribute or non-existing field, this
            # will fail during the upcoming pydantic validation
            continue

        if value is None:
            continue

        if not secret_utils.is_secret_reference(value):
            if (
                secret_utils.is_secret_field(field)
                and warn_about_plain_text_secrets
            ):
                logger.warning(
                    "You specified a plain-text value for the sensitive "
                    f"attribute `{key}` for a `{self.__class__.__name__}` "
                    "stack component. This is currently only a warning, "
                    "but future versions of ZenML will require you to pass "
                    "in sensitive information as secrets. Check out the "
                    "documentation on how to configure your stack "
                    "components with secrets here: "
                    "https://docs.zenml.io/deploying-zenml/deploying-zenml/secret-management"
                )
            continue

        if pydantic_utils.has_validators(
            pydantic_class=self.__class__, field_name=key
        ):
            raise ValueError(
                f"Passing the stack component attribute `{key}` as a "
                "secret reference is not allowed as additional validation "
                "is required for this attribute."
            )

    super().__init__(**kwargs)
BaseArtifactStoreFlavor

Bases: Flavor

Base class for artifact store flavors.

Attributes
config_class: Type[StackComponentConfig] property

Config class for this flavor.

Returns:

Type Description
Type[StackComponentConfig]

The config class.
implementation_class: Type[BaseArtifactStore] abstractmethod property

Implementation class.

Returns:

Type Description
Type[BaseArtifactStore]

The implementation class.
type: StackComponentType property

Returns the flavor type.

Returns:

Type Description
StackComponentType

The flavor type.
Functions
Modules

local_artifact_store

The local artifact store is a local implementation of the artifact store.

In ZenML, the inputs and outputs which go through any step is treated as an artifact and as its name suggests, an ArtifactStore is a place where these artifacts get stored.

Classes
LocalArtifactStore(*args: Any, **kwargs: Any)

Bases: LocalFilesystem, BaseArtifactStore

Artifact Store for local artifacts.

All methods are inherited from the default LocalFilesystem.

Source code in src/zenml/artifact_stores/base_artifact_store.py 
457
458
459
460
461
462
463
464
465
466
467
468
469
470
def __init__(self, *args: Any, **kwargs: Any) -> None:
    """Initiate the Pydantic object and register the corresponding filesystem.

    Args:
        *args: The positional arguments to pass to the Pydantic object.
        **kwargs: The keyword arguments to pass to the Pydantic object.
    """
    super(BaseArtifactStore, self).__init__(*args, **kwargs)
    self._add_path_sanitization()

    # If running in a ZenML server environment, we don't register
    # the filesystems. We always use the artifact stores directly.
    if ENV_ZENML_SERVER not in os.environ:
        self._register()
Attributes
custom_cache_key: Optional[bytes] property

Custom cache key.

The client ID is returned here to invalidate caching when using the same local artifact store on multiple client machines.

Returns:

Type Description
Optional[bytes]

Custom cache key.
local_path: Optional[str] property

Returns the local path of the artifact store.

Returns:

Type Description
Optional[str]

The local path of the artifact store.
path: str property

Returns the path to the local artifact store.

If the user has not defined a path in the config, this will create a sub-folder in the global config directory.

Returns:

Type Description
str

The path to the local artifact store.
Functions
get_default_local_path(id_: UUID) -> str staticmethod

Returns the default local path for a local artifact store.

Parameters:

Name Type Description Default
id_ UUID

The id of the local artifact store.

 required

Returns:

Name Type Description
str str

The default local path.
Source code in src/zenml/artifact_stores/local_artifact_store.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
@staticmethod
def get_default_local_path(id_: "UUID") -> str:
    """Returns the default local path for a local artifact store.

    Args:
        id_: The id of the local artifact store.

    Returns:
        str: The default local path.
    """
    return os.path.join(
        GlobalConfiguration().local_stores_path,
        str(id_),
    )
LocalArtifactStoreConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)

Bases: BaseArtifactStoreConfig

Config class for the local artifact store.

Attributes:

Name Type Description
path str

The path to the local artifact store.
Source code in src/zenml/stack/stack_component.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def __init__(
    self, warn_about_plain_text_secrets: bool = False, **kwargs: Any
) -> None:
    """Ensures that secret references don't clash with pydantic validation.

    StackComponents allow the specification of all their string attributes
    using secret references of the form `{{secret_name.key}}`. This however
    is only possible when the stack component does not perform any explicit
    validation of this attribute using pydantic validators. If this were
    the case, the validation would run on the secret reference and would
    fail or in the worst case, modify the secret reference and lead to
    unexpected behavior. This method ensures that no attributes that require
    custom pydantic validation are set as secret references.

    Args:
        warn_about_plain_text_secrets: If true, then warns about using
            plain-text secrets.
        **kwargs: Arguments to initialize this stack component.

    Raises:
        ValueError: If an attribute that requires custom pydantic validation
            is passed as a secret reference, or if the `name` attribute
            was passed as a secret reference.
    """
    for key, value in kwargs.items():
        try:
            field = self.__class__.model_fields[key]
        except KeyError:
            # Value for a private attribute or non-existing field, this
            # will fail during the upcoming pydantic validation
            continue

        if value is None:
            continue

        if not secret_utils.is_secret_reference(value):
            if (
                secret_utils.is_secret_field(field)
                and warn_about_plain_text_secrets
            ):
                logger.warning(
                    "You specified a plain-text value for the sensitive "
                    f"attribute `{key}` for a `{self.__class__.__name__}` "
                    "stack component. This is currently only a warning, "
                    "but future versions of ZenML will require you to pass "
                    "in sensitive information as secrets. Check out the "
                    "documentation on how to configure your stack "
                    "components with secrets here: "
                    "https://docs.zenml.io/deploying-zenml/deploying-zenml/secret-management"
                )
            continue

        if pydantic_utils.has_validators(
            pydantic_class=self.__class__, field_name=key
        ):
            raise ValueError(
                f"Passing the stack component attribute `{key}` as a "
                "secret reference is not allowed as additional validation "
                "is required for this attribute."
            )

    super().__init__(**kwargs)
Attributes
is_local: bool property

Checks if this stack component is running locally.

Returns:

Type Description
bool

True if this config is for a local component, False otherwise.
Functions
ensure_path_local(path: str) -> str classmethod

Pydantic validator which ensures that the given path is a local path.

Parameters:

Name Type Description Default
path str

The path to validate.

 required

Returns:

Name Type Description
str str

The validated (local) path.

Raises:

Type Description
ArtifactStoreInterfaceError

If the given path is not a local path.
Source code in src/zenml/artifact_stores/local_artifact_store.py 
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
@field_validator("path")
@classmethod
def ensure_path_local(cls, path: str) -> str:
    """Pydantic validator which ensures that the given path is a local path.

    Args:
        path: The path to validate.

    Returns:
        str: The validated (local) path.

    Raises:
        ArtifactStoreInterfaceError: If the given path is not a local path.
    """
    remote_prefixes = ["gs://", "hdfs://", "s3://", "az://", "abfs://"]
    if any(path.startswith(prefix) for prefix in remote_prefixes):
        raise ArtifactStoreInterfaceError(
            f"The path '{path}' you defined for your local artifact store "
            f"starts with a remote prefix."
        )
    return path
LocalArtifactStoreFlavor

Bases: BaseArtifactStoreFlavor

Class for the LocalArtifactStoreFlavor.

Attributes
config_class: Type[LocalArtifactStoreConfig] property

Config class for this flavor.

Returns:

Type Description
Type[LocalArtifactStoreConfig]

The config class.
docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type Description
Optional[str]

A flavor docs url.
implementation_class: Type[LocalArtifactStore] property

Implementation class.

Returns:

Type Description
Type[LocalArtifactStore]

The implementation class.
logo_url: str property

A url to represent the flavor in the dashboard.

Returns:

Type Description
str

The flavor logo.
name: str property

Returns the name of the artifact store flavor.

Returns:

Name Type Description
str str

The name of the artifact store flavor.
sdk_docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type Description
Optional[str]

A flavor docs url.
Modules