S3
zenml.integrations.s3
Initialization of the S3 integration.
The S3 integration allows the use of cloud artifact stores and file operations on S3 buckets.
Attributes
S3 = 's3'
module-attribute
S3_ARTIFACT_STORE_FLAVOR = 's3'
module-attribute
Classes
Flavor
Class for ZenML Flavors.
Attributes
config_class: Type[StackComponentConfig]
abstractmethod
property
Returns StackComponentConfig
config class.
Returns:
Type | Description |
---|---|
Type[StackComponentConfig]
|
The config class. |
config_schema: Dict[str, Any]
property
The config schema for a flavor.
Returns:
Type | Description |
---|---|
Dict[str, Any]
|
The config schema. |
docs_url: Optional[str]
property
A url to point at docs explaining this flavor.
Returns:
Type | Description |
---|---|
Optional[str]
|
A flavor docs url. |
implementation_class: Type[StackComponent]
abstractmethod
property
Implementation class for this flavor.
Returns:
Type | Description |
---|---|
Type[StackComponent]
|
The implementation class for this flavor. |
logo_url: Optional[str]
property
A url to represent the flavor in the dashboard.
Returns:
Type | Description |
---|---|
Optional[str]
|
The flavor logo. |
name: str
abstractmethod
property
The flavor name.
Returns:
Type | Description |
---|---|
str
|
The flavor name. |
sdk_docs_url: Optional[str]
property
A url to point at SDK docs explaining this flavor.
Returns:
Type | Description |
---|---|
Optional[str]
|
A flavor SDK docs url. |
service_connector_requirements: Optional[ServiceConnectorRequirements]
property
Service connector resource requirements for service connectors.
Specifies resource requirements that are used to filter the available service connector types that are compatible with this flavor.
Returns:
Type | Description |
---|---|
Optional[ServiceConnectorRequirements]
|
Requirements for compatible service connectors, if a service |
Optional[ServiceConnectorRequirements]
|
connector is required for this flavor. |
type: StackComponentType
abstractmethod
property
Functions
from_model(flavor_model: FlavorResponse) -> Flavor
classmethod
Loads a flavor from a model.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
flavor_model
|
FlavorResponse
|
The model to load from. |
required |
Raises:
Type | Description |
---|---|
CustomFlavorImportError
|
If the custom flavor can't be imported. |
ImportError
|
If the flavor can't be imported. |
Returns:
Type | Description |
---|---|
Flavor
|
The loaded flavor. |
Source code in src/zenml/stack/flavor.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 |
|
generate_default_docs_url() -> str
Generate the doc urls for all inbuilt and integration flavors.
Note that this method is not going to be useful for custom flavors, which do not have any docs in the main zenml docs.
Returns:
Type | Description |
---|---|
str
|
The complete url to the zenml documentation |
Source code in src/zenml/stack/flavor.py
206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 |
|
generate_default_sdk_docs_url() -> str
Generate SDK docs url for a flavor.
Returns:
Type | Description |
---|---|
str
|
The complete url to the zenml SDK docs |
Source code in src/zenml/stack/flavor.py
232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 |
|
to_model(integration: Optional[str] = None, is_custom: bool = True) -> FlavorRequest
Converts a flavor to a model.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
integration
|
Optional[str]
|
The integration to use for the model. |
None
|
is_custom
|
bool
|
Whether the flavor is a custom flavor. |
True
|
Returns:
Type | Description |
---|---|
FlavorRequest
|
The model. |
Source code in src/zenml/stack/flavor.py
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 |
|
Integration
Base class for integration in ZenML.
Functions
activate() -> None
classmethod
Abstract method to activate the integration.
Source code in src/zenml/integrations/integration.py
175 176 177 |
|
check_installation() -> bool
classmethod
Method to check whether the required packages are installed.
Returns:
Type | Description |
---|---|
bool
|
True if all required packages are installed, False otherwise. |
Source code in src/zenml/integrations/integration.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
|
flavors() -> List[Type[Flavor]]
classmethod
Abstract method to declare new stack component flavors.
Returns:
Type | Description |
---|---|
List[Type[Flavor]]
|
A list of new stack component flavors. |
Source code in src/zenml/integrations/integration.py
179 180 181 182 183 184 185 186 |
|
get_requirements(target_os: Optional[str] = None, python_version: Optional[str] = None) -> List[str]
classmethod
Method to get the requirements for the integration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
target_os
|
Optional[str]
|
The target operating system to get the requirements for. |
None
|
python_version
|
Optional[str]
|
The Python version to use for the requirements. |
None
|
Returns:
Type | Description |
---|---|
List[str]
|
A list of requirements. |
Source code in src/zenml/integrations/integration.py
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 |
|
get_uninstall_requirements(target_os: Optional[str] = None) -> List[str]
classmethod
Method to get the uninstall requirements for the integration.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
target_os
|
Optional[str]
|
The target operating system to get the requirements for. |
None
|
Returns:
Type | Description |
---|---|
List[str]
|
A list of requirements. |
Source code in src/zenml/integrations/integration.py
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 |
|
plugin_flavors() -> List[Type[BasePluginFlavor]]
classmethod
Abstract method to declare new plugin flavors.
Returns:
Type | Description |
---|---|
List[Type[BasePluginFlavor]]
|
A list of new plugin flavors. |
Source code in src/zenml/integrations/integration.py
188 189 190 191 192 193 194 195 |
|
S3Integration
Bases: Integration
Definition of S3 integration for ZenML.
Functions
flavors() -> List[Type[Flavor]]
classmethod
Declare the stack component flavors for the s3 integration.
Returns:
Type | Description |
---|---|
List[Type[Flavor]]
|
List of stack component flavors for this integration. |
Source code in src/zenml/integrations/s3/__init__.py
43 44 45 46 47 48 49 50 51 52 |
|
Modules
artifact_stores
Initialization of the S3 Artifact Store.
Classes
S3ArtifactStore(*args: Any, **kwargs: Any)
Bases: BaseArtifactStore
, AuthenticationMixin
Artifact Store for S3 based artifacts.
Initializes the artifact store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
*args
|
Any
|
Additional positional arguments. |
()
|
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
|
config: S3ArtifactStoreConfig
property
Get the config of this artifact store.
Returns:
Type | Description |
---|---|
S3ArtifactStoreConfig
|
The config of this artifact store. |
filesystem: ZenMLS3Filesystem
property
The s3 filesystem to access this artifact store.
Returns:
Type | Description |
---|---|
ZenMLS3Filesystem
|
The s3 filesystem. |
cleanup() -> None
Close the filesystem.
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
238 239 240 241 |
|
copyfile(src: PathType, dst: PathType, overwrite: bool = False) -> None
Copy a file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
src
|
PathType
|
The path to copy from. |
required |
dst
|
PathType
|
The path to copy to. |
required |
overwrite
|
bool
|
If a file already exists at the destination, this
method will overwrite it if overwrite= |
False
|
Raises:
Type | Description |
---|---|
FileExistsError
|
If a file already exists at the destination
and overwrite is not set to |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 |
|
exists(path: PathType) -> bool
Check whether a path exists.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to check. |
required |
Returns:
Type | Description |
---|---|
bool
|
True if the path exists, False otherwise. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
282 283 284 285 286 287 288 289 290 291 |
|
get_credentials() -> Tuple[Optional[str], Optional[str], Optional[str], Optional[str]]
Gets authentication credentials.
If an authentication secret is configured, the secret values are returned. Otherwise, we fall back to the plain text component attributes.
Returns:
Type | Description |
---|---|
Optional[str]
|
Tuple (key, secret, token, region) of credentials used to |
Optional[str]
|
authenticate with the S3 filesystem. |
Raises:
Type | Description |
---|---|
RuntimeError
|
If the AWS connector behaves unexpectedly. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 |
|
glob(pattern: PathType) -> List[PathType]
Return all paths that match the given glob pattern.
The glob pattern may include: - '' to match any number of characters - '?' to match a single character - '[...]' to match one of the characters inside the brackets - '' as the full name of a path component to match to search in subdirectories of any depth (e.g. '/some_dir/*/some_file)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
pattern
|
PathType
|
The glob pattern to match, see details above. |
required |
Returns:
Type | Description |
---|---|
List[PathType]
|
A list of paths that match the given glob pattern. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 |
|
isdir(path: PathType) -> bool
Check whether a path is a directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to check. |
required |
Returns:
Type | Description |
---|---|
bool
|
True if the path is a directory, False otherwise. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
311 312 313 314 315 316 317 318 319 320 |
|
listdir(path: PathType) -> List[PathType]
Return a list of files in a directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to list. |
required |
Returns:
Type | Description |
---|---|
List[PathType]
|
A list of paths that are files in the given directory. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 |
|
makedirs(path: PathType) -> None
Create a directory at the given path.
If needed also create missing parent directories.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to create. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
358 359 360 361 362 363 364 365 366 |
|
mkdir(path: PathType) -> None
Create a directory at the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to create. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
368 369 370 371 372 373 374 |
|
open(path: PathType, mode: str = 'r') -> Any
Open a file at the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
Path of the file to open. |
required |
mode
|
str
|
Mode in which to open the file. Currently, only 'rb' and 'wb' to read and write binary files are supported. |
'r'
|
Returns:
Type | Description |
---|---|
Any
|
A file-like object. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
243 244 245 246 247 248 249 250 251 252 253 254 |
|
remove(path: PathType) -> None
Remove the file at the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path of the file to remove. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
376 377 378 379 380 381 382 |
|
rename(src: PathType, dst: PathType, overwrite: bool = False) -> None
Rename source file to destination file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
src
|
PathType
|
The path of the file to rename. |
required |
dst
|
PathType
|
The path to rename the source file to. |
required |
overwrite
|
bool
|
If a file already exists at the destination, this
method will overwrite it if overwrite= |
False
|
Raises:
Type | Description |
---|---|
FileExistsError
|
If a file already exists at the destination
and overwrite is not set to |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 |
|
rmtree(path: PathType) -> None
Remove the given directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path of the directory to remove. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
410 411 412 413 414 415 416 |
|
size(path: PathType) -> int
Get the size of a file in bytes.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to the file. |
required |
Returns:
Type | Description |
---|---|
int
|
The size of the file in bytes. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
429 430 431 432 433 434 435 436 437 438 |
|
stat(path: PathType) -> Dict[str, Any]
Return stat info for the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to get stat info for. |
required |
Returns:
Type | Description |
---|---|
Dict[str, Any]
|
A dictionary containing the stat info. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
418 419 420 421 422 423 424 425 426 427 |
|
walk(top: PathType, topdown: bool = True, onerror: Optional[Callable[..., None]] = None) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]]
Return an iterator that walks the contents of the given directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
top
|
PathType
|
Path of directory to walk. |
required |
topdown
|
bool
|
Unused argument to conform to interface. |
True
|
onerror
|
Optional[Callable[..., None]]
|
Unused argument to conform to interface. |
None
|
Yields:
Type | Description |
---|---|
Iterable[Tuple[PathType, List[PathType], List[PathType]]]
|
An Iterable of Tuples, each of which contain the path of the current directory path, a list of directories inside the current directory and a list of files inside the current directory. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 |
|
Modules
s3_artifact_store
Implementation of the S3 Artifact Store.
S3ArtifactStore(*args: Any, **kwargs: Any)
Bases: BaseArtifactStore
, AuthenticationMixin
Artifact Store for S3 based artifacts.
Initializes the artifact store.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
*args
|
Any
|
Additional positional arguments. |
()
|
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
|
config: S3ArtifactStoreConfig
property
Get the config of this artifact store.
Returns:
Type | Description |
---|---|
S3ArtifactStoreConfig
|
The config of this artifact store. |
filesystem: ZenMLS3Filesystem
property
The s3 filesystem to access this artifact store.
Returns:
Type | Description |
---|---|
ZenMLS3Filesystem
|
The s3 filesystem. |
cleanup() -> None
Close the filesystem.
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
238 239 240 241 |
|
copyfile(src: PathType, dst: PathType, overwrite: bool = False) -> None
Copy a file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
src
|
PathType
|
The path to copy from. |
required |
dst
|
PathType
|
The path to copy to. |
required |
overwrite
|
bool
|
If a file already exists at the destination, this
method will overwrite it if overwrite= |
False
|
Raises:
Type | Description |
---|---|
FileExistsError
|
If a file already exists at the destination
and overwrite is not set to |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 |
|
exists(path: PathType) -> bool
Check whether a path exists.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to check. |
required |
Returns:
Type | Description |
---|---|
bool
|
True if the path exists, False otherwise. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
282 283 284 285 286 287 288 289 290 291 |
|
get_credentials() -> Tuple[Optional[str], Optional[str], Optional[str], Optional[str]]
Gets authentication credentials.
If an authentication secret is configured, the secret values are returned. Otherwise, we fall back to the plain text component attributes.
Returns:
Type | Description |
---|---|
Optional[str]
|
Tuple (key, secret, token, region) of credentials used to |
Optional[str]
|
authenticate with the S3 filesystem. |
Raises:
Type | Description |
---|---|
RuntimeError
|
If the AWS connector behaves unexpectedly. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 |
|
glob(pattern: PathType) -> List[PathType]
Return all paths that match the given glob pattern.
The glob pattern may include: - '' to match any number of characters - '?' to match a single character - '[...]' to match one of the characters inside the brackets - '' as the full name of a path component to match to search in subdirectories of any depth (e.g. '/some_dir/*/some_file)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
pattern
|
PathType
|
The glob pattern to match, see details above. |
required |
Returns:
Type | Description |
---|---|
List[PathType]
|
A list of paths that match the given glob pattern. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 |
|
isdir(path: PathType) -> bool
Check whether a path is a directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to check. |
required |
Returns:
Type | Description |
---|---|
bool
|
True if the path is a directory, False otherwise. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
311 312 313 314 315 316 317 318 319 320 |
|
listdir(path: PathType) -> List[PathType]
Return a list of files in a directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to list. |
required |
Returns:
Type | Description |
---|---|
List[PathType]
|
A list of paths that are files in the given directory. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 |
|
makedirs(path: PathType) -> None
Create a directory at the given path.
If needed also create missing parent directories.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to create. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
358 359 360 361 362 363 364 365 366 |
|
mkdir(path: PathType) -> None
Create a directory at the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to create. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
368 369 370 371 372 373 374 |
|
open(path: PathType, mode: str = 'r') -> Any
Open a file at the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
Path of the file to open. |
required |
mode
|
str
|
Mode in which to open the file. Currently, only 'rb' and 'wb' to read and write binary files are supported. |
'r'
|
Returns:
Type | Description |
---|---|
Any
|
A file-like object. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
243 244 245 246 247 248 249 250 251 252 253 254 |
|
remove(path: PathType) -> None
Remove the file at the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path of the file to remove. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
376 377 378 379 380 381 382 |
|
rename(src: PathType, dst: PathType, overwrite: bool = False) -> None
Rename source file to destination file.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
src
|
PathType
|
The path of the file to rename. |
required |
dst
|
PathType
|
The path to rename the source file to. |
required |
overwrite
|
bool
|
If a file already exists at the destination, this
method will overwrite it if overwrite= |
False
|
Raises:
Type | Description |
---|---|
FileExistsError
|
If a file already exists at the destination
and overwrite is not set to |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 |
|
rmtree(path: PathType) -> None
Remove the given directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path of the directory to remove. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
410 411 412 413 414 415 416 |
|
size(path: PathType) -> int
Get the size of a file in bytes.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to the file. |
required |
Returns:
Type | Description |
---|---|
int
|
The size of the file in bytes. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
429 430 431 432 433 434 435 436 437 438 |
|
stat(path: PathType) -> Dict[str, Any]
Return stat info for the given path.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
path
|
PathType
|
The path to get stat info for. |
required |
Returns:
Type | Description |
---|---|
Dict[str, Any]
|
A dictionary containing the stat info. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
418 419 420 421 422 423 424 425 426 427 |
|
walk(top: PathType, topdown: bool = True, onerror: Optional[Callable[..., None]] = None) -> Iterable[Tuple[PathType, List[PathType], List[PathType]]]
Return an iterator that walks the contents of the given directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
top
|
PathType
|
Path of directory to walk. |
required |
topdown
|
bool
|
Unused argument to conform to interface. |
True
|
onerror
|
Optional[Callable[..., None]]
|
Unused argument to conform to interface. |
None
|
Yields:
Type | Description |
---|---|
Iterable[Tuple[PathType, List[PathType], List[PathType]]]
|
An Iterable of Tuples, each of which contain the path of the current directory path, a list of directories inside the current directory and a list of files inside the current directory. |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 |
|
ZenMLS3Filesystem
Bases: S3FileSystem
Modified s3fs.S3FileSystem to disable caching.
The original s3fs.S3FileSystem caches all class instances based on the constructor input arguments and it never releases them. This is problematic in the context of the ZenML server, because the server is a long-running process that instantiates many S3 filesystems with different credentials, especially when the credentials are generated by service connectors.
The caching behavior of s3fs causes the server to slowly consume more and
more memory over time until it crashes. This class disables the caching
behavior of s3fs by setting the cachable
attribute to False
.
In addition to disabling instance caching, this class also provides a
correct cleanup implementation by overriding the close_session
method
the S3 aiobotocore client. The original one provided by s3fs was causing
memory leaks by creating a new event loop in the destructor instead of
using the existing one.
A close
method is also provided to allow for synchronous on-demand cleanup
of the S3 client.
close_session(loop: Any, s3: Any) -> None
staticmethod
Close the S3 client session.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
loop
|
Any
|
The event loop to use for closing the session. |
required |
s3
|
Any
|
The S3 client to close. |
required |
Source code in src/zenml/integrations/s3/artifact_stores/s3_artifact_store.py
84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
|
flavors
Amazon S3 integration flavors.
Classes
S3ArtifactStoreConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)
Bases: BaseArtifactStoreConfig
, AuthenticationConfigMixin
Configuration for the S3 Artifact Store.
All attributes of this class except path
will be passed to the
s3fs.S3FileSystem
initialization. See
here for more information on how
to use those configuration options to connect to any S3-compatible storage.
When you want to register an S3ArtifactStore from the CLI and need to pass
client_kwargs
, config_kwargs
or s3_additional_kwargs
, you should pass
them as a json string:
zenml artifact-store register my_s3_store --flavor=s3 --path=s3://my_bucket --client_kwargs='{"endpoint_url": "http://my-s3-endpoint"}'
Source code in src/zenml/stack/stack_component.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
|
bucket: str
property
The bucket name of the artifact store.
Returns:
Type | Description |
---|---|
str
|
The bucket name of the artifact store. |
S3ArtifactStoreFlavor
Bases: BaseArtifactStoreFlavor
Flavor of the S3 artifact store.
config_class: Type[S3ArtifactStoreConfig]
property
The config class of the flavor.
Returns:
Type | Description |
---|---|
Type[S3ArtifactStoreConfig]
|
The config class of the flavor. |
docs_url: Optional[str]
property
A URL to point at docs explaining this flavor.
Returns:
Type | Description |
---|---|
Optional[str]
|
A flavor docs url. |
implementation_class: Type[S3ArtifactStore]
property
Implementation class for this flavor.
Returns:
Type | Description |
---|---|
Type[S3ArtifactStore]
|
The implementation class for this flavor. |
logo_url: str
property
A URL to represent the flavor in the dashboard.
Returns:
Type | Description |
---|---|
str
|
The flavor logo. |
name: str
property
Name of the flavor.
Returns:
Type | Description |
---|---|
str
|
The name of the flavor. |
sdk_docs_url: Optional[str]
property
A URL to point at SDK docs explaining this flavor.
Returns:
Type | Description |
---|---|
Optional[str]
|
A flavor SDK docs url. |
service_connector_requirements: Optional[ServiceConnectorRequirements]
property
Service connector resource requirements for service connectors.
Specifies resource requirements that are used to filter the available service connector types that are compatible with this flavor.
Returns:
Type | Description |
---|---|
Optional[ServiceConnectorRequirements]
|
Requirements for compatible service connectors, if a service |
Optional[ServiceConnectorRequirements]
|
connector is required for this flavor. |
Modules
s3_artifact_store_flavor
Amazon S3 artifact store flavor.
S3ArtifactStoreConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)
Bases: BaseArtifactStoreConfig
, AuthenticationConfigMixin
Configuration for the S3 Artifact Store.
All attributes of this class except path
will be passed to the
s3fs.S3FileSystem
initialization. See
here for more information on how
to use those configuration options to connect to any S3-compatible storage.
When you want to register an S3ArtifactStore from the CLI and need to pass
client_kwargs
, config_kwargs
or s3_additional_kwargs
, you should pass
them as a json string:
zenml artifact-store register my_s3_store --flavor=s3 --path=s3://my_bucket --client_kwargs='{"endpoint_url": "http://my-s3-endpoint"}'
Source code in src/zenml/stack/stack_component.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
|
bucket: str
property
The bucket name of the artifact store.
Returns:
Type | Description |
---|---|
str
|
The bucket name of the artifact store. |
S3ArtifactStoreFlavor
Bases: BaseArtifactStoreFlavor
Flavor of the S3 artifact store.
config_class: Type[S3ArtifactStoreConfig]
property
The config class of the flavor.
Returns:
Type | Description |
---|---|
Type[S3ArtifactStoreConfig]
|
The config class of the flavor. |
docs_url: Optional[str]
property
A URL to point at docs explaining this flavor.
Returns:
Type | Description |
---|---|
Optional[str]
|
A flavor docs url. |
implementation_class: Type[S3ArtifactStore]
property
Implementation class for this flavor.
Returns:
Type | Description |
---|---|
Type[S3ArtifactStore]
|
The implementation class for this flavor. |
logo_url: str
property
A URL to represent the flavor in the dashboard.
Returns:
Type | Description |
---|---|
str
|
The flavor logo. |
name: str
property
Name of the flavor.
Returns:
Type | Description |
---|---|
str
|
The name of the flavor. |
sdk_docs_url: Optional[str]
property
A URL to point at SDK docs explaining this flavor.
Returns:
Type | Description |
---|---|
Optional[str]
|
A flavor SDK docs url. |
service_connector_requirements: Optional[ServiceConnectorRequirements]
property
Service connector resource requirements for service connectors.
Specifies resource requirements that are used to filter the available service connector types that are compatible with this flavor.
Returns:
Type | Description |
---|---|
Optional[ServiceConnectorRequirements]
|
Requirements for compatible service connectors, if a service |
Optional[ServiceConnectorRequirements]
|
connector is required for this flavor. |
utils
Utility methods for S3.
Functions
split_s3_path(s3_path: str) -> Tuple[str, str]
Split S3 URI into bucket and key.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
s3_path
|
str
|
S3 URI (e.g. "s3://bucket/path") |
required |
Returns:
Type | Description |
---|---|
Tuple[str, str]
|
A tuple of bucket and key, for "s3://bucket/path/path2" it will return ("bucket","path/path2") |
Raises:
Type | Description |
---|---|
ValueError
|
if the S3 URI is invalid |
Source code in src/zenml/integrations/s3/utils.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
|