Step Operators
zenml.step_operators
Step operators allow you to run steps on custom infrastructure.
While an orchestrator defines how and where your entire pipeline runs, a step operator defines how and where an individual step runs. This can be useful in a variety of scenarios. An example could be if one step within a pipeline should run on a separate environment equipped with a GPU (like a trainer step).
Attributes
__all__ = ['BaseStepOperator', 'BaseStepOperatorFlavor', 'BaseStepOperatorConfig']
module-attribute
Classes
BaseStepOperator(name: str, id: UUID, config: StackComponentConfig, flavor: str, type: StackComponentType, user: Optional[UUID], created: datetime, updated: datetime, labels: Optional[Dict[str, Any]] = None, connector_requirements: Optional[ServiceConnectorRequirements] = None, connector: Optional[UUID] = None, connector_resource_id: Optional[str] = None, *args: Any, **kwargs: Any)
Bases: StackComponent
, ABC
Base class for all ZenML step operators.
Source code in src/zenml/stack/stack_component.py
328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 |
|
Attributes
config: BaseStepOperatorConfig
property
Returns the config of the step operator.
Returns:
Type | Description |
---|---|
BaseStepOperatorConfig
|
The config of the step operator. |
entrypoint_config_class: Type[StepOperatorEntrypointConfiguration]
property
Returns the entrypoint configuration class for this step operator.
Concrete step operator implementations may override this property to return a custom entrypoint configuration class if they need to customize the entrypoint configuration.
Returns:
Type | Description |
---|---|
Type[StepOperatorEntrypointConfiguration]
|
The entrypoint configuration class for this step operator. |
Functions
launch(info: StepRunInfo, entrypoint_command: List[str], environment: Dict[str, str]) -> None
abstractmethod
Abstract method to execute a step.
Subclasses must implement this method and launch a synchronous
job that executes the entrypoint_command
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
info
|
StepRunInfo
|
Information about the step run. |
required |
entrypoint_command
|
List[str]
|
Command that executes the step. |
required |
environment
|
Dict[str, str]
|
Environment variables to set in the step operator environment. |
required |
Source code in src/zenml/step_operators/base_step_operator.py
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
|
BaseStepOperatorConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)
Bases: StackComponentConfig
Base config for step operators.
Source code in src/zenml/stack/stack_component.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
|
BaseStepOperatorFlavor
Bases: Flavor
Base class for all ZenML step operator flavors.
Attributes
config_class: Type[BaseStepOperatorConfig]
property
Returns the config class for this flavor.
Returns:
Type | Description |
---|---|
Type[BaseStepOperatorConfig]
|
The config class for this flavor. |
implementation_class: Type[BaseStepOperator]
abstractmethod
property
Returns the implementation class for this flavor.
Returns:
Type | Description |
---|---|
Type[BaseStepOperator]
|
The implementation class for this flavor. |
type: StackComponentType
property
Modules
base_step_operator
Base class for ZenML step operators.
Classes
BaseStepOperator(name: str, id: UUID, config: StackComponentConfig, flavor: str, type: StackComponentType, user: Optional[UUID], created: datetime, updated: datetime, labels: Optional[Dict[str, Any]] = None, connector_requirements: Optional[ServiceConnectorRequirements] = None, connector: Optional[UUID] = None, connector_resource_id: Optional[str] = None, *args: Any, **kwargs: Any)
Bases: StackComponent
, ABC
Base class for all ZenML step operators.
Source code in src/zenml/stack/stack_component.py
328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 |
|
config: BaseStepOperatorConfig
property
Returns the config of the step operator.
Returns:
Type | Description |
---|---|
BaseStepOperatorConfig
|
The config of the step operator. |
entrypoint_config_class: Type[StepOperatorEntrypointConfiguration]
property
Returns the entrypoint configuration class for this step operator.
Concrete step operator implementations may override this property to return a custom entrypoint configuration class if they need to customize the entrypoint configuration.
Returns:
Type | Description |
---|---|
Type[StepOperatorEntrypointConfiguration]
|
The entrypoint configuration class for this step operator. |
launch(info: StepRunInfo, entrypoint_command: List[str], environment: Dict[str, str]) -> None
abstractmethod
Abstract method to execute a step.
Subclasses must implement this method and launch a synchronous
job that executes the entrypoint_command
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
info
|
StepRunInfo
|
Information about the step run. |
required |
entrypoint_command
|
List[str]
|
Command that executes the step. |
required |
environment
|
Dict[str, str]
|
Environment variables to set in the step operator environment. |
required |
Source code in src/zenml/step_operators/base_step_operator.py
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
|
BaseStepOperatorConfig(warn_about_plain_text_secrets: bool = False, **kwargs: Any)
Bases: StackComponentConfig
Base config for step operators.
Source code in src/zenml/stack/stack_component.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
|
BaseStepOperatorFlavor
Bases: Flavor
Base class for all ZenML step operator flavors.
config_class: Type[BaseStepOperatorConfig]
property
Returns the config class for this flavor.
Returns:
Type | Description |
---|---|
Type[BaseStepOperatorConfig]
|
The config class for this flavor. |
implementation_class: Type[BaseStepOperator]
abstractmethod
property
Returns the implementation class for this flavor.
Returns:
Type | Description |
---|---|
Type[BaseStepOperator]
|
The implementation class for this flavor. |
type: StackComponentType
property
Functions
step_operator_entrypoint_configuration
Abstract base class for entrypoint configurations that run a single step.
Classes
StepOperatorEntrypointConfiguration(arguments: List[str])
Bases: StepEntrypointConfiguration
Base class for step operator entrypoint configurations.
Source code in src/zenml/entrypoints/base_entrypoint_configuration.py
60 61 62 63 64 65 66 |
|
get_entrypoint_arguments(**kwargs: Any) -> List[str]
classmethod
Gets all arguments that the entrypoint command should be called with.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
**kwargs
|
Any
|
Kwargs, must include the step run id. |
{}
|
Returns:
Type | Description |
---|---|
List[str]
|
The superclass arguments as well as arguments for the step run id. |
Source code in src/zenml/step_operators/step_operator_entrypoint_configuration.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
|
get_entrypoint_options() -> Set[str]
classmethod
Gets all options required for running with this configuration.
Returns:
Type | Description |
---|---|
Set[str]
|
The superclass options as well as an option for the step run id. |
Source code in src/zenml/step_operators/step_operator_entrypoint_configuration.py
38 39 40 41 42 43 44 45 46 47 |
|