Stack Deployments
zenml.stack_deployments
special
ZenML Stack Deployments.
aws_stack_deployment
Functionality to deploy a ZenML stack to AWS.
AWSZenMLCloudStackDeployment (ZenMLCloudStackDeployment)
AWS ZenML Cloud Stack Deployment.
Source code in zenml/stack_deployments/aws_stack_deployment.py
class AWSZenMLCloudStackDeployment(ZenMLCloudStackDeployment):
"""AWS ZenML Cloud Stack Deployment."""
provider: ClassVar[StackDeploymentProvider] = StackDeploymentProvider.AWS
deployment: ClassVar[str] = AWS_DEPLOYMENT_TYPE
@classmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
return """
Provision and register a basic AWS ZenML stack authenticated and connected to
all the necessary cloud infrastructure resources required to run pipelines in
AWS.
"""
@classmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
return """
You will be redirected to the AWS console in your browser where you'll be asked
to log into your AWS account and create a CloudFormation ZenML stack. The stack
parameters will be pre-filled with the necessary information to connect ZenML to
your AWS account, so you should only need to review and confirm the stack.
**NOTE**: The CloudFormation stack will create the following new resources in
your AWS account. Please ensure you have the necessary permissions and are aware
of any potential costs:
- An S3 bucket registered as a [ZenML artifact store](https://docs.zenml.io/stack-components/artifact-stores/s3).
- An ECR repository registered as a [ZenML container registry](https://docs.zenml.io/stack-components/container-registries/aws).
- Sagemaker registered as a [ZenML orchestrator](https://docs.zenml.io/stack-components/orchestrators/sagemaker)
as well as a [ZenML step operator](https://docs.zenml.io/stack-components/step-operators/sagemaker).
- A CodeBuild project registered as a [ZenML image builder](https://docs.zenml.io/stack-components/image-builder/aws).
- An IAM user and IAM role with the minimum necessary permissions to access the
above resources.
- An AWS access key used to give access to ZenML to connect to the above
resources through a [ZenML service connector](https://docs.zenml.io/how-to/auth-management/aws-service-connector).
The CloudFormation stack will automatically create an AWS secret key and
will share it with ZenML to give it permission to access the resources created
by the stack. You can revoke these permissions at any time by deleting the
CloudFormation stack.
**Estimated costs**
A small training job would cost around: $0.60
These are rough estimates and actual costs may vary based on your usage and specific AWS pricing.
Some services may be eligible for the AWS Free Tier. Use [the AWS Pricing Calculator](https://calculator.aws)
for a detailed estimate based on your usage.
💡 **After the CloudFormation stack is deployed, you can return to the CLI to
view details about the associated ZenML stack automatically registered with
ZenML.**
"""
@classmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
return """
The ZenML stack has been successfully deployed and registered. You can delete
the CloudFormation at any time to revoke ZenML's access to your AWS account and
to clean up the resources created by the stack by using the AWS CloudFormation
console.
"""
@classmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
return [
"aws",
"s3",
]
@classmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
return {
"S3 Bucket": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetBucketVersioning",
"s3:ListBucketVersions",
"s3:DeleteObjectVersion",
],
"ECR Repository": [
"ecr:DescribeRepositories",
"ecr:ListRepositories",
"ecr:DescribeRegistry",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage",
"ecr:GetAuthorizationToken",
],
"CloudBuild (Client)": [
"codebuild:CreateProject",
"codebuild:BatchGetBuilds",
],
"CloudBuild (Service)": [
"s3:GetObject",
"s3:GetObjectVersion",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage",
"ecr:GetAuthorizationToken",
],
"SageMaker (Client)": [
"sagemaker:CreatePipeline",
"sagemaker:StartPipelineExecution",
"sagemaker:DescribePipeline",
"sagemaker:DescribePipelineExecution",
],
"SageMaker (Jobs)": [
"AmazonSageMakerFullAccess",
],
}
@classmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
# Return a list of all possible AWS regions
# Based on the AWS regions listed at
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
return {
"US East (Ohio)": "us-east-2",
"US East (N. Virginia)": "us-east-1",
"US West (N. California)": "us-west-1",
"US West (Oregon)": "us-west-2",
"Africa (Cape Town)": "af-south-1",
"Asia Pacific (Hong Kong)": "ap-east-1",
"Asia Pacific (Hyderabad)": "ap-south-2",
"Asia Pacific (Jakarta)": "ap-southeast-3",
"Asia Pacific (Melbourne)": "ap-southeast-4",
"Asia Pacific (Mumbai)": "ap-south-1",
"Asia Pacific (Osaka)": "ap-northeast-3",
"Asia Pacific (Seoul)": "ap-northeast-2",
"Asia Pacific (Singapore)": "ap-southeast-1",
"Asia Pacific (Sydney)": "ap-southeast-2",
"Asia Pacific (Tokyo)": "ap-northeast-1",
"Canada (Central)": "ca-central-1",
"Canada West (Calgary)": "ca-west-1",
"Europe (Frankfurt)": "eu-central-1",
"Europe (Ireland)": "eu-west-1",
"Europe (London)": "eu-west-2",
"Europe (Milan)": "eu-south-1",
"Europe (Paris)": "eu-west-3",
"Europe (Spain)": "eu-south-2",
"Europe (Stockholm)": "eu-north-1",
"Europe (Zurich)": "eu-central-2",
"Israel (Tel Aviv)": "il-central-1",
"Middle East (Bahrain)": "me-south-1",
"Middle East (UAE)": "me-central-1",
"South America (São Paulo)": "sa-east-1",
}
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* a Terraform script used to deploy the ZenML stack
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
params = dict(
stackName=self.stack_name,
templateURL="https://zenml-cf-templates.s3.eu-central-1.amazonaws.com/aws-ecr-s3-sagemaker.yaml",
param_ResourceName=f"zenml-{random_str(6).lower()}",
param_ZenMLServerURL=self.zenml_server_url,
param_ZenMLServerAPIToken=self.zenml_server_api_token,
param_CodeBuild="true",
)
# Encode the parameters as URL query parameters
query_params = "&".join([f"{k}={v}" for k, v in params.items()])
region = ""
if self.location:
region = f"region={self.location}"
url = (
f"https://console.aws.amazon.com/cloudformation/home?"
f"{region}#/stacks/create/review?{query_params}"
)
config: Optional[str] = None
if self.deployment_type == STACK_DEPLOYMENT_TERRAFORM:
config = f"""terraform {{
required_providers {{
aws = {{
source = "hashicorp/aws"
}}
zenml = {{
source = "zenml-io/zenml"
}}
}}
}}
provider "aws" {{
region = "{self.location or "eu-central-1"}"
}}
provider "zenml" {{
server_url = "{self.zenml_server_url}"
api_token = "{self.zenml_server_api_token}"
}}
module "zenml_stack" {{
source = "zenml-io/zenml-stack/aws"
zenml_stack_name = "{self.stack_name}"
zenml_stack_deployment = "{self.deployment_type}"
}}
output "zenml_stack_id" {{
value = module.zenml_stack.zenml_stack_id
}}
output "zenml_stack_name" {{
value = module.zenml_stack.zenml_stack_name
}}"""
return StackDeploymentConfig(
deployment_url=url,
deployment_url_text="AWS CloudFormation Console",
configuration=config,
)
description()
classmethod
Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
A MarkDown description of the ZenML Cloud Stack Deployment. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
@classmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
return """
Provision and register a basic AWS ZenML stack authenticated and connected to
all the necessary cloud infrastructure resources required to run pipelines in
AWS.
"""
get_deployment_config(self)
Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
- a cloud provider console URL where the user will be redirected to deploy the ZenML stack. The URL should include as many pre-filled URL query parameters as possible.
- a textual description of the URL
- a Terraform script used to deploy the ZenML stack
- some deployment providers may require additional configuration parameters or scripts to be passed to the cloud provider in addition to the deployment URL query parameters. Where that is the case, this method should also return a string that the user can copy and paste into the cloud provider console to deploy the ZenML stack (e.g. a set of environment variables, YAML configuration snippet, bash or Terraform script etc.).
Returns:
| Type | Description |
|---|---|
StackDeploymentConfig |
The configuration or script to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* a Terraform script used to deploy the ZenML stack
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
params = dict(
stackName=self.stack_name,
templateURL="https://zenml-cf-templates.s3.eu-central-1.amazonaws.com/aws-ecr-s3-sagemaker.yaml",
param_ResourceName=f"zenml-{random_str(6).lower()}",
param_ZenMLServerURL=self.zenml_server_url,
param_ZenMLServerAPIToken=self.zenml_server_api_token,
param_CodeBuild="true",
)
# Encode the parameters as URL query parameters
query_params = "&".join([f"{k}={v}" for k, v in params.items()])
region = ""
if self.location:
region = f"region={self.location}"
url = (
f"https://console.aws.amazon.com/cloudformation/home?"
f"{region}#/stacks/create/review?{query_params}"
)
config: Optional[str] = None
if self.deployment_type == STACK_DEPLOYMENT_TERRAFORM:
config = f"""terraform {{
required_providers {{
aws = {{
source = "hashicorp/aws"
}}
zenml = {{
source = "zenml-io/zenml"
}}
}}
}}
provider "aws" {{
region = "{self.location or "eu-central-1"}"
}}
provider "zenml" {{
server_url = "{self.zenml_server_url}"
api_token = "{self.zenml_server_api_token}"
}}
module "zenml_stack" {{
source = "zenml-io/zenml-stack/aws"
zenml_stack_name = "{self.stack_name}"
zenml_stack_deployment = "{self.deployment_type}"
}}
output "zenml_stack_id" {{
value = module.zenml_stack.zenml_stack_id
}}
output "zenml_stack_name" {{
value = module.zenml_stack.zenml_stack_name
}}"""
return StackDeploymentConfig(
deployment_url=url,
deployment_url_text="AWS CloudFormation Console",
configuration=config,
)
instructions()
classmethod
Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on how to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
@classmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
return """
You will be redirected to the AWS console in your browser where you'll be asked
to log into your AWS account and create a CloudFormation ZenML stack. The stack
parameters will be pre-filled with the necessary information to connect ZenML to
your AWS account, so you should only need to review and confirm the stack.
**NOTE**: The CloudFormation stack will create the following new resources in
your AWS account. Please ensure you have the necessary permissions and are aware
of any potential costs:
- An S3 bucket registered as a [ZenML artifact store](https://docs.zenml.io/stack-components/artifact-stores/s3).
- An ECR repository registered as a [ZenML container registry](https://docs.zenml.io/stack-components/container-registries/aws).
- Sagemaker registered as a [ZenML orchestrator](https://docs.zenml.io/stack-components/orchestrators/sagemaker)
as well as a [ZenML step operator](https://docs.zenml.io/stack-components/step-operators/sagemaker).
- A CodeBuild project registered as a [ZenML image builder](https://docs.zenml.io/stack-components/image-builder/aws).
- An IAM user and IAM role with the minimum necessary permissions to access the
above resources.
- An AWS access key used to give access to ZenML to connect to the above
resources through a [ZenML service connector](https://docs.zenml.io/how-to/auth-management/aws-service-connector).
The CloudFormation stack will automatically create an AWS secret key and
will share it with ZenML to give it permission to access the resources created
by the stack. You can revoke these permissions at any time by deleting the
CloudFormation stack.
**Estimated costs**
A small training job would cost around: $0.60
These are rough estimates and actual costs may vary based on your usage and specific AWS pricing.
Some services may be eligible for the AWS Free Tier. Use [the AWS Pricing Calculator](https://calculator.aws)
for a detailed estimate based on your usage.
💡 **After the CloudFormation stack is deployed, you can return to the CLI to
view details about the associated ZenML stack automatically registered with
ZenML.**
"""
integrations()
classmethod
Return the ZenML integrations required for the stack.
Returns:
| Type | Description |
|---|---|
List[str] |
The list of ZenML integrations that need to be installed for the stack to be usable. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
@classmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
return [
"aws",
"s3",
]
locations()
classmethod
Return the locations where the ZenML stack can be deployed.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions where the ZenML stack can be deployed as a map of region names to region descriptions. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
@classmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
# Return a list of all possible AWS regions
# Based on the AWS regions listed at
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
return {
"US East (Ohio)": "us-east-2",
"US East (N. Virginia)": "us-east-1",
"US West (N. California)": "us-west-1",
"US West (Oregon)": "us-west-2",
"Africa (Cape Town)": "af-south-1",
"Asia Pacific (Hong Kong)": "ap-east-1",
"Asia Pacific (Hyderabad)": "ap-south-2",
"Asia Pacific (Jakarta)": "ap-southeast-3",
"Asia Pacific (Melbourne)": "ap-southeast-4",
"Asia Pacific (Mumbai)": "ap-south-1",
"Asia Pacific (Osaka)": "ap-northeast-3",
"Asia Pacific (Seoul)": "ap-northeast-2",
"Asia Pacific (Singapore)": "ap-southeast-1",
"Asia Pacific (Sydney)": "ap-southeast-2",
"Asia Pacific (Tokyo)": "ap-northeast-1",
"Canada (Central)": "ca-central-1",
"Canada West (Calgary)": "ca-west-1",
"Europe (Frankfurt)": "eu-central-1",
"Europe (Ireland)": "eu-west-1",
"Europe (London)": "eu-west-2",
"Europe (Milan)": "eu-south-1",
"Europe (Paris)": "eu-west-3",
"Europe (Spain)": "eu-south-2",
"Europe (Stockholm)": "eu-north-1",
"Europe (Zurich)": "eu-central-2",
"Israel (Tel Aviv)": "il-central-1",
"Middle East (Bahrain)": "me-south-1",
"Middle East (UAE)": "me-central-1",
"South America (São Paulo)": "sa-east-1",
}
permissions()
classmethod
Return the permissions granted to ZenML to access the cloud resources.
Returns:
| Type | Description |
|---|---|
Dict[str, List[str]] |
The permissions granted to ZenML to access the cloud resources, as a dictionary grouping permissions by resource. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
@classmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
return {
"S3 Bucket": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetBucketVersioning",
"s3:ListBucketVersions",
"s3:DeleteObjectVersion",
],
"ECR Repository": [
"ecr:DescribeRepositories",
"ecr:ListRepositories",
"ecr:DescribeRegistry",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage",
"ecr:GetAuthorizationToken",
],
"CloudBuild (Client)": [
"codebuild:CreateProject",
"codebuild:BatchGetBuilds",
],
"CloudBuild (Service)": [
"s3:GetObject",
"s3:GetObjectVersion",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage",
"ecr:GetAuthorizationToken",
],
"SageMaker (Client)": [
"sagemaker:CreatePipeline",
"sagemaker:StartPipelineExecution",
"sagemaker:DescribePipeline",
"sagemaker:DescribePipelineExecution",
],
"SageMaker (Jobs)": [
"AmazonSageMakerFullAccess",
],
}
post_deploy_instructions()
classmethod
Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on what to do after the deployment is complete. |
Source code in zenml/stack_deployments/aws_stack_deployment.py
@classmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
return """
The ZenML stack has been successfully deployed and registered. You can delete
the CloudFormation at any time to revoke ZenML's access to your AWS account and
to clean up the resources created by the stack by using the AWS CloudFormation
console.
"""
azure_stack_deployment
Functionality to deploy a ZenML stack to Azure.
AZUREZenMLCloudStackDeployment (ZenMLCloudStackDeployment)
Azure ZenML Cloud Stack Deployment.
Source code in zenml/stack_deployments/azure_stack_deployment.py
class AZUREZenMLCloudStackDeployment(ZenMLCloudStackDeployment):
"""Azure ZenML Cloud Stack Deployment."""
provider: ClassVar[StackDeploymentProvider] = StackDeploymentProvider.AZURE
deployment: ClassVar[str] = AZURE_DEPLOYMENT_TYPE
@classmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
return """
Provision and register a basic Azure ZenML stack authenticated and connected to
all the necessary cloud infrastructure resources required to run pipelines in
Azure.
"""
@classmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
return """
You will be redirected to an Azure Cloud Shell console in your browser where
you'll be asked to log into your Azure project and then use
[the Azure ZenML Stack Terraform module](https://registry.terraform.io/modules/zenml-io/zenml-stack/azure)
to provision the necessary cloud resources for ZenML.
**NOTE**: The Azure ZenML Stack Terraform module will create the following new
resources in your Azure subscription. Please ensure you have the necessary
permissions and are aware of any potential costs:
- An Azure Resource Group to contain all the resources required for the ZenML stack
- An Azure Storage Account and Blob Storage Container registered as a [ZenML artifact store](https://docs.zenml.io/stack-components/artifact-stores/azure).
- An Azure Container Registry registered as a [ZenML container registry](https://docs.zenml.io/stack-components/container-registries/azure).
- An AzureML Workspace registered as both a [ZenML orchestrator](https://docs.zenml.io/stack-components/orchestrators/azureml) and a
[ZenML step operator](https://docs.zenml.io/stack-components/step-operators/azureml) and used to run pipelines.
A Key Vault and Application Insights instance will also be created in the same Resource Group and used to construct the AzureML Workspace.
- An Azure Service Principal with the minimum necessary permissions to access
the above resources.
- An Azure Service Principal client secret used to give access to ZenML to
connect to the above resources through a [ZenML service connector](https://docs.zenml.io/how-to/auth-management/azure-service-connector).
The Azure ZenML Stack Terraform module will automatically create an Azure
Service Principal client secret and will share it with ZenML to give it
permission to access the resources created by the stack. You can revoke these
permissions at any time by deleting the Service Principal in your Azure
subscription.
**Estimated costs**
A small training job would cost around: $0.60
These are rough estimates and actual costs may vary based on your usage and specific Azure pricing.
Some services may be eligible for the Azure Free Tier. Use [the Azure Pricing Calculator](https://azure.microsoft.com/en-us/pricing/calculator)
for a detailed estimate based on your usage.
💡 **After the Terraform deployment is complete, you can close the Cloud
Shell session and return to the CLI to view details about the associated ZenML
stack automatically registered with ZenML.**
"""
@classmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
return """
The ZenML stack has been successfully deployed and registered. You can delete
the provisioned Service Principal and Resource Group at any time to revoke
ZenML's access to your Azure subscription.
"""
@classmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
return ["azure"]
@classmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
return {
"Storage Account": [
"Storage Blob Data Contributor",
],
"Container Registry": [
"AcrPull",
"AcrPush",
"Contributor",
],
"AzureML Workspace": [
"AzureML Compute Operator",
"AzureML Data Scientist",
],
}
@classmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
# Based on `az account list-locations -o table` on 16.07.2024
return {
"(US) East US": "eastus",
"(US) South Central US": "southcentralus",
"(US) West US 2": "westus2",
"(US) West US 3": "westus3",
"(Asia Pacific) Australia East": "australiaeast",
"(Asia Pacific) Southeast Asia": "southeastasia",
"(Europe) North Europe": "northeurope",
"(Europe) Sweden Central": "swedencentral",
"(Europe) UK South": "uksouth",
"(Europe) West Europe": "westeurope",
"(US) Central US": "centralus",
"(Africa) South Africa North": "southafricanorth",
"(Asia Pacific) Central India": "centralindia",
"(Asia Pacific) East Asia": "eastasia",
"(Asia Pacific) Japan East": "japaneast",
"(Asia Pacific) Korea Central": "koreacentral",
"(Canada) Canada Central": "canadacentral",
"(Europe) France Central": "francecentral",
"(Europe) Germany West Central": "germanywestcentral",
"(Europe) Italy North": "italynorth",
"(Europe) Norway East": "norwayeast",
"(Europe) Poland Central": "polandcentral",
"(Europe) Spain Central": "spaincentral",
"(Europe) Switzerland North": "switzerlandnorth",
"(Mexico) Mexico Central": "mexicocentral",
"(Middle East) UAE North": "uaenorth",
"(South America) Brazil South": "brazilsouth",
"(Middle East) Israel Central": "israelcentral",
"(Middle East) Qatar Central": "qatarcentral",
"(US) Central US (Stage)": "centralusstage",
"(US) East US (Stage)": "eastusstage",
"(US) East US 2 (Stage)": "eastus2stage",
"(US) North Central US (Stage)": "northcentralusstage",
"(US) South Central US (Stage)": "southcentralusstage",
"(US) West US (Stage)": "westusstage",
"(US) West US 2 (Stage)": "westus2stage",
"(Asia Pacific) East Asia (Stage)": "eastasiastage",
"(Asia Pacific) Southeast Asia (Stage)": "southeastasiastage",
"(South America) Brazil US": "brazilus",
"(US) East US 2": "eastus2",
"(US) East US STG": "eastusstg",
"(US) North Central US": "northcentralus",
"(US) West US": "westus",
"(Asia Pacific) Japan West": "japanwest",
"(Asia Pacific) Jio India West": "jioindiawest",
"(US) Central US EUAP": "centraluseuap",
"(US) East US 2 EUAP": "eastus2euap",
"(US) West Central US": "westcentralus",
"(Africa) South Africa West": "southafricawest",
"(Asia Pacific) Australia Central": "australiacentral",
"(Asia Pacific) Australia Central 2": "australiacentral2",
"(Asia Pacific) Australia Southeast": "australiasoutheast",
"(Asia Pacific) Jio India Central": "jioindiacentral",
"(Asia Pacific) Korea South": "koreasouth",
"(Asia Pacific) South India": "southindia",
"(Asia Pacific) West India": "westindia",
"(Canada) Canada East": "canadaeast",
"(Europe) France South": "francesouth",
"(Europe) Germany North": "germanynorth",
"(Europe) Norway West": "norwaywest",
"(Europe) Switzerland West": "switzerlandwest",
"(Europe) UK West": "ukwest",
"(Middle East) UAE Central": "uaecentral",
"(South America) Brazil Southeast": "brazilsoutheast",
}
@classmethod
def skypilot_default_regions(cls) -> Dict[str, str]:
"""Returns the regions supported by default for the Skypilot.
Returns:
The regions supported by default for the Skypilot.
"""
matcher = re.compile(r".*us\d*( |$)")
return {
k: v
for k, v in cls.locations().items()
if "(US)" in k and matcher.match(v)
}
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
config = f"""terraform {{
required_providers {{
azurerm = {{
source = "hashicorp/azurerm"
}}
azuread = {{
source = "hashicorp/azuread"
}}
zenml = {{
source = "zenml-io/zenml"
}}
}}
}}
provider "azurerm" {{
features {{
resource_group {{
prevent_deletion_if_contains_resources = false
}}
}}
}}
provider "zenml" {{
server_url = "{self.zenml_server_url}"
api_token = "{self.zenml_server_api_token}"
}}
module "zenml_stack" {{
source = "zenml-io/zenml-stack/azure"
location = "{self.location or "eastus"}"
zenml_stack_name = "{self.stack_name}"
zenml_stack_deployment = "{self.deployment_type}"
}}
output "zenml_stack_id" {{
value = module.zenml_stack.zenml_stack_id
}}
output "zenml_stack_name" {{
value = module.zenml_stack.zenml_stack_name
}}"""
instructions = """
1. The Azure Cloud Shell console will open in your browser.
2. Create a file named `main.tf` in the Cloud Shell and copy and paste the
Terraform configuration below into it.
3. Run `terraform init --upgrade` to initialize the Terraform configuration.
4. Run `terraform apply` to deploy the ZenML stack to Azure.
"""
return StackDeploymentConfig(
deployment_url="https://shell.azure.com",
deployment_url_text="Azure Cloud Shell Console",
configuration=config,
instructions=instructions,
)
description()
classmethod
Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
A MarkDown description of the ZenML Cloud Stack Deployment. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
return """
Provision and register a basic Azure ZenML stack authenticated and connected to
all the necessary cloud infrastructure resources required to run pipelines in
Azure.
"""
get_deployment_config(self)
Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
- a cloud provider console URL where the user will be redirected to deploy the ZenML stack. The URL should include as many pre-filled URL query parameters as possible.
- a textual description of the URL
- some deployment providers may require additional configuration parameters or scripts to be passed to the cloud provider in addition to the deployment URL query parameters. Where that is the case, this method should also return a string that the user can copy and paste into the cloud provider console to deploy the ZenML stack (e.g. a set of environment variables, YAML configuration snippet, bash or Terraform script etc.).
Returns:
| Type | Description |
|---|---|
StackDeploymentConfig |
The configuration or script to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
config = f"""terraform {{
required_providers {{
azurerm = {{
source = "hashicorp/azurerm"
}}
azuread = {{
source = "hashicorp/azuread"
}}
zenml = {{
source = "zenml-io/zenml"
}}
}}
}}
provider "azurerm" {{
features {{
resource_group {{
prevent_deletion_if_contains_resources = false
}}
}}
}}
provider "zenml" {{
server_url = "{self.zenml_server_url}"
api_token = "{self.zenml_server_api_token}"
}}
module "zenml_stack" {{
source = "zenml-io/zenml-stack/azure"
location = "{self.location or "eastus"}"
zenml_stack_name = "{self.stack_name}"
zenml_stack_deployment = "{self.deployment_type}"
}}
output "zenml_stack_id" {{
value = module.zenml_stack.zenml_stack_id
}}
output "zenml_stack_name" {{
value = module.zenml_stack.zenml_stack_name
}}"""
instructions = """
1. The Azure Cloud Shell console will open in your browser.
2. Create a file named `main.tf` in the Cloud Shell and copy and paste the
Terraform configuration below into it.
3. Run `terraform init --upgrade` to initialize the Terraform configuration.
4. Run `terraform apply` to deploy the ZenML stack to Azure.
"""
return StackDeploymentConfig(
deployment_url="https://shell.azure.com",
deployment_url_text="Azure Cloud Shell Console",
configuration=config,
instructions=instructions,
)
instructions()
classmethod
Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on how to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
return """
You will be redirected to an Azure Cloud Shell console in your browser where
you'll be asked to log into your Azure project and then use
[the Azure ZenML Stack Terraform module](https://registry.terraform.io/modules/zenml-io/zenml-stack/azure)
to provision the necessary cloud resources for ZenML.
**NOTE**: The Azure ZenML Stack Terraform module will create the following new
resources in your Azure subscription. Please ensure you have the necessary
permissions and are aware of any potential costs:
- An Azure Resource Group to contain all the resources required for the ZenML stack
- An Azure Storage Account and Blob Storage Container registered as a [ZenML artifact store](https://docs.zenml.io/stack-components/artifact-stores/azure).
- An Azure Container Registry registered as a [ZenML container registry](https://docs.zenml.io/stack-components/container-registries/azure).
- An AzureML Workspace registered as both a [ZenML orchestrator](https://docs.zenml.io/stack-components/orchestrators/azureml) and a
[ZenML step operator](https://docs.zenml.io/stack-components/step-operators/azureml) and used to run pipelines.
A Key Vault and Application Insights instance will also be created in the same Resource Group and used to construct the AzureML Workspace.
- An Azure Service Principal with the minimum necessary permissions to access
the above resources.
- An Azure Service Principal client secret used to give access to ZenML to
connect to the above resources through a [ZenML service connector](https://docs.zenml.io/how-to/auth-management/azure-service-connector).
The Azure ZenML Stack Terraform module will automatically create an Azure
Service Principal client secret and will share it with ZenML to give it
permission to access the resources created by the stack. You can revoke these
permissions at any time by deleting the Service Principal in your Azure
subscription.
**Estimated costs**
A small training job would cost around: $0.60
These are rough estimates and actual costs may vary based on your usage and specific Azure pricing.
Some services may be eligible for the Azure Free Tier. Use [the Azure Pricing Calculator](https://azure.microsoft.com/en-us/pricing/calculator)
for a detailed estimate based on your usage.
💡 **After the Terraform deployment is complete, you can close the Cloud
Shell session and return to the CLI to view details about the associated ZenML
stack automatically registered with ZenML.**
"""
integrations()
classmethod
Return the ZenML integrations required for the stack.
Returns:
| Type | Description |
|---|---|
List[str] |
The list of ZenML integrations that need to be installed for the stack to be usable. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
return ["azure"]
locations()
classmethod
Return the locations where the ZenML stack can be deployed.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions where the ZenML stack can be deployed as a map of region names to region descriptions. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
# Based on `az account list-locations -o table` on 16.07.2024
return {
"(US) East US": "eastus",
"(US) South Central US": "southcentralus",
"(US) West US 2": "westus2",
"(US) West US 3": "westus3",
"(Asia Pacific) Australia East": "australiaeast",
"(Asia Pacific) Southeast Asia": "southeastasia",
"(Europe) North Europe": "northeurope",
"(Europe) Sweden Central": "swedencentral",
"(Europe) UK South": "uksouth",
"(Europe) West Europe": "westeurope",
"(US) Central US": "centralus",
"(Africa) South Africa North": "southafricanorth",
"(Asia Pacific) Central India": "centralindia",
"(Asia Pacific) East Asia": "eastasia",
"(Asia Pacific) Japan East": "japaneast",
"(Asia Pacific) Korea Central": "koreacentral",
"(Canada) Canada Central": "canadacentral",
"(Europe) France Central": "francecentral",
"(Europe) Germany West Central": "germanywestcentral",
"(Europe) Italy North": "italynorth",
"(Europe) Norway East": "norwayeast",
"(Europe) Poland Central": "polandcentral",
"(Europe) Spain Central": "spaincentral",
"(Europe) Switzerland North": "switzerlandnorth",
"(Mexico) Mexico Central": "mexicocentral",
"(Middle East) UAE North": "uaenorth",
"(South America) Brazil South": "brazilsouth",
"(Middle East) Israel Central": "israelcentral",
"(Middle East) Qatar Central": "qatarcentral",
"(US) Central US (Stage)": "centralusstage",
"(US) East US (Stage)": "eastusstage",
"(US) East US 2 (Stage)": "eastus2stage",
"(US) North Central US (Stage)": "northcentralusstage",
"(US) South Central US (Stage)": "southcentralusstage",
"(US) West US (Stage)": "westusstage",
"(US) West US 2 (Stage)": "westus2stage",
"(Asia Pacific) East Asia (Stage)": "eastasiastage",
"(Asia Pacific) Southeast Asia (Stage)": "southeastasiastage",
"(South America) Brazil US": "brazilus",
"(US) East US 2": "eastus2",
"(US) East US STG": "eastusstg",
"(US) North Central US": "northcentralus",
"(US) West US": "westus",
"(Asia Pacific) Japan West": "japanwest",
"(Asia Pacific) Jio India West": "jioindiawest",
"(US) Central US EUAP": "centraluseuap",
"(US) East US 2 EUAP": "eastus2euap",
"(US) West Central US": "westcentralus",
"(Africa) South Africa West": "southafricawest",
"(Asia Pacific) Australia Central": "australiacentral",
"(Asia Pacific) Australia Central 2": "australiacentral2",
"(Asia Pacific) Australia Southeast": "australiasoutheast",
"(Asia Pacific) Jio India Central": "jioindiacentral",
"(Asia Pacific) Korea South": "koreasouth",
"(Asia Pacific) South India": "southindia",
"(Asia Pacific) West India": "westindia",
"(Canada) Canada East": "canadaeast",
"(Europe) France South": "francesouth",
"(Europe) Germany North": "germanynorth",
"(Europe) Norway West": "norwaywest",
"(Europe) Switzerland West": "switzerlandwest",
"(Europe) UK West": "ukwest",
"(Middle East) UAE Central": "uaecentral",
"(South America) Brazil Southeast": "brazilsoutheast",
}
permissions()
classmethod
Return the permissions granted to ZenML to access the cloud resources.
Returns:
| Type | Description |
|---|---|
Dict[str, List[str]] |
The permissions granted to ZenML to access the cloud resources, as a dictionary grouping permissions by resource. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
return {
"Storage Account": [
"Storage Blob Data Contributor",
],
"Container Registry": [
"AcrPull",
"AcrPush",
"Contributor",
],
"AzureML Workspace": [
"AzureML Compute Operator",
"AzureML Data Scientist",
],
}
post_deploy_instructions()
classmethod
Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on what to do after the deployment is complete. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
return """
The ZenML stack has been successfully deployed and registered. You can delete
the provisioned Service Principal and Resource Group at any time to revoke
ZenML's access to your Azure subscription.
"""
skypilot_default_regions()
classmethod
Returns the regions supported by default for the Skypilot.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions supported by default for the Skypilot. |
Source code in zenml/stack_deployments/azure_stack_deployment.py
@classmethod
def skypilot_default_regions(cls) -> Dict[str, str]:
"""Returns the regions supported by default for the Skypilot.
Returns:
The regions supported by default for the Skypilot.
"""
matcher = re.compile(r".*us\d*( |$)")
return {
k: v
for k, v in cls.locations().items()
if "(US)" in k and matcher.match(v)
}
gcp_stack_deployment
Functionality to deploy a ZenML stack to GCP.
GCPZenMLCloudStackDeployment (ZenMLCloudStackDeployment)
GCP ZenML Cloud Stack Deployment.
Source code in zenml/stack_deployments/gcp_stack_deployment.py
class GCPZenMLCloudStackDeployment(ZenMLCloudStackDeployment):
"""GCP ZenML Cloud Stack Deployment."""
provider: ClassVar[StackDeploymentProvider] = StackDeploymentProvider.GCP
deployment: ClassVar[str] = GCP_DEPLOYMENT_TYPE
@classmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
return """
Provision and register a basic GCP ZenML stack authenticated and connected to
all the necessary cloud infrastructure resources required to run pipelines in
GCP.
"""
@classmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
return """
You will be redirected to a GCP Cloud Shell console in your browser where you'll
be asked to log into your GCP project and then create a Deployment Manager
deployment to provision the necessary cloud resources for ZenML.
**NOTE**: The Deployment Manager deployment will create the following new
resources in your GCP project. Please ensure you have the necessary permissions
and are aware of any potential costs:
- A GCS bucket registered as a [ZenML artifact store](https://docs.zenml.io/stack-components/artifact-stores/gcp).
- A Google Artifact Registry registered as a [ZenML container registry](https://docs.zenml.io/stack-components/container-registries/gcp).
- Vertex AI registered as a [ZenML orchestrator](https://docs.zenml.io/stack-components/orchestrators/vertex)
and as a [ZenML step operator](https://docs.zenml.io/stack-components/step-operators/vertex).
- GCP Cloud Build registered as a [ZenML image builder](https://docs.zenml.io/stack-components/image-builders/gcp).
- A GCP Service Account with the minimum necessary permissions to access the
above resources.
- A GCP Service Account access key used to give access to ZenML to connect to
the above resources through a [ZenML service connector](https://docs.zenml.io/how-to/auth-management/gcp-service-connector).
The Deployment Manager deployment will automatically create a GCP Service
Account secret key and will share it with ZenML to give it permission to access
the resources created by the stack. You can revoke these permissions at any time
by deleting the Deployment Manager deployment in the GCP Cloud Console.
**Estimated costs**
A small training job would cost around: $0.60
These are rough estimates and actual costs may vary based on your usage and specific GCP pricing.
Some services may be eligible for the GCP Free Tier. Use [the GCP Pricing Calculator](https://cloud.google.com/products/calculator)
for a detailed estimate based on your usage.
⚠️ **The Cloud Shell session will warn you that the ZenML GitHub repository is
untrusted. We recommend that you review [the contents of the repository](https://github.com/zenml-io/zenml/tree/main/infra/gcp)
and then check the `Trust repo` checkbox to proceed with the deployment,
otherwise the Cloud Shell session will not be authenticated to access your
GCP projects. You will also get a chance to review the scripts that will be
executed in the Cloud Shell session before proceeding.**
💡 **After the Deployment Manager deployment is complete, you can close the Cloud
Shell session and return to the CLI to view details about the associated ZenML
stack automatically registered with ZenML.**
"""
@classmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
return """
The ZenML stack has been successfully deployed and registered. You can delete
the Deployment Manager deployment at any time to revoke ZenML's access to your
GCP project and to clean up the resources created by the stack by using
[the GCP Cloud Console](https://console.cloud.google.com/dm/deployments).
"""
@classmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
return [
"gcp",
]
@classmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
return {
"GCS Bucket": [
"roles/storage.objectUser",
],
"GCP Artifact Registry": [
"roles/artifactregistry.createOnPushWriter",
],
"Vertex AI (Client)": [
"roles/aiplatform.user",
],
"Vertex AI (Jobs)": [
"roles/aiplatform.serviceAgent",
],
"Cloud Build (Client)": [
"roles/cloudbuild.builds.editor",
],
}
@classmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
# Return a list of all possible GCP regions
# Based on the AWS regions listed at
# https://cloud.google.com/about/locations
return {
"Africa (Johannesburg)": "africa-south1",
"Asia Pacific (Taiwan)": "asia-east1",
"Asia Pacific (Hong Kong)": "asia-east2",
"Asia Pacific (Tokyo)": "asia-northeast1",
"Asia Pacific (Osaka)": "asia-northeast2",
"Asia Pacific (Seoul)": "asia-northeast3",
"Asia Pacific (Mumbai)": "asia-south1",
"Asia Pacific (Delhi)": "asia-south2",
"Asia Pacific (Singapore)": "asia-southeast1",
"Asia Pacific (Jakarta)": "asia-southeast2",
"Australia (Sydney)": "australia-southeast1",
"Australia (Melbourne)": "australia-southeast2",
"Europe (Belgium)": "europe-west1",
"Europe (London)": "europe-west2",
"Europe (Frankfurt)": "europe-west3",
"Europe (Netherlands)": "europe-west4",
"Europe (Zurich)": "europe-west6",
"Europe (Milan)": "europe-west8",
"Europe (Paris)": "europe-west9",
"Europe (Berlin)": "europe-west10",
"Europe (Turin)": "europe-west12",
"Europe (Warsaw)": "europe-central2",
"Europe (Finland)": "europe-north1",
"Europe (Madrid)": "europe-southwest1",
"Middle East (Doha)": "me-central1",
"Middle East (Dubai)": "me-central2",
"Middle East (Tel Aviv)": "me-west1",
"North America (Montreal)": "northamerica-northeast1",
"North America (Toronto)": "northamerica-northeast2",
"South America (Sao Paulo)": "southamerica-east1",
"South America (Santiago)": "southamerica-west1",
"US Central (Iowa)": "us-central1",
"US East (South Carolina)": "us-east1",
"US East (Northern Virginia)": "us-east4",
"US East (Columbus)": "us-east5",
"US South (Dallas)": "us-south1",
"US West (Oregon)": "us-west1",
"US West (Los Angeles)": "us-west2",
"US West (Salt Lake City)": "us-west3",
"US West (Las Vegas)": "us-west4",
}
@classmethod
def skypilot_default_regions(cls) -> Dict[str, str]:
"""Returns the regions supported by default for the Skypilot.
Returns:
The regions supported by default for the Skypilot.
"""
matcher = re.compile(r"us-.*")
return {k: v for k, v in cls.locations().items() if matcher.match(v)}
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
params = dict(
cloudshell_git_repo="https://github.com/zenml-io/zenml",
cloudshell_workspace="infra/gcp",
cloudshell_open_in_editor="gcp-gar-gcs-vertex.jinja,gcp-gar-gcs-vertex-deploy.sh",
cloudshell_tutorial="gcp-gar-gcs-vertex.md",
ephemeral="true",
)
# Encode the parameters as URL query parameters
query_params = "&".join([f"{k}={v}" for k, v in params.items()])
url = (
f"https://shell.cloud.google.com/cloudshell/editor?{query_params}"
)
if self.deployment_type == STACK_DEPLOYMENT_TERRAFORM:
config = f"""terraform {{
required_providers {{
google = {{
source = "hashicorp/google"
}}
zenml = {{
source = "zenml-io/zenml"
}}
}}
}}
provider "google" {{
region = "{self.location or "europe-west3"}"
project = your GCP project name
}}
provider "zenml" {{
server_url = "{self.zenml_server_url}"
api_token = "{self.zenml_server_api_token}"
}}
module "zenml_stack" {{
source = "zenml-io/zenml-stack/gcp"
zenml_stack_name = "{self.stack_name}"
zenml_stack_deployment = "{self.deployment_type}"
}}
output "zenml_stack_id" {{
value = module.zenml_stack.zenml_stack_id
}}
output "zenml_stack_name" {{
value = module.zenml_stack.zenml_stack_name
}}"""
else:
config = f"""
### BEGIN CONFIGURATION ###
ZENML_STACK_NAME={self.stack_name}
ZENML_STACK_REGION={self.location or "europe-west3"}
ZENML_SERVER_URL={self.zenml_server_url}
ZENML_SERVER_API_TOKEN={self.zenml_server_api_token}
### END CONFIGURATION ###"""
instructions = (
"You will be asked to provide the following configuration values "
"during the deployment process:"
)
return StackDeploymentConfig(
deployment_url=url,
deployment_url_text="GCP Cloud Shell Console",
configuration=config,
instructions=instructions,
)
description()
classmethod
Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
A MarkDown description of the ZenML Cloud Stack Deployment. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
return """
Provision and register a basic GCP ZenML stack authenticated and connected to
all the necessary cloud infrastructure resources required to run pipelines in
GCP.
"""
get_deployment_config(self)
Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
- a cloud provider console URL where the user will be redirected to deploy the ZenML stack. The URL should include as many pre-filled URL query parameters as possible.
- a textual description of the URL
- some deployment providers may require additional configuration parameters or scripts to be passed to the cloud provider in addition to the deployment URL query parameters. Where that is the case, this method should also return a string that the user can copy and paste into the cloud provider console to deploy the ZenML stack (e.g. a set of environment variables, YAML configuration snippet, bash or Terraform script etc.).
Returns:
| Type | Description |
|---|---|
StackDeploymentConfig |
The configuration or script to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
params = dict(
cloudshell_git_repo="https://github.com/zenml-io/zenml",
cloudshell_workspace="infra/gcp",
cloudshell_open_in_editor="gcp-gar-gcs-vertex.jinja,gcp-gar-gcs-vertex-deploy.sh",
cloudshell_tutorial="gcp-gar-gcs-vertex.md",
ephemeral="true",
)
# Encode the parameters as URL query parameters
query_params = "&".join([f"{k}={v}" for k, v in params.items()])
url = (
f"https://shell.cloud.google.com/cloudshell/editor?{query_params}"
)
if self.deployment_type == STACK_DEPLOYMENT_TERRAFORM:
config = f"""terraform {{
required_providers {{
google = {{
source = "hashicorp/google"
}}
zenml = {{
source = "zenml-io/zenml"
}}
}}
}}
provider "google" {{
region = "{self.location or "europe-west3"}"
project = your GCP project name
}}
provider "zenml" {{
server_url = "{self.zenml_server_url}"
api_token = "{self.zenml_server_api_token}"
}}
module "zenml_stack" {{
source = "zenml-io/zenml-stack/gcp"
zenml_stack_name = "{self.stack_name}"
zenml_stack_deployment = "{self.deployment_type}"
}}
output "zenml_stack_id" {{
value = module.zenml_stack.zenml_stack_id
}}
output "zenml_stack_name" {{
value = module.zenml_stack.zenml_stack_name
}}"""
else:
config = f"""
### BEGIN CONFIGURATION ###
ZENML_STACK_NAME={self.stack_name}
ZENML_STACK_REGION={self.location or "europe-west3"}
ZENML_SERVER_URL={self.zenml_server_url}
ZENML_SERVER_API_TOKEN={self.zenml_server_api_token}
### END CONFIGURATION ###"""
instructions = (
"You will be asked to provide the following configuration values "
"during the deployment process:"
)
return StackDeploymentConfig(
deployment_url=url,
deployment_url_text="GCP Cloud Shell Console",
configuration=config,
instructions=instructions,
)
instructions()
classmethod
Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on how to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
return """
You will be redirected to a GCP Cloud Shell console in your browser where you'll
be asked to log into your GCP project and then create a Deployment Manager
deployment to provision the necessary cloud resources for ZenML.
**NOTE**: The Deployment Manager deployment will create the following new
resources in your GCP project. Please ensure you have the necessary permissions
and are aware of any potential costs:
- A GCS bucket registered as a [ZenML artifact store](https://docs.zenml.io/stack-components/artifact-stores/gcp).
- A Google Artifact Registry registered as a [ZenML container registry](https://docs.zenml.io/stack-components/container-registries/gcp).
- Vertex AI registered as a [ZenML orchestrator](https://docs.zenml.io/stack-components/orchestrators/vertex)
and as a [ZenML step operator](https://docs.zenml.io/stack-components/step-operators/vertex).
- GCP Cloud Build registered as a [ZenML image builder](https://docs.zenml.io/stack-components/image-builders/gcp).
- A GCP Service Account with the minimum necessary permissions to access the
above resources.
- A GCP Service Account access key used to give access to ZenML to connect to
the above resources through a [ZenML service connector](https://docs.zenml.io/how-to/auth-management/gcp-service-connector).
The Deployment Manager deployment will automatically create a GCP Service
Account secret key and will share it with ZenML to give it permission to access
the resources created by the stack. You can revoke these permissions at any time
by deleting the Deployment Manager deployment in the GCP Cloud Console.
**Estimated costs**
A small training job would cost around: $0.60
These are rough estimates and actual costs may vary based on your usage and specific GCP pricing.
Some services may be eligible for the GCP Free Tier. Use [the GCP Pricing Calculator](https://cloud.google.com/products/calculator)
for a detailed estimate based on your usage.
⚠️ **The Cloud Shell session will warn you that the ZenML GitHub repository is
untrusted. We recommend that you review [the contents of the repository](https://github.com/zenml-io/zenml/tree/main/infra/gcp)
and then check the `Trust repo` checkbox to proceed with the deployment,
otherwise the Cloud Shell session will not be authenticated to access your
GCP projects. You will also get a chance to review the scripts that will be
executed in the Cloud Shell session before proceeding.**
💡 **After the Deployment Manager deployment is complete, you can close the Cloud
Shell session and return to the CLI to view details about the associated ZenML
stack automatically registered with ZenML.**
"""
integrations()
classmethod
Return the ZenML integrations required for the stack.
Returns:
| Type | Description |
|---|---|
List[str] |
The list of ZenML integrations that need to be installed for the stack to be usable. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
return [
"gcp",
]
locations()
classmethod
Return the locations where the ZenML stack can be deployed.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions where the ZenML stack can be deployed as a map of region names to region descriptions. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
# Return a list of all possible GCP regions
# Based on the AWS regions listed at
# https://cloud.google.com/about/locations
return {
"Africa (Johannesburg)": "africa-south1",
"Asia Pacific (Taiwan)": "asia-east1",
"Asia Pacific (Hong Kong)": "asia-east2",
"Asia Pacific (Tokyo)": "asia-northeast1",
"Asia Pacific (Osaka)": "asia-northeast2",
"Asia Pacific (Seoul)": "asia-northeast3",
"Asia Pacific (Mumbai)": "asia-south1",
"Asia Pacific (Delhi)": "asia-south2",
"Asia Pacific (Singapore)": "asia-southeast1",
"Asia Pacific (Jakarta)": "asia-southeast2",
"Australia (Sydney)": "australia-southeast1",
"Australia (Melbourne)": "australia-southeast2",
"Europe (Belgium)": "europe-west1",
"Europe (London)": "europe-west2",
"Europe (Frankfurt)": "europe-west3",
"Europe (Netherlands)": "europe-west4",
"Europe (Zurich)": "europe-west6",
"Europe (Milan)": "europe-west8",
"Europe (Paris)": "europe-west9",
"Europe (Berlin)": "europe-west10",
"Europe (Turin)": "europe-west12",
"Europe (Warsaw)": "europe-central2",
"Europe (Finland)": "europe-north1",
"Europe (Madrid)": "europe-southwest1",
"Middle East (Doha)": "me-central1",
"Middle East (Dubai)": "me-central2",
"Middle East (Tel Aviv)": "me-west1",
"North America (Montreal)": "northamerica-northeast1",
"North America (Toronto)": "northamerica-northeast2",
"South America (Sao Paulo)": "southamerica-east1",
"South America (Santiago)": "southamerica-west1",
"US Central (Iowa)": "us-central1",
"US East (South Carolina)": "us-east1",
"US East (Northern Virginia)": "us-east4",
"US East (Columbus)": "us-east5",
"US South (Dallas)": "us-south1",
"US West (Oregon)": "us-west1",
"US West (Los Angeles)": "us-west2",
"US West (Salt Lake City)": "us-west3",
"US West (Las Vegas)": "us-west4",
}
permissions()
classmethod
Return the permissions granted to ZenML to access the cloud resources.
Returns:
| Type | Description |
|---|---|
Dict[str, List[str]] |
The permissions granted to ZenML to access the cloud resources, as a dictionary grouping permissions by resource. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
return {
"GCS Bucket": [
"roles/storage.objectUser",
],
"GCP Artifact Registry": [
"roles/artifactregistry.createOnPushWriter",
],
"Vertex AI (Client)": [
"roles/aiplatform.user",
],
"Vertex AI (Jobs)": [
"roles/aiplatform.serviceAgent",
],
"Cloud Build (Client)": [
"roles/cloudbuild.builds.editor",
],
}
post_deploy_instructions()
classmethod
Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on what to do after the deployment is complete. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
return """
The ZenML stack has been successfully deployed and registered. You can delete
the Deployment Manager deployment at any time to revoke ZenML's access to your
GCP project and to clean up the resources created by the stack by using
[the GCP Cloud Console](https://console.cloud.google.com/dm/deployments).
"""
skypilot_default_regions()
classmethod
Returns the regions supported by default for the Skypilot.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions supported by default for the Skypilot. |
Source code in zenml/stack_deployments/gcp_stack_deployment.py
@classmethod
def skypilot_default_regions(cls) -> Dict[str, str]:
"""Returns the regions supported by default for the Skypilot.
Returns:
The regions supported by default for the Skypilot.
"""
matcher = re.compile(r"us-.*")
return {k: v for k, v in cls.locations().items() if matcher.match(v)}
stack_deployment
Functionality to deploy a ZenML stack to a cloud provider.
ZenMLCloudStackDeployment (BaseModel)
ZenML Cloud Stack CLI Deployment base class.
Source code in zenml/stack_deployments/stack_deployment.py
class ZenMLCloudStackDeployment(BaseModel):
"""ZenML Cloud Stack CLI Deployment base class."""
provider: ClassVar[StackDeploymentProvider]
deployment: ClassVar[str]
terraform: bool = False
stack_name: str
zenml_server_url: str
zenml_server_api_token: str
location: Optional[str] = None
@classmethod
@abstractmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
@classmethod
@abstractmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
@classmethod
@abstractmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
@classmethod
@abstractmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
@classmethod
@abstractmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
@classmethod
@abstractmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
@property
def deployment_type(self) -> str:
"""Return the type of deployment.
Returns:
The type of deployment.
"""
if self.terraform:
return STACK_DEPLOYMENT_TERRAFORM
return self.deployment
@classmethod
def skypilot_default_regions(cls) -> Dict[str, str]:
"""Returns the regions supported by default for the Skypilot.
Returns:
The regions supported by default for the Skypilot.
"""
return cls.locations()
@classmethod
def get_deployment_info(cls) -> StackDeploymentInfo:
"""Return information about the ZenML Cloud Stack Deployment.
Returns:
Information about the ZenML Cloud Stack Deployment.
"""
return StackDeploymentInfo(
provider=cls.provider,
description=cls.description(),
instructions=cls.instructions(),
post_deploy_instructions=cls.post_deploy_instructions(),
integrations=cls.integrations(),
permissions=cls.permissions(),
locations=cls.locations(),
skypilot_default_regions=cls.skypilot_default_regions(),
)
@abstractmethod
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
def get_stack(
self,
date_start: Optional[datetime.datetime] = None,
) -> Optional[DeployedStack]:
"""Return the ZenML stack that was deployed and registered.
This method is called to retrieve a ZenML stack matching the deployment
provider.
Args:
date_start: The date when the deployment started.
Returns:
The ZenML stack that was deployed and registered or None if a
matching stack was not found.
"""
client = Client()
# It's difficult to find a stack that matches the CloudFormation
# deployment 100% because the user can change the stack name before they
# deploy the stack in GCP.
#
# We try to find a full GCP stack that matches the deployment provider
# that was registered after this deployment was created.
# Get all stacks created after the start date
stacks = client.list_stacks(
created=f"gt:{str(date_start.replace(microsecond=0))}"
if date_start
else None,
sort_by="desc:created",
size=50,
)
if not stacks.items:
return None
# Find a stack that best matches the deployment provider
for stack in stacks.items:
if not stack.labels:
continue
if stack.labels.get("zenml:provider") != self.provider.value:
continue
if stack.labels.get("zenml:deployment") != self.deployment_type:
continue
orchestrator = stack.components[StackComponentType.ORCHESTRATOR][0]
if not orchestrator.connector:
continue
return DeployedStack(
stack=stack,
service_connector=orchestrator.connector,
)
return None
deployment_type: str
property
readonly
Return the type of deployment.
Returns:
| Type | Description |
|---|---|
str |
The type of deployment. |
description()
classmethod
Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
A MarkDown description of the ZenML Cloud Stack Deployment. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
@abstractmethod
def description(cls) -> str:
"""Return a description of the ZenML Cloud Stack Deployment.
This will be displayed when the user is prompted to deploy
the ZenML stack.
Returns:
A MarkDown description of the ZenML Cloud Stack Deployment.
"""
get_deployment_config(self)
Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
- a cloud provider console URL where the user will be redirected to deploy the ZenML stack. The URL should include as many pre-filled URL query parameters as possible.
- a textual description of the URL
- some deployment providers may require additional configuration parameters or scripts to be passed to the cloud provider in addition to the deployment URL query parameters. Where that is the case, this method should also return a string that the user can copy and paste into the cloud provider console to deploy the ZenML stack (e.g. a set of environment variables, YAML configuration snippet, bash or Terraform script etc.).
Returns:
| Type | Description |
|---|---|
StackDeploymentConfig |
The configuration or script to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/stack_deployment.py
@abstractmethod
def get_deployment_config(
self,
) -> StackDeploymentConfig:
"""Return the configuration to deploy the ZenML stack to the specified cloud provider.
The configuration should include:
* a cloud provider console URL where the user will be redirected to
deploy the ZenML stack. The URL should include as many pre-filled
URL query parameters as possible.
* a textual description of the URL
* some deployment providers may require additional configuration
parameters or scripts to be passed to the cloud provider in addition to
the deployment URL query parameters. Where that is the case, this method
should also return a string that the user can copy and paste into the
cloud provider console to deploy the ZenML stack (e.g. a set of
environment variables, YAML configuration snippet, bash or Terraform
script etc.).
Returns:
The configuration or script to deploy the ZenML stack to the
specified cloud provider.
"""
get_deployment_info()
classmethod
Return information about the ZenML Cloud Stack Deployment.
Returns:
| Type | Description |
|---|---|
StackDeploymentInfo |
Information about the ZenML Cloud Stack Deployment. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
def get_deployment_info(cls) -> StackDeploymentInfo:
"""Return information about the ZenML Cloud Stack Deployment.
Returns:
Information about the ZenML Cloud Stack Deployment.
"""
return StackDeploymentInfo(
provider=cls.provider,
description=cls.description(),
instructions=cls.instructions(),
post_deploy_instructions=cls.post_deploy_instructions(),
integrations=cls.integrations(),
permissions=cls.permissions(),
locations=cls.locations(),
skypilot_default_regions=cls.skypilot_default_regions(),
)
get_stack(self, date_start=None)
Return the ZenML stack that was deployed and registered.
This method is called to retrieve a ZenML stack matching the deployment provider.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
date_start |
Optional[datetime.datetime] |
The date when the deployment started. |
None |
Returns:
| Type | Description |
|---|---|
Optional[zenml.models.v2.misc.stack_deployment.DeployedStack] |
The ZenML stack that was deployed and registered or None if a matching stack was not found. |
Source code in zenml/stack_deployments/stack_deployment.py
def get_stack(
self,
date_start: Optional[datetime.datetime] = None,
) -> Optional[DeployedStack]:
"""Return the ZenML stack that was deployed and registered.
This method is called to retrieve a ZenML stack matching the deployment
provider.
Args:
date_start: The date when the deployment started.
Returns:
The ZenML stack that was deployed and registered or None if a
matching stack was not found.
"""
client = Client()
# It's difficult to find a stack that matches the CloudFormation
# deployment 100% because the user can change the stack name before they
# deploy the stack in GCP.
#
# We try to find a full GCP stack that matches the deployment provider
# that was registered after this deployment was created.
# Get all stacks created after the start date
stacks = client.list_stacks(
created=f"gt:{str(date_start.replace(microsecond=0))}"
if date_start
else None,
sort_by="desc:created",
size=50,
)
if not stacks.items:
return None
# Find a stack that best matches the deployment provider
for stack in stacks.items:
if not stack.labels:
continue
if stack.labels.get("zenml:provider") != self.provider.value:
continue
if stack.labels.get("zenml:deployment") != self.deployment_type:
continue
orchestrator = stack.components[StackComponentType.ORCHESTRATOR][0]
if not orchestrator.connector:
continue
return DeployedStack(
stack=stack,
service_connector=orchestrator.connector,
)
return None
instructions()
classmethod
Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML stack.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on how to deploy the ZenML stack to the specified cloud provider. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
@abstractmethod
def instructions(cls) -> str:
"""Return instructions on how to deploy the ZenML stack to the specified cloud provider.
This will be displayed before the user is prompted to deploy the ZenML
stack.
Returns:
MarkDown instructions on how to deploy the ZenML stack to the
specified cloud provider.
"""
integrations()
classmethod
Return the ZenML integrations required for the stack.
Returns:
| Type | Description |
|---|---|
List[str] |
The list of ZenML integrations that need to be installed for the stack to be usable. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
@abstractmethod
def integrations(cls) -> List[str]:
"""Return the ZenML integrations required for the stack.
Returns:
The list of ZenML integrations that need to be installed for the
stack to be usable.
"""
locations()
classmethod
Return the locations where the ZenML stack can be deployed.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions where the ZenML stack can be deployed as a map of region names to region descriptions. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
@abstractmethod
def locations(cls) -> Dict[str, str]:
"""Return the locations where the ZenML stack can be deployed.
Returns:
The regions where the ZenML stack can be deployed as a map of region
names to region descriptions.
"""
permissions()
classmethod
Return the permissions granted to ZenML to access the cloud resources.
Returns:
| Type | Description |
|---|---|
Dict[str, List[str]] |
The permissions granted to ZenML to access the cloud resources, as a dictionary grouping permissions by resource. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
@abstractmethod
def permissions(cls) -> Dict[str, List[str]]:
"""Return the permissions granted to ZenML to access the cloud resources.
Returns:
The permissions granted to ZenML to access the cloud resources, as
a dictionary grouping permissions by resource.
"""
post_deploy_instructions()
classmethod
Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
| Type | Description |
|---|---|
str |
MarkDown instructions on what to do after the deployment is complete. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
@abstractmethod
def post_deploy_instructions(cls) -> str:
"""Return instructions on what to do after the deployment is complete.
This will be displayed after the deployment is complete.
Returns:
MarkDown instructions on what to do after the deployment is
complete.
"""
skypilot_default_regions()
classmethod
Returns the regions supported by default for the Skypilot.
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
The regions supported by default for the Skypilot. |
Source code in zenml/stack_deployments/stack_deployment.py
@classmethod
def skypilot_default_regions(cls) -> Dict[str, str]:
"""Returns the regions supported by default for the Skypilot.
Returns:
The regions supported by default for the Skypilot.
"""
return cls.locations()
utils
Functionality to deploy a ZenML stack to a cloud provider.
get_stack_deployment_class(provider)
Get the ZenML Cloud Stack Deployment class for the specified provider.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
provider |
StackDeploymentProvider |
The stack deployment provider. |
required |
Returns:
| Type | Description |
|---|---|
Type[zenml.stack_deployments.stack_deployment.ZenMLCloudStackDeployment] |
The ZenML Cloud Stack Deployment class for the specified provider. |
Source code in zenml/stack_deployments/utils.py
def get_stack_deployment_class(
provider: StackDeploymentProvider,
) -> Type[ZenMLCloudStackDeployment]:
"""Get the ZenML Cloud Stack Deployment class for the specified provider.
Args:
provider: The stack deployment provider.
Returns:
The ZenML Cloud Stack Deployment class for the specified provider.
"""
return STACK_DEPLOYMENT_PROVIDERS[provider]