Evidently

zenml.integrations.evidently

Initialization of the Evidently integration.

The Evidently integration provides a way to monitor your models in production. It includes a way to detect data drift and different kinds of model performance issues.

The results of Evidently calculations can either be exported as an interactive dashboard (visualized as an html file or in your Jupyter notebook), or as a JSON file.

Attributes

EVIDENTLY = 'evidently' module-attribute

EVIDENTLY_DATA_VALIDATOR_FLAVOR = 'evidently' module-attribute

numba_logger = logging.getLogger('numba') module-attribute

Classes

EvidentlyIntegration

Bases: Integration

Evidently integration for ZenML.

Functions

flavors() -> List[Type[Flavor]] classmethod

Declare the stack component flavors for the Great Expectations integration.

Returns:

Type	Description
List[Type[Flavor]]	List of stack component flavors for this integration.

Source code in src/zenml/integrations/evidently/__init__.py

@classmethod
def flavors(cls) -> List[Type[Flavor]]:
    """Declare the stack component flavors for the Great Expectations integration.

    Returns:
        List of stack component flavors for this integration.
    """
    from zenml.integrations.evidently.flavors import (
        EvidentlyDataValidatorFlavor,
    )

    return [EvidentlyDataValidatorFlavor]

get_requirements(target_os: Optional[str] = None, python_version: Optional[str] = None) -> List[str] classmethod

Method to get the requirements for the integration.

Parameters:

Name	Type	Description	Default
target_os	Optional[str]	The target operating system to get the requirements for.	None
python_version	Optional[str]	The Python version to use for the requirements.	None

Returns:

Type	Description
List[str]	A list of requirements.

Source code in src/zenml/integrations/evidently/__init__.py

@classmethod
def get_requirements(
    cls, target_os: Optional[str] = None, python_version: Optional[str] = None
) -> List[str]:
    """Method to get the requirements for the integration.

    Args:
        target_os: The target operating system to get the requirements for.
        python_version: The Python version to use for the requirements.

    Returns:
        A list of requirements.
    """
    from zenml.integrations.pandas import PandasIntegration

    return cls.REQUIREMENTS + \
        PandasIntegration.get_requirements(target_os=target_os, python_version=python_version)

Flavor

Class for ZenML Flavors.

Attributes

config_class: Type[StackComponentConfig] abstractmethod property

Returns StackComponentConfig config class.

Returns:

Type	Description
Type[StackComponentConfig]	The config class.

config_schema: Dict[str, Any] property

The config schema for a flavor.

Returns:

Type	Description
Dict[str, Any]	The config schema.

docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type	Description
Optional[str]	A flavor docs url.

implementation_class: Type[StackComponent] abstractmethod property

Implementation class for this flavor.

Returns:

Type	Description
Type[StackComponent]	The implementation class for this flavor.

logo_url: Optional[str] property

A url to represent the flavor in the dashboard.

Returns:

Type	Description
Optional[str]	The flavor logo.

name: str abstractmethod property

The flavor name.

Returns:

Type	Description
str	The flavor name.

sdk_docs_url: Optional[str] property

A url to point at SDK docs explaining this flavor.

Returns:

Type	Description
Optional[str]	A flavor SDK docs url.

service_connector_requirements: Optional[ServiceConnectorRequirements] property

Service connector resource requirements for service connectors.

Specifies resource requirements that are used to filter the available service connector types that are compatible with this flavor.

Returns:

Type	Description
Optional[ServiceConnectorRequirements]	Requirements for compatible service connectors, if a service
Optional[ServiceConnectorRequirements]	connector is required for this flavor.

type: StackComponentType abstractmethod property

The stack component type.

Returns:

Type	Description
StackComponentType	The stack component type.

Functions

from_model(flavor_model: FlavorResponse) -> Flavor classmethod

Loads a flavor from a model.

Parameters:

Name	Type	Description	Default
flavor_model	FlavorResponse	The model to load from.	required

Raises:

Type	Description
CustomFlavorImportError	If the custom flavor can't be imported.
ImportError	If the flavor can't be imported.

Returns:

Type	Description
Flavor	The loaded flavor.

Source code in src/zenml/stack/flavor.py

@classmethod
def from_model(cls, flavor_model: FlavorResponse) -> "Flavor":
    """Loads a flavor from a model.

    Args:
        flavor_model: The model to load from.

    Raises:
        CustomFlavorImportError: If the custom flavor can't be imported.
        ImportError: If the flavor can't be imported.

    Returns:
        The loaded flavor.
    """
    try:
        flavor = source_utils.load(flavor_model.source)()
    except (ModuleNotFoundError, ImportError, NotImplementedError) as err:
        if flavor_model.is_custom:
            flavor_module, _ = flavor_model.source.rsplit(".", maxsplit=1)
            expected_file_path = os.path.join(
                source_utils.get_source_root(),
                flavor_module.replace(".", os.path.sep),
            )
            raise CustomFlavorImportError(
                f"Couldn't import custom flavor {flavor_model.name}: "
                f"{err}. Make sure the custom flavor class "
                f"`{flavor_model.source}` is importable. If it is part of "
                "a library, make sure it is installed. If "
                "it is a local code file, make sure it exists at "
                f"`{expected_file_path}.py`."
            )
        else:
            raise ImportError(
                f"Couldn't import flavor {flavor_model.name}: {err}"
            )
    return cast(Flavor, flavor)

generate_default_docs_url() -> str

Generate the doc urls for all inbuilt and integration flavors.

Note that this method is not going to be useful for custom flavors, which do not have any docs in the main zenml docs.

Returns:

Type	Description
str	The complete url to the zenml documentation

Source code in src/zenml/stack/flavor.py

def generate_default_docs_url(self) -> str:
    """Generate the doc urls for all inbuilt and integration flavors.

    Note that this method is not going to be useful for custom flavors,
    which do not have any docs in the main zenml docs.

    Returns:
        The complete url to the zenml documentation
    """
    from zenml import __version__

    component_type = self.type.plural.replace("_", "-")
    name = self.name.replace("_", "-")

    try:
        is_latest = is_latest_zenml_version()
    except RuntimeError:
        # We assume in error cases that we are on the latest version
        is_latest = True

    if is_latest:
        base = "https://docs.zenml.io"
    else:
        base = f"https://zenml-io.gitbook.io/zenml-legacy-documentation/v/{__version__}"
    return f"{base}/stack-components/{component_type}/{name}"

generate_default_sdk_docs_url() -> str

Generate SDK docs url for a flavor.

Returns:

Type	Description
str	The complete url to the zenml SDK docs

Source code in src/zenml/stack/flavor.py

def generate_default_sdk_docs_url(self) -> str:
    """Generate SDK docs url for a flavor.

    Returns:
        The complete url to the zenml SDK docs
    """
    from zenml import __version__

    base = f"https://sdkdocs.zenml.io/{__version__}"

    component_type = self.type.plural

    if "zenml.integrations" in self.__module__:
        # Get integration name out of module path which will look something
        #  like this "zenml.integrations.<integration>....
        integration = self.__module__.split(
            "zenml.integrations.", maxsplit=1
        )[1].split(".")[0]

        return (
            f"{base}/integration_code_docs"
            f"/integrations-{integration}/#{self.__module__}"
        )

    else:
        return (
            f"{base}/core_code_docs/core-{component_type}/"
            f"#{self.__module__}"
        )

to_model(integration: Optional[str] = None, is_custom: bool = True) -> FlavorRequest

Converts a flavor to a model.

Parameters:

Name	Type	Description	Default
integration	Optional[str]	The integration to use for the model.	None
is_custom	bool	Whether the flavor is a custom flavor.	True

Returns:

Type	Description
FlavorRequest	The model.

Source code in src/zenml/stack/flavor.py

def to_model(
    self,
    integration: Optional[str] = None,
    is_custom: bool = True,
) -> FlavorRequest:
    """Converts a flavor to a model.

    Args:
        integration: The integration to use for the model.
        is_custom: Whether the flavor is a custom flavor.

    Returns:
        The model.
    """
    connector_requirements = self.service_connector_requirements
    connector_type = (
        connector_requirements.connector_type
        if connector_requirements
        else None
    )
    resource_type = (
        connector_requirements.resource_type
        if connector_requirements
        else None
    )
    resource_id_attr = (
        connector_requirements.resource_id_attr
        if connector_requirements
        else None
    )

    model = FlavorRequest(
        name=self.name,
        type=self.type,
        source=source_utils.resolve(self.__class__).import_path,
        config_schema=self.config_schema,
        connector_type=connector_type,
        connector_resource_type=resource_type,
        connector_resource_id_attr=resource_id_attr,
        integration=integration,
        logo_url=self.logo_url,
        docs_url=self.docs_url,
        sdk_docs_url=self.sdk_docs_url,
        is_custom=is_custom,
    )
    return model

Integration

Base class for integration in ZenML.

Functions

activate() -> None classmethod

Abstract method to activate the integration.

Source code in src/zenml/integrations/integration.py

@classmethod
def activate(cls) -> None:
    """Abstract method to activate the integration."""

check_installation() -> bool classmethod

Method to check whether the required packages are installed.

Returns:

Type	Description
bool	True if all required packages are installed, False otherwise.

Source code in src/zenml/integrations/integration.py

@classmethod
def check_installation(cls) -> bool:
    """Method to check whether the required packages are installed.

    Returns:
        True if all required packages are installed, False otherwise.
    """
    for r in cls.get_requirements():
        try:
            # First check if the base package is installed
            dist = pkg_resources.get_distribution(r)

            # Next, check if the dependencies (including extras) are
            # installed
            deps: List[Requirement] = []

            _, extras = parse_requirement(r)
            if extras:
                extra_list = extras[1:-1].split(",")
                for extra in extra_list:
                    try:
                        requirements = dist.requires(extras=[extra])  # type: ignore[arg-type]
                    except pkg_resources.UnknownExtra as e:
                        logger.debug(f"Unknown extra: {str(e)}")
                        return False
                    deps.extend(requirements)
            else:
                deps = dist.requires()

            for ri in deps:
                try:
                    # Remove the "extra == ..." part from the requirement string
                    cleaned_req = re.sub(
                        r"; extra == \"\w+\"", "", str(ri)
                    )
                    pkg_resources.get_distribution(cleaned_req)
                except pkg_resources.DistributionNotFound as e:
                    logger.debug(
                        f"Unable to find required dependency "
                        f"'{e.req}' for requirement '{r}' "
                        f"necessary for integration '{cls.NAME}'."
                    )
                    return False
                except pkg_resources.VersionConflict as e:
                    logger.debug(
                        f"Package version '{e.dist}' does not match "
                        f"version '{e.req}' required by '{r}' "
                        f"necessary for integration '{cls.NAME}'."
                    )
                    return False

        except pkg_resources.DistributionNotFound as e:
            logger.debug(
                f"Unable to find required package '{e.req}' for "
                f"integration {cls.NAME}."
            )
            return False
        except pkg_resources.VersionConflict as e:
            logger.debug(
                f"Package version '{e.dist}' does not match version "
                f"'{e.req}' necessary for integration {cls.NAME}."
            )
            return False

    logger.debug(
        f"Integration {cls.NAME} is installed correctly with "
        f"requirements {cls.get_requirements()}."
    )
    return True

flavors() -> List[Type[Flavor]] classmethod

Abstract method to declare new stack component flavors.

Returns:

Type	Description
List[Type[Flavor]]	A list of new stack component flavors.

Source code in src/zenml/integrations/integration.py

@classmethod
def flavors(cls) -> List[Type[Flavor]]:
    """Abstract method to declare new stack component flavors.

    Returns:
        A list of new stack component flavors.
    """
    return []

get_requirements(target_os: Optional[str] = None, python_version: Optional[str] = None) -> List[str] classmethod

Method to get the requirements for the integration.

Parameters:

Name	Type	Description	Default
target_os	Optional[str]	The target operating system to get the requirements for.	None
python_version	Optional[str]	The Python version to use for the requirements.	None

Returns:

Type	Description
List[str]	A list of requirements.

Source code in src/zenml/integrations/integration.py

@classmethod
def get_requirements(
    cls,
    target_os: Optional[str] = None,
    python_version: Optional[str] = None,
) -> List[str]:
    """Method to get the requirements for the integration.

    Args:
        target_os: The target operating system to get the requirements for.
        python_version: The Python version to use for the requirements.

    Returns:
        A list of requirements.
    """
    return cls.REQUIREMENTS

get_uninstall_requirements(target_os: Optional[str] = None) -> List[str] classmethod

Method to get the uninstall requirements for the integration.

Parameters:

Name	Type	Description	Default
target_os	Optional[str]	The target operating system to get the requirements for.	None

Returns:

Type	Description
List[str]	A list of requirements.

Source code in src/zenml/integrations/integration.py

@classmethod
def get_uninstall_requirements(
    cls, target_os: Optional[str] = None
) -> List[str]:
    """Method to get the uninstall requirements for the integration.

    Args:
        target_os: The target operating system to get the requirements for.

    Returns:
        A list of requirements.
    """
    ret = []
    for each in cls.get_requirements(target_os=target_os):
        is_ignored = False
        for ignored in cls.REQUIREMENTS_IGNORED_ON_UNINSTALL:
            if each.startswith(ignored):
                is_ignored = True
                break
        if not is_ignored:
            ret.append(each)
    return ret

plugin_flavors() -> List[Type[BasePluginFlavor]] classmethod

Abstract method to declare new plugin flavors.

Returns:

Type	Description
List[Type[BasePluginFlavor]]	A list of new plugin flavors.

Source code in src/zenml/integrations/integration.py

@classmethod
def plugin_flavors(cls) -> List[Type["BasePluginFlavor"]]:
    """Abstract method to declare new plugin flavors.

    Returns:
        A list of new plugin flavors.
    """
    return []

Modules

column_mapping

ZenML representation of an Evidently column mapping.

Classes

EvidentlyColumnMapping

Bases: BaseModel

Column mapping configuration for Evidently.

This class is a 1-to-1 serializable analogue of Evidently's ColumnMapping data type that can be used as a step configuration field (see https://docs.evidentlyai.com/user-guide/input-data/column-mapping).

Attributes:

Name	Type	Description
target	Optional[str]	target column
prediction	Optional[Union[str, Sequence[str]]]	target column
datetime	Optional[str]	datetime column
id	Optional[str]	id column
numerical_features	Optional[List[str]]	numerical features
categorical_features	Optional[List[str]]	categorical features
datetime_features	Optional[List[str]]	datetime features
target_names	Optional[List[str]]	target column names
task	Optional[str]	model task
pos_label	Optional[Union[str, int]]	positive label
text_features	Optional[List[str]]	text features

Functions

to_evidently_column_mapping() -> ColumnMapping

Convert this Pydantic object to an Evidently ColumnMapping object.

Returns:

Type	Description
ColumnMapping	An Evidently column mapping converted from this Pydantic object.

Source code in src/zenml/integrations/evidently/column_mapping.py

def to_evidently_column_mapping(self) -> ColumnMapping:
    """Convert this Pydantic object to an Evidently ColumnMapping object.

    Returns:
        An Evidently column mapping converted from this Pydantic object.
    """
    column_mapping = ColumnMapping()

    # preserve the Evidently defaults where possible
    column_mapping.target = self.target or column_mapping.target
    column_mapping.prediction = (
        self.prediction or column_mapping.prediction
    )
    column_mapping.datetime = self.datetime or column_mapping.datetime
    column_mapping.id = self.id or column_mapping.id
    column_mapping.numerical_features = (
        self.numerical_features or column_mapping.numerical_features
    )
    column_mapping.datetime_features = (
        self.datetime_features or column_mapping.datetime_features
    )
    column_mapping.target_names = (
        self.target_names or column_mapping.target_names
    )
    column_mapping.task = self.task or column_mapping.task
    column_mapping.pos_label = self.pos_label or column_mapping.pos_label
    column_mapping.text_features = (
        self.text_features or column_mapping.text_features
    )

    return column_mapping

data_validators

Initialization of the Evidently data validator for ZenML.

Classes

EvidentlyDataValidator(name: str, id: UUID, config: StackComponentConfig, flavor: str, type: StackComponentType, user: Optional[UUID], created: datetime, updated: datetime, labels: Optional[Dict[str, Any]] = None, connector_requirements: Optional[ServiceConnectorRequirements] = None, connector: Optional[UUID] = None, connector_resource_id: Optional[str] = None, *args: Any, **kwargs: Any)

Bases: BaseDataValidator

Evidently data validator stack component.

Source code in src/zenml/stack/stack_component.py

def __init__(
    self,
    name: str,
    id: UUID,
    config: StackComponentConfig,
    flavor: str,
    type: StackComponentType,
    user: Optional[UUID],
    created: datetime,
    updated: datetime,
    labels: Optional[Dict[str, Any]] = None,
    connector_requirements: Optional[ServiceConnectorRequirements] = None,
    connector: Optional[UUID] = None,
    connector_resource_id: Optional[str] = None,
    *args: Any,
    **kwargs: Any,
):
    """Initializes a StackComponent.

    Args:
        name: The name of the component.
        id: The unique ID of the component.
        config: The config of the component.
        flavor: The flavor of the component.
        type: The type of the component.
        user: The ID of the user who created the component.
        created: The creation time of the component.
        updated: The last update time of the component.
        labels: The labels of the component.
        connector_requirements: The requirements for the connector.
        connector: The ID of a connector linked to the component.
        connector_resource_id: The custom resource ID to access through
            the connector.
        *args: Additional positional arguments.
        **kwargs: Additional keyword arguments.

    Raises:
        ValueError: If a secret reference is passed as name.
    """
    if secret_utils.is_secret_reference(name):
        raise ValueError(
            "Passing the `name` attribute of a stack component as a "
            "secret reference is not allowed."
        )

    self.id = id
    self.name = name
    self._config = config
    self.flavor = flavor
    self.type = type
    self.user = user
    self.created = created
    self.updated = updated
    self.labels = labels
    self.connector_requirements = connector_requirements
    self.connector = connector
    self.connector_resource_id = connector_resource_id
    self._connector_instance: Optional[ServiceConnector] = None

Functions

data_profiling(dataset: pd.DataFrame, comparison_dataset: Optional[pd.DataFrame] = None, profile_list: Optional[Sequence[EvidentlyMetricConfig]] = None, column_mapping: Optional[ColumnMapping] = None, report_options: Sequence[Tuple[str, Dict[str, Any]]] = [], download_nltk_data: bool = False, **kwargs: Any) -> Report

Analyze a dataset and generate a data report with Evidently.

The method takes in an optional list of Evidently options to be passed to the report constructor (report_options). Each element in the list must be composed of two items: the first is a full class path of an Evidently option dataclass, the second is a dictionary of kwargs with the actual option parameters, e.g.:

options = [
    (
        "evidently.options.ColorOptions",{
            "primary_color": "#5a86ad",
            "fill_color": "#fff4f2",
            "zero_line_color": "#016795",
            "current_data_color": "#c292a1",
            "reference_data_color": "#017b92",
        }
    ),
]

Parameters:

Name	Type	Description	Default
dataset	DataFrame	Target dataset to be profiled. When a comparison dataset is provided, this dataset is considered the reference dataset.	required
comparison_dataset	Optional[DataFrame]	Optional dataset to be used for data profiles that require a current dataset for comparison (e.g data drift profiles).	None
profile_list	Optional[Sequence[EvidentlyMetricConfig]]	List of Evidently metric configurations to be included in the report. If not provided, all available metric presets will be included.	None
column_mapping	Optional[ColumnMapping]	Properties of the DataFrame columns used	None
report_options	Sequence[Tuple[str, Dict[str, Any]]]	List of Evidently options to be passed to the report constructor.	[]
download_nltk_data	bool	Whether to download NLTK data for text metrics. Defaults to False.	False
**kwargs	Any	Extra keyword arguments (unused).	{}

Returns:

Type	Description
Report	The Evidently Report as JSON object and as HTML.

Source code in src/zenml/integrations/evidently/data_validators/evidently_data_validator.py

def data_profiling(
    self,
    dataset: pd.DataFrame,
    comparison_dataset: Optional[pd.DataFrame] = None,
    profile_list: Optional[Sequence[EvidentlyMetricConfig]] = None,
    column_mapping: Optional[ColumnMapping] = None,
    report_options: Sequence[Tuple[str, Dict[str, Any]]] = [],
    download_nltk_data: bool = False,
    **kwargs: Any,
) -> Report:
    """Analyze a dataset and generate a data report with Evidently.

    The method takes in an optional list of Evidently options to be passed
    to the report constructor (`report_options`). Each element in the list must be
    composed of two items: the first is a full class path of an Evidently
    option `dataclass`, the second is a dictionary of kwargs with the actual
    option parameters, e.g.:

    ```python
    options = [
        (
            "evidently.options.ColorOptions",{
                "primary_color": "#5a86ad",
                "fill_color": "#fff4f2",
                "zero_line_color": "#016795",
                "current_data_color": "#c292a1",
                "reference_data_color": "#017b92",
            }
        ),
    ]
    ```

    Args:
        dataset: Target dataset to be profiled. When a comparison dataset
            is provided, this dataset is considered the reference dataset.
        comparison_dataset: Optional dataset to be used for data profiles
            that require a current dataset for comparison (e.g data drift
            profiles).
        profile_list: List of Evidently metric configurations to
            be included in the report. If not provided, all available
            metric presets will be included.
        column_mapping: Properties of the DataFrame columns used
        report_options: List of Evidently options to be passed to the
            report constructor.
        download_nltk_data: Whether to download NLTK data for text metrics.
            Defaults to False.
        **kwargs: Extra keyword arguments (unused).

    Returns:
        The Evidently Report as JSON object and as HTML.
    """
    self._set_nltk_data_path()
    if download_nltk_data:
        self._download_nltk_data()

    profile_list = profile_list or EvidentlyMetricConfig.default_metrics()
    metrics = [metric.to_evidently_metric() for metric in profile_list]

    unpacked_report_options = self._unpack_options(report_options)

    report = Report(metrics=metrics, options=unpacked_report_options)

    report.run(
        reference_data=dataset,
        current_data=comparison_dataset,
        column_mapping=column_mapping,
    )

    return report

data_validation(dataset: Any, comparison_dataset: Optional[Any] = None, check_list: Optional[Sequence[EvidentlyTestConfig]] = None, test_options: Sequence[Tuple[str, Dict[str, Any]]] = [], column_mapping: Optional[ColumnMapping] = None, download_nltk_data: bool = False, **kwargs: Any) -> TestSuite

Validate a dataset with Evidently.

Parameters:

Name	Type	Description	Default
dataset	Any	Target dataset to be validated.	required
comparison_dataset	Optional[Any]	Optional dataset to be used for data validation that require a baseline for comparison (e.g data drift validation).	None
check_list	Optional[Sequence[EvidentlyTestConfig]]	List of Evidently test configurations to be included in the test suite. If not provided, all available test presets will be included.	None
test_options	Sequence[Tuple[str, Dict[str, Any]]]	List of Evidently options to be passed to the test suite constructor.	[]
column_mapping	Optional[ColumnMapping]	Properties of the DataFrame columns used	None
download_nltk_data	bool	Whether to download NLTK data for text tests. Defaults to False.	False
**kwargs	Any	Extra keyword arguments (unused).	{}

Returns:

Type	Description
TestSuite	The Evidently Test Suite as JSON object and as HTML.

Source code in src/zenml/integrations/evidently/data_validators/evidently_data_validator.py

def data_validation(
    self,
    dataset: Any,
    comparison_dataset: Optional[Any] = None,
    check_list: Optional[Sequence[EvidentlyTestConfig]] = None,
    test_options: Sequence[Tuple[str, Dict[str, Any]]] = [],
    column_mapping: Optional[ColumnMapping] = None,
    download_nltk_data: bool = False,
    **kwargs: Any,
) -> TestSuite:
    """Validate a dataset with Evidently.

    Args:
        dataset: Target dataset to be validated.
        comparison_dataset: Optional dataset to be used for data validation
            that require a baseline for comparison (e.g data drift
            validation).
        check_list: List of Evidently test configurations to be
            included in the test suite. If not provided, all available
            test presets will be included.
        test_options: List of Evidently options to be passed to the
            test suite constructor.
        column_mapping: Properties of the DataFrame columns used
        download_nltk_data: Whether to download NLTK data for text tests.
            Defaults to False.
        **kwargs: Extra keyword arguments (unused).

    Returns:
        The Evidently Test Suite as JSON object and as HTML.
    """
    if download_nltk_data:
        self._download_nltk_data()

    check_list = check_list or EvidentlyTestConfig.default_tests()
    tests = [test.to_evidently_test() for test in check_list]

    unpacked_test_options = self._unpack_options(test_options)

    test_suite = TestSuite(tests=tests, options=unpacked_test_options)
    test_suite.run(
        reference_data=dataset,
        current_data=comparison_dataset,
        column_mapping=column_mapping,
    )

    return test_suite

Modules

evidently_data_validator

Implementation of the Evidently data validator.

Classes

EvidentlyDataValidator(name: str, id: UUID, config: StackComponentConfig, flavor: str, type: StackComponentType, user: Optional[UUID], created: datetime, updated: datetime, labels: Optional[Dict[str, Any]] = None, connector_requirements: Optional[ServiceConnectorRequirements] = None, connector: Optional[UUID] = None, connector_resource_id: Optional[str] = None, *args: Any, **kwargs: Any)

Bases: BaseDataValidator

Evidently data validator stack component.

Source code in src/zenml/stack/stack_component.py

def __init__(
    self,
    name: str,
    id: UUID,
    config: StackComponentConfig,
    flavor: str,
    type: StackComponentType,
    user: Optional[UUID],
    created: datetime,
    updated: datetime,
    labels: Optional[Dict[str, Any]] = None,
    connector_requirements: Optional[ServiceConnectorRequirements] = None,
    connector: Optional[UUID] = None,
    connector_resource_id: Optional[str] = None,
    *args: Any,
    **kwargs: Any,
):
    """Initializes a StackComponent.

    Args:
        name: The name of the component.
        id: The unique ID of the component.
        config: The config of the component.
        flavor: The flavor of the component.
        type: The type of the component.
        user: The ID of the user who created the component.
        created: The creation time of the component.
        updated: The last update time of the component.
        labels: The labels of the component.
        connector_requirements: The requirements for the connector.
        connector: The ID of a connector linked to the component.
        connector_resource_id: The custom resource ID to access through
            the connector.
        *args: Additional positional arguments.
        **kwargs: Additional keyword arguments.

    Raises:
        ValueError: If a secret reference is passed as name.
    """
    if secret_utils.is_secret_reference(name):
        raise ValueError(
            "Passing the `name` attribute of a stack component as a "
            "secret reference is not allowed."
        )

    self.id = id
    self.name = name
    self._config = config
    self.flavor = flavor
    self.type = type
    self.user = user
    self.created = created
    self.updated = updated
    self.labels = labels
    self.connector_requirements = connector_requirements
    self.connector = connector
    self.connector_resource_id = connector_resource_id
    self._connector_instance: Optional[ServiceConnector] = None

Functions

data_profiling(dataset: pd.DataFrame, comparison_dataset: Optional[pd.DataFrame] = None, profile_list: Optional[Sequence[EvidentlyMetricConfig]] = None, column_mapping: Optional[ColumnMapping] = None, report_options: Sequence[Tuple[str, Dict[str, Any]]] = [], download_nltk_data: bool = False, **kwargs: Any) -> Report

Analyze a dataset and generate a data report with Evidently.

The method takes in an optional list of Evidently options to be passed to the report constructor (report_options). Each element in the list must be composed of two items: the first is a full class path of an Evidently option dataclass, the second is a dictionary of kwargs with the actual option parameters, e.g.:

options = [
    (
        "evidently.options.ColorOptions",{
            "primary_color": "#5a86ad",
            "fill_color": "#fff4f2",
            "zero_line_color": "#016795",
            "current_data_color": "#c292a1",
            "reference_data_color": "#017b92",
        }
    ),
]

Parameters:

Name	Type	Description	Default
dataset	DataFrame	Target dataset to be profiled. When a comparison dataset is provided, this dataset is considered the reference dataset.	required
comparison_dataset	Optional[DataFrame]	Optional dataset to be used for data profiles that require a current dataset for comparison (e.g data drift profiles).	None
profile_list	Optional[Sequence[EvidentlyMetricConfig]]	List of Evidently metric configurations to be included in the report. If not provided, all available metric presets will be included.	None
column_mapping	Optional[ColumnMapping]	Properties of the DataFrame columns used	None
report_options	Sequence[Tuple[str, Dict[str, Any]]]	List of Evidently options to be passed to the report constructor.	[]
download_nltk_data	bool	Whether to download NLTK data for text metrics. Defaults to False.	False
**kwargs	Any	Extra keyword arguments (unused).	{}

Returns:

Type	Description
Report	The Evidently Report as JSON object and as HTML.

Source code in src/zenml/integrations/evidently/data_validators/evidently_data_validator.py

def data_profiling(
    self,
    dataset: pd.DataFrame,
    comparison_dataset: Optional[pd.DataFrame] = None,
    profile_list: Optional[Sequence[EvidentlyMetricConfig]] = None,
    column_mapping: Optional[ColumnMapping] = None,
    report_options: Sequence[Tuple[str, Dict[str, Any]]] = [],
    download_nltk_data: bool = False,
    **kwargs: Any,
) -> Report:
    """Analyze a dataset and generate a data report with Evidently.

    The method takes in an optional list of Evidently options to be passed
    to the report constructor (`report_options`). Each element in the list must be
    composed of two items: the first is a full class path of an Evidently
    option `dataclass`, the second is a dictionary of kwargs with the actual
    option parameters, e.g.:

    ```python
    options = [
        (
            "evidently.options.ColorOptions",{
                "primary_color": "#5a86ad",
                "fill_color": "#fff4f2",
                "zero_line_color": "#016795",
                "current_data_color": "#c292a1",
                "reference_data_color": "#017b92",
            }
        ),
    ]
    ```

    Args:
        dataset: Target dataset to be profiled. When a comparison dataset
            is provided, this dataset is considered the reference dataset.
        comparison_dataset: Optional dataset to be used for data profiles
            that require a current dataset for comparison (e.g data drift
            profiles).
        profile_list: List of Evidently metric configurations to
            be included in the report. If not provided, all available
            metric presets will be included.
        column_mapping: Properties of the DataFrame columns used
        report_options: List of Evidently options to be passed to the
            report constructor.
        download_nltk_data: Whether to download NLTK data for text metrics.
            Defaults to False.
        **kwargs: Extra keyword arguments (unused).

    Returns:
        The Evidently Report as JSON object and as HTML.
    """
    self._set_nltk_data_path()
    if download_nltk_data:
        self._download_nltk_data()

    profile_list = profile_list or EvidentlyMetricConfig.default_metrics()
    metrics = [metric.to_evidently_metric() for metric in profile_list]

    unpacked_report_options = self._unpack_options(report_options)

    report = Report(metrics=metrics, options=unpacked_report_options)

    report.run(
        reference_data=dataset,
        current_data=comparison_dataset,
        column_mapping=column_mapping,
    )

    return report

data_validation(dataset: Any, comparison_dataset: Optional[Any] = None, check_list: Optional[Sequence[EvidentlyTestConfig]] = None, test_options: Sequence[Tuple[str, Dict[str, Any]]] = [], column_mapping: Optional[ColumnMapping] = None, download_nltk_data: bool = False, **kwargs: Any) -> TestSuite

Validate a dataset with Evidently.

Parameters:

Name	Type	Description	Default
dataset	Any	Target dataset to be validated.	required
comparison_dataset	Optional[Any]	Optional dataset to be used for data validation that require a baseline for comparison (e.g data drift validation).	None
check_list	Optional[Sequence[EvidentlyTestConfig]]	List of Evidently test configurations to be included in the test suite. If not provided, all available test presets will be included.	None
test_options	Sequence[Tuple[str, Dict[str, Any]]]	List of Evidently options to be passed to the test suite constructor.	[]
column_mapping	Optional[ColumnMapping]	Properties of the DataFrame columns used	None
download_nltk_data	bool	Whether to download NLTK data for text tests. Defaults to False.	False
**kwargs	Any	Extra keyword arguments (unused).	{}

Returns:

Type	Description
TestSuite	The Evidently Test Suite as JSON object and as HTML.

Source code in src/zenml/integrations/evidently/data_validators/evidently_data_validator.py

def data_validation(
    self,
    dataset: Any,
    comparison_dataset: Optional[Any] = None,
    check_list: Optional[Sequence[EvidentlyTestConfig]] = None,
    test_options: Sequence[Tuple[str, Dict[str, Any]]] = [],
    column_mapping: Optional[ColumnMapping] = None,
    download_nltk_data: bool = False,
    **kwargs: Any,
) -> TestSuite:
    """Validate a dataset with Evidently.

    Args:
        dataset: Target dataset to be validated.
        comparison_dataset: Optional dataset to be used for data validation
            that require a baseline for comparison (e.g data drift
            validation).
        check_list: List of Evidently test configurations to be
            included in the test suite. If not provided, all available
            test presets will be included.
        test_options: List of Evidently options to be passed to the
            test suite constructor.
        column_mapping: Properties of the DataFrame columns used
        download_nltk_data: Whether to download NLTK data for text tests.
            Defaults to False.
        **kwargs: Extra keyword arguments (unused).

    Returns:
        The Evidently Test Suite as JSON object and as HTML.
    """
    if download_nltk_data:
        self._download_nltk_data()

    check_list = check_list or EvidentlyTestConfig.default_tests()
    tests = [test.to_evidently_test() for test in check_list]

    unpacked_test_options = self._unpack_options(test_options)

    test_suite = TestSuite(tests=tests, options=unpacked_test_options)
    test_suite.run(
        reference_data=dataset,
        current_data=comparison_dataset,
        column_mapping=column_mapping,
    )

    return test_suite

Functions Modules

flavors

Evidently integration flavors.

Classes

EvidentlyDataValidatorFlavor

Bases: BaseDataValidatorFlavor

Evidently data validator flavor.

Attributes

docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type	Description
Optional[str]	A flavor docs url.

implementation_class: Type[EvidentlyDataValidator] property

Implementation class.

Returns:

Type	Description
Type[EvidentlyDataValidator]	The implementation class.

logo_url: str property

A url to represent the flavor in the dashboard.

Returns:

Type	Description
str	The flavor logo.

name: str property

Name of the flavor.

Returns:

Type	Description
str	The name of the flavor.

sdk_docs_url: Optional[str] property

A url to point at SDK docs explaining this flavor.

Returns:

Type	Description
Optional[str]	A flavor SDK docs url.

Modules

evidently_data_validator_flavor

Evidently data validator flavor.

Classes

EvidentlyDataValidatorFlavor

Bases: BaseDataValidatorFlavor

Evidently data validator flavor.

Attributes

docs_url: Optional[str] property

A url to point at docs explaining this flavor.

Returns:

Type	Description
Optional[str]	A flavor docs url.

implementation_class: Type[EvidentlyDataValidator] property

Implementation class.

Returns:

Type	Description
Type[EvidentlyDataValidator]	The implementation class.

logo_url: str property

A url to represent the flavor in the dashboard.

Returns:

Type	Description
str	The flavor logo.

name: str property

Name of the flavor.

Returns:

Type	Description
str	The name of the flavor.

sdk_docs_url: Optional[str] property

A url to point at SDK docs explaining this flavor.

Returns:

Type	Description
Optional[str]	A flavor SDK docs url.

metrics

ZenML declarative representation of Evidently Metrics.

Classes

EvidentlyMetricConfig

Bases: BaseModel

Declarative Evidently Metric configuration.

This is a declarative representation of the configuration that goes into an Evidently Metric, MetricPreset or Metric generator instance. We need this to be able to store the configuration as part of a ZenML step parameter and later instantiate the Evidently Metric from it.

This representation covers all 3 possible ways of configuring an Evidently Metric or Metric-like object that can later be used in an Evidently Report:

1. A Metric (derived from the Metric class).
2. A MetricPreset (derived from the MetricPreset class).
3. A column Metric generator (derived from the BaseGenerator class).

Ideally, it should be possible to just pass a Metric or Metric-like object to this class and have it automatically derive the configuration used to instantiate it. Unfortunately, this is not possible because the Evidently Metric classes are not designed in a way that allows us to extract the constructor parameters from them in a generic way.

Attributes:

Name	Type	Description
class_path	str	The full class path of the Evidently Metric class.
parameters	Dict[str, Any]	The parameters of the Evidently Metric.
is_generator	bool	Whether this is an Evidently column Metric generator.
columns	Optional[Union[str, List[str]]]	The columns that the Evidently column Metric generator is applied to. Only used if generator is True.
skip_id_column	bool	Whether to skip the ID column when applying the Evidently Metric generator. Only used if generator is True.

Functions

default_metrics() -> List[EvidentlyMetricConfig] classmethod

Default Evidently metric configurations.

Call this to fetch a default list of Evidently metrics to use in cases where no metrics are explicitly configured for a data validator. All available Evidently MetricPreset classes are used, except for the TextOverviewPreset which requires a text column, which we don't have by default.

Returns:

Type	Description
List[EvidentlyMetricConfig]	A list of EvidentlyMetricConfig objects to use as default metrics.

Source code in src/zenml/integrations/evidently/metrics.py

@classmethod
def default_metrics(cls) -> List["EvidentlyMetricConfig"]:
    """Default Evidently metric configurations.

    Call this to fetch a default list of Evidently metrics to use in cases
    where no metrics are explicitly configured for a data validator.
    All available Evidently MetricPreset classes are used, except for the
    `TextOverviewPreset` which requires a text column, which we don't have
    by default.

    Returns:
        A list of EvidentlyMetricConfig objects to use as default metrics.
    """
    return [
        cls.metric(metric=metric_preset_class_name)
        for metric_preset_class_name in metric_preset.__all__
        # TextOverviewPreset requires a text column, which we don't
        # have by default
        if metric_preset_class_name != "TextOverviewPreset"
    ]

get_metric_class(metric_name: str) -> Union[Metric, MetricPreset] staticmethod

Get the Evidently metric or metric preset class from a string.

Parameters:

Name	Type	Description	Default
metric_name	str	The metric or metric preset class or full class path.	required

Returns:

Type	Description
Union[Metric, MetricPreset]	The Evidently metric or metric preset class.

Raises:

Type	Description
ValueError	If the name cannot be converted into a valid Evidently metric or metric preset class.

Source code in src/zenml/integrations/evidently/metrics.py

@staticmethod
def get_metric_class(metric_name: str) -> Union[Metric, MetricPreset]:
    """Get the Evidently metric or metric preset class from a string.

    Args:
        metric_name: The metric or metric preset class or full class
            path.

    Returns:
        The Evidently metric or metric preset class.

    Raises:
        ValueError: If the name cannot be converted into a valid Evidently
            metric or metric preset class.
    """
    # First, try to interpret the metric name as a full class path.
    if "." in metric_name:
        try:
            metric_class = source_utils.load(metric_name)
        except (ImportError, AttributeError) as e:
            raise ValueError(
                f"Could not import Evidently Metric or MetricPreset "
                f"`{metric_name}`: {str(e)}"
            )

    else:
        # Next, try to interpret the metric as a Metric or MetricPreset
        # class name
        if hasattr(metrics, metric_name):
            metric_class = getattr(metrics, metric_name)
        elif hasattr(metric_preset, metric_name):
            metric_class = getattr(metric_preset, metric_name)
        else:
            raise ValueError(
                f"Could not import Evidently Metric or MetricPreset "
                f"`{metric_name}`"
            )

    if not issubclass(metric_class, (Metric, MetricPreset)):
        raise ValueError(
            f"Class `{metric_name}` is not a valid Evidently "
            f"Metric or MetricPreset."
        )

    return metric_class

metric(metric: Union[Type[Metric], Type[MetricPreset], str], **parameters: Any) -> EvidentlyMetricConfig classmethod

Create a declarative configuration for an Evidently Metric.

Call this method to get a declarative representation for the configuration of an Evidently Metric.

Some examples

from zenml.integrations.evidently.data_validators import EvidentlyMetric

# Configure an Evidently MetricPreset using its class name
config = EvidentlyMetric.metric("DataDriftPreset")

from zenml.integrations.evidently.data_validators import EvidentlyMetric

# Configure an Evidently MetricPreset using its full class path
config = EvidentlyMetric.metric(
    "evidently.metric_preset.DataDriftPreset"
)

from zenml.integrations.evidently.data_validators import EvidentlyMetric

# Configure an Evidently Metric using its class and pass additional
# parameters
from evidently.metrics import ColumnSummaryMetric
config = EvidentlyMetric.metric(
    ColumnSummaryMetric, column_name="age"
)

Parameters:

Name	Type	Description	Default
metric	Union[Type[Metric], Type[MetricPreset], str]	The Evidently Metric or MetricPreset class, class name or class path.	required
parameters	Any	Additional optional parameters needed to instantiate the Evidently Metric or MetricPreset.	{}

Returns:

Type	Description
EvidentlyMetricConfig	The EvidentlyMetric declarative representation of the Evidently
EvidentlyMetricConfig	Metric configuration.

Raises:

Type	Description
ValueError	If metric does not point to a valid Evidently Metric or MetricPreset class.

Source code in src/zenml/integrations/evidently/metrics.py

@classmethod
def metric(
    cls,
    metric: Union[Type[Metric], Type[MetricPreset], str],
    **parameters: Any,
) -> "EvidentlyMetricConfig":
    """Create a declarative configuration for an Evidently Metric.

    Call this method to get a declarative representation for the
    configuration of an Evidently Metric.

    Some examples
    -------------

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyMetric

    # Configure an Evidently MetricPreset using its class name
    config = EvidentlyMetric.metric("DataDriftPreset")
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyMetric

    # Configure an Evidently MetricPreset using its full class path
    config = EvidentlyMetric.metric(
        "evidently.metric_preset.DataDriftPreset"
    )
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyMetric

    # Configure an Evidently Metric using its class and pass additional
    # parameters
    from evidently.metrics import ColumnSummaryMetric
    config = EvidentlyMetric.metric(
        ColumnSummaryMetric, column_name="age"
    )
    ```

    Args:
        metric: The Evidently Metric or MetricPreset class, class name or
            class path.
        parameters: Additional optional parameters needed to instantiate the
            Evidently Metric or MetricPreset.

    Returns:
        The EvidentlyMetric declarative representation of the Evidently
        Metric configuration.

    Raises:
        ValueError: If `metric` does not point to a valid Evidently Metric
            or MetricPreset class.
    """
    if isinstance(metric, str):
        metric_class = cls.get_metric_class(metric)
    elif issubclass(metric, (Metric, MetricPreset)):
        metric_class = metric
    else:
        raise ValueError(
            f"Invalid Evidently Metric or MetricPreset class: {metric}"
        )

    class_path = f"{metric_class.__module__}.{metric_class.__name__}"

    config = cls(class_path=class_path, parameters=parameters)

    # Try to instantiate the configuration to check if the parameters are
    # valid
    config.to_evidently_metric()

    return config

metric_generator(metric: Union[Type[Metric], str], columns: Optional[Union[str, List[str]]] = None, skip_id_column: bool = False, **parameters: Any) -> EvidentlyMetricConfig classmethod

Create a declarative configuration for an Evidently column Metric generator.

Call this method to get a declarative representation for the configuration of an Evidently column Metric generator.

The columns, skip_id_column and parameters arguments will be passed to the Evidently generate_column_metrics function:

• if columns is a list, it is interpreted as a list of column names.
• if columns is a string, it can be one of values:
  • "all" - use all columns, including target/prediction columns
  • "num" - for numeric features
  • "cat" - for category features
  • "text" - for text features
  • "features" - for all features, not target/prediction columns.
• a None value is the same as "all".

Some examples

from zenml.integrations.evidently.data_validators import EvidentlyMetric

# Configure an Evidently Metric generator using a Metric class name
# and pass additional parameters
config = EvidentlyMetric.metric_generator(
    "ColumnQuantileMetric", columns="num", quantile=0.5
)

from zenml.integrations.evidently.data_validators import EvidentlyMetric

# Configure an Evidently Metric generator using a full Metric class
# path
config = EvidentlyMetric.metric_generator(
    "evidently.metrics.ColumnSummaryMetric", columns=["age", "name"]
)

from zenml.integrations.evidently.data_validators import EvidentlyMetric

# Configure an Evidently Metric generator using a Metric class
from evidently.metrics import ColumnDriftMetric
config = EvidentlyMetric.metric_generator(
    ColumnDriftMetric, columns="all", skip_id_column=True
)

Parameters:

Name	Type	Description	Default
metric	Union[Type[Metric], str]	The Evidently Metric class, class name or class path to use for the generator.	required
columns	Optional[Union[str, List[str]]]	The columns to apply the generator to. Takes the same values that the Evidently generate_column_metrics function takes.	None
skip_id_column	bool	Whether to skip the ID column when applying the generator.	False
parameters	Any	Additional optional parameters needed to instantiate the Evidently Metric. These will be passed to the Evidently generate_column_metrics function.	{}

Returns:

Type	Description
EvidentlyMetricConfig	The EvidentlyMetric declarative representation of the Evidently
EvidentlyMetricConfig	Metric generator configuration.

Raises:

Type	Description
ValueError	If metric does not point to a valid Evidently Metric or MetricPreset class.

Source code in src/zenml/integrations/evidently/metrics.py

@classmethod
def metric_generator(
    cls,
    metric: Union[Type[Metric], str],
    columns: Optional[Union[str, List[str]]] = None,
    skip_id_column: bool = False,
    **parameters: Any,
) -> "EvidentlyMetricConfig":
    """Create a declarative configuration for an Evidently column Metric generator.

    Call this method to get a declarative representation for the
    configuration of an Evidently column Metric generator.

    The `columns`, `skip_id_column` and `parameters` arguments will be
    passed to the Evidently `generate_column_metrics` function:

    - if `columns` is a list, it is interpreted as a list of column names.
    - if `columns` is a string, it can be one of values:
        - "all" - use all columns, including target/prediction columns
        - "num" - for numeric features
        - "cat" - for category features
        - "text" - for text features
        - "features" - for all features, not target/prediction columns.
    - a None value is the same as "all".

    Some examples
    -------------

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyMetric

    # Configure an Evidently Metric generator using a Metric class name
    # and pass additional parameters
    config = EvidentlyMetric.metric_generator(
        "ColumnQuantileMetric", columns="num", quantile=0.5
    )
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyMetric

    # Configure an Evidently Metric generator using a full Metric class
    # path
    config = EvidentlyMetric.metric_generator(
        "evidently.metrics.ColumnSummaryMetric", columns=["age", "name"]
    )
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyMetric

    # Configure an Evidently Metric generator using a Metric class
    from evidently.metrics import ColumnDriftMetric
    config = EvidentlyMetric.metric_generator(
        ColumnDriftMetric, columns="all", skip_id_column=True
    )
    ```

    Args:
        metric: The Evidently Metric class, class name or class path to use
            for the generator.
        columns: The columns to apply the generator to. Takes the same
            values that the Evidently `generate_column_metrics` function
            takes.
        skip_id_column: Whether to skip the ID column when applying the
            generator.
        parameters: Additional optional parameters needed to instantiate the
            Evidently Metric. These will be passed to the Evidently
            `generate_column_metrics` function.

    Returns:
        The EvidentlyMetric declarative representation of the Evidently
        Metric generator configuration.

    Raises:
        ValueError: If `metric` does not point to a valid Evidently Metric
            or MetricPreset class.
    """
    if isinstance(metric, str):
        metric_class = cls.get_metric_class(metric)
    elif issubclass(metric, (Metric, MetricPreset)):
        metric_class = metric
    else:
        raise ValueError(f"Invalid Evidently Metric class: {metric}")

    class_path = f"{metric_class.__module__}.{metric_class.__name__}"

    config = cls(
        class_path=class_path,
        parameters=parameters,
        columns=columns,
        skip_id_column=skip_id_column,
        is_generator=True,
    )

    # Try to instantiate the configuration to check if the parameters are
    # valid
    config.to_evidently_metric()

    return config

to_evidently_metric() -> Union[Metric, MetricPreset, BaseGenerator]

Create an Evidently Metric, MetricPreset or metric generator object.

Call this method to create an Evidently Metric, MetricPreset or metric generator instance from its declarative representation.

Returns:

Type	Description
Union[Metric, MetricPreset, BaseGenerator]	The Evidently Metric, MetricPreset or metric generator object.

Raises:

Type	Description
ValueError	If the Evidently Metric, MetricPreset or column metric generator could not be instantiated.

Source code in src/zenml/integrations/evidently/metrics.py

def to_evidently_metric(
    self,
) -> Union[Metric, MetricPreset, BaseGenerator]:
    """Create an Evidently Metric, MetricPreset or metric generator object.

    Call this method to create an Evidently Metric, MetricPreset or metric
    generator instance from its declarative representation.

    Returns:
        The Evidently Metric, MetricPreset or metric generator object.

    Raises:
        ValueError: If the Evidently Metric, MetricPreset or column metric
            generator could not be instantiated.
    """
    metric_class = self.get_metric_class(self.class_path)

    if self.is_generator:
        try:
            return generate_column_metrics(
                metric_class=metric_class,
                columns=self.columns,
                skip_id_column=self.skip_id_column,
                parameters=self.parameters,
            )
        except Exception as e:
            raise ValueError(
                f"Could not instantiate Evidently column Metric generator "
                f"`{self.class_path}`: {str(e)}"
            )

    try:
        return metric_class(**self.parameters)
    except Exception as e:
        raise ValueError(
            f"Could not instantiate Evidently Metric or MetricPreset "
            f"`{self.class_path}`: {str(e)}"
        )

Functions

Modules

steps

Initialization of the Evidently Standard Steps.

Classes

Functions

Modules

evidently_report

Implementation of the Evidently Report Step.

Classes Functions

evidently_report_step(reference_dataset: pd.DataFrame, comparison_dataset: Optional[pd.DataFrame] = None, column_mapping: Optional[EvidentlyColumnMapping] = None, ignored_cols: Optional[List[str]] = None, metrics: Optional[List[EvidentlyMetricConfig]] = None, report_options: Optional[Sequence[Tuple[str, Dict[str, Any]]]] = None, download_nltk_data: bool = False) -> Tuple[Annotated[str, report_json], Annotated[HTMLString, report_html]]

Generate an Evidently report on one or two pandas datasets.

Parameters:

Name	Type	Description	Default
reference_dataset	DataFrame	a Pandas DataFrame	required
comparison_dataset	Optional[DataFrame]	a Pandas DataFrame of new data you wish to compare against the reference data	None
column_mapping	Optional[EvidentlyColumnMapping]	properties of the DataFrame columns used	None
ignored_cols	Optional[List[str]]	columns to ignore during the Evidently report step	None
metrics	Optional[List[EvidentlyMetricConfig]]	a list of Evidently metric configurations to use for the report.	None
report_options	Optional[Sequence[Tuple[str, Dict[str, Any]]]]	a list of tuples containing the name of the report and a dictionary of options for the report.	None
download_nltk_data	bool	whether to download the NLTK data for the report step. Defaults to False.	False

Returns:

Type	Description
Annotated[str, report_json]	A tuple containing the Evidently report in JSON and HTML
Annotated[HTMLString, report_html]	formats.

Source code in src/zenml/integrations/evidently/steps/evidently_report.py

@step
def evidently_report_step(
    reference_dataset: pd.DataFrame,
    comparison_dataset: Optional[pd.DataFrame] = None,
    column_mapping: Optional[EvidentlyColumnMapping] = None,
    ignored_cols: Optional[List[str]] = None,
    metrics: Optional[List[EvidentlyMetricConfig]] = None,
    report_options: Optional[Sequence[Tuple[str, Dict[str, Any]]]] = None,
    download_nltk_data: bool = False,
) -> Tuple[
    Annotated[str, "report_json"], Annotated[HTMLString, "report_html"]
]:
    """Generate an Evidently report on one or two pandas datasets.

    Args:
        reference_dataset: a Pandas DataFrame
        comparison_dataset: a Pandas DataFrame of new data you wish to
            compare against the reference data
        column_mapping: properties of the DataFrame columns used
        ignored_cols: columns to ignore during the Evidently report step
        metrics: a list of Evidently metric configurations to use for the
            report.
        report_options: a list of tuples containing the name of the report
            and a dictionary of options for the report.
        download_nltk_data: whether to download the NLTK data for the report
            step. Defaults to False.

    Returns:
        A tuple containing the Evidently report in JSON and HTML
        formats.
    """
    if not metrics:
        metrics = EvidentlyMetricConfig.default_metrics()

    data_validator = cast(
        EvidentlyDataValidator,
        EvidentlyDataValidator.get_active_data_validator(),
    )

    if ignored_cols:
        exception_msg = (
            "Columns {extra_cols} configured in the `ignored_cols` "
            "parameter are not found in the {dataset} dataset. "
        )
        extra_cols = set(ignored_cols) - set(reference_dataset.columns)
        if extra_cols:
            logger.warning(
                exception_msg.format(
                    extra_cols=extra_cols, dataset="reference"
                )
            )
        reference_dataset = reference_dataset.drop(
            labels=list(set(ignored_cols) - extra_cols), axis=1
        )

        if comparison_dataset is not None:
            extra_cols = set(ignored_cols) - set(comparison_dataset.columns)
            if extra_cols:
                logger.warning(
                    exception_msg.format(
                        extra_cols=extra_cols, dataset="comparison"
                    )
                )

            comparison_dataset = comparison_dataset.drop(
                labels=list(set(ignored_cols) - extra_cols), axis=1
            )

    if column_mapping:
        evidently_column_mapping = column_mapping.to_evidently_column_mapping()
    else:
        evidently_column_mapping = None
    report = data_validator.data_profiling(
        dataset=reference_dataset,
        comparison_dataset=comparison_dataset,
        profile_list=metrics,
        column_mapping=evidently_column_mapping,
        report_options=report_options or [],
        download_nltk_data=download_nltk_data,
    )
    return report.json(), HTMLString(report.show(mode="inline").data)

evidently_test

Implementation of the Evidently Test Step.

Classes Functions

evidently_test_step(reference_dataset: pd.DataFrame, comparison_dataset: Optional[pd.DataFrame], column_mapping: Optional[EvidentlyColumnMapping] = None, ignored_cols: Optional[List[str]] = None, tests: Optional[List[EvidentlyTestConfig]] = None, test_options: Optional[Sequence[Tuple[str, Dict[str, Any]]]] = None, download_nltk_data: bool = False) -> Tuple[Annotated[str, test_json], Annotated[HTMLString, test_html]]

Run an Evidently test suite on one or two pandas datasets.

Parameters:

Name	Type	Description	Default
reference_dataset	DataFrame	a Pandas DataFrame	required
comparison_dataset	Optional[DataFrame]	a Pandas DataFrame of new data you wish to compare against the reference data	required
column_mapping	Optional[EvidentlyColumnMapping]	properties of the DataFrame columns used	None
ignored_cols	Optional[List[str]]	columns to ignore during the Evidently profile step	None
tests	Optional[List[EvidentlyTestConfig]]	a list of Evidently test configuration to use for the test suite.	None
test_options	Optional[Sequence[Tuple[str, Dict[str, Any]]]]	a list of tuples containing the name of the test and a dictionary of options for the test.	None
download_nltk_data	bool	whether to download the NLTK data for the report step. Defaults to False.	False

Returns:

Type	Description
Tuple[Annotated[str, test_json], Annotated[HTMLString, test_html]]	A tuple containing the TestSuite in JSON and HTML formats.

Raises:

Type	Description
ValueError	If ignored_cols is an empty list
ValueError	If column is not found in reference or comparison dataset

Source code in src/zenml/integrations/evidently/steps/evidently_test.py

@step
def evidently_test_step(
    reference_dataset: pd.DataFrame,
    comparison_dataset: Optional[pd.DataFrame],
    column_mapping: Optional[EvidentlyColumnMapping] = None,
    ignored_cols: Optional[List[str]] = None,
    tests: Optional[List[EvidentlyTestConfig]] = None,
    test_options: Optional[Sequence[Tuple[str, Dict[str, Any]]]] = None,
    download_nltk_data: bool = False,
) -> Tuple[Annotated[str, "test_json"], Annotated[HTMLString, "test_html"]]:
    """Run an Evidently test suite on one or two pandas datasets.

    Args:
        reference_dataset: a Pandas DataFrame
        comparison_dataset: a Pandas DataFrame of new data you wish to
            compare against the reference data
        column_mapping: properties of the DataFrame columns used
        ignored_cols: columns to ignore during the Evidently profile step
        tests: a list of Evidently test configuration to use for the test suite.
        test_options: a list of tuples containing the name of the test
            and a dictionary of options for the test.
        download_nltk_data: whether to download the NLTK data for the report
            step. Defaults to False.

    Returns:
        A tuple containing the TestSuite in JSON and HTML formats.

    Raises:
        ValueError: If ignored_cols is an empty list
        ValueError: If column is not found in reference or comparison
            dataset
    """
    if not tests:
        tests = EvidentlyTestConfig.default_tests()

    data_validator = cast(
        EvidentlyDataValidator,
        EvidentlyDataValidator.get_active_data_validator(),
    )

    if ignored_cols:
        extra_cols = set(ignored_cols) - set(reference_dataset.columns)
        if extra_cols:
            raise ValueError(
                f"Columns {extra_cols} configured in the ignored_cols "
                "parameter are not found in the reference dataset."
            )
        reference_dataset = reference_dataset.drop(
            labels=list(ignored_cols), axis=1
        )

        if comparison_dataset is not None:
            extra_cols = set(ignored_cols) - set(comparison_dataset.columns)
            if extra_cols:
                raise ValueError(
                    f"Columns {extra_cols} configured in the ignored_cols "
                    "parameter are not found in the comparison dataset."
                )

            comparison_dataset = comparison_dataset.drop(
                labels=list(ignored_cols), axis=1
            )

    if column_mapping:
        evidently_column_mapping = column_mapping.to_evidently_column_mapping()
    else:
        evidently_column_mapping = None
    test_suite = data_validator.data_validation(
        dataset=reference_dataset,
        comparison_dataset=comparison_dataset,
        check_list=tests,
        column_mapping=evidently_column_mapping,
        test_options=test_options or [],
        download_nltk_data=download_nltk_data,
    )
    return (
        test_suite.json(),
        HTMLString(test_suite.show(mode="inline").data),
    )

tests

ZenML declarative representation of Evidently Tests.

Classes

EvidentlyTestConfig

Bases: BaseModel

Declarative Evidently Test configuration.

This is a declarative representation of the configuration that goes into an Evidently Test, TestPreset or Test generator instance. We need this to be able to store the configuration as part of a ZenML step parameter and later instantiate the Evidently Test from it.

This representation covers all 3 possible ways of configuring an Evidently Test or Test-like object that can later be used in an Evidently TestSuite:

1. A Test (derived from the Test class).
2. A TestPreset (derived from the TestPreset class).
3. A column Test generator (derived from the BaseGenerator class).

Ideally, it should be possible to just pass a Test or Test-like object to this class and have it automatically derive the configuration used to instantiate it. Unfortunately, this is not possible because the Evidently Test classes are not designed in a way that allows us to extract the constructor parameters from them in a generic way.

Attributes:

Name	Type	Description
class_path	str	The full class path of the Evidently Test class.
parameters	Dict[str, Any]	The parameters of the Evidently Test.
is_generator	bool	Whether this is an Evidently column Test generator.
columns	Optional[Union[str, List[str]]]	The columns that the Evidently column Test generator is applied to. Only used if generator is True.

Functions

default_tests() -> List[EvidentlyTestConfig] classmethod

Default Evidently test configurations.

Call this to fetch a default list of Evidently tests to use in cases where no tests are explicitly configured for a data validator. All available Evidently TestPreset classes are used.

Returns:

Type	Description
List[EvidentlyTestConfig]	A list of EvidentlyTestConfig objects to use as default tests.

Source code in src/zenml/integrations/evidently/tests.py

@classmethod
def default_tests(cls) -> List["EvidentlyTestConfig"]:
    """Default Evidently test configurations.

    Call this to fetch a default list of Evidently tests to use in cases
    where no tests are explicitly configured for a data validator.
    All available Evidently TestPreset classes are used.

    Returns:
        A list of EvidentlyTestConfig objects to use as default tests.
    """
    return [
        cls.test(test=test_preset_class_name)
        for test_preset_class_name in test_preset.__all__
    ]

get_test_class(test_name: str) -> Union[Test, TestPreset] staticmethod

Get the Evidently test or test preset class from a string.

Parameters:

Name	Type	Description	Default
test_name	str	The test or test preset class or full class path.	required

Returns:

Type	Description
Union[Test, TestPreset]	The Evidently test or test preset class.

Raises:

Type	Description
ValueError	If the name cannot be converted into a valid Evidently test or test preset class.

Source code in src/zenml/integrations/evidently/tests.py

@staticmethod
def get_test_class(test_name: str) -> Union[Test, TestPreset]:
    """Get the Evidently test or test preset class from a string.

    Args:
        test_name: The test or test preset class or full class
            path.

    Returns:
        The Evidently test or test preset class.

    Raises:
        ValueError: If the name cannot be converted into a valid Evidently
            test or test preset class.
    """
    # First, try to interpret the test name as a full class path.
    if "." in test_name:
        try:
            test_class = source_utils.load(test_name)
        except (ImportError, AttributeError) as e:
            raise ValueError(
                f"Could not import Evidently Test or TestPreset "
                f"`{test_name}`: {str(e)}"
            )

    else:
        # Next, try to interpret the test as a Test or TestPreset
        # class name
        if hasattr(tests, test_name):
            test_class = getattr(tests, test_name)
        elif hasattr(test_preset, test_name):
            test_class = getattr(test_preset, test_name)
        else:
            raise ValueError(
                f"Could not import Evidently Test or TestPreset "
                f"`{test_name}`"
            )

    if not issubclass(test_class, (Test, TestPreset)):
        raise ValueError(
            f"Class `{test_name}` is not a valid Evidently "
            f"Test or TestPreset."
        )

    return test_class

test(test: Union[Type[Test], Type[TestPreset], str], **parameters: Any) -> EvidentlyTestConfig classmethod

Create a declarative configuration for an Evidently Test.

Call this method to get a declarative representation for the configuration of an Evidently Test.

Some examples

from zenml.integrations.evidently.data_validators import EvidentlyTest

# Configure an Evidently TestPreset using its class name
config = EvidentlyTest.test("DataDriftPreset")

from zenml.integrations.evidently.data_validators import EvidentlyTest

# Configure an Evidently TestPreset using its full class path
config = EvidentlyTest.test(
    "evidently.test_preset.DataDriftPreset"
)

from zenml.integrations.evidently.data_validators import EvidentlyTest

# Configure an Evidently Test using its class and pass additional
# parameters
from evidently.tests import ColumnSummaryTest
config = EvidentlyTest.test(
    ColumnSummaryTest, column_name="age"
)

Parameters:

Name	Type	Description	Default
test	Union[Type[Test], Type[TestPreset], str]	The Evidently Test or TestPreset class, class name or class path.	required
parameters	Any	Additional optional parameters needed to instantiate the Evidently Test or TestPreset.	{}

Returns:

Type	Description
EvidentlyTestConfig	The EvidentlyTest declarative representation of the Evidently
EvidentlyTestConfig	Test configuration.

Raises:

Type	Description
ValueError	If test does not point to a valid Evidently Test or TestPreset class.

Source code in src/zenml/integrations/evidently/tests.py

@classmethod
def test(
    cls,
    test: Union[Type[Test], Type[TestPreset], str],
    **parameters: Any,
) -> "EvidentlyTestConfig":
    """Create a declarative configuration for an Evidently Test.

    Call this method to get a declarative representation for the
    configuration of an Evidently Test.

    Some examples
    -------------

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyTest

    # Configure an Evidently TestPreset using its class name
    config = EvidentlyTest.test("DataDriftPreset")
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyTest

    # Configure an Evidently TestPreset using its full class path
    config = EvidentlyTest.test(
        "evidently.test_preset.DataDriftPreset"
    )
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyTest

    # Configure an Evidently Test using its class and pass additional
    # parameters
    from evidently.tests import ColumnSummaryTest
    config = EvidentlyTest.test(
        ColumnSummaryTest, column_name="age"
    )
    ```

    Args:
        test: The Evidently Test or TestPreset class, class name or
            class path.
        parameters: Additional optional parameters needed to instantiate the
            Evidently Test or TestPreset.

    Returns:
        The EvidentlyTest declarative representation of the Evidently
        Test configuration.

    Raises:
        ValueError: If `test` does not point to a valid Evidently Test
            or TestPreset class.
    """
    if isinstance(test, str):
        test_class = cls.get_test_class(test)
    elif issubclass(test, (Test, TestPreset)):
        test_class = test
    else:
        raise ValueError(
            f"Invalid Evidently Test or TestPreset class: {test}"
        )

    class_path = f"{test_class.__module__}.{test_class.__name__}"
    config = cls(class_path=class_path, parameters=parameters)

    # Try to instantiate the configuration to check if the parameters are
    # valid
    config.to_evidently_test()

    return config

test_generator(test: Union[Type[Test], str], columns: Optional[Union[str, List[str]]] = None, **parameters: Any) -> EvidentlyTestConfig classmethod

Create a declarative configuration for an Evidently column Test generator.

Call this method to get a declarative representation for the configuration of an Evidently column Test generator.

The columns, parameters arguments will be passed to the Evidently generate_column_tests function:

• if columns is a list, it is interpreted as a list of column names.
• if columns is a string, it can be one of values:
  • "all" - use all columns, including target/prediction columns
  • "num" - for numeric features
  • "cat" - for category features
  • "text" - for text features
  • "features" - for all features, not target/prediction columns.
• a None value is the same as "all".

Some examples

from zenml.integrations.evidently.data_validators import EvidentlyTest

# Configure an Evidently Test generator using a Test class name
# and pass additional parameters
config = EvidentlyTest.test_generator(
    "TestColumnValueMin", columns="num", gt=0.5
)

from zenml.integrations.evidently.data_validators import EvidentlyTest

# Configure an Evidently Test generator using a full Test class
# path
config = EvidentlyTest.test_generator(
    "evidently.tests.TestColumnShareOfMissingValues", columns=["age", "name"]
)

from zenml.integrations.evidently.data_validators import EvidentlyTest

# Configure an Evidently Test generator using a Test class
from evidently.tests import TestColumnQuantile
config = EvidentlyTest.test_generator(
    TestColumnQuantile, columns="all", quantile=0.5
)

Parameters:

Name	Type	Description	Default
test	Union[Type[Test], str]	The Evidently Test class, class name or class path to use for the generator.	required
columns	Optional[Union[str, List[str]]]	The columns to apply the generator to. Takes the same values that the Evidently generate_column_tests function takes.	None
parameters	Any	Additional optional parameters needed to instantiate the Evidently Test. These will be passed to the Evidently generate_column_tests function.	{}

Returns:

Type	Description
EvidentlyTestConfig	The EvidentlyTest declarative representation of the Evidently
EvidentlyTestConfig	Test generator configuration.

Raises:

Type	Description
ValueError	If test does not point to a valid Evidently Test or TestPreset class.

Source code in src/zenml/integrations/evidently/tests.py

@classmethod
def test_generator(
    cls,
    test: Union[Type[Test], str],
    columns: Optional[Union[str, List[str]]] = None,
    **parameters: Any,
) -> "EvidentlyTestConfig":
    """Create a declarative configuration for an Evidently column Test generator.

    Call this method to get a declarative representation for the
    configuration of an Evidently column Test generator.

    The `columns`, `parameters` arguments will be
    passed to the Evidently `generate_column_tests` function:

    - if `columns` is a list, it is interpreted as a list of column names.
    - if `columns` is a string, it can be one of values:
        - "all" - use all columns, including target/prediction columns
        - "num" - for numeric features
        - "cat" - for category features
        - "text" - for text features
        - "features" - for all features, not target/prediction columns.
    - a None value is the same as "all".

    Some examples
    -------------

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyTest

    # Configure an Evidently Test generator using a Test class name
    # and pass additional parameters
    config = EvidentlyTest.test_generator(
        "TestColumnValueMin", columns="num", gt=0.5
    )
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyTest

    # Configure an Evidently Test generator using a full Test class
    # path
    config = EvidentlyTest.test_generator(
        "evidently.tests.TestColumnShareOfMissingValues", columns=["age", "name"]
    )
    ```

    ```python
    from zenml.integrations.evidently.data_validators import EvidentlyTest

    # Configure an Evidently Test generator using a Test class
    from evidently.tests import TestColumnQuantile
    config = EvidentlyTest.test_generator(
        TestColumnQuantile, columns="all", quantile=0.5
    )
    ```

    Args:
        test: The Evidently Test class, class name or class path to use
            for the generator.
        columns: The columns to apply the generator to. Takes the same
            values that the Evidently `generate_column_tests` function
            takes.
        parameters: Additional optional parameters needed to instantiate the
            Evidently Test. These will be passed to the Evidently
            `generate_column_tests` function.

    Returns:
        The EvidentlyTest declarative representation of the Evidently
        Test generator configuration.

    Raises:
        ValueError: If `test` does not point to a valid Evidently Test
            or TestPreset class.
    """
    if isinstance(test, str):
        test_class = cls.get_test_class(test)
    elif issubclass(test, (Test, TestPreset)):
        test_class = test
    else:
        raise ValueError(f"Invalid Evidently Test class: {test}")

    class_path = f"{test_class.__module__}.{test_class.__name__}"

    config = cls(
        class_path=class_path,
        parameters=parameters,
        columns=columns,
        is_generator=True,
    )

    # Try to instantiate the configuration to check if the parameters are
    # valid
    config.to_evidently_test()

    return config

to_evidently_test() -> Union[Test, TestPreset, BaseGenerator]

Create an Evidently Test, TestPreset or test generator object.

Call this method to create an Evidently Test, TestPreset or test generator instance from its declarative representation.

Returns:

Type	Description
Union[Test, TestPreset, BaseGenerator]	The Evidently Test, TestPreset or test generator object.

Raises:

Type	Description
ValueError	If the Evidently Test, TestPreset or column test generator could not be instantiated.

Source code in src/zenml/integrations/evidently/tests.py

def to_evidently_test(self) -> Union[Test, TestPreset, BaseGenerator]:
    """Create an Evidently Test, TestPreset or test generator object.

    Call this method to create an Evidently Test, TestPreset or test
    generator instance from its declarative representation.

    Returns:
        The Evidently Test, TestPreset or test generator object.

    Raises:
        ValueError: If the Evidently Test, TestPreset or column test
            generator could not be instantiated.
    """
    test_class = self.get_test_class(self.class_path)

    if self.is_generator:
        try:
            return generate_column_tests(
                test_class=test_class,
                columns=self.columns,
                parameters=self.parameters,
            )
        except Exception as e:
            raise ValueError(
                f"Could not instantiate Evidently column Test generator "
                f"`{self.class_path}`: {str(e)}"
            )

    try:
        return test_class(**self.parameters)
    except Exception as e:
        raise ValueError(
            f"Could not instantiate Evidently Test or TestPreset "
            f"`{self.class_path}`: {str(e)}"
        )

Functions

Modules