evidently.test_preset

Predefined Test Presets for Test Suite

class BinaryClassificationTestPreset(stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, stattest_threshold: Optional[float] = None, probas_threshold: Optional[float] = None)

Bases: TestPreset

Binary Classification Tests. :param stattest: statistical test for target drift test. :param stattest_threshold: threshold for statistical test for target drift test. :param probas_threshold: threshold for label calculation for prediction.

Contains:

• TestColumnValueDrift for target

• TestPrecisionScore

• TestRecallScore

• TestF1Score

• TestAccuracyScore

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class BinaryClassificationTopKTestPreset(k: Union[float, int], stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, stattest_threshold: Optional[float] = None)

Bases: TestPreset

Binary Classification Tests for Top K threshold. :param stattest: stattest for TestColumnDrift :param stattest_threshold: threshold for stattest

Contains:

• TestColumnDrift for target

• TestPrecisionScore

• TestRecallScore

• TestF1Score

• TestAccuracyScore

Attributes:

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

stattest_threshold : Optional[float]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class DataDriftTestPreset(columns: Optional[List[str]] = None, drift_share: Optional[float] = None, stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, cat_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, num_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, per_column_stattest: Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]] = None, stattest_threshold: Optional[float] = None, cat_stattest_threshold: Optional[float] = None, num_stattest_threshold: Optional[float] = None, per_column_stattest_threshold: Optional[Dict[str, float]] = None)

Bases: TestPreset

Data Drift tests.

Contains tests:

• TestShareOfDriftedColumns

• TestColumnValueDrift

• TestAllFeaturesValueDrift

Attributes:

cat_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

cat_stattest_threshold : Optional[float]

columns : Optional[List[str]]

drift_share : Optional[float]

num_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

num_stattest_threshold : Optional[float]

per_column_stattest : Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]]

per_column_stattest_threshold : Optional[Dict[str, float]]

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

stattest_threshold : Optional[float]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class DataQualityTestPreset(columns: Optional[List[str]] = None)

Bases: TestPreset

Data Quality tests.

Contains tests:

• TestAllColumnsShareOfMissingValues

• TestAllColumnsMostCommonValueShare

• TestNumberOfConstantColumns

• TestNumberOfDuplicatedColumns

• TestNumberOfDuplicatedRows

• TestHighlyCorrelatedColumns

Attributes:

columns : Optional[List[str]]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class DataStabilityTestPreset(columns: Optional[List[str]] = None)

Bases: TestPreset

Data Stability tests.

Contains tests:

• TestNumberOfRows

• TestNumberOfColumns

• TestColumnsType

• TestAllColumnsShareOfMissingValues

• TestNumColumnsOutOfRangeValues

• TestCatColumnsOutOfListValues

• TestNumColumnsMeanInNSigmas

Attributes:

columns : Optional[List[str]]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class MulticlassClassificationTestPreset(stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, stattest_threshold: Optional[float] = None)

Bases: TestPreset

Multiclass Classification tests.

• Parameters

  • stattest – statistical test for target drift test.

  • stattest_threshold – threshold for statistical test for target drift test.

Contains tests:

• TestAccuracyScore

• TestF1Score

• TestPrecisionByClass for each class in data

• TestRecallByClass for each class in data

• TestNumberOfRows

• TestColumnValueDrift

• TestRocAuc

• TestLogLoss

Attributes:

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

stattest_threshold : Optional[float]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class NoTargetPerformanceTestPreset(columns: Optional[List[str]] = None, drift_share: Optional[float] = None, stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, cat_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, num_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, per_column_stattest: Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]] = None, stattest_threshold: Optional[float] = None, cat_stattest_threshold: Optional[float] = None, num_stattest_threshold: Optional[float] = None, per_column_stattest_threshold: Optional[Dict[str, float]] = None)

Bases: TestPreset

No Target Performance tests.

• Parameters

  columns – list of columns include to tests

Contains tests:

• TestColumnValueDrift

• TestShareOfDriftedColumns

• TestColumnsType

• TestAllColumnsShareOfMissingValues

• TestNumColumnsOutOfRangeValues

• TestCatColumnsOutOfListValues

• TestNumColumnsMeanInNSigmas

• TestCustomFeaturesValueDrift

Attributes:

cat_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None

cat_stattest_threshold : Optional[float] = None

columns : Optional[List[str]]

drift_share : Optional[float]

num_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None

num_stattest_threshold : Optional[float] = None

per_column_stattest : Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]] = None

per_column_stattest_threshold : Optional[Dict[str, float]] = None

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None

stattest_threshold : Optional[float] = None

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

class RegressionTestPreset()

Bases: TestPreset

Regression performance tests.

Contains tests:

• TestValueMeanError

• TestValueMAE

• TestValueRMSE

• TestValueMAPE

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

Submodules

classification_binary module

class BinaryClassificationTestPreset(stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, stattest_threshold: Optional[float] = None, probas_threshold: Optional[float] = None)

Bases: TestPreset

Binary Classification Tests. :param stattest: statistical test for target drift test. :param stattest_threshold: threshold for statistical test for target drift test. :param probas_threshold: threshold for label calculation for prediction.

Contains:

• TestColumnValueDrift for target

• TestPrecisionScore

• TestRecallScore

• TestF1Score

• TestAccuracyScore

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

classification_binary_topk module

class BinaryClassificationTopKTestPreset(k: Union[float, int], stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, stattest_threshold: Optional[float] = None)

Bases: TestPreset

Binary Classification Tests for Top K threshold. :param stattest: stattest for TestColumnDrift :param stattest_threshold: threshold for stattest

Contains:

• TestColumnDrift for target

• TestPrecisionScore

• TestRecallScore

• TestF1Score

• TestAccuracyScore

Attributes:

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

stattest_threshold : Optional[float]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

classification_multiclass module

class MulticlassClassificationTestPreset(stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, stattest_threshold: Optional[float] = None)

Bases: TestPreset

Multiclass Classification tests.

• Parameters

  • stattest – statistical test for target drift test.

  • stattest_threshold – threshold for statistical test for target drift test.

Contains tests:

• TestAccuracyScore

• TestF1Score

• TestPrecisionByClass for each class in data

• TestRecallByClass for each class in data

• TestNumberOfRows

• TestColumnValueDrift

• TestRocAuc

• TestLogLoss

Attributes:

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

stattest_threshold : Optional[float]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

data_drift module

class DataDriftTestPreset(columns: Optional[List[str]] = None, drift_share: Optional[float] = None, stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, cat_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, num_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, per_column_stattest: Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]] = None, stattest_threshold: Optional[float] = None, cat_stattest_threshold: Optional[float] = None, num_stattest_threshold: Optional[float] = None, per_column_stattest_threshold: Optional[Dict[str, float]] = None)

Bases: TestPreset

Data Drift tests.

Contains tests:

• TestShareOfDriftedColumns

• TestColumnValueDrift

• TestAllFeaturesValueDrift

Attributes:

cat_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

cat_stattest_threshold : Optional[float]

columns : Optional[List[str]]

drift_share : Optional[float]

num_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

num_stattest_threshold : Optional[float]

per_column_stattest : Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]]

per_column_stattest_threshold : Optional[Dict[str, float]]

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]

stattest_threshold : Optional[float]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

data_quality module

class DataQualityTestPreset(columns: Optional[List[str]] = None)

Bases: TestPreset

Data Quality tests.

Contains tests:

• TestAllColumnsShareOfMissingValues

• TestAllColumnsMostCommonValueShare

• TestNumberOfConstantColumns

• TestNumberOfDuplicatedColumns

• TestNumberOfDuplicatedRows

• TestHighlyCorrelatedColumns

Attributes:

columns : Optional[List[str]]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

data_stability module

class DataStabilityTestPreset(columns: Optional[List[str]] = None)

Bases: TestPreset

Data Stability tests.

Contains tests:

• TestNumberOfRows

• TestNumberOfColumns

• TestColumnsType

• TestAllColumnsShareOfMissingValues

• TestNumColumnsOutOfRangeValues

• TestCatColumnsOutOfListValues

• TestNumColumnsMeanInNSigmas

Attributes:

columns : Optional[List[str]]

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

no_target_performance module

class NoTargetPerformanceTestPreset(columns: Optional[List[str]] = None, drift_share: Optional[float] = None, stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, cat_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, num_stattest: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None, per_column_stattest: Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]] = None, stattest_threshold: Optional[float] = None, cat_stattest_threshold: Optional[float] = None, num_stattest_threshold: Optional[float] = None, per_column_stattest_threshold: Optional[Dict[str, float]] = None)

Bases: TestPreset

No Target Performance tests.

• Parameters

  columns – list of columns include to tests

Contains tests:

• TestColumnValueDrift

• TestShareOfDriftedColumns

• TestColumnsType

• TestAllColumnsShareOfMissingValues

• TestNumColumnsOutOfRangeValues

• TestCatColumnsOutOfListValues

• TestNumColumnsMeanInNSigmas

• TestCustomFeaturesValueDrift

Attributes:

cat_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None

cat_stattest_threshold : Optional[float] = None

columns : Optional[List[str]]

drift_share : Optional[float]

num_stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None

num_stattest_threshold : Optional[float] = None

per_column_stattest : Optional[Dict[str, Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]]] = None

per_column_stattest_threshold : Optional[Dict[str, float]] = None

stattest : Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]] = None

stattest_threshold : Optional[float] = None

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

regression module

class RegressionTestPreset()

Bases: TestPreset

Regression performance tests.

Contains tests:

• TestValueMeanError

• TestValueMAE

• TestValueRMSE

• TestValueMAPE

Methods:

generate_tests(data: InputData, columns: DatasetColumns)

test_preset module

class TestPreset()

Bases: object

Methods:

abstract generate_tests(data: InputData, columns: DatasetColumns)