evidently.calculations.stattests

Available statistical tests. For detailed information about statistical tests see module documentation.

Submodules

anderson_darling_stattest module

Anderson-Darling test of two samples.

Name: “anderson”

Import:

>>> from evidently.calculations.stattests import anderson_darling_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import anderson_darling_test
>>> options = DataDriftOptions(all_features_stattest=anderson_darling_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="anderson")

chisquare_stattest module

Chisquare test of two samples.

Name: “chisquare”

Import:

>>> from evidently.calculations.stattests import chi_stat_test

Properties:

• only for categorical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import chi_stat_test
>>> options = DataDriftOptions(all_features_stattest=chi_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="chisquare")

cramer_von_mises_stattest module

Cramer-Von-mises test of two samples.

Name: “cramer_von_mises”

Import:

>>> from evidently.calculations.stattests import cramer_von_mises

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import cramer_von_mises
>>> options = DataDriftOptions(all_features_stattest=cramer_von_mises)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="cramer_von_mises")

class CramerVonMisesResult(statistic, pvalue)

Bases: object

energy_distance module

Energy-distance test of two samples.

Name: “ed”

Import:

>>> from evidently.calculations.stattests import energy_dist_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import energy_dist_test
>>> options = DataDriftOptions(all_features_stattest=energy_dist_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="ed")

epps_singleton_stattest module

Epps-Singleton test of two samples.

Name: “es”

Import:

>>> from evidently.calculations.stattests import epps_singleton_test

Properties:

• only for numerical features

• returns p-value

• default threshold 0.05

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import epps_singleton_test
>>> options = DataDriftOptions(all_features_stattest=epps_singleton_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="es")

fisher_exact_stattest module

Fisher’s exact test of two samples.

Name: “fisher_exact”

Import:

>>> from evidently.calculations.stattests import fisher_exact_test

Properties:

• only for categorical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import fisher_exact_test
>>> options = DataDriftOptions(all_features_stattest=fisher_exact_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="fisher_exact")

g_stattest module

G-test of two samples.

Name: “g_test”

Import:

>>> from evidently.calculations.stattests import g_test

Properties:

• only for categorical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import g_test
>>> options = DataDriftOptions(all_features_stattest=g_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="g_test")

hellinger_distance module

Hellinger distance of two samples.

Name: “hellinger”

Import:

>>> from evidently.calculations.stattests import hellinger_stat_test

Properties:

• only for categorical and numerical features

• returns distance

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import hellinger_stat_test
>>> options = DataDriftOptions(all_features_stattest=hellinger_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="hellinger")

jensenshannon module

Jensen-Shannon distance of two samples.

Name: “jensenshannon”

Import:

>>> from evidently.calculations.stattests import jensenshannon_stat_test

Properties:

• only for categorical and numerical features

• returns distance

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import jensenshannon_stat_test
>>> options = DataDriftOptions(all_features_stattest=jensenshannon_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="jensenshannon")

kl_div module

Kullback-Leibler divergence of two samples.

Name: “kl_div”

Import:

>>> from evidently.calculations.stattests import kl_div_stat_test

Properties:

• only for categorical and numerical features

• returns divergence

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import kl_div_stat_test
>>> options = DataDriftOptions(all_features_stattest=kl_div_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="kl_div")

ks_stattest module

Kolmogorov-Smirnov test of two samples.

Name: “ks”

Import:

>>> from evidently.calculations.stattests import ks_stat_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import ks_stat_test
>>> options = DataDriftOptions(all_features_stattest=ks_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="ks")

mann_whitney_urank_stattest module

Mann-Whitney U-rank test of two samples.

Name: “mannw”

Import:

>>> from evidently.calculations.stattests import mann_whitney_u_stat_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import mann_whitney_u_stat_test
>>> options = DataDriftOptions(all_features_stattest=mann_whitney_u_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="mannw")

psi module

PSI of two samples.

Name: “psi”

Import:

>>> from evidently.calculations.stattests import psi_stat_test

Properties:

• only for categorical and numerical features

• returns PSI value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import psi_stat_test
>>> options = DataDriftOptions(all_features_stattest=psi_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="psi")

registry module

class StatTest(name: str, display_name: str, func: Callable[[pandas.core.series.Series, pandas.core.series.Series, str, float], Tuple[float, bool]], allowed_feature_types: List[str], default_threshold: float = 0.05)

Bases: object

Attributes:

allowed_feature_types : List[str]

default_threshold : float = 0.05

display_name : str

func : Callable[[Series, Series, str, float], Tuple[float, bool]]

name : str

exception StatTestInvalidFeatureTypeError(stattest_name: str, feature_type: str)

Bases: ValueError

exception StatTestNotFoundError(stattest_name: str)

Bases: ValueError

class StatTestResult(drift_score: float, drifted: bool, actual_threshold: float)

Bases: object

Attributes:

actual_threshold : float

drift_score : float

drifted : bool

get_stattest(reference_data: Series, current_data: Series, feature_type: str, stattest_func: Optional[Union[str, Callable[[Series, Series, str, float], Tuple[float, bool]], StatTest]])

register_stattest(stat_test: StatTest)

t_test module

T test of two samples.

Name: “t_test”

Import:

>>> from evidently.calculations.stattests import t_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import t_test
>>> options = DataDriftOptions(all_features_stattest=t_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="t_test")

tvd_stattest module

Total variation distance of two samples.

Name: “TVD”

Import:

>>> from evidently.calculations.stattests import tvd_test

Properties:

• only for numerical features

• returns distance

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import tvd_test
>>> options = DataDriftOptions(all_features_stattest=tvd_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="TVD")

utils module

generate_fisher2x2_contingency_table(reference_data: Series, current_data: Series)

Generate 2x2 contingency matrix for fisher exact test :param reference_data: reference data :param current_data: current data

• Raises

  ValueError – if reference_data and current_data are not of equal length

• Returns

  contingency_matrix for binary data

• Return type

  contingency_matrix

get_binned_data(reference_data: Series, current_data: Series, feature_type: str, n: int, feel_zeroes: bool = True)

Split variable into n buckets based on reference quantiles :param reference_data: reference data :param current_data: current data :param feature_type: feature type :param n: number of quantiles

• Returns

  % of records in each bucket for reference current_percents: % of records in each bucket for current

• Return type

  reference_percents

get_unique_not_nan_values_list_from_series(current_data: Series, reference_data: Series)

Get unique values from current and reference series, drop NaNs

permutation_test(reference_data, current_data, observed, test_statistic_func, iterations=100)

Perform a two-sided permutation test :param reference_data: reference data :param current_data: current data :param observed: observed value :param test_statistic_func: the test statistic function :param iterations: number of times to permute

• Returns

  two-sided p_value

• Return type

  p_value

wasserstein_distance_norm module

Wasserstein distance of two samples.

Name: “wasserstein”

Import:

>>> from evidently.calculations.stattests import wasserstein_stat_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import wasserstein_stat_test
>>> options = DataDriftOptions(all_features_stattest=wasserstein_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="wasserstein")

z_stattest module

Mann-Whitney U-rank test of two samples.

Name: “mannw”

Import:

>>> from evidently.calculations.stattests import mann_whitney_u_stat_test

Properties:

• only for numerical features

• returns p-value

Example

Using by object:

>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import mann_whitney_u_stat_test
>>> options = DataDriftOptions(all_features_stattest=mann_whitney_u_stat_test)

Using by name:

>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="mannw")

proportions_diff_z_stat_ind(ref: DataFrame, curr: DataFrame)

proportions_diff_z_test(z_stat, alternative='two-sided')