# What is Evidently?

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Evidently helps evaluate, test, and monitor data and ML-powered systems.

* Predictive tasks: classification, regression, ranking, recommendations.
* Generative tasks: chatbots, RAGs, Q\&A, summarization.
* Data monitoring: data quality and data drift for text, tabular data, embeddings.

Evidently is available both as an open-source Python library and Evidently Cloud platform.

## Get started

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Evidently Cloud</strong></td><td>AI evaluation and observability platform built on top of Evidently Python library. Includes advanced features, collaboration and support.</td><td><a href="/get-started/quickstart-cloud/cloud_quickstart_llm">→ LLM Evals Quickstart</a><br><a href="/get-started/quickstart-cloud/cloud_quickstart_tabular">→ Data and ML Quickstart</a><br><a href="/get-started/quickstart-cloud/cloud_quickstart_tracing">→ LLM Tracing Quickstart</a></td></tr><tr><td><strong>Evidently Open-Source</strong></td><td>An open-source Python library with 20m+ downloads. Helps evaluate, test and monitor data, ML and LLM-powered systems.</td><td><a href="/get-started/hello-world/oss_quickstart_llm">→ LLM Evals Quickstart</a><br><a href="/get-started/hello-world/oss_quickstart_tabular">→ Data and ML Quickstart</a><br><a href="/tutorials-and-examples/tutorial-monitoring">→ Self-hosted dashboard</a></td></tr></tbody></table>

You can explore more in-depth [Examples and Tutorials](/tutorials-and-examples/examples).

## How it works

Evidently helps evaluate and track quality of ML-based systems, from experimentation to production.

Evidently is both a library of 100+ ready-made evaluations, and a framework to easily implement yours: from Python functions to LLM judges.

Evidently has a modular architecture, and you can start with ad hoc checks without complex installations. There are 3 interfaces: you can get a visual `Report` to see a summary of evaluation metrics, run conditional checks with a `TestSuite` to get a pass/fail outcome, or plot the evaluation results over time on a Monitoring `Dashboard`.

### Reports

Reports compute different metrics on data and ML quality. You can use Reports for visual analysis and debugging, or as a computation layer for the monitoring dashboard.

You can be as hands-off or hands-on as you like: start with Presets, and customize metrics as you go.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3a9e8034a127297a8e5db990e6414107cfa282a2%2Freports-min.png?alt=media)

<details>

<summary>More on Reports</summary>

* You can pass a single dataset or two for **side-by-side comparison**.
  * Pass data as a CSV, pandas or Spark dataframe.
* You can get **pre-built Reports** with [Presets](/presets/all-presets), or combine [individual Metrics](/reference/all-metrics).
* You can use Reports as a **standalone tool**:
  * For exploration and debugging: view results in Python or export as HTML.
  * As a computation layer: export results to Python dictionary, JSON or dataframe.
  * For documentation: add text comments and save Model Card.
* You can also use Reports as a **component of ML Monitoring system**:
  * Compute Reports on a cadence over live data and save as JSON snapshots.
  * Visualize results from multiple Reports over time on the Monitoring Dashboard.
  * Configure alerts when metrics are out of bounds.

**Docs**:

* [Reference: available Metrics](/reference/all-metrics)
* [User guide: how to get Reports](/user-guide/tests-and-reports/get-reports)

</details>

### Tests suites

Tests verify whether computed metrics satisfy defined conditions. Each Test returns a pass or fail result.

This interface helps automate your evaluations for regression testing, checks during CI/CD, or validation steps in data pipelines.&#x20;

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-96b355f8fc850d9e3aec7c6d6805303dc0001a32%2Ftests.gif?alt=media)

<details>

<summary>More on Test Suites</summary>

* You can set Test conditions manually or **auto-generate conditions** from a reference dataset.&#x20;
* You can get **pre-built Test Suites** with [Presets](/presets/all-presets), or combine [individual Tests](/reference/all-tests).
* You can see Test results in a visual report or get a JSON or Python export.
* You can use Test Suites as a **standalone tool**:
  * Regression testing during experimentation.
  * Automated CI/CD checks after you get new labeled data or update models.
  * Pipeline testing: add a validation step to your data pipelines. &#x20;
* You can also use Test Suites as a **component of ML Monitoring system**:
  * Run automated Test Suites and save results as JSON snapshots.
  * Show test outcomes and metrics metrics on the Monitoring Dashboard.&#x20;
  * Configure alerts on failed Tests.

**Docs**:

* [Reference: available Tests](/reference/all-tests)
* [User guide: how to generate Tests](/user-guide/tests-and-reports/run-tests)

</details>

### ML monitoring dashboard

The monitoring dashboard helps visualize ML system performance over time and detect issues. You can track key metrics and test outcomes.

You can use Evidently Cloud or self-host. Evidently Cloud offers extra features like user authentication and roles, built-in alerting, and a no-code interface.&#x20;

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b5fa709bf1a88a5e8e60fbc61163626d69bed224%2Fdashboard.gif?alt=media)

<details>

<summary>More on Monitoring Dashboard</summary>

* You save Reports or Test Suites as JSON snapshots. The monitoring dashboard runs over these evaluation results as a data source.
* You can create custom combinations of Panels and choose what exactly to plot.
* You can get dashboards as code for version control and reproducibility.&#x20;
* You can send data in near real-time using a Collector service or in batches.&#x20;
* For Evidently Cloud: send alerts to Slack, Discord, and email.&#x20;
* For Evidently Cloud: get pre-built Tabs and manage everything in the UI.

**Docs**:

* [Monitoring user guide](/user-guide/monitoring/monitoring_overview)

</details>

## What can you evaluate?

Evidently Reports, Test Suites and ML Monitoring dashboard rely on the shared set of metrics. Here are some examples of what you can evaluate.

| Evaluation group                | Examples                                                                                                                                            |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Tabular Data Quality**        | Missing values, duplicates, empty rows or columns, min-max ranges, new categorical values, correlation changes, etc.                                |
| **Text Descriptors**            | Text length, out-of-vocabulary words, share of special symbols, regular expressions matches.                                                        |
| **Data Distribution Drift**     | Statistical tests and distance metrics to compare distributions of model predictions, numerical and categorical features, text data, or embeddings. |
| **Classification Quality**      | Accuracy, precision, recall, ROC AUC, confusion matrix, class separation quality, classification bias.                                              |
| **Regression Quality**          | MAE, ME, RMSE, error distribution, error normality, error bias per group and feature.                                                               |
| **Ranking and Recommendations** | NDCG, MAP, MRR, Hit Rate, recommendation serendipity, novelty, diversity, popularity bias.                                                          |
| **LLM Output Quality**          | Model-based scoring with external models and LLMs to detect toxicity, sentiment, evaluate retrieval relevance, etc.                                 |

You can also implement custom checks as Python functions or define your prompts for LLM-as-a-judge.

**See more**:

* [Reference: available Metrics](/reference/all-metrics)
* [Reference: available Tests](/reference/all-tests)
* [Presets: pre-built evaluation suites](/presets/all-presets)

## Community and support

Evidently is in active development, and we are happy to receive and incorporate feedback. If you have any questions, ideas or want to hang out and chat about doing ML in production, [join our Discord community](https://discord.com/invite/xZjKRaNp8b)!

## User newsletter

To get updates on new features, integrations and code tutorials, sign up for the [Evidently User Newsletter](https://www.evidentlyai.com/user-newsletter).


# Get Started


# Evidently Cloud

Get started with Evidently Cloud.

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Tracing</strong></td><td>Instrument your LLM application to send inputs and outputs to Evidently Cloud.</td><td><a href="/get-started/quickstart-cloud/cloud_quickstart_tracing">→ Tracing Quickstart</a></td></tr><tr><td><strong>LLM evaluations</strong></td><td>Evaluate and monitor the quality of LLM system inputs and outputs.</td><td><a href="/get-started/quickstart-cloud/cloud_quickstart_llm">→ LLM Quickstart</a></td></tr><tr><td><strong>Data and ML monitoring</strong></td><td>Evaluate and monitor data drift, data quality, or ML model performance.</td><td><a href="/get-started/quickstart-cloud/cloud_quickstart_tabular">→ Tabular Quickstart</a></td></tr><tr><td><strong>No-code evaluations</strong></td><td>Upload a CSV file to run evaluations directly in the Evidently Cloud.</td><td><a href="/user-guide/evaluations/no_code_evals">→ No-code Quickstart</a></td></tr></tbody></table>


# Quickstart - LLM tracing

LLM tracing "Hello world."

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This quickstart shows how to instrument a simple LLM app to send inputs and outputs to Evidently Cloud. You will use the open-source **Tracely** library.

You will need an OpenAI key to create a toy LLM app.

Need help? Ask on [Discord](https://discord.com/invite/xZjKRaNp8b).

## 1. Set up Evidently Cloud

Set up your Evidently Cloud workspace:

* **Sign up**. If you do not have one yet, sign up for a free [Evidently Cloud account](https://app.evidently.cloud/signup).
* **Create an Organization**. When you log in the first time, create and name your Organization.
* **Create a Project**. Click **+** button under Project List. Create a Project, copy and save the Project ID. ([Projects page](https://app.evidently.cloud/))
* **Get your API token**. Click the **Key** icon in the left menu. Generate and save the token. ([Token page](https://app.evidently.cloud/token)).

You can now go to your Python environment.

## 2. Installation

Install the Tracely library to instrument your app:

```python
!pip install tracely
```

Install the Evidently library to interact with Evidently Cloud:

```python
!pip install evidently
```

Install the OpenAI library to create a toy app:

```python
!pip install openai
```

Imports:

```python
import os
import openai
import time
from tracely import init_tracing
from tracely import trace_event
```

## 2. Initialize Tracing

Initialize the OpenAI client. Pass the token as an environment variable:

```python
# os.environ["OPENAI_API_KEY"] = "YOUR_KEY"
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
```

Set up tracing parameters. Give it a name to identify your tracing dataset.

```python
init_tracing(
    address="https://app.evidently.cloud/",
    api_key="EVIDENTLY_API_KEY",
    project_id="YOUR_PROJECT_ID"
    export_name="LLM tracing example"
    )
```

## 3. Trace a simple function

Create a simple function to send questions to Open AI API and receive a completion. Set the questions list:

```python
question_list = [
    "What is Evidently Python library?",
    "What is LLM observability?",
    "How is MLOps different from LLMOps?"
]
```

Create a function and use the `trace_event()` decorator to trace it:

```python
@trace_event()
def pseudo_assistant(question):
    system_prompt = "You are a helpful assistant. Please answer the following question concisely."
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": question},
    ]
    return client.chat.completions.create(model="gpt-4o-mini", messages=messages).choices[0].message.content

# Iterate over the list of questions and pass each to the assistant
for question in question_list:
    response = pseudo_assistant(question=question)
    time.sleep(1)
```

## 4. View Traces

Go to the Evidently Cloud, open Datasets in the left menu ([Datasets Page](https://app.evidently.cloud/datasets)), and view your Traces.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-4f7134458e3621bc63408694cec64cbbd26131e6%2Fqs_tracing_dataset.png?alt=media)

## What's next?

Want to run evaluations over this data? See a Quickstart.

{% content-ref url="/pages/82JD0PYdxbIagT53baIy" %}
[Quickstart - LLM evaluations](/get-started/quickstart-cloud/cloud_quickstart_llm)
{% endcontent-ref %}

Check out a more in-depth tutorial to learn more about tracing:

{% content-ref url="/pages/x0swdPZMJgaRc2QR1Byp" %}
[Tutorial - Tracing](/tutorials-and-examples/tutorial_tracing)
{% endcontent-ref %}


# Quickstart - LLM evaluations

LLM evaluation "Hello world."

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower and Evidently Cloud v1. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This quickstart shows how to evaluate text data, such as inputs and outputs from your LLM system.

You will run evals locally in Python and send results to Evidently Cloud for analysis and monitoring.

Need help? Ask on [Discord](https://discord.com/invite/xZjKRaNp8b).

## 1. Set up Evidently Cloud

Set up your Evidently Cloud workspace:

* **Sign up** for a free [Evidently Cloud account](https://app.evidently.cloud/signup).
* **Create an Organization** when you log in for the first time. Get an ID of your organization. [Organizations page](https://app.evidently.cloud/organizations).
* **Get your API token**. Click the **Key** icon in the left menu. Generate and save the token. ([Token page](https://app.evidently.cloud/token)).

Now, switch to your Python environment.

## 2. Installation

Install the Evidently Python library:

```python
!pip install evidently[llm]
```

Import the components to run the evals:

```python
import pandas as pd
from evidently.report import Report
from evidently.metric_preset import TextEvals
from evidently.descriptors import *
```

Import the components to connect with Evidently Cloud:

```python
from evidently.ui.workspace.cloud import CloudWorkspace
```

## 3. Create a Project

Connect to Evidently Cloud using your API token:

```python
ws = CloudWorkspace(token="YOUR_API_TOKEN", url="https://app.evidently.cloud")
```

Create a Project within your Organization:

```python
project = ws.create_project("My test project", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

## 4. Import the toy dataset

Prepare your data as a pandas dataframe with texts and metadata columns. Here’s a toy chatbot dataset with "Questions" and "Answers".

```python
data = [
    ["What is the chemical symbol for gold?", "The chemical symbol for gold is Au."],
    ["What is the capital of Japan?", "The capital of Japan is Tokyo."],
    ["Tell me a joke.", "Why don't programmers like nature? It has too many bugs!"],
    ["What is the boiling point of water?", "The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit)."],
    ["Who painted the Mona Lisa?", "Leonardo da Vinci painted the Mona Lisa."],
    ["What’s the fastest animal on land?", "The cheetah is the fastest land animal, capable of running up to 75 miles per hour."],
    ["Can you help me with my math homework?", "I'm sorry, but I can't assist with homework. You might want to consult your teacher for help."],
    ["How many states are there in the USA?", "There are 50 states in the USA."],
    ["What’s the primary function of the heart?", "The primary function of the heart is to pump blood throughout the body."],
    ["Can you tell me the latest stock market trends?", "I'm sorry, but I can't provide real-time stock market trends. You might want to check a financial news website or consult a financial advisor."]
]

# Columns
columns = ["question", "answer"]

# Creating the DataFrame
evaluation_dataset = pd.DataFrame(data, columns=columns)

```

{% hint style="info" %}
**Collecting live data**: use the open-source `tracely` library to collect the inputs and outputs from your LLM app. Check the [Tracing Quickstart](/get-started/quickstart-cloud/cloud_quickstart_tracing). You can then download the traced dataset for evaluation.
{% endhint %}

## 5. Run your first eval

You have two options:

* Run evals that work locally.
* Use LLM-as-a-judge (requires an OpenAI token).

{% tabs %}
{% tab title="Only local methods" %}
**Define your evals**. You will evaluate all "Answers" for:

* Sentiment: from -1 for negative to 1 for positive.
* Text length: character count.
* Presence of "sorry" or "apologize": True/False.

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="answer", descriptors=[
        Sentiment(),
        TextLength(),
        IncludesWords(words_list=['sorry', 'apologize'], display_name="Denials"),
        ]
    ),
])

text_evals_report.run(reference_data=None, current_data=evaluation_dataset)
```

{% endtab %}

{% tab title="LLM as a judge" %}
**Set the OpenAI key**. It's best to set an environment variable: [see Open AI docs](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety) for tips.

```python
## import os
## os.environ["OPENAI_API_KEY"] = "YOUR KEY"
```

**Define your evals**. Evaluate all "Answers" for:

* Sentiment: from -1 for negative to 1 for positive.
* Text length: character count.
* Whether the chatbot denied an answer: returns "OK" / "Denial" labels with explanations. This uses LLM-as-a-judge (defaults to `gpt-4o-mini`) with a template Evidently prompt.

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="answer", descriptors=[
        Sentiment(),
        TextLength(),
        DeclineLLMEval(),
        ]
    ),
])

text_evals_report.run(reference_data=None, current_data=evaluation_dataset)
```

{% endtab %}
{% endtabs %}

Each evaluation is a `descriptor`. You can choose from multiple built-in evaluations or create custom ones, including LLM-as-a-judge.

## 6. Send results to Evidently Cloud

**Upload the Report** and include raw data for detailed analysis:

```python
ws.add_report(project.id, text_evals_report, include_data=True)
```

**View the Report**. Go to [Evidently Cloud](https://app.evidently.cloud/), open your Project, and navigate to "Reports" in the left.

You will see the scores summary, and the dataset with new descriptor columns. For example, you can sort to find all answers with "Denials".

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-7c04c1417b460f6780739167172d88842d0fa96d%2Fqs_denials.png?alt=media)

## 7. Get a dashboard

Go to the "Dashboard" tab and enter the "Edit" mode. Add a new tab, and select the "Descriptors" template.

You'll see a set of panels that show descriptor values. Each has a single data point. As you log ongoing evaluation results, you can track trends and set up alerts.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-abebd63baa2883692a8c3a578ad0179c99668702%2Fadd_descriptor_tab.gif?alt=media)

## What's next?

Explore the full tutorial for advanced workflows: custom LLM judges, conditional test suites, monitoring, and more.

{% content-ref url="/pages/7XC3sQCPEUuWxfy7PhCK" %}
[Tutorial - LLM Evaluation](/tutorials-and-examples/tutorial-llm)
{% endcontent-ref %}


# Quickstart - Data and ML checks

ML Monitoring “Hello world.” From data to dashboard in a couple of minutes.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## 1. Set up Evidently Cloud

Set up your Evidently Cloud workspace:

* **Sign up**. If you do not have one yet, sign up for a free [Evidently Cloud account](https://app.evidently.cloud/signup).
* **Create an Organization** when you log in for the first time. Get an ID of your organization. [Organizations page](https://app.evidently.cloud/organizations).
* **Get your API token**. Click the **Key** icon in the left menu. Generate and save the token. ([Token page](https://app.evidently.cloud/token)).

You can now go to your Python environment.

## 2. Install the Python library

Install the Evidently Python library. You can run this example in Colab or another Python environment.

```
!pip install evidently
```

Import the components to work with the dataset and send the metrics.

```python
import pandas as pd
from sklearn import datasets

from evidently.ui.workspace.cloud import CloudWorkspace
from evidently.report import Report
from evidently.metric_preset import DataQualityPreset
```

## 3. Create a new Project

Connect to Evidently Cloud using your access token.

```python
ws = CloudWorkspace(token="YOUR_TOKEN_HERE", url="https://app.evidently.cloud")
```

Create a new Project inside your Organization. Pass the `org_id`.

```python
project = ws.create_project("My test project", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

## 4. Import a toy dataset

Import the demo "adult" dataset as a pandas DataFrame.

```python
adult_data = datasets.fetch_openml(name="adult", version=2, as_frame="auto")
adult = adult_data.frame
```

## 5. Run an evaluation

Run a Data Quality Report and upload it to the Project.

```
data_report = Report(
       metrics=[
           DataQualityPreset(),
       ],
    )
data_report.run(reference_data=None, current_data=adult)
ws.add_report(project.id, data_report)
```

We call each such evaluation result a `snapshot`.

## 6. View the Report

Visit Evidently Cloud, open your Project, and navigate to the "Report" tab to see the data stats.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c3cf178c63938c4e409b402d561c7183e3f45e25%2Fqs_view_reports.gif?alt=media)

## 7. Add a monitoring panel

Go to the "Dashboard" tab and enter the "Edit" mode. Add a new tab, and select the "Data quality" template.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-08545232dd208adddbb1437d9e16b44d25d97466%2Fqs_add_data_quality_tab_2.gif?alt=media)

You'll see a set of panels with a single data point. As you send more snapshots, you can track trends and set up alerts. You can choose from 100+ metrics and tests on data quality, data drift, ML quality (regression, classification, ranking, recsys), LLM quality and text data, and add your own metrics.

## Want to see more?

Check out a more in-depth tutorial to learn the key workflows:

{% content-ref url="/pages/xkPOcA1OQZ9ugNXJ8kNv" %}
[Tutorial - Data & ML Monitoring](/tutorials-and-examples/tutorial-cloud)
{% endcontent-ref %}

Working with LLMs? See a Quickstart:

{% content-ref url="/pages/82JD0PYdxbIagT53baIy" %}
[Quickstart - LLM evaluations](/get-started/quickstart-cloud/cloud_quickstart_llm)
{% endcontent-ref %}

Need help? Ask in our [Discord community](https://discord.com/invite/xZjKRaNp8b).


# Quickstart - No-code evaluations

Check [this Guide](/user-guide/evaluations/no_code_evals) on how to upload a CSV file and run an eval.


# Evidently OSS

Get started with Evidently OSS.

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>LLM evaluations</strong></td><td>Run checks for text data and generative LLM outputs.</td><td><a href="/get-started/hello-world/oss_quickstart_llm">→ LLM Quickstart</a></td></tr><tr><td><strong>Tabular data checks</strong></td><td>Run evaluations for tabular data (data drift, data quality) and ML models (classification, regression, ranking).</td><td><a href="/get-started/hello-world/oss_quickstart_tabular">→ Tabular Quickstart</a></td></tr></tbody></table>


# OSS Quickstart - LLM evals

Run your first LLM evaluation using Evidently open-source.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This quickstart shows how to evaluate text data, such as inputs and outputs from your LLM system.

It's best to run this example in Jupyter Notebook or Google Colab so that you can render HTML Reports directly in a notebook cell.

## 1. Installation

Install the Evidently library.

```python
!pip install evidently[llm]
```

Import the required modules:

```python
import pandas as pd
from evidently.report import Report
from evidently.metric_preset import TextEvals
from evidently.descriptors import *
```

## 2. Create a toy dataset

Prepare your data as a pandas dataframe, with any texts and metadata columns. Here’s a toy example with chatbot "Questions" and "Answers":

```python
data = [
    ["What's the capital of France?", "The capital of France is Paris."],
    ["Who wrote 1984?", "George Orwell wrote 1984."], 
    ["How does photosynthesis work?", "Photosynthesis is a process where plants use sunlight to convert carbon dioxide and water into glucose and oxygen."],
    ["Can you give me the recipe for making pancakes?", "Sure! Here's a simple recipe: mix 1 cup flour, 1 cup milk, 1 egg, and a pinch of salt. Cook on a hot griddle until golden brown."],
    ["What is the largest planet in our solar system?", "Jupiter is the largest planet in our solar system."],
    ["Tell me a joke.", "Why don't scientists trust atoms? Because they make up everything!"],
    ["Can you translate 'Hello' into Spanish?", "'Hello' in Spanish is 'Hola'."],
    ["What's the code to the universe?", "I'm sorry, I can't provide that information."],
    ["What's the distance between Earth and the Sun?", "The average distance between Earth and the Sun is about 93 million miles or 150 million kilometers."],
    ["How do I fix a flat tire?", "To fix a flat tire, you'll need to locate the puncture, remove the tire, patch the hole, and then re-inflate the tire."]
]

columns = ["question", "answer"]

eval_dataset = pd.DataFrame(data, columns=columns)
```

**Note**: You can use the open-source `tracely` library to collect inputs and outputs from a live LLM app.

## 3. Run your first eval

Run evaluations for the "Answer" column:

* Sentiment (from -1 for negative to 1 for positive)
* Text length (number of symbols))
* Presence of "sorry" or "apologize" (True/False)

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="answer", descriptors=[
        Sentiment(),
        TextLength(),
        IncludesWords(words_list=['sorry', 'apologize'], display_name="Denials"),        
        ]
    ),
])

text_evals_report.run(reference_data=None, current_data=eval_dataset)
```

Each evaluation is a `descriptor`. You can choose from many built-in evaluations or create custom ones.

View the Report in Python to see the distribution of scores:

```
text_evals_report
```

You can also export the dataset with added descriptors for each row.

```
text_evals_report.datasets().current
```

Or get a dictionary with results:

```
text_evals_report.datasets().as_dict
```

## 4. Use LLM as a judge (Optional)

To run this, you'll need an OpenAI key.

Set the OpenAI key (it's best to pass it as an environment variable). [See Open AI docs](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety) for best practices.

```python
## import os
## os.environ["OPENAI_API_KEY"] = "YOUR KEY"
```

Run a Report with the new `DeclineLLMEval`. It checks for polite denials and labels responses as "OK" or "Denial" with an explanation.

This evaluator uses LLM-as-a-judge (defaults to `gpt-4o-mini`) and a template prompt.

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="answer", descriptors=[
        Sentiment(),
        TextLength(),
        DeclineLLMEval(),
        ]
    ),
])

text_evals_report.run(reference_data=None, current_data=evaluation_dataset)
```

View the Report in Python:

```
text_evals_report
```

View the dataset with scores and explanation:

```
text_evals_report.datasets().current
```

Or get a dictionary with results:

```
text_evals_report.datasets().as_dict
```

## What's next?

Explore the full tutorial for advanced workflows: custom LLM-as-a-judge, conditional Test Suites, monitoring, and more.

{% content-ref url="/pages/7XC3sQCPEUuWxfy7PhCK" %}
[Tutorial - LLM Evaluation](/tutorials-and-examples/tutorial-llm)
{% endcontent-ref %}

You can also send evaluation results to Evidently Cloud to analyze and track them. See the Quickstart:

{% content-ref url="/pages/82JD0PYdxbIagT53baIy" %}
[Quickstart - LLM evaluations](/get-started/quickstart-cloud/cloud_quickstart_llm)
{% endcontent-ref %}

Need help? Ask in our [Discord community](https://discord.com/invite/xZjKRaNp8b).


# OSS Quickstart - Data and ML monitoring

Run your first evaluation using Evidently open-source, for tabular data.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower and Evidently Cloud v1. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

It's best to run this example in Jupyter Notebook or Google Colab so that you can render HTML Reports directly in a notebook cell.

## Installation

Install **Evidently** using the pip package manager:

```python
!pip install evidently
```

## Imports

Import the Evidently components and a toy “Iris” dataset:

```python
import pandas as pd

from sklearn import datasets

from evidently.test_suite import TestSuite
from evidently.test_preset import DataStabilityTestPreset

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

iris_data = datasets.load_iris(as_frame='auto')
iris_frame = iris_data.frame
```

## Run a Test Suite

Split the data into two batches. Run a set of pre-built data quality Tests to evaluate the quality of the `current_data`:

```python
data_stability= TestSuite(tests=[
    DataStabilityTestPreset(),
])
data_stability.run(current_data=iris_frame.iloc[:60], reference_data=iris_frame.iloc[60:], column_mapping=None)
data_stability 
```

This will automatically generate tests on share of nulls, out-of-range values, etc. – with test conditions generated based on the first "reference" dataset.

## Get a Report

Get a Data Drift Report to see if the data distributions shifted between two datasets:

```python
data_drift_report = Report(metrics=[
    DataDriftPreset(),
])

data_drift_report.run(current_data=iris_frame.iloc[:60], reference_data=iris_frame.iloc[60:], column_mapping=None)
data_drift_report
```

## What's next?

Want more details on Reports and Test Suites? See an in-depth tutorial.

{% content-ref url="/pages/H7AWGQhMl4Rp5YGRfCUT" %}
[Tutorial - Reports and Tests](/tutorials-and-examples/tutorial_reports_tests)
{% endcontent-ref %}

Want to set up monitoring? Send the evaluation results to Evidently Cloud for analysis and tracking. See the Quickstart:

{% content-ref url="/pages/82JD0PYdxbIagT53baIy" %}
[Quickstart - LLM evaluations](/get-started/quickstart-cloud/cloud_quickstart_llm)
{% endcontent-ref %}

Working with LLMs? Check the Quickstart:

{% content-ref url="/pages/82JD0PYdxbIagT53baIy" %}
[Quickstart - LLM evaluations](/get-started/quickstart-cloud/cloud_quickstart_llm)
{% endcontent-ref %}

Need help? Ask in our [Discord community](https://discord.com/invite/xZjKRaNp8b).


# Presets


# All Presets

An overview of the evaluations you can do with Evidently.

{% hint style="info" %}
**You are looking at the old Evidently documentation**. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Evidently has several pre-built reports and test suites. We call them **Presets**. Each preset evaluates or tests a particular aspect of the data or model quality.

This page links to the **description** of each preset. To see the code and interactive examples, head to [example notebooks](/tutorials-and-examples/examples) instead.

## Metric Presets

Metric presets are **pre-built reports** that help with visual exploration, debugging and documentation of the data and model performance. You can also use them to calculate and log metrics as JSON or Python dictionary.

|                                                                                                                                                                                                                        |                                                                                                                                                                                                             |                                                                                                                                                                                                                                       |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><a href="/presets/data-quality"><strong>Data Quality</strong></a><br><br>Shows the dataset statistics and feature behavior.<br><br><strong>Requirements</strong>: model inputs.</p>                                 | <p><a href="/presets/data-drift"><strong>Data Drift</strong></a><br><br>Explores the distribution shift in the model features.<br><br><strong>Requirements</strong>: model inputs, a reference dataset.</p> | <p><a href="/presets/target-drift"><strong>Target Drift</strong></a><br><br>Explores the distribution shift in the model predictions.<br><br><strong>Requirements:</strong> model predictions and/or target, a reference dataset.</p> |
| <p><a href="/presets/class-performance"><strong>Classification</strong></a><br><br>Evaluates the classification model quality and errors.<br><br><strong>Requirements</strong>: model predictions and true labels.</p> | <p><a href="/presets/reg-performance"><strong>Regression</strong></a><br><br>Evaluates the regression model quality and errors.<br><br><strong>Requirements</strong>: model predictions and actuals.</p>    | <p><a href="/presets/text-overview"><strong>Text Overview</strong></a><br><br>Evaluates text data drift and descriptive statistics.<br><br><strong>Requirements</strong>: model inputs (raw text data)</p>                            |

## Test Presets

Test presets are **pre-built test suites** that perform structured data and model checks as part of the pipeline.

|                                                                                                                                                                                                                                                                                       |                                                                                                                                                                                                                                                                                  |                                                                                                                                                                                                                                                                                                 |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><a href="/presets/no-target-performance"><strong>NoTargetPerformance</strong></a><br><br>Tests the model performance without ground truth or actuals.<br><br><strong>Requirements</strong>: model inputs, predictions, a reference dataset.</p>                                    | <p><a href="/presets/data-quality#data-quality-test-suite"><strong>Data Quality</strong></a><br><br>Tests if the data quality is suitable for (re)training. Checks nulls, duplicates, etc.<br><br><strong>Requirements</strong>: model inputs.</p>                               | <p><a href="/presets/data-quality#data-stability-test-suite"><strong>Data Stability</strong></a><br><br>Tests if a data batch is similar to reference. Checks schema, data ranges, etc.<br><br><strong>Requirements</strong>: inputs, a reference dataset.</p>                                  |
| <p><a href="/presets/data-drift#data-drift-test-suite"><strong>Data Drift</strong></a><br><br>Tests for distribution drift per column and overall dataset drift.<br><br><strong>Requirements</strong>: model inputs, a reference dataset.</p>                                         | <p><a href="/presets/reg-performance#regression-performance-test-suite"><strong>Regression</strong></a><br><br>Tests the performance of the regression model against expectation.<br><br><strong>Requirements</strong>: model predictions and actuals.</p>                       | <p><a href="/presets/class-performance#classification-performance-test-suite"><strong>Multi-class Classification</strong></a><br><br>Tests the performance of a multi-class classification model against expectation.<br><br><strong>Requirements</strong>: model predictions, true labels.</p> |
| <p><a href="/presets/class-performance#classification-performance-test-suite"><strong>Binary Classification</strong></a><br><br>Tests the performance of a binary classification model against expectation.<br><br><strong>Requirements</strong>: model predictions, true labels.</p> | <p><a href="/presets/class-performance#classification-performance-test-suite"><strong>Binary Classification top-K</strong></a><br><br>Tests the performance of a binary classification model at top-K.<br><br><strong>Requirements</strong>: model predictions, true labels.</p> |                                                                                                                                                                                                                                                                                                 |

## Individual Metrics and Tests

You can also create custom test suites and reports from individual metrics and tests. You can explore 100+ [available tests](/reference/all-tests) and [metrics](/reference/all-metrics).


# Data Drift

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can detect and analyze changes in the input feature distributions.

* **Report**: for visual analysis or metrics export, use the `DataDriftPreset`.
* **Test Suite**: for pipeline checks, use the `DataDriftTestPreset`.

## Use Case

You can evaluate data drift in different scenarios.

1. **To monitor the model performance without ground truth.** When you do not have true labels or actuals, you can monitor the feature drift to check if the model operates in a familiar environment. You can combine it with the [Prediction Drift](https://github.com/evidentlyai/docs-old/blob/main/presets/prediction-drift.md). If you detect drift, you can trigger labeling and retraining, or decide to pause and switch to a different decision method.
2. **When you are debugging the model quality decay.** If you observe a drop in the model quality, you can evaluate Data Drift to explore the change in the feature patterns, e.g., to understand the change in the environment or discover the appearance of a new segment.
3. **To understand model drift in an offline environment.** You can explore the historical data drift to understand past changes in the input data and define the optimal drift detection approach and retraining strategy.
4. **To decide on the model retraining.** Before feeding fresh data into the model, you might want to verify whether it even makes sense. If there is no data drift, the environment is stable, and retraining might not be necessary.

To run drift checks as part of the pipeline, use the Test Suite. To explore and debug, use the Report.

## Data Drift Report

If you want to get a visual report, you can create a new Report object and use the `DataDriftPreset`.

### Code example

```python
data_drift_report = Report(metrics=[
    DataDriftPreset(),
])

data_drift_report.run(reference_data=ref, current_data=cur)
data_drift_report
```

### How it works

The **Data Drift** report helps detect and explore changes in the input data.

* Applies as suitable **drift detection method** for numerical, categorical or text features.
* Plots **feature values and distributions** for the two datasets.

### Data Requirements

* You will need **two** datasets. The **reference** dataset serves as a benchmark. Evidently analyzes the change by comparing the **current** production data to the **reference** data to detect distribution drift.
* **Input features**. The dataset should include the features you want to evaluate for drift. The schema of both datasets should be identical. If your dataset contains target or prediction column, they will also be analyzed for drift.
* **Column mapping**. Evidently can evaluate drift both for numerical, categorical and text features. You can explicitly specify the type of each column using [column mapping object](/user-guide/input-data/column-mapping). If it is not specified, Evidently will try to identify the numerical and categorical features automatically. It is recommended to use column mapping to avoid errors. If you have text data, you must always specify it.

### How it looks

The default report includes 4 components. All plots are interactive.

**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.

#### 1. Data Drift Summary

The report returns **the share of drifting features** and an aggregate **Dataset Drift** result.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-21b4dee477a4479f24377ef1216e032094b167a4%2Fmetric_dataset_drift-min.png?alt=media\&token=0d6b7e27-aa99-49b5-ba1f-a7412abd0b9e)

Dataset Drift sets a rule on top of the results of the statistical tests for individual features. By default, Dataset Drift is detected if at least 50% of features drift.

Evidently uses the default [data drift detection algorithm](/reference/data-drift-algorithm) to select the drift detection method based on feature type and the number of observations in the reference dataset.

{% hint style="info" %}
You can modify the drift detection logic by selecting a different method, including PSI, K–L divergence, Jensen-Shannon distance, Wasserstein distance, setting a different threshold and condition for the dataset drift. See more details about [setting data drift parameters](/user-guide/customization/options-for-statistical-tests). You can also implement a [custom drift detection method](/user-guide/customization/add-custom-metric-or-test).
{% endhint %}

To build up a better intuition for which tests are better in different kinds of use cases, visit our blog to read [an in-depth guide](https://evidentlyai.com/blog/data-drift-detection-large-datasets) to the tradeoffs when choosing the statistical test for data drift.

#### 2. Data Drift Table

The table shows the drifting features first. You can also choose to sort the rows by the feature name or type.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e35a4e9ab5fc22dda4a15294f6ea9f1bf15e2f3c%2Fmetric_data_drift_table-min.png?alt=media\&token=e0e8ead0-390c-4714-a32e-75a046b2e386)

#### 3. Data Distribution by Feature

By clicking on each feature, you can explore the distributions or top characteristic words (for text features).

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-ac3b16fbc47bdaa86fbfef56a788c0c18646ba2f%2Fmetric_data_drift_table_expand_1-min.png?alt=media\&token=0684578d-4344-414c-917c-f1cdd2a7f1b0)

#### 4. Data Drift by Feature

For numerical features, you can also explore the values mapped in a plot.

* The dark green line is the **mean**, as seen in the reference dataset.
* The green area covers **one standard deviation** from the mean.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-5da9d8a6b99fed4cb562666143a0919b59e992ac%2Fmetric_data_drift_table_expand_2-min.png?alt=media\&token=30c94d19-9b32-4dea-bf59-4216f021fbd6)

Note: by default, the visualization is aggregated. In this case, the index is binned into 150 bins, and the y-axis shows the mean value. You can enable the [raw data option](/user-guide/customization/report-data-aggregation) to see the individual data points.

### Metrics output

You can get the report output as a JSON or a Python dictionary:

<details>

<summary>See JSON example</summary>

```yaml
{
  "data_drift": {
    "name": "data_drift",
    "datetime": "datetime",
    "data": {
      "utility_columns": {
        "date": null,
        "id": null,
        "target": null,
        "prediction": null,
        "drift_conf_level": value,
        "drift_features_share": value,
        "nbinsx": {
          "feature_name": value,
          "feature_name": value
        },
        "xbins": null
      },
      },
      "cat_feature_names": [],
      "num_feature_names": [],
      "metrics": {
        "feature_name" :{
          "prod_small_hist": [
            [],
            []
          ],
          "ref_small_hist": [
            [],
            []
          ],
          "feature_type": "num",
          "p_value": p_value
      },
      "n_features": value,
      "n_drifted_features": value,
      "share_drifted_features": value,
      "dataset_drift": false
    }
  },
  "timestamp": "timestamp"
}
```

</details>

### Report customization

* You can [specify the drift detection methods and thresholds](/user-guide/customization/options-for-statistical-tests).
* You can add a [custom drift detection method](/user-guide/customization/add-custom-metric-or-test).
* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* You can create a different report from scratch taking this one as an inspiration.
* You can apply the report only to selected columns, for example, the most important features.

## Data Drift Test Suite

If you want to run data drift checks as part of the pipeline, you can create a Test Suite and use the `DataDriftTestPreset`.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1bee7be20d15727f4abf20dc1b6a9aa03f14814a%2Ftest_preset_data_drift-min.png?alt=media\&token=a45bb3b3-e2ed-4da3-b8b8-911f61946e55)

### Code example

```python
data_drift_test_suite = TestSuite(tests=[
   DataDriftTestPreset(),
])
 
data_drift_test_suite.run(reference_data=ref, current_data=curr)
data_drift_test_suite
```

### How it works

You can use the `DataDriftTestPreset` to test features for drift when you receive a new batch of input data or generate a new set of predictions.

The test preset works similarly to the metric preset. It will perform two types of tests:

* test the **share of drifted columns** to detect dataset drift;
* test distribution **drift in the individual columns** (all or from a defined list).

Head here to the [All tests](/reference/all-tests) table to see the description of individual tests and parameters.

### Test Suite customization

* You can [specify the drift detection methods and thresholds](/user-guide/customization/options-for-statistical-tests).
* You can add a [custom drift detection method](/user-guide/customization/add-custom-metric-or-test).
* You can apply the preset only to selected columns.
* You can create a different test suite from scratch taking this one as an inspiration.

If you want to compare descriptive statistics between the two datasets, you can also use the [Data Stability test preset](/presets/data-quality).

## Examples

* Browse the [examples](/tutorials-and-examples/examples) for sample Jupyter notebooks and Colabs.

You can also explore [blog posts](https://www.evidentlyai.com/tags/data-drift) about drift detection, including [How to handle drift](https://www.evidentlyai.com/blog/ml-monitoring-data-drift-how-to-handle) or [how to analyze historical drift patterns](https://evidentlyai.com/blog/tutorial-3-historical-data-drift).


# Data Quality

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can explore and track various dataset and feature statistics.

* **Report**: for visual analysis or metrics export, use the `DataQualityPreset`.
* **Test Suite**: for pipeline checks, use the `DataQualityTestPreset`or `DataStabilityTestPreset`.

## Use Cases

You might need to track and evaluate data quality and integrity in different scenarios.

1. **Data quality tests in production.** You can check the quality and stability of the input data before you generate the predictions, every time you perform a certain transformation, add a new data source, etc.
2. **Data profiling in production.** You can log and store JSON snapshots of your production data stats for future analysis and visualization.
3. **Exploratory data analysis.** You can use the visual report to explore your training dataset and understand which features are stable and useful enough to use in modeling.
4. **Dataset comparison.** You can use the report to compare two datasets to confirm similarities or understand the differences. For example, you might compare training and test dataset, subgroups in the same dataset (e.g., customers from Region 1 and Region 2), or current production data against training.
5. **Production model debugging.** If your model is underperforming, you can use this report to explore and interpret the details of changes in the input data or debug the quality issues.

For production pipeline tests, use Test Suites. For exploratory analysis and debugging, use Report.

## Data Quality Report

If you want to get a visual report, you can create a new Report object and use the `DataQualityPreset`.

### Code example

```python
data_quality_report = Report(metrics=[
    DataQualityPreset(),
])

data_quality_report.run(reference_data=adult_ref, current_data=adult_cur)
data_quality_report
```

### How it works

The Data Quality report provides detailed feature statistics and a feature behavior overview.

* The report works for a **single dataset** or **compares the two**.
* Calculates base **statistics** for numerical, categorical and datetime features
* Displays **interactive plots** with data distribution and behavior in time
* Plots **interactions and correlations** between features and target

### Data Requirements

* **Input features**. You need to pass only the input features. Target and prediction are optional.
* **One or two datasets**. If you want to perform a side-by-side comparison, pass two datasets with identical schema. You can also pass a single dataset.
* **Column mapping**. Feature types (numerical, categorical, datetime) will be parsed based on pandas column type. If you want to specify a different feature mapping strategy, you can explicitly set the feature type using `column_mapping`.

You might also need to specify additional column mapping:

* If you have a **datetime** index column and want to learn how features change with time, specify the datetime column in the `column_mapping`.
* If you have a **target** column and want to see features distribution by target, specify the target column in the `column_mapping`.
* Specify the **task** if you want to explore interactions between the features and the target. This section looks slightly different for classification and regression tasks. By default, if the target has a numeric type and has >5 unique values, Evidently will treat it as a regression problem. Everything else is treated as a classification problem. If you want to explicitly define your task as `regression` or `classification`, you should set the `task` parameter in the `column_mapping` object.
* If you have **text** features, you should specify it in the column mapping to generate descriptive statistics specific to text.

{% hint style="info" %}
You can read more to understand [column mapping](/user-guide/input-data/column-mapping) and [data requirements](/user-guide/input-data/data-requirements) for Evidently reports in the corresponding sections of documentation.
{% endhint %}

### How it looks

The default report includes 3 widgets. All plots are interactive.

**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.

#### 1. Summary widget

The table gives an overview of the dataset, including missing or empty features and other general information. It also shows the share of almost empty and almost constant features. This applies to cases when 95% or more features are missing or constant.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-893e96b92c369e8ad43eddc86aba54f79ce46b27%2Freports_data_quality_summary.png?alt=media\&token=850dae4e-a48e-4dca-a5e0-6e9a9f182572)

#### 2. Features widget

For each feature, this widget generates a set of visualizations. They vary depending on the feature type. There are 3 components:

**2.1. Feature overview table**

The table shows relevant statistical summaries for each feature based on its type and a visualization of feature distribution.

**Example for a categorical feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-31dbb98333b62c917c7b2cbba158445774508baa%2Freports_data_quality_overview_cat.png?alt=media\&token=37466c04-04d4-45e2-89e9-322a0932a12b)

**Example for a numerical feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c7affc8d3dd2d0bee0b70eafdeb5867e5c52b30e%2Freports_data_quality_overview_num.png?alt=media\&token=1a9f62e2-5e02-4bc1-8894-36dfba2c9bbc)

**Example for a datetime feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-ebcc131d1b824e4754edd777804343a4ea5dfdb1%2Freports_data_quality_overview_datetime.png?alt=media\&token=18638a85-f2ca-47a3-82ca-ca7a4555b715)

**Example for a text feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-7074e49da73257ed1bd5ef7b675b8f63836ed15d%2Fmetric_column_summary_text-min.png?alt=media\&token=fec9c87d-e56c-4d01-bfe9-0cf14614d2e7)

**2.2. Feature in time**

If you click on "details", each feature would include additional visualization to show feature behavior in time.

**Example for a categorical feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-71b9ba91a04fa177dffbbb2b25984be292f15283%2Freports_data_quality_in_time_cat.png?alt=media\&token=2ee2a063-4c5d-48d9-8e95-c3d63c3361a9)

**Example for a numerical feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a06be74e97d4d2fae56e8686520d0059161f36eb%2Freports_data_quality_in_time_num.png?alt=media\&token=013fadd4-2d52-4783-b8b2-b8320dd36b42)

**Example for a datetime feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-9b4017bb0fd6a38fae0b868c2cf016a2ec8e2250%2Freports_data_quality_in_time_datetime.png?alt=media\&token=bb8e5bcc-cda9-4dff-89cc-c54b91abc299)

**2.3. Feature by target**

Categorical and numerical features include an additional visualization that plots the interaction between a given feature and the target.

**Example for a categorical feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-70efaf4625facf8c22e1821af9494b070322ca60%2Freports_data_quality_by_target_cat.png?alt=media\&token=3886a230-3029-4156-8256-110fc90ec577)

**Example for a numerical feature:**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-dae7161fc71ab08b8569678ce02dc8d58420f7c9%2Freports_data_quality_by_target_num.png?alt=media\&token=247b5844-7e34-4cbe-970d-cd91b44486b4)

#### 3. Correlation widget

This widget shows the correlations between different features.

**3.1. Insights**

This table shows a summary of pairwise feature correlations.

For a single dataset, it lists the top-5 highly correlated variables from Cramer's v correlation matrix (categorical features) and from Spearman correlation matrix (numerical features).

For two datasets, it lists the top-5 pairs of variables **where correlation changes** the most between the reference and current datasets. Similarly, it uses categorical features from Cramer's v correlation matrix and numerical features from Spearman correlation matrix.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-cb472b98063f4e2f57029b8ea4e450d7c8a453c7%2Freports_data_quality_correlations.png?alt=media\&token=d2f1307e-2956-4241-b149-9fc19b2b4f9d)

**3.2. Correlation heatmaps**

{% hint style="info" %}
**This widget has been removed from Preset in versions above 0.4.31**. You can add it to your Report as `DatasetCorrelationsMetric()`.
{% endhint %}

This section includes four heatmaps.

For categorical features, Evidently calculates the [Cramer's v](https://en.wikipedia.org/wiki/Cramér%27s_V) correlation matrix.\
For numerical features, Evidently calculates the [Pearson](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient), [Spearman](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) and [Kendall](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient) matrices.

If your dataset includes the target, the target will be also shown in the matrix according to its type.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b8e9570d01a6413c4c8eeabd91252bf263516e21%2Freports_data_quality_correlation_heatmaps.png?alt=media\&token=a2ebcd95-93ce-42f2-8c27-92f576db0231)

### Metrics outputs

You can get the report output as a JSON or a Python dictionary:

<details>

<summary>See JSON example</summary>

```yaml
{
  'data_quality': {
    'data': {
      'cat_feature_names': [],
      'datetime_feature_names': [],
      'metrics': {
        'current': {
          'feature_name': {
            'count': count,
            'feature_type': 'num',
            'infinite_count': 0,
            'infinite_percentage': 0.0,
            'max': max,
            'mean': mean,
            'min': min,
            'missing_count': 0,
            'missing_percentage': 0.0,
            'most_common_value': most_common_value,
            'most_common_value_percentage': most_common_value_percentage,
            'percentile_25': percentile_25,
            'percentile_50': percentile_50,
            'percentile_75': percentile_75,
            'std': std,
            'unique_count': unique_count,
            'unique_percentage': unique_percentage
            },
          },
        },
        'num_feature_names': [],
        'target_names': None,
        'utility_columns': {
          'date': 'dteday',
          'id': None,
          'prediction': 'prediction',
          'target': 'target'
          }
        },
      'correlations': {
        'current': {
          'pearson': {
            'feature_name_1': {
              'feature_name_2': value,
              'feature_name_3': value
              },
            'feature_name_2': {
              'feature_name_1': value,
              'feature_name_3': value
              },
            'feature_name_3': {
              'feature_name_1': value,
              'feature_name_2': value
              }
            },
          'spearman': {
            'feature_name_1': {
              'feature_name_2': value,
              'feature_name_3': value
              },
            'feature_name_2': {
              'feature_name_1': value,
              'feature_name_3': value
              },
            'feature_name_3': {
              'feature_name_1': value,
              'feature_name_2': value
              }
            },
          'kendall': {
            'feature_name_1': {
              'feature_name_2': value,
              'feature_name_3': value
              },
            'feature_name_2': {
              'feature_name_1': value,
              'feature_name_3': value
              },
            'feature_name_3': {
              'feature_name_1': value,
              'feature_name_2': value
              }
            },
          'cramer_v': {
            'feature_name_4': {
              'feature_name_5': value
              },
            'feature_name_5': {
              'feature_name_4': value
              }
            }  
          }
        }  
    'datetime': '2022-02-22 16:35:15.529404',
    'name': 'data_quality'
    },
  'timestamp': 'timestamp'
}
```

</details>

### Report customization

* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* You can create a different report from scratch taking this one as an inspiration by combining chosen metrics.
* You can apply the report only to selected columns, for example, the most important features.

## Data Quality Test Suite

If you want to run data quality checks as part of the pipeline, you can create a new Test Suite and include the `DataQualityTestPreset`.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c1014ebe7580ee30936ab5f74a5eb5e45bf6faa2%2Ftest_preset_data_quality-min.png?alt=media\&token=6363d3b9-940c-4116-97c7-ba4eacca73f5)

### Code example

```python
data_quality_test_suite = TestSuite(tests=[
   DataQualityTestPreset(),
])
 
data_quality_test_suite.run(reference_data=ref, current_data=curr)
data_quality_test_suite
```

### How it works

You can use the `DataQualityTestPreset` when you want to evaluate the quality of a new data batch, even **without a reference dataset**. It will help assess whether a data batch is e.g. **suitable for training or retraining**.

* It calculates various descriptive statistics for the dataset.
* It helps detect issues like **missing data**, **duplicates**, or **almost constant** features.
* Evidently will **generate the test conditions automatically** based on the set of heuristics or provided reference dataset. You can also pass custom conditions.

Head here to the [All tests](/reference/all-tests) table to see the composition of the preset and default parameters for each test.

### Test Suite customization

* You can pass custom test conditions.
* You can apply the tests only to chosen columns.
* If you want to exclude or add some tests, you can create a custom test suite. See the complete [test list](/reference/all-tests).

## Data Stability Test Suite

If you want to run data stability checks as part of the pipeline, you can create a new Test Suite and include the `DataStabilityTestPreset`.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-95d77b0e3157cd4cb086d1c0a4ac2d04ce9899fc%2Ftest_preset_data_stability-min.png?alt=media\&token=9fc4e51a-730c-42f1-a948-2de842ec2990)

### How it works

You can use the `DataStabilityTestPreset` when you receive a new batch of input data that you **expect to be similar to the previous batch** and want to compare them.

* It calculates various dataset and feature statistics.
* It helps detect issues like **new categorical values**, **values out of range**, or change in the **data volume**.
* Evidently will **generate the test conditions automatically** based on the provided reference dataset. You can also pass custom conditions.

### Test Suite customization

* You can pass custom test conditions.
* You can apply the tests only to chosen columns.
* If you want to exclude or add some tests, you can create a custom test suite. See the complete [test list](/reference/all-tests).

### Examples

* Browse our [example](/tutorials-and-examples/examples) notebooks to see sample Reports and Test Suites.


# Target Drift

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can detect and explore changes in the target function (prediction) and detect distribution drift.

* **Report**: for visual analysis or metrics export, use the`TargetDriftPreset`.
* **Test Suite**: for pipeline checks, use a `TestColumnDrift` test and apply it to the prediction or target column. Since it is a single test, there is no need for a Preset.

## Use case

You can analyze target or prediction drift:

**1. To monitor the model performance without ground truth.** When you do not have true labels or actuals, you can monitor Prediction Drift to react to meaningful changes. For example, to detect when there is a distribution shift in predicted values, probabilities, or classes. You can often combine it with the [Data Drift analysis.](/presets/data-drift)

**2. When you are debugging the model decay.** If you observe a drop in performance, you can evaluate Target Drift to see how the behavior of the target changed and explore the shift in the relationship between the features and prediction (target).

**3. Before model retraining.** Before feeding fresh data into the model, you might want to verify whether it even makes sense. If there is no target drift and no data drift, the retraining might not be necessary.

To run drift checks as part of the pipeline, use the Test Suite. To explore and debug, use the Report.

## Target (Prediction) Drift Report

If you want to visually explore the prediction or target drift, you can create a new Report object and use the `TargetDriftPreset`.

**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.

### Code example

```python
num_target_drift_report = Report(metrics=[
    TargetDriftPreset(),
])

num_target_drift_report.run(reference_data=ref, current_data=cur)
num_target_drift_report
```

### How it works

The `TargetDriftPreset` helps detect and explore changes in the target function and/or model predictions:

* Performs a suitable **statistical test** to compare target (prediction) **distribution**.
* For numerical targets, calculates the **correlations** between the feature and the target (prediction)
* **Plots the relations** between each individual feature and the target (prediction)

You can generate this preset both for numerical targets (e.g. if you have a regression problem) or categorical targets (e.g. if you have a classification problem). You can explicitly specify the type of the target column in column mapping. If it is not specified, Evidently will define the column type automatically.

### Data Requirements

* You will need **two** datasets. The **reference** dataset serves as a benchmark. Evidently analyzes the change by comparing the **current** production data to the **reference** data.
* To run this preset, you need to have **target and/or prediction** columns available. Input features are optional. Pass them if you want to analyze the correlations between the features and target (prediction). Evidently estimates the drift for the **target** and **predictions** in the same manner. If you pass both columns, Evidently will generate two sets of plots. If you pass only one of them (either target or predictions), Evidently will build one set of plots.
* **Column mapping**. Evidently can evaluate drift both for numerical and categorical targets. You can explicitly specify the type of target using the task parameter in [column mapping](/user-guide/input-data/column-mapping). If it is not specified, Evidently will try to identify the target type automatically. It is recommended to use column mapping to avoid errors.

### How it looks

The report includes 4 components. All plots are interactive.

#### 1. Target (Prediction) Drift

The report first shows the **comparison of target (prediction) distributions** in the current and reference datasets. You can see the result of the statistical test or the value of a distance metric.

Evidently uses the default [data drift detection algorithm](/reference/data-drift-algorithm) to select the drift detection method based on target type and the number of observations in the reference dataset.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-dfc5461a199e975447af7cb691a74b58036222ae%2Fnum_targ_drift%20\(1\).png?alt=media)

{% hint style="info" %}
You can modify the drift detection logic by selecting a different method already available in the library, including PSI, K–L divergence, Jensen-Shannon distance, Wasserstein distance, and/or by setting a different threshold. See more details about [setting data drift parameters](/user-guide/customization/options-for-statistical-tests). You can also implement a [custom drift detection method](/user-guide/customization/add-custom-metric-or-test).
{% endhint %}

#### 2. Target (Prediction) Correlations

For numerical targets, the report calculates the [Pearson correlation](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) between the target (prediction) and each individual feature in the two datasets to detect a **change in the relationship.**

The report shows the **correlations between individual features and the target (prediction)** in the current and reference dataset. It helps detects shifts in the relationship.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-663e41054bfae1f8c1ac7c5ef0d3fdc654d1ce37%2Fnum_targ_drift_target_correlations%20\(1\).png?alt=media)

#### 3. Target (Prediction) Values

For numerical targets, the report visualizes the **target (prediction) values by index or time** (if the`datetime` column is available or defined in the `column_mapping` dictionary). This plot helps explore the target behavior and compare it between the datasets.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b806f021a5361524e9fe99af54eee2eaecec997b%2Fnum_targ_drift_target_values%20\(1\).png?alt=media)

#### 4. Target (Prediction) Behavior By Feature

Finally, it generates an interactive table with the **visualizations of dependencies between the target and each feature**.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-354c9622fc712c99386e0c811ea32a7dcbc0459f%2Fnum_targ_drift_behavior_by_feature%20\(1\).png?alt=media)

If you click on any feature in the table, you get an overview of its behavior. The plot shows how **feature values relate to the target (prediction) values** and if there are differences between the datasets. It helps explore if they can explain the target (prediction) shift.

For numerical targets:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-de6d24640d3860918430e929a8c91c0fa774bb40%2Fnum_targ_drift_behavior_by_feature_example_tax.png?alt=media\&token=4b6cf896-3ad3-47ba-ae6d-ad8b55458ff3)

We recommend paying attention to the behavior of the **most important features** since significant changes might confuse the model and cause higher errors. For example, in a Boston house pricing dataset, we can see a new segment with values of TAX above 600 but the low value of the target (house price).

For categorical targets:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-cb8f13329c6a6bd2ebf6a82e78fc46d38265e9ab%2Fcat_target_drift_behavior_by_feature_example.png?alt=media\&token=6c20a719-5607-4ba1-8f7a-4d65c36cfcf9)

### Metrics output

You can get the report output as a JSON or a Python dictionary:

<details>

<summary>See JSON example</summary>

```yaml
{
 "num_target_drift": {
    "name": "num_target_drift",
    "datetime": "datetime",
    "data": {
      "utility_columns": {
        "date": null,
        "id": null,
        "target": "target",
        "prediction": null
      },
      "cat_feature_names": [],
      "num_feature_names": [],
      "metrics": {
        "target_name": "target",
        "target_type": "num",
        "target_drift": p_value,
        "target_correlations": {
          "reference": {
            "feature_name": corr_coefficient
          },
          "current": {
            "feature_name": corr_coefficient
          }
        }
      }
    }
  },
  "timestamp": "timestamp"
}
```

</details>

### Report customization

* You can [specify the drift detection methods and thresholds](/user-guide/customization/options-for-statistical-tests).
* You can add a [custom drift detection method](/user-guide/customization/add-custom-metric-or-test).
* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* You can create a different report or test suite from scratch taking this one as an inspiration.

## Examples

* Browse the [examples](/tutorials-and-examples/examples) for sample Jupyter notebooks and Colabs.


# Regression Performance

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can monitor and analyze the performance of a regression model.

* **Report**: for visual analysis or metrics export, use the `RegressionPreset`.
* **Test Suite**: for pipeline checks, use the `RegressionTestPreset`.

## Use case

These presets help evaluate and test the quality of classification models. You can use them in different scenarios.

**1. To monitor the performance of a regression model in production.** You can run the test suite as a regular job (e.g., weekly or every time you get the labels) to contrast the model performance against the expectation. You can generate visual reports for documentation and sharing with stakeholders.

**2. To trigger or decide on the model retraining.** You can use the test suite to check if the model performance is below the threshold to initiate a model update.

**3. To debug or improve model performance.** If you detect a quality drop, you can use the visual report to explore the model errors. You can use the Error Bias table to identify the groups with high error where the model under- or over-estimates the target function. By manipulating the input data frame, you can explore the performance on different data segments (e.g., users from a specific region). You can also combine it with the [Data Drift](/presets/data-drift) report.

**4. To analyze the results of the model test.** You can explore the results of an online or offline test and contrast it to the performance in training. You can use this report to compare the model performance in an A/B test or during a shadow model deployment.

## Regression Performance Report

If you want to visually explore the model performance, create a new Report object and include the `RegressionPreset`.

### Code example

```python
reg_performance_report = Report(metrics=[
    RegressionPreset(),
])

ref_performance_report.run(reference_data=ref, current_data=bcur)

reg_performance_report
```

### How it works

The **Regression Performance** report evaluates the quality of a regression model.

It can also compare the performance against the past, or the performance of an alternative model.

* Works for a **single model** or helps compare the **two**
* Displays a variety of plots related to the **performance** and **errors**
* Helps explore areas of **under-** and **overestimation**

### Data Requirements

* To run this report, you need to have input features, and **both target and prediction** columns available. Input features are optional. Pass them if you want to explore the relations between features and target.
* To generate a comparative report, you will need **two** datasets. The **reference** dataset serves as a benchmark. Evidently analyzes the change by comparing the **current** production data to the **reference** data.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8bd9a2018ea10add3c55f3b143884457e8bf6c2a%2Ftwo_datasets_regression.png?alt=media\&token=1f648618-132c-4332-9560-97ee71c2c1d0)

You can also run this report for a **single** dataset, with no comparison performed.

### How it looks

The report includes multiple components. All plots are interactive.

**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.

#### **1. Model Quality Summary Metrics**

Evidently calculate a few standard model quality metrics: Mean Error (ME), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE).

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0592d3c8a92179e411b172cd8eb8884138505050%2Freg_perf_model_quality_summary%20\(1\).png?alt=media)

For each quality metric, Evidently also shows one standard deviation of its value (in brackets) to estimate the stability of the performance.

**To support the model performance analysis, Evidently also generates interactive visualizations. They help analyze where the model makes mistakes and come up with improvement ideas.**

#### 2. **Predicted vs Actual**

Predicted versus actual values in a scatter plot.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1269097ed5a2ea2a6bed82561daf473d2de81e0e%2Freg_perf_predicted_actual%20\(1\).png?alt=media)

#### 3. **Predicted vs Actual in Time**

Predicted and Actual values over time or by index, if no datetime is provided.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-9159f11410f69aded7e280be6f85aa9c1f8cf00d%2Freg_perf_predicted_actual_in_time%20\(1\)%20\(1\).png?alt=media)

#### 4. Error (Predicted - Actual)

Model error values over time or by index, if no datetime is provided.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-712477dee018c0e0c8f64bcc4cec3506a2e4bece%2Freg_perf_error%20\(1\).png?alt=media)

#### 5. Absolute Percentage Error

Absolute percentage error values over time or by index, if no datetime is provided.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b7221d5d2c9be3177055289ee1ba043539b9e10b%2Freg_perf_abs_per_error.png?alt=media\&token=6871f6f1-f463-49db-843e-65bb1e0fe13e)

#### 6. Error Distribution

Distribution of the model error values.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e9ac059e8fd627aa632cfad0e1a6096376205a83%2Freg_perf_error_distribution%20\(1\).png?alt=media)

#### 7. Error Normality

Quantile-quantile plot ([Q-Q plot](https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot)) to estimate value normality.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-acfd422bc65e1153cd554783d72e8b5f26086365%2Freg_perf_error_normality%20\(1\).png?alt=media)

**Next, Evidently explore in detail the two segments in the dataset: 5% of predictions with the highest negative and positive errors. We refer to them as "underestimation" and "overestimation" groups. We refer to the rest of the predictions as "majority".**

#### **8. Mean Error per Group**

A summary of the model quality metrics for each of the two segments: mean Error (ME), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE).

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2fb0b43dbd352188ebb4ef8693a07cc53fdddd17%2Freg_perf_mean_error_per_group.png?alt=media\&token=51afc892-c513-40b7-8c1c-55ca311341cc)

#### **9. Predicted vs Actual per Group**

Prediction plots that visualize the regions where the model underestimates and overestimates the target function.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-6a3efa2e1605602e0056354603d6b198210029c9%2Freg_perf_predicted_actual_per_group%20\(1\).png?alt=media)

#### **10. Error Bias: Mean/Most Common Feature Value per Group**

This table helps quickly see the differences in feature values between the 3 groups:

* **OVER** (top-5% of predictions with overestimation)
* **UNDER** (top-5% of the predictions with underestimation)
* **MAJORITY** (the rest 90%)

For the numerical features, it shows the mean value per group. For the categorical features, it shows the most common value.

If you have two datasets, the table displays the values for both REF (reference) and CURR (current).

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-cda35cf799db0f2431780cb4f03340da6e13d239%2Freg_perf_error_bias_table.png?alt=media\&token=0d48ba4c-f024-47f5-a501-35b6f1a65867)

If you observe a large difference between the groups, it means that the model error is sensitive to the values of a given feature.

**To search for cases like this, you can sort the table using the column "Range(%)".** It increases when either or both of the "extreme" groups are different from the majority.

Here is the formula used to calculate the Range %:

$$
Range = 100\*|(Vover-Vunder)/(Vmax-Vmin)|
$$

***Where:*** ***V**over = average feature value in the OVER group; **V**under = average feature value in the UNDER group; **V**max = maximum feature value; **V**min = minimum feature value*

#### **11. Error Bias per Feature**

For each feature, Evidently shows a histogram to visualize the **distribution of its values in the segments with extreme errors** and in the rest of the data. You can visually explore if there is a relationship between the high error and the values of a given feature.

Here is an example where extreme errors are dependent on the "temperature" feature.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0f36fde30923f9a29d1c62512e42aae47a53ef54%2Freg_perf_error_bias_per_feature.png?alt=media\&token=00d541b0-611d-404e-bd65-04ddb6beeeba)

#### 12. Predicted vs Actual per Feature

For each feature, Evidently also show the Predicted vs Actual scatterplot. It helps visually detect and explore underperforming segments which might be sensitive to the values of the given feature.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-12264ff6301d03477160f5b9f004d7af52476d88%2Freg_perf_error_bias_predicted_actual_per_feature.png?alt=media\&token=86a2deba-d146-4817-9b9f-448a86acbb34)

### Metrics output

You can get the report output as a JSON or a Python dictionary:

<details>

<summary>See JSON example</summary>

```yaml
{
  "regression_performance": {
    "name": "regression_performance",
    "datetime": "datetime",
    "data": {
      "utility_columns": {
        "date": "date",
        "id": null,
        "target": "target",
        "prediction": "prediction"
      },
      "cat_feature_names": [],
      "num_feature_names": [],
      "metrics": {
        "reference": {
          "mean_error": mean_error,
          "mean_abs_error": mean_abs_error,
          "mean_abs_perc_error": mean_abs_perc_error,
          "error_std": error_std,
          "abs_error_std": abs_error_std,
          "abs_perc_error_std": abs_perc_error_std,
          "error_normality": {
            "order_statistic_medians": [],
            "slope": slope,
            "intercept": intercept,
            "r": r
          },
          "underperformance": {
            "majority": {
              "mean_error": mean_error,
              "std_error": std_error
            },
            "underestimation": {
              "mean_error": mean_error,
              "std_error": std_error
            },
            "overestimation": {
              "mean_error": mean_error,
              "std_error": std_error
            }
          }
        },
        "current": {
          "mean_error": mean_error,
          "mean_abs_error": mean_abs_error,
          "mean_abs_perc_error": mean_abs_perc_error,
          "error_std": error_std,
          "abs_error_std": abs_error_std,
          "abs_perc_error_std": abs_perc_error_std,
          "error_normality": {
            "order_statistic_medians": [],
            "slope": slope,
            "intercept": intercept,
            "r": r
          },
          "underperformance": {
            "majority": {
              "mean_error": mean_error,
              "std_error": std_error
            },
            "underestimation": {
              "mean_error": mean_error,
              "std_error": std_error
            },
            "overestimation": {
              "mean_error": mean_error,
              "std_error": std_error
            }
          }
        },
        "error_bias": {
          "feature_name": {
            "feature_type": "num",
            "ref_majority": ref_majority,
            "ref_under": ref_under,
            "ref_over": ref_over,
            "ref_range": ref_range,
            "prod_majority": prod_majority,
            "prod_under": prod_under,
            "prod_over": prod_over,
            "prod_range": prod_range
          },
          
          "holiday": {
            "feature_type": "cat",
            "ref_majority": 0,
            "ref_under": 0,
            "ref_over": 0,
            "ref_range": 0,
            "prod_majority": 0,
            "prod_under": 0,
            "prod_over": 1,
            "prod_range": 1
          },
        }
      }
    }
  },
  "timestamp": "timestamp"
}
```

</details>

### Report customization

* You can perform the Error bias analysis only for selected columns.
* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* If you want to exclude some of the metrics, you can create a custom report by combining the chosen metrics. See the complete list [here](/reference/all-metrics)

## Regression Performance Test Suite

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-73ed815a4cc0e1ad5680d39b62aa80fa5ae3497d%2Ftest_preset_regression-min.png?alt=media\&token=7f2159fe-4bdd-45aa-9c3e-979c0a4e999f)

If you want to run regression performance checks as part of a pipeline, you can create a Test Suite and include the `RegressionTestPreset`.

### Code example

```python
regression_performance = TestSuite(tests=[
   RegressionTestPreset(),
])
 
regression_performance.run(reference_data=ref, current_data=curr)
regression_performance
```

### How it works

You can use the `RegressionTestPreset` to evaluate the quality of a regression model, when you have the ground truth data (actuals).

* It compares **regression quality metrics** against the defined expectation.
* For Evidently to **generate the test conditions automatically**, you should pass the reference dataset (e.g., performance during model validation or a previous period). You can also set the performance expectations manually by passing a custom test condition.
* If you neither pass the reference dataset nor set custom test conditions, Evidently will compare the model performance to a **dummy model**.

Head here to the [All tests](/reference/all-tests) table to see the composition of each preset and default parameters.

### Test Suite customization

* You can set custom test conditions.
* If you want to exclude some tests or add additional ones, you can create a custom test suite by combining the chosen tests. See the complete list [here](/reference/all-tests).

## Examples

* Browse the [examples](/tutorials-and-examples/examples) for sample Jupyter notebooks and Colabs.
* See a tutorial "[How to break a model in 20 days](https://evidentlyai.com/blog/tutorial-1-model-analytics-in-production)" where we create a demand prediction model and analyze its gradual decay.


# Classification Performance

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can use the pre-built Reports and Test suites to analyze the performance of a classification model. The Presets work for binary and multi-class classification, probabilistic and non-probabilistic classification.

* **Report**: for visual analysis or metrics export, use the `ClassificationPreset`.
* **Test Suite**: for pipeline checks, use the `MulticlassClassificationTestPreset`, `BinaryClassificationTopKTestPreset` or `BinaryClassificationTestPreset`.

## Use Case

These presets help evaluate and test the quality of classification models. You can use them:

**1. To monitor the performance of a classification model in production.** You can run the test suite as a regular job (e.g., weekly or when you get the labels) to contrast the model performance against the expectation. You can generate visual reports for documentation and sharing with stakeholders.

**2. To trigger or decide on the model retraining.** You can use the test suite to check if the model performance is below the threshold to initiate a model update.

**3. To debug or improve model performance.** If you detect a quality drop, you can use the visual report to explore the model errors and underperforming segments. By manipulating the input data frame, you can explore how the model performs on different data segments (e.g., users from a specific region). You can also combine it with the [Data Drift](/presets/data-drift) report.

**4. To analyze the results of the model test.** You can explore the results of an online or offline test and contrast it to the performance in training. You can also use this report to compare the model performance in an A/B test or during a shadow model deployment.

To run performance checks as part of the pipeline, use the Test Suite. To explore and debug, use the Report.

## Classification Performance Report

If you want to visually explore the model performance, create a new Report object and include the `ClassificationPreset`.

### Code example

```python
classification_performance_report = Report(metrics=[
    ClassificationPreset(),
])

classification_performance_report.run(reference_data=bcancer_ref, current_data=bcancer_cur)

classification_performance_report
```

### How it works

This report evaluates the quality of a classification model.

* Can be generated for a **single dataset**, or compare it **against the reference** (e.g. past performance or alternative model).
* Works for **binary** and **multi-class**, **probabilistic** and non-probabilistic classification.
* Displays a variety of metrics and plots related to the model **performance**.
* Helps **explore regions** where the model makes different types of **errors**.

### Data Requirements

To run this report, you need to have **both target and prediction** columns available. Input features are optional. Pass them if you want to explore the relations between features and target.

Refer to the [column mapping section](/user-guide/input-data/column-mapping) to see how to pass model predictions and labels in different cases.

The tool does not yet work for multi-label classification. It expects a single true label.

To generate a comparative report, you will need **two** datasets.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-eabc7007b02ce25aabc99de56cc8db8121e87e42%2Ftwo_datasets_classification.png?alt=media\&token=89edb6a4-2d09-4a3b-a387-9b08292074b7)

You can also run this report for a **single** dataset, with no comparison performed.

### How it looks

The report includes multiple components. The composition might vary based on problem type (there are more plots in the case of probabilistic classification). All plots are interactive.

**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.

#### **1. Model Quality Summary Metrics**

Evidently calculates a few standard model quality metrics: Accuracy, Precision, Recall, F1-score, ROC AUC, and LogLoss.

**To support the model performance analysis, Evidently also generates interactive visualizations. They help analyze where the model makes mistakes and come up with improvement ideas.**

#### 2. Class Representation

Shows the number of objects of each class.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-014d6141c52c22ea8c3367805211fd8d40e31849%2Fprob_class_perf_class_representation%20\(1\).png?alt=media)

#### 3. Confusion Matrix

Visualizes the classification errors and their type.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-d44aceed79a4f98cde410058c12367a63037f2ce%2Fprob_class_perf_confusion_matrix%20\(1\).png?alt=media)

#### 4. Quality Metrics by Class

Shows the model quality metrics for the individual classes. In the case of multi-class problems, it will also include ROC AUC.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-54512715bad59a70038f7c168822fd087f1d2719%2Fprob_class_perf_quality_by_class%20\(1\).png?alt=media)

#### **5. Class Separation Quality**

A scatter plot of the predicted probabilities shows correct and incorrect predictions for each class.

It serves as a representation of both model accuracy and the quality of its calibration. It also helps visually **choose the best probability threshold for each class.**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e16ca5618f050c77396f80958fecebf330c24c72%2Fprob_class_perf_class_separation_quality%20\(1\).png?alt=media)

#### 6. Probability Distribution

A similar view as above, it shows the distribution of predicted probabilities.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-41006102d2e027f1b046d083bfd9fe097dee5563%2Fprob_class_perf_probability_distr%20\(1\).png?alt=media)

#### **7. ROC Curve**

ROC Curve (**receiver operating characteristic curve**) shows the share of true positives and true negatives at different classification thresholds.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-756dcacc9d462ad45745c6bb8f6dd23c5a18d9d2%2Fprob_class_perf_roc.png?alt=media\&token=b47338eb-8e52-41f8-9b2e-bc8bfb69856b)

#### 8. **Precision-Recall Curve**

The **precision**-**recall curve** shows the trade-off between **precision** and **recall** for different classification thresholds.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-f558cc77f29cde0d60ffddc81a622232ca022c41%2Fprob_class_perf_pr%20\(1\).png?alt=media)

#### 9. Precision-Recall Table

The table shows possible **outcomes for different classification thresholds** and **prediction coverage**. If you have two datasets, the table is generated for both.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-23088b547ed1b7126b1584e67acd2bf42715582b%2Fprob_class_perf_pr_table_current%20\(1\).png?alt=media)

Each line in the table defines a case when only *top-X%* predictions are considered, with a 5% step. It shows the absolute number of predictions *(Count)* and the probability threshold *(Prob)* that correspond to this combination.

The table then shows the quality metrics for a given combination. It includes *Precision*, *Recall*, the share of *True Positives (TP)*, and *False Positives (FP)*.

This helps explore the quality of the model if you choose to act only on some of the predictions.

#### 10. Classification Quality by Feature

In this table, we show a number of plots for each feature. To expand the plots, click on the feature name.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-89edd11c3d8fe70b6e8d65cb7e37ca4f246b067f%2Fprob_class_perf_classification_quality_by_feature%20\(1\).png?alt=media)

In the tab “ALL”, you can see the distribution of classes against the values of the feature. If you compare the two datasets, it visually shows the changes in the feature distribution and in the relationship between the values of the feature and the target.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-84fc78dd31a5732be416239154f2c90515d78c42%2Fprob_class_perf_classification_quality_by_feature_example_all%20\(1\).png?alt=media)

For each class, you can see the predicted probabilities alongside the values of the feature.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-95a87ddd8e3296f900797f822e06b5c623de08a4%2Fprob_class_perf_classification_quality_by_feature_example_class%20\(1\).png?alt=media)

It visualizes the regions where the model makes errors of each type and reveals the low-performance segments. You can compare the distributions and see **if the errors are sensitive to the values of a given feature**.

### Metrics output

You can get the report output as a JSON or a Python dictionary:

<details>

<summary>See JSON example</summary>

```yaml
{
  "probabilistic_classification_performance": {
    "name": "probabilistic_classification_performance",
    "datetime": "datetime",
    "data": {
      "utility_columns": {
        "date": null,
        "id": null,
        "target": "target",
        "prediction": [
          "label1",
          "label2",
          "label3"
        ]
      },
      "cat_feature_names": [],
      "num_feature_names": [],
      "metrics": {
        "reference": {
          "accuracy": accuracy,
          "precision": precision,
          "recall": recall,
          "f1": f1,
          "roc_auc": roc_auc,
          "log_loss": log_loss,
          "metrics_matrix": {
            "label1": {
              "precision": precision,
              "recall": recall,
              "f1-score": f1,
              "support": support
            },
            "accuracy": accuracy,
            "macro avg": {
              "precision": precision,
              "recall": recall,
              "f1-score": f1,
              "support": support
            },
            "weighted avg": {
              "precision": precision,
              "recall": recall,
              "f1-score": f1,
              "support": support
            }
          },
          "roc_aucs": [
            roc_auc_label_1,
            roc_auc_label_2,
            roc_auc_label_3
          ],
          "confusion_matrix": {
            "labels": [],
            "values": []
          },
          "roc_curve": {
            "label1": {
              "fpr": [],
              "tpr": [],
              "thrs": []
          },  
          "pr_curve": {
            "label1": []
        },
        "current": {
          "accuracy": accuracy,
          "precision": precision,
          "recall": recall,
          "f1": f1,
          "roc_auc": roc_auc,
          "log_loss": log_loss,
          "metrics_matrix": {
            "label1": {
              "precision": precision,
              "recall": recall,
              "f1-score": f1,
              "support": support
          },
          "roc_aucs": [
            roc_auc_label_1,
            roc_auc_label_2,
            roc_auc_label_3
          ],
          "confusion_matrix": {
            "labels": [],
            "values": [],
          },
          "roc_curve": {
            "label1": {
              "fpr": [],
              "tpr": [],
              "thrs": []
          },
          "pr_curve": {
            "label1": []
          }
        }
      }
    }
  },
  "timestamp": "timestamp"
}
```

</details>

### Report customization

* You can perform the analysis of relations between features and target only for selected columns.
* You can pass relevant parameters to change the way some of the metrics are calculated, such as decision threshold or K to evaluate precision\@K. See the available parameters [here](/reference/all-metrics)
* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* If you want to exclude some of the metrics, you can create a custom report by combining the chosen metrics. See the complete list [here](/reference/all-metrics)

## Classification Performance Test Suite

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-d67364a99fcd713e710d193d08ed4e983c1c0826%2Ftest_preset_binarytopKclass-min.png?alt=media\&token=194712ce-1311-4fc8-b219-8fbd51b0c29f)

If you want to run classification performance checks as part of a pipeline, you can create a Test Suite and use one of the classification presets. There are several presets for different classification tasks. They apply to Multiclass Classification, Binary Classification, and Binary Classification at topK accordingly:

```python
MulticlassClassificationTestPreset
BinaryClassificationTopKTestPreset
BinaryClassificationTestPreset
```

### Code example

```python
binary_topK_classification_performance = TestSuite(tests=[
    BinaryClassificationTopKTestPreset(k=10),
])

binary_topK_classification_performance.run(reference_data=ref, current_data=cur)
binary_topK_classification_performance
```

### How it works

You can use the test presets to evaluate the quality of a classification model when you have the ground truth labels.

* Each preset compares **relevant quality metrics** for the model type and **against the defined expectation**.
* They also test for the **target drift** to detect shift in the distribution of classes and/or probabilities. It might indicate emerging concept drift.
* For Evidently to **generate the test conditions automatically**, you should pass the reference dataset (e.g., performance during model validation or a previous period). You can also set the performance expectations manually by passing a custom test condition.
* If you neither pass the reference dataset nor set custom test conditions, Evidently will compare the model performance to a **dummy model**.

Head here to the [All tests](/reference/all-tests) table to see the composition of each preset and default parameters.

### Test Suite customization

* You can set custom test conditions.
* You can pass relevant parameters to change how some of the metrics are calculated, such as classification decision threshold or K to evaluate precision\@K. See the [available parameters](/reference/all-tests).
* If you want to exclude some tests or add additional ones, you can create a custom test suite by combining the chosen tests. See the complete list [here](/reference/all-tests).

### Examples

* Browse the [examples](/tutorials-and-examples/examples) for sample Jupyter notebooks and Colabs.
* See a blog post and a tutorial "[What is your model hiding](https://evidentlyai.com/blog/tutorial-2-model-evaluation-hr-attrition)" where we analyze the performance of two models with identical ROC AUC to choose between the two.


# NoTargetPerformance

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can combine different checks to test data quality, stability, and drift when you have a model with delayed feedback.

* **Test Suite**: for pipeline checks, use the `NoTargetPerformanceTestPreset`.

## Use Case

This test suite is designed for a specific scenario:

**To monitor the model performance without ground truth.** You can use it to perform batch checks for a model that has delayed feedback (when true labels or actuals come days, weeks or months after the prediction is made). This preset helps evaluate the production model quality through proxy metrics. It combines several metrics that check for data quality, data integrity, and data and prediction drift.

## NoTargetPerformance Test Suite

To perform the check, create a new Test Suite object and include `NoTargetPerformanceTestPreset`.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-bb0cd186cefa1525ef1b1a48e584c0a00ddfc310%2Ftest_preset_notargetperformance-min.png?alt=media\&token=b7b938df-49cc-448e-a724-1aa02736b655)

### Code example

```python
no_target_performance = TestSuite(tests=[
   NoTargetPerformanceTestPreset()
])
 
no_target_performance.run(reference_data=past_week, current_data=curr_week)
no_target_performance
```

### Data requirements

* You need to provide **two** datasets with identical schema. The **reference** dataset serves as a benchmark (e.g., previous data batch). Evidently analyzes the change by comparing the **current** production data to the **reference** data.
* They should include **input features and predictions**.
* **Column mapping**. You can explicitly specify the types input columns and target using [column mapping](/user-guide/input-data/column-mapping). If it is not specified, Evidently will try to guess the types automatically. It is recommended to use column mapping to avoid errors. If you have text features, you must always specify this in column mapping.

### How it works

Evidently will generate the test conditions automatically based on the set of heuristics or provided reference dataset. You can also pass custom conditions for data drift.

The preset combines several checks that go well together.

**Data stability**. They verify if the input column types match the reference and whether you have features out of range (for numerical columns) or out of the list (for categorical data). The preset also checks for missing data.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e62c6e26a18d3dc15e22ad0584cd13700fe49611%2Ftest_column_type.png?alt=media\&token=71d1eecc-a5f1-48d7-9dd9-744d5b72a5e1)

**Prediction drift**. This test checks if there is a distribution shift in the model prediction. The default [drift detection algorithm](/reference/data-drift-algorithm) is used.

**Input data drift**. The preset also tests for distribution shifts in the model input features. It returns the overall share of drifting features (alerting if more than ⅓ of the features drifted).

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8e5877cc9784574d22db585ae36fcbfab5ea8907%2Ftest_share_of_drifted_features.png?alt=media\&token=ce61a04c-7b2e-4957-b681-6b38905e1dc5)

**Input data stability**. One more check verifies the change in the mean values of numerical features. It automatically tests if the means of all numerical columns are within 2 standard deviations from the reference mean. This test helps detect significant changes.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-680d58d0d7f2e0a920be242e05f8ed68edab565b%2Ftest_mean_in_sigmas.png?alt=media\&token=463eacc5-cc14-48bd-a0b9-aac5a38cf171)

### Report customization

* You can [specify the drift detection methods and thresholds](/user-guide/customization/options-for-statistical-tests).
* You can add a [custom drift detection method](/user-guide/customization/add-custom-metric-or-test).
* You can apply data quality and stability checks only to selected columns.
* You can add extra tests to this preset, e.g., to perform column-level drift checks for the most important features.
* You can create a different test suite from scratch, taking this one as an inspiration.

Head here to the [All tests](/reference/all-tests) table to see the description of individual tests and default parameters.

### Examples

* Browse the [example](/tutorials-and-examples/examples) notebooks to see a sample Test Suite.


# Text Evals

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can explore and compare text datasets.

* **Report**: for visual analysis or metrics export, use the `TextEvals`.

## Text Evals Report

To visually explore the descriptive properties of text data, you can create a new Report object and generate `TextEvals` preset for the column containing the text data. It's best to define your own set of `descriptors` by passing them as a list to the `TextEvals` preset. For more details, see [how descriptors work](/user-guide/tests-and-reports/text-descriptors).

If you don’t specify descriptors, the Preset will use default statistics.

### Code example

```python
text_overview_report = Report(metrics=[
    TextEvals(column_name="Review_Text")
])

text_overview_report.run(reference_data=ref, current_data=cur)
text_overview_report
```

Note that to calculate some text-related metrics, you may also need to also import additional libraries:

```
import nltk
nltk.download('words')
nltk.download('wordnet')
nltk.download('omw-1.4')
```

### Data Requirements

* You can pass **one or two** datasets. Evidently will compute descriptors both for the **current** production data and the **reference** data. If you pass a single dataset, there will be no comparison.
* To run this preset, you must have **text columns** in your dataset. Additional features and prediction/target are optional. Pass them if you want to analyze the correlations with text descriptors.
* **Column mapping**. Specify the columns that contain text features in [column mapping](/user-guide/input-data/column-mapping).

### How it looks

The report includes 5 components. All plots are interactive.

**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.

#### Text Descriptors Distribution

The report generates several features that describe different text properties and shows the distributions of these text descriptors.

**Text length**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-83bcc2b634692787edd2ef67c3148661b6569217%2Fmetric_text_descriptors_distribution_text_length-min.png?alt=media\&token=23dbb5fb-26aa-4261-9e6b-776bba1d1a19)

**Non-letter characters**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-ac682f2507f57302d98a0c0f09949ec9dbb34b3d%2Fmetric_text_descriptors_distribution_nlc-min.png?alt=media\&token=3f29df1f-f9da-4c02-b2d0-83725fd71ba5)

**Out-of-vocabulary words**

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-9c9c9bdf9d88ff534e8801f98d1919e44b0a0789%2Fmetric_text_descriptors_distribution_oov-min.png?alt=media\&token=cfecc20a-2f34-4b29-9be7-799ea85ac5c4)

**Sentiment**

Shows the distribution of text sentiment (-1 negative to 1 positive).

**Sentence Count**

Shows the sentence count.

### Metrics output

You can also get the report output as a JSON or a Python dictionary.

### Report customization

* You can [choose your own descriptors](/user-guide/tests-and-reports/text-descriptors).
* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* You can create a different report or test suite from scratch, taking this one as an inspiration.

## Examples

* Head to an [example how-to notebook](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_run_calculations_over_text_data.ipynb) to see an example Text Overview preset and other metrics and tests for text data.


# Recommender System

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**TL;DR:** You can monitor and analyze the performance of a recommender system.

* **Report**: for visual analysis or metrics export, use the `RecsysPreset`.
* **Test Suite**: for pipeline checks, use the `RecsysTestPreset`.

## Recommendation Quality Report

If you want to visually explore the model performance, create a new Report object and include the `RecsysPreset`.

### Code example

```python
report = Report(metrics=[
    RecsysPreset(k=5),
])
column_mapping = ColumnMapping(recommendations_type='rank', target='rating', prediction='rank', item_id='title', user_id='user_id')
report.run(
    reference_data=reference,
    current_data=current,
    column_mapping=column_mapping,
    additional_data={'current_train_data': train_data}
  )
report
```

Check the [How-to example](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_run_recsys_metrics.ipynb) for an end-to-end example on a sample dataset.

### How it works

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2f45f72abc613e41675bcb146bf935a5b7f8aafa%2Fmetric_popularity_bias-min.png?alt=media\&token=6a16323c-76ac-4310-9c2d-08f4d97d0562)

The **Recommender System Performance** report evaluates the quality of a recommender system. It can also compare the performance against the past, or the performance of an alternative model.

* Works for a **single model** or helps compare the **two**.
* Displays 10+ plots related to the **predictive quality**, **ranking quality** and recommendation **diversity**.
* Helps explore recommendation bias and examples of recommendations.

Check the [All Metrics Table](/reference/all-metrics) for a complete list of metrics and parameters. Check the [Ranking Metrics Guide](/reference/all-metrics/ranking-metrics) for an explanation of each metric.

### Data Requirements

* To run this report, you need to have both **target** (true relevance) and **prediction** (rank or score) columns available. Additional **training data** and **input features** are required for some of the metrics.
* To generate a comparative report, you will need **two** datasets. The **reference** dataset serves as a benchmark, e.g. past model performance or a performance of a different model. Evidently analyzes the change by comparing the **current** production data to the **reference** data.
* You can also run this report for a **single** dataset, with no comparison performed.
* You must perform **column mapping** to map the data inputs correctly.

Check the [Column Mapping](/user-guide/input-data/column-mapping) guide and instructions on using [Additional Data](/user-guide/input-data/recsys_data) in recommendations.

### Report customization

* You can pass `user_bias_columns` columns or `item_bias_columns` you want to evaluate for bias.
* You can pass `display_features` you want to display in the recommendation cases table.
* You can pass `user_ids: List` to specify which users you want to display in the recommendation cases table.
* You can use a [different color schema for the report](/user-guide/customization/options-for-color-schema).
* If you want to exclude some metrics, you can create a custom report by combining the chosen metrics. See the complete list [here](/reference/all-metrics)

Check the [All Metrics Table](/reference/all-metrics) for a complete list of metrics and parameters.

## Recommender System Test Suite

If you want to run recommender system performance checks as part of a pipeline, you can create a Test Suite and include the `RecsysTestPreset`.

### Code example

```python

tests = TestSuite(tests=[
    RecsysTestPreset(k=5)
])

column_mapping = ColumnMapping(recommendations_type='rank', target='rating', prediction='rank', item_id='title', user_id='user_id')
tests.run(
    reference_data=reference,
    current_data=current,
    column_mapping=column_mapping,
    additional_data={'current_train_data': train_data}
  )
tests
```

### How it works

You can use the `RecsysTestPreset` to evaluate the quality of a recommender or ranking system, when you have the ground truth data (relevance scores).

* It compares **ranking quality metrics** against the defined expectation.
* For Evidently to **generate the test conditions automatically**, you should pass the reference dataset (e.g., performance during the previous period). You can also set the performance expectations manually by passing custom test conditions.

Head here to the [All tests](/reference/all-tests) table to see the composition of each preset and default parameters.

### Test Suite customization

* You can set custom test conditions.
* If you want to exclude some tests or add additional ones, you can create a custom test suite by combining the chosen tests. See the complete list [here](/reference/all-tests).


# Tutorials and Examples


# All Tutorials

Code examples and tutorials.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Quick Start

Check the short Quickstart examples [here](https://github.com/evidentlyai/docs-old/blob/main/get-started/README.MD).

## Get Started Tutorials

Introductory tutorials that walk you through the basic functionality step by step.

| Title                                   | Guide                                                      | Code                                                                                                                                                                       |
| --------------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LLM Evaluation                          | [Tutorial](/tutorials-and-examples/tutorial-llm)           | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/llm_evaluation_tutorial.ipynb)         |
| Data & ML Monitoring                    | [Tutorial](/tutorials-and-examples/tutorial-cloud)         | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/data_and_ml_monitoring_tutorial.ipynb) |
| LLM Tracing                             | [Tutorial](/tutorials-and-examples/tutorial_tracing)       | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/llm_tracing_tutorial.ipynb)            |
| Intro to Reports & Test Suites (OSS)    | [Tutorial](/tutorials-and-examples/tutorial_reports_tests) | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/getting_started_tutorial.ipynb)        |
| Self-host ML monitoring Dashboard (OSS) | [Tutorial](/tutorials-and-examples/tutorial-monitoring)    | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/get_started_monitoring.py)             |

## Example Reports and Tests

Simple examples show different local evaluations (Metrics, Tests and Presets) for tabular data and ML.

| Title                    | Code example                                                                                                                                                                         | Contents                                                                                                                                                                                                                        |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Evidently Test Presets   | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/evidently_test_presets.ipynb)                    | <p>Pre-built Test Suites on tabular data:</p><ul><li>Data Drift</li><li>Data Stability</li><li>Data Quality</li><li>NoTargetPerformance</li><li>Regression</li><li>Classification (Multi-class, binary, binary top-K)</li></ul> |
| Evidently Tests          | <p><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/evidently_tests.ipynb">Jupyter notebook</a><br></p>     | <ul><li>All individual Tests (50+) that one can use to create a custom Test Suite. Tabular data examples.</li><li>How to set test conditions and parameters.</li></ul>                                                          |
| Evidently Metric Presets | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/evidently_metric_presets.ipynb)                  | <p>All pre-built Reports:</p><ul><li>Data Drift</li><li>Target Drift</li><li>Data Quality</li><li>Regression</li><li>Classification</li></ul>                                                                                   |
| Evidently Metrics        | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/evidently_metrics.ipynb)                         | <ul><li>All individual metrics (30+) that one can use to create a custom Report.</li><li>How to set simple metric parameters.</li></ul>                                                                                         |
| Evidently LLM Metrics    | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_evaluate_llm_with_text_descriptors.ipynb) | <ul><li>Evaluations for Text Data and LLMs</li></ul>                                                                                                                                                                            |

For LLM and text metrics, check the [LLM evaluation tutorial](/tutorials-and-examples/tutorial-llm).

## Tutorials - LLM

| Title                                          | Tutorial                                                            |
| ---------------------------------------------- | ------------------------------------------------------------------- |
| How to create LLM judge evaluator              | [Tutorial](/tutorials-and-examples/cookbook_llm_judge)              |
| How to run regression testing for LLM products | [Tutorial](/tutorials-and-examples/cookbook_llm_regression_testing) |

## Tutorials - ML

To better understand the Evidently use cases, refer to the **detailed tutorials** accompanied by the blog posts.

| Title                                                            | Code example                                                                                                                                                                         | Blog post                                                                                                                                                 |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Understand ML model decay in production (regression example)     | [Jupyter notebook](https://github.com/evidentlyai/community-examples/blob/main/tutorials/bicycle_demand_monitoring.ipynb)                                                            | [How to break a model in 20 days. A tutorial on production model analytics.](https://evidentlyai.com/blog/tutorial-1-model-analytics-in-production)       |
| Compare two ML models before deployment (classification example) | [Jupyter notebook](https://github.com/evidentlyai/community-examples/blob/main/tutorials/ibm_hr_attrition_model_validation.ipynb)                                                    | [What Is Your Model Hiding? A Tutorial on Evaluating ML Models.](https://evidentlyai.com/blog/tutorial-2-model-evaluation-hr-attrition)                   |
| Evaluate and visualize historical data drift                     | [Jupyter notebook](https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/integrations/mlflow_logging/historical_drift_visualization.ipynb) | [How to detect, evaluate and visualize historical drifts in the data.](https://evidentlyai.com/blog/tutorial-3-historical-data-drift)                     |
| Monitor NLP models in production                                 | [Colab](https://colab.research.google.com/drive/15ON-Ub_1QUYkDbdLpyt-XyEx34MD28E1)                                                                                                   | [Monitoring NLP models in production: a tutorial on detecting drift in text data](https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data) |
| Create ML model cards                                            | [Jupyter notebook](https://github.com/evidentlyai/community-examples/tree/main/tutorials/How_to_create_an_ML_model_card.ipynb)                                                       | [A simple way to create ML Model Cards in Python](https://www.evidentlyai.com/blog/ml-model-card-tutorial)                                                |
| Use descriptors to monitor text data                             | [Jupyter notebook](https://github.com/evidentlyai/community-examples/tree/main/tutorials/How_to_add_a_custom_text_descriptor.ipynb)                                                  | [Monitoring unstructured data for LLM and NLP with text descriptors](https://www.evidentlyai.com/blog/unstructured-data-monitoring)                       |

You can find more examples in the [Community Examples](https://github.com/evidentlyai/community-examples) repository.

### How to examples

For code examples on specific functionality, check the How-To examples:

{% content-ref url="<https://github.com/evidentlyai/evidently/tree/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions>" %}
<https://github.com/evidentlyai/evidently/tree/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions>
{% endcontent-ref %}

### Integrations

To see how to integrate Evidently in your prediction pipelines and use it with other tools, refer to the **integrations**.

{% content-ref url="/pages/C83MWVyC3hbvfQawVkV5" %}
[Evidently integrations](/integrations/integrations/evidently-integrations)
{% endcontent-ref %}


# Tutorial - Tracing

How to capture LLM inputs and outputs and view them in Evidently Cloud.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In this tutorial, you will learn how to set up tracing for an LLM application to collect inputs and outputs of your generative AI application and view the collected traces in Evidently Cloud. You can later run evaluations on the created datasets.&#x20;

You will use the following tools:

* **Tracely**: An open-source library based on OpenTelemetry to track events in your LLM application.
* **Evidently**: An open-source library to run LLM evaluations and interact with Evidently Cloud.&#x20;
* **Evidently Cloud**: A web platform to view traces and run evaluations.
* **OpenAI**: Used to simulate an LLM application.

You will go through the following steps:

* Install libraries&#x20;
* Set up and initialize tracing
* Create a simple question-answer LLM function
* Collect and send traces to Evidently Cloud&#x20;
* (Optional) Download the resulting dataset to run local evals

To complete the tutorial, use the provided code snippets or run a sample notebook.

Jupyter notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/llm_tracing_tutorial.ipynb>" %}

If you're having problems or getting stuck, reach out on [Discord](https://discord.com/invite/xZjKRaNp8b).

### 1. Installation

Install the necessary libraries:

```python
pip install evidently 
pip install tracely 
pip install openai
```

Import the required modules:

```python
import pandas as pd
import openai
from tracely import init_tracing
from tracely import trace_event
```

**Optional**. To load the traced dataset back to Python and run evals.

```python
from evidently.ui.workspace.cloud import CloudWorkspace
from evidently.report import Report
from evidently.metric_preset import TextEvals
from evidently.descriptors import *

```

## 2. Get the API keys

Obtain your API keys from Evidently Cloud and OpenAI.

* **Evidently Cloud**: Create an account, set up an Organization and Team. Get the API key from the [Token page](https://app.evidently.cloud/token). (Check the [step by step instructions](/setup/cloud_account) if you need help).
* **OpenAI:** Get your API key from OpenAI. ([Token page](https://platform.openai.com/api-keys)).

Set your API keys:

```python
openai_api_key = "YOUR_OPEN_AI_API_KEY" 
my_token = "YOUR_EVIDENTLY_TOKEN"
```

It is recommended to pass the key as an environment variable. [See Open AI docs](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety) for best practices.

## 3. Configure tracing

Set up configuration details:

* The `address` is the destination backend to store collected traces. In this case, it is Evidently Cloud.
* `Project_id` is the identification of the Evidently Project. Go to the [Projects page](https://app.evidently.cloud/), enter the selected Project and copy its ID.
* `Dataset_name` helps identify the resulting Tracing dataset. All data with the same ID would be grouped into single dataset.

```python
address="https://app.evidently.cloud/"
project_id="YOUR_PROJECT_ID"
dataset_name="YOUR_TRACING_DATASET_NAME"
```

Initialize tracing:

```python
init_tracing(
 address=address,
 api_key=my_token,
 project_id=project_id,
 export_name=dataset_name
 )
```

## 4. Trace a simple LLM app&#x20;

You will now create a simple function that sends a list of questions to the LLM and gets the responses.

Initialize the OpenAI client with the API key:

```python
client = openai.OpenAI(api_key=openai_api_key)
```

Define the list of questions to answer:

```python
question_list = [
   "What are the main differences between climate change and global warming?",
   "What are the key principles of quantum mechanics?",
   "Can you explain the theory of relativity in simple terms?",
   "How do vaccines work?",
   "What is blockchain technology and how does it work?",
   "What are the potential benefits and risks of artificial intelligence?",
   "What are the distinguishing features of Baroque art?",
   "What are some effective ways to manage stress?",
   "What is the difference between stocks and bonds?",
   "Can you explain the concept of cryptocurrency and how it works?",

]
```

Create a template for the questions you will pass to the LLM.

```python
question_prompt = """
Please answer the following question nicely with a clear structure of response and some conclusions at the end.
Here is a question: {{text_input}}
"""
```

Use the `@trace_event()` decorator from `Tracely` to trace the execution of the function. This captures input arguments and outputs, sending the trace data to Evidently Cloud.&#x20;

```python
@trace_event()
def pseudo_assistant(prompt, question):
 model = "gpt-4o-mini"
 system_prompt = "You are a nice and helpful assistant "
 user_prompt = prompt.replace("{{text_input}}", question)

 choices = client.chat.completions.create(
 model=model,
 messages=[
 {"role": "system", "content": system_prompt},
 {"role": "user", "content": user_prompt},
 ]
 )

 response = choices.choices[0].message.content

 return response
```

Loop through the list of questions and call the traced function `pseudo_assistant` to get responses while `Tracely` captures all relevant data.

```python
for question in question_list:
 pseudo_assistant(prompt=question_prompt, question=question)
```

## 5. View traces

Go to the Evidently Cloud, navigate to the datasets in the left menu, and open the traces you just sent. It might take a few moments until OpenAI processes all the inputs.&#x20;

You can now view, sort, export, and work with the traced dataset. You can run evaluations on this dataset both in the Cloud and locally.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b24151004a6a1fdab636353fd92895523311c88d%2Ftracing_tutorial_dataset.png?alt=media)

## 6. Load the dataset&#x20;

**This is an optional step**. If you want to access your traced dataset locally, for example, to run evaluations, you can do that by loading your dataset from Evidently Cloud.

Connect to the Evidently Cloud workspace:

```python
ws = CloudWorkspace(token=my_token, url=address)
```

Specify the dataset ID. You can copy it from the dataset page in the UI.

```python
dataset_id = "YOUR_DATASET_ID"
```

Load the dataset to pandas:

```python
traced_data = ws.load_dataset(dataset_id = dataset_id)
```

Preview the dataset with `traced_data.head()`.

## 7. Run an evaluation&#x20;

You can run evaluations on this dataset using the Evidently Python library. You can generate the Reports to view them locally or send them to Evidently Cloud.\
For example, let’s evaluate the length and sentiment of the responses, and whether they include the word "Certainly".

```python
project = ws.get_project(project_id)
```

Define the evaluations:

```python
evals_report = Report(
    metrics=[
        TextEvals(
            column_name="pseudo_assistant.result",
            descriptors=[
                TextLength(display_name="Response length"),
                Sentiment(display_name="Response sentiment"),
                IncludesWords(words_list=["certainly"], display_name="Says 'Certainly'"),
            ],
        ),
    ]
)
```

Run the Report on the `traced_data`:

```python
evals_report.run(reference_data=None, current_data=traced_data)
```

Send the results to Evidently Cloud:

```python
ws.add_report(project.id, evals_report, include_data=True)
```

To explore the evaluation results, go to Evidently Cloud, enter your Project and navigate to "Reports" in the left menu.

You can view and brows the results. For example, find the longest responses or all responses that say "Certainly".

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-bc577a8821b043c628b4baf514998cb660b1b4bb%2Ftracing_tutorial_eval_example.png?alt=media)

To view the evals locally, run `evals_report` for the Report and `evals_report.datasets().current` for the Dataset with added scores.&#x20;

## What's next?

Check the complete [LLM evaluation tutorial](/tutorials-and-examples/tutorial-llm) for more details: how to run other evaluation methods, including LLM as a judge, or test for specific conditions.&#x20;

Need help? Ask in our [Discord community](https://discord.com/invite/xZjKRaNp8b).


# Tutorial - Reports and Tests

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In this tutorial, you will use the Evidently open-source Python library to evaluate **data stability** and **data drift** on tabular data. You will run batch checks on a toy dataset and generate visual Reports and Test Suites in your Python environment.

We recommend going through this tutorial once to understand the basic functionality. Once you complete it, you will be ready to use all Evidently evaluations, including checks for ML model quality or text data.

You can later run the Evidently Reports and Test Suites independently or use them as a logging layer for Evidently ML Monitoring. You can choose between self-hosting [ML monitoring dashboard](/tutorials-and-examples/tutorial-monitoring) or sending the Reports and Test Suite to [Evidently Cloud platform](/tutorials-and-examples/tutorial-cloud) to monitor metrics over time.

To complete the tutorial, you need basic knowledge of Python. You should be able to complete it in **about 15 minutes**.

You can reproduce the steps in Jupyter notebooks or Colab or open and run a sample notebook from the links below.

Jupyter notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/getting_started_tutorial.ipynb>" %}

Video version:

{% embed url="<https://youtu.be/2d5AgPKbthE>" %}

You will go through the following steps:

* Install Evidently
* Prepare the input data
* Generate a pre-built data drift report
* Customize the report
* Run and customize data stability tests

If you're having problems or getting stuck, reach out on [Discord](https://discord.com/invite/xZjKRaNp8b).

### 1. Install Evidently

#### MAC OS and Linux

To install Evidently using the pip package manager, run:

```bash
$ pip install evidently
```

#### Hosted notebooks

If you are using Google Colab, Kaggle Kernel, Deepnote or Databricks notebooks, run the following command in the notebook cell:

```
!pip install evidently
```

#### Windows

To install Evidently in Jupyter notebook on Windows, run:

```bash
$ pip install evidently
```

### 2. Import Evidently

After installing the tool, import `evidently` and the required components. In this tutorial, you will use several **test suites** and **reports**. Each corresponds to a specific type of analysis.

You should also import `pandas`, `numpy`, and the toy `california_housing` dataset.

```python
import pandas as pd
import numpy as np

from sklearn.datasets import fetch_california_housing

from evidently import ColumnMapping

from evidently.report import Report
from evidently.metrics.base_metric import generate_column_metrics
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.metrics import *

from evidently.test_suite import TestSuite
from evidently.tests.base_test import generate_column_tests
from evidently.test_preset import DataStabilityTestPreset, NoTargetPerformanceTestPreset
from evidently.tests import *
```

### 3. Prepare the data

In the example, you will work with a toy dataset. In practice, you should use the model prediction logs. They can include input data, model predictions, and true labels or actuals, if available.

To prepare the data for analysis, create a `pandas.DataFrame`:

```python
data = fetch_california_housing(as_frame=True)
housing_data = data.frame
```

Rename one of the columns to “target” and create a “prediction” column. This way, the dataset will resemble the model application logs with known labels.

```python
housing_data.rename(columns={'MedHouseVal': 'target'}, inplace=True)
housing_data['prediction'] = housing_data['target'].values + np.random.normal(0, 5, housing_data.shape[0])
```

Split the dataset by taking 5000 objects for **reference** and **current** datasets.

```python
reference = housing_data.sample(n=5000, replace=False)
current = housing_data.sample(n=5000, replace=False)
```

The first **reference** dataset is the baseline. This is often the data used in model training or earlier production data. The second dataset is the **current** production data. Evidently will compare the current data to the reference.

If you work with your own data, you can prepare two datasets with an identical schema. You can also take a single dataset and explicitly identify rows for reference and current data.

{% hint style="info" %}
**Column mapping.** In this example, we directly proceed to analysis. In other cases, you might need a `ColumnMapping` object to help Evidently process the input data correctly. For example, you can point to the encoded categorical features or specify the name of the target column. Consult the [Column Mapping section](/user-guide/input-data/column-mapping) section for help.
{% endhint %}

### 4. Get the Data Drift report

Evidently **Reports** help explore and debug data and model quality. They calculate various metrics and generate a dashboard with rich visuals.

To start, you can use **Metric Presets**. These are pre-built Reports that group relevant metrics to evaluate a specific aspect of the model performance.

Let’s start with the **Data Drift**. This Preset compares the distributions of the model features and show which have drifted. When you do not have ground truth labels or actuals, evaluating input data drift can help understand if an ML model still operates in a familiar environment.

To get the Report, create a corresponding `Report` object, list the `preset` to include, and point to the reference and current datasets created in the previous step:

```python
report = Report(metrics=[
    DataDriftPreset(), 
])

report.run(reference_data=reference, current_data=current)
report
```

It will display the HTML report directly in the notebook.

First, you can see the Data Drift summary.

![Data Drift report overview](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-406f76cde15bb1a2cd9646a479238b7baa46896d%2Fget_started_3_data_drift_summary-min.png?alt=media\&token=8b029b4a-d7e7-4daa-aedc-532564214d0d)

If you click on individual features, it will show additional plots to explore.

![Data Drift report details](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b9f23b559ce24da180bda8c7740ada86fb084811%2Fget_started_4_data_drift_expand-min.png?alt=media\&token=5f402acc-d2d7-4d29-b3e6-bb542bea6e2f)

<details>

<summary>How does it work?</summary>

The data drift report compares the distributions of each feature in the two datasets. It [automatically picks](/reference/data-drift-algorithm) an appropriate statistical test or metric based on the feature type and volume. It then returns p-values or distances and visually plots the distributions. You can also [adjust the drift detection method or thresholds](/user-guide/customization/options-for-statistical-tests), or pass your own.

</details>

{% hint style="info" %}
**Aggregated visuals in plots.** Starting from v 0.3.2, all visuals in the Evidently Reports are aggregated by default. This helps decrease the load time and report size for larger datasets. If you work with smaller datasets or samples, you can pass an [option to generate plots with raw data](/user-guide/customization/report-data-aggregation). You can choose whether you want it on not based on the size of your dataset.
{% endhint %}

### 5. Customize the Report

Evidently Reports are very configurable. You can define which Metrics to include and how to calculate them.

To create a custom Report, you need to list individual **Metrics**. Evidently has dozens of Metrics that evaluate anything from descriptive feature statistics to model quality. You can calculate Metrics on the column level (e.g., mean value of a specific column) or dataset-level (e.g., share of drifted features in the dataset).

In this example, you can list several Metrics that evaluate individual statistics for the defined column.

```python
report = Report(metrics=[
    ColumnSummaryMetric(column_name='AveRooms'),
    ColumnQuantileMetric(column_name='AveRooms', quantile=0.25),
    ColumnDriftMetric(column_name='AveRooms')
])

report.run(reference_data=reference, current_data=current)
report
```

You will see a combined report that includes multiple Metrics:

![Part of the custom report, ColumnSummaryMetric.](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a1cef9066369bfebcfcf9bf3fdba200a358df902%2Fget-started-column-summary_metric-min.png?alt=media\&token=e3516c79-d950-4fbd-9979-655576e46ead)

If you want to generate multiple column-level Metrics, there is a helper function. For example, in order to calculate the same quantile value for all the columns in the list, you can use the generator:

```
report = Report(metrics=[
    generate_column_metrics(ColumnQuantileMetric, parameters={'quantile':0.25}, columns=['AveRooms', 'AveBedrms']),
])

report.run(reference_data=reference, current_data=current)
report
```

You can easily combine individual Metrics, Presets and metric generators in a single list:

```
report = Report(metrics=[
    ColumnSummaryMetric(column_name='AveRooms'),
    generate_column_metrics(ColumnQuantileMetric, parameters={'quantile':0.25}, columns='num'),
    DataDriftPreset()
])

report.run(reference_data=reference, current_data=current)
report
```

{% hint style="info" %}
**Available Metrics and Presets**. You can refer to the All Metrics [reference table](/reference/all-metrics) to browse available Metrics and Presets or use one of the example notebooks in the [Examples](/tutorials-and-examples/examples) section.
{% endhint %}

### 6. Define the Report output format

You can render the visualizations in the notebook as shown above. There are also alternative options.

If you only want to log the metrics and test results, you can get the output as a Python dictionary.

```python
report.as_dict()
```

You can also get the output as JSON.

```python
report.json()
```

You can also save HTML or JSON externally and specify a path and file name:

```python
report.save_html("file.html")
```

You can also save the output as an Evidently JSON `snapshot`. This will allow you to visualize the model or data quality over time using the Evidently ML monitoring dashboard.

```python
report.save("snapshot.json")
```

{% hint style="info" %}
**Building a live ML monitoring dashboard**. To better understand how the ML monitoring dashboard works, we recommend going through the [ML Monitoring Quickstart](/tutorials-and-examples/tutorial-monitoring) after completing this tutorial.
{% endhint %}

### 7. Run data stability tests

Reports help visually explore the data or model quality or share results with the team. However, it is less convenient if you want to run your checks automatically and only react to meaningful issues.

To integrate Evidently checks in the prediction pipeline, you can use the **Test Suites** functionality. They are also better suited to handle large datasets.

Test Suites help compare the two datasets in a structured way. A **Test Suite** contains several individual tests. Each **Test** compares a specific value against a defined condition and returns an explicit pass/fail result. You can apply Tests to the whole dataset or individual columns.

Just like with Reports, you can create a custom Test Suite or use one of the **Presets**.

Let's create a custom one! Imagine you received a new batch of data. Before generating the predictions, you want to check if the quality is good enough to run your model. You can combine several Tests to check missing values, duplicate columns, and so on.

You need to create a `TestSuite` object and specify the `preset`:

```python
tests = TestSuite(tests=[
    TestNumberOfColumnsWithMissingValues(),
    TestNumberOfRowsWithMissingValues(),
    TestNumberOfConstantColumns(),
    TestNumberOfDuplicatedRows(),
    TestNumberOfDuplicatedColumns(),
    TestColumnsType(),
    TestNumberOfDriftedColumns(),
])

tests.run(reference_data=reference, current_data=current)
tests
```

You will get a summary with the test results:

![Part of the custom Test Suite.](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-74b6613ecf7f5d837615e435bd093efc3ebdbb93%2Fget-started-test-output-min.png?alt=media\&token=0fef24d7-de3c-4ef1-b59a-7609bc57713b)

<details>

<summary>How does it work?</summary>

Evidently automatically generates the test conditions based on the provided reference dataset. They are based on heuristics. For example, the test for column types fails if the column types do not match the reference. The test for the number of columns with missing values fails if the number is higher than in reference. The test for the share of drifting features fails if over 50% are drifting. You can easily [pass custom conditions](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/custom-test-suite.md) to set your own expectations.

</details>

You can also use **Test Presets**. For example, `NoTargetPerformance` preset combines multiple checks related to data stability, drift and data quality to help evaluate the model without ground truth labels.

```python
suite = TestSuite(tests=[
    NoTargetPerformanceTestPreset(),
])

suite.run(reference_data=reference, current_data=current)
suite
```

You can group the outputs by test status, feature, test group, and type. By clicking on “details,” you will see related plots or tables.

![Details on Mean Value Stability test](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e693c37cd3da05d4274eb1491ae23a3742d1bd32%2Fget_started_2_mean_value_stability-min.png?alt=media\&token=5b9cb6e5-26f8-45cc-9be5-8e9429ae0323)

If some of the Tests fail, you can use the supporting visuals for debugging:

![Failed tests](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-42a8b256bcbdd274eb549061122cbdedbad55658%2Ftest-notargetperformance-min.png?alt=media\&token=7ff8e816-b078-4314-9c07-086ba0689e2d)

Just like with Reports, you can combine individual Tests and Presets in a single Test Suite and use column test generator to generate multiple column-level tests:

```python
suite = TestSuite(tests=[
    TestColumnDrift('Population'),
    TestShareOfOutRangeValues('Population'),
    generate_column_tests(TestMeanInNSigmas, columns='num'),
    
])

suite.run(reference_data=reference, current_data=current)
suite
```

{% hint style="info" %}
**Available Tests and Presets**. You can refer to the All Tests [reference table](/reference/all-tests) to browse available Tests and Presets. To see interactive examples, refer to the notebooks in the [examples](/tutorials-and-examples/examples) section.
{% endhint %}

You can also export the output in other formats.

To integrate Evidently checks in the prediction pipeline, you can get the output as JSON or a Python dictionary:

```python
suite.as_dict()
```

You can extract necessary information from the JSON or Python dictionary output and design a conditional workflow around it. For example, if tests fail, you can trigger an alert, retrain the model or generate the Report.

You can also save the output as an Evidently JSON `snapshot`. This will allow you to visualize the test results over time the Evidently ML monitoring dashboard.

```python
suite.save("snapshot.json")
```

### 8. What should I do next?

* **Explore available evaluations**

In this tutorial, you explored some of the data quality and data drift checks on tabular data. Evidently also support evaluations on text data and model quality checks.

The easiest way to understand what else is there is to look at **Presets**. Both Tests and Reports have multiple Presets. Some, like Data Quality, require only input data. You can use them even without the reference dataset. When you have the true labels, you can run Presets like **Regression Performance**, **Ranking Performance** and **Classification Performance** to evaluate the model quality and errors.

To understand the contents of each Preset, head to the [Preset overview](/presets/all-presets). If you want to see the pre-rendered examples of the reports, browse Colab notebooks in the [Examples](/tutorials-and-examples/examples) section. You can also design custom Reports and Test Suites from individual Metrics and Tests.

* **Learn how to get a Monitoring Dashboard**

If you want to track the results of different checks over time, you get an ML monitoring dashboard. Go through the [ML monitoring quickstart (Evidently Cloud) - Recommended](/tutorials-and-examples/tutorial-cloud) or [ML monitoring quickstart (Self-hosting)](/tutorials-and-examples/tutorial-monitoring) to see how to monitor metrics over time.

* **Explore available integrations**

To explore how to integrate Evidently with other tools, refer to the [Integrations](/integrations/integrations). For example, if you run predictions in batches, you can use a tool like [Airflow](/integrations/integrations/evidently-and-airflow) to orchestrate the process.

* **Go through the steps in more detail**

To better understand working with Reports and Test Suites, refer to the **User Guide** section of the docs. A good next step is to explore how to pass custom test parameters to define your own [test conditions](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/custom-test-suite.md).

### Join our Community!

Evidently is in active development, so expect things to change and evolve. You can subscribe to the [user newsletter](https://www.evidentlyai.com/user-newsletter) or follow our [releases on GitHub](https://github.com/evidentlyai/evidently/releases) to stay updated about the latest functionality.

We run a [Discord community](https://discord.gg/xZjKRaNp8b) to connect with our users and chat about ML in production topics.

In case you have feedback or need help, just ask in Discord or open a GitHub issue.

And if you want to support a project, give us a star on [GitHub](https://github.com/evidentlyai/evidently)!


# Tutorial - Data & ML Monitoring

Get started with Evidently Cloud. Run checks and customize a Dashboard in 15 minutes.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In this tutorial, you'll set up production data and ML monitoring for a toy ML model. You'll run evaluations in Python and access a web dashboard in Evidently Cloud.

The tutorial consists of three parts:

* Overview of the architecture (2 min).
* Launching a pre-built demo Dashboard (2-3 min).
* Setting up monitoring for a new toy dataset (10 min).

You'll need basic knowledge of Python. Once you connect the data, you can continue in the web interface.

{% hint style="success" %}
**Want a very simple example first?** Check this [Evidently Cloud "Hello World"](https://github.com/evidentlyai/docs-old/blob/main/examples/quickstart-cloud.md) instead.
{% endhint %}

Video version:

{% embed url="<https://youtu.be/ivhhS7extgE>" %}

If you're having problems or getting stuck, reach out on [Discord](https://discord.com/invite/xZjKRaNp8b).

## How it all works

Evidently Cloud helps you monitor the performance of ML-powered systems in production: from tracking the quality of incoming data to the accuracy of model predictions.

<details>

<summary>What can you evaluate with Evidently Cloud?</summary>

In Evidently Cloud, you can:

* Monitor tabular and text **data quality** in your production pipelines and services.
* Track **data and prediction drift** over time for text, tabular data and **embeddings**.
* Monitor **text data** (e.g., sentiment, drift, trigger words) for NLP and **LLM models**.
* Monitor **ML model quality** for classification, regression, ranking, and recommendations, including champion/challenger and shadow models and delayed ground truth.
* Track the results of **Test Suites** that bundle multiple evaluations.

Evidently supports over 100 pre-built Metrics and Tests. You can also add custom metrics.

</details>

The monitoring setup consists of two components:

* **Open-source Evidently Python library**. You perform evaluations in your environment. Each run produces a JSON `snapshot` with statistics, metrics, or test results for a specific period. You then send these `snapshots` to Evidently Cloud using an API key.
* **Evidently Cloud web app**. After sending the data, you can access it in the Evidently Cloud UI. You can view individual evaluation results, build a Dashboard with trends over time, and set up alerts to notify on issues.

You can run batch monitoring jobs (e.g., hourly, daily, weekly) or use Evidently Collector for near real-time checks. This tutorial shows a batch workflow.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c73bd199e07b1f3be3ce451ed9227455f7d8342e%2Fcloud_service_overview-min.png?alt=media)

{% hint style="success" %}
**Data security by design**. By default, Evidently Cloud does not store raw data or model inferences. Snapshots contain only data aggregates (e.g., histograms of data distributions, descriptive stats, etc.) and metadata with test results. This hybrid architecture helps avoid data duplication and preserves its privacy.
{% endhint %}

## Demo Dashboard

Let's quickly look at an example monitoring Dashboard.

### 1. Create an account

If you do not have one yet, [create an Evidently Cloud account](https://app.evidently.cloud/signup).

### 2. View a demo project

View an example Demo Project for a Regression Model for bike demand forecasting. Dashboards Tabs will show data quality, data drift, and model quality over time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-ad05e3558672c8a9024d02e2bab714a0721188b4%2Fdemo_dashboard.gif?alt=media\&token=7d544547-c1ad-4cdd-a7d9-06d60640305f)

You can customize the choice of Panels and Tabs for your Project – this is just an example.

You can also see individual snapshots if you navigate to the "Reports" or "Test Suites" section using the left menu. They display the performance on a given day and act as a data source for the monitoring Panels.

Now, let's see how you can create something similar for your dataset!

## Add a new Project

You'll use a toy dataset to mimic a production ML model. You'll follow these steps:

* Prepare a tabular dataset.
* Run data quality and data drift Reports in daily batches.
* Send them to Evidently Cloud.
* Get a Dashboard to track metrics over time.
* (Optional) Add custom monitoring panels.
* (Optional) Run Test Suites for continuous testing.

In the example, you'll track data quality and drift. ML monitoring often starts here because true labels for assessing model quality come with a delay. Until then, you monitor the incoming data and predictions.

However, the core workflow tutorial covers will work for any evaluation. You can later expand it to monitor ML model quality and text-based LLM models.

To complete the tutorial, use the provided code snippets or run a sample notebook.

Jupyter notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/data_and_ml_monitoring_tutorial.ipynb>" %}

### 1. Installation and imports

Evidently is available as a PyPi package. Run the command to install it:

```python
pip install evidently
```

You can also install Evidently from Conda:

```python
conda install -c conda-forge evidently
```

You'll need several components to complete the tutorial. Import the components to prepare the toy data:

```python
import pandas as pd
import datetime
from sklearn import datasets
```

Import the components to compute and send the snapshots:

```python
from evidently.ui.workspace.cloud import CloudWorkspace

from evidently.report import Report
from evidently.metric_preset import DataQualityPreset
from evidently.metric_preset import DataDriftPreset
from evidently.metrics import *
from evidently.test_suite import TestSuite
from evidently.tests import *
from evidently.test_preset import DataDriftTestPreset
from evidently.tests.base_test import TestResult, TestStatus
```

**Optional**. Import the components to design monitoring Panels via API. This is entirely optional: you can also add the Panels using the UI.

```python
from evidently import metrics
from evidently.ui.dashboards import DashboardPanelPlot
from evidently.ui.dashboards import DashboardPanelTestSuite
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import PlotType
from evidently.ui.dashboards import ReportFilter
from evidently.ui.dashboards import TestFilter
from evidently.ui.dashboards import TestSuitePanelType
from evidently.renderers.html_widgets import WidgetSize
```

### 2. Prepare toy data

You'll use the `adult` dataset from OpenML.

* Import it as a pandas `DataFrame`.
* Split it into two datasets: `adult_ref` (reference dataset) and `adult_prod` (current production data).

We'll base the split on the "education" feature to introduce some artificial drift for demo purposes. Current data will include people with education levels unseen in the reference dataset. Here's how you can do it:

```python
adult_data = datasets.fetch_openml(name="adult", version=2, as_frame="auto")
adult = adult_data.frame
adult_ref = adult[~adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
adult_prod = adult[adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
```

{% hint style="info" %}
**What is a reference dataset?** You need one to evaluate distribution drift. Here, you compare the current data against a past period, like an earlier data batch. You must provide this reference to compute the distance between two datasets. A reference dataset is optional when you compute descriptive stats or model quality metrics.
{% endhint %}

Preview the dataset. It resembles a binary classification use case with "class" as the prediction column.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-97c2a71f3a27be3ddd5f054c075e9f0a30c653e2%2Fdata_preview-min.png?alt=media)

### 3. Create a Project

Now, let's start monitoring!

**Get the API token**. To connect to Evidently Cloud, you need an access token. Use the "key" sign in the left menu to get to the token page, and click "generate token."

To connect to the Evidently Cloud workspace, run:

```python
ws = CloudWorkspace(
token="YOUR_TOKEN_HERE",
url="https://app.evidently.cloud")
```

Now, you need to create a new Project. You can do this programmatically or in the UI.

{% tabs %}
{% tab title="UI" %}
Click on the “plus” sign on the home page. Type your Project name and description.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-62015ecba8c22cd19bba1b4f8c6ab2a846450bc2%2Fadd_project_wide-min.png?alt=media)

After creating a Project, click its name to open the Dashboard. Since there's no data yet, it will be empty.

To send data to this Project, you'll need to connect to it from your Python environment using `get_project` method. You can find your Project ID above the monitoring Dashboard.

```python
project = ws.get_project("PROJECT_ID")
```

{% endtab %}

{% tab title="API" %}
Use the `create_project` command to create a new Project. Add a name and description. Copy the ID of your organization from the [organizations page](https://app.evidently.cloud/organizations).

```python
project = ws.create_project("My test project", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
**What is a Project?** Projects help organize monitoring for different use cases. Each Project has a shared Dashboard and alerting. You can create a Project for a single ML model or dataset or put related models together and use Tags to distinguish them.
{% endhint %}

### 4. Run first evaluation

To send snapshots, first compute them using the Evidently Python library. Here's the process:

* Prepare the data batch to evaluate.
* Create a `Report` or `TestSuite` object.
* Define `Metrics` or `Tests` to include.
* Pass optional parameters, like data drift detection method or test conditions.
* Compute and send the snapshot to the Project.

{% hint style="info" %}
**What are Reports and Test Suites?** These are pre-built evaluations available in the open-source Evidently Python library. They cover 100+ checks for data quality, data drift, and model quality. You can check out the [open-source Evidently Tutorial](https://docs.evidentlyai.com/get-started/tutorial) for an introduction. A `snapshot` is a "JSON version" of a Report or Test Suite.
{% endhint %}

Let’s start with data quality and drift checks using `Presets`. This will help observe how model inputs and outputs are changing. For each batch of data, you'll generate:

* **Data Quality Preset**. It captures stats like feature ranges and missing values.
* **Data Drift Preset**. This compares current and reference data distributions. You will use PSI (Population Stability Index) method, with a 0.3 threshold for significant drift.

To create a single Report using the first 100 rows of our "production" data:

```python
data_report = Report(
        metrics=[
            DataDriftPreset(stattest='psi', stattest_threshold='0.3'),
            DataQualityPreset(),
        ],
    )

data_report.run(reference_data=adult_ref, current_data=adult_prod.iloc[0 : 100, :])
```

{% hint style="info" %}
**Defining the dataset.** To specify the dataset to evaluate, you pass it as the `current_dataset` inside the `run` method. Our example uses a slice function `adult_prod.iloc[0 : 100, :]` to select 100 rows from the `adult_prod` dataset. In practice, simply pass your data: `current_data=your_batch_name`.
{% endhint %}

To send this Report to the Evidently Cloud, use the `add_report` method.

```python
ws.add_report(project.id, data_report)
```

You can now view the Report in the Evidently Cloud web app. Go to the "Reports" section via the left menu and click to open the first Report. You can also download it as an HTML or JSON.

### 5. Send multiple snapshots

In production, you can run evaluations on a schedule (e.g., daily or hourly) each time passing a new batch of data. Once you have multiple snapshots in the Project, you can plot trends on a monitoring Dashboard.

To simulate production use, let’s create a script to compute multiple Reports, taking 100 rows per "day":

```python
def create_report(i: int):
    data_report = Report(
        metrics=[
            DataDriftPreset(stattest='psi', stattest_threshold='0.3'),
            DataQualityPreset(),
        ],
        timestamp=datetime.datetime.now() + datetime.timedelta(days=i),
    )

    data_report.run(reference_data=adult_ref, current_data=adult_prod.iloc[100 * i : 100 * (i + 1), :])
    return data_report
```

You can set the loop variable i to 10 to generate and send Reports for 10 days.

```python
for i in range(0, 10):
        report = create_report(i=i)
        ws.add_report(project.id, report)
```

<details>

<summary>What happens in this code?</summary>

* You create a `Report` with two preset metric combinations. The [**Data Quality**](https://docs.evidentlyai.com/presets/data-quality) preset includes summary statistics like nulls, min-max, etc. The [**Data Drift**](https://docs.evidentlyai.com/presets/data-drift) preset compares new data to reference, using the Population Stability Index (PSI) test as an [optional parameter](https://docs.evidentlyai.com/user-guide/customization/options-for-statistical-tests).
* You add a **timestamp** to each `Report`: `datetime.now` for the first data batch, and the next day's date for subsequent batches. In real use, you can skip this step – the current timestamp will be automatically assigned. We do it for demo purposes.
* You compute each `Report` changing the `current_data` and keeping the `reference_data` static. For current data, you pass 100 new rows for `i` days to simulate batch inference.
* The `add_report` method computes the Report in the JSON snapshot format and sends it to the workspace `ws` (which is Evidently Cloud).

We use the script only to imitate multiple batch checks. In real use, you should create Reports sequentially, passing the true `current_data` for each run.

</details>

Run the script to compute and send 10 daily snapshots. Go to the "Reports" section to view them.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-786c6198607e9c9dbf6002b1e61d31880bf6d4bd%2Fview-reports-min.gif?alt=media\&token=4fa9401c-465d-43e7-8c36-acfc01bb088a)

However, each such Report is static. To see trends, you need a monitoring Dashboard!

{% hint style="info" %}
**Want to reuse this script for your data?** If you try replacing the toy dataset for your data, increasing the `i`, or adding more metrics, it's best to send Reports one by one instead of running a script. Otherwise, you might hit Rate Limits when sending many Reports together. For free trial users, the limit on the single data upload is 50MB; for paying users, it is 500MB. Snapshot size varies based on metrics and tests included.
{% endhint %}

### 6. Add monitoring Tabs

Monitoring Dashboard helps observe trends. It pulls selected values from individual Reports to show them over time. You can add multiple monitoring Panels and organize them by Tabs.

For a simple start, you can use Tab templates, which are pre-built combinations of monitoring Panels:

* **Data Quality Tab**: displays data quality metrics (nulls, duplicates, etc.).
* **Columns Tab**: shows descriptive statistics for each column over time.
* **Data Drift Tab**: shows the share of drifting features over time.

To add pre-built Tabs, enter "Edit" mode in the top right corner of the Dashboard. Click the plus sign to add a new Tab and choose the template.

### 7. Add custom Panels \[OPTIONAL]

You can also add individual monitoring Panels one by one. You can:

* Add them to an existing or a new Tab.
* Choose the Panel type, including Line Plot, Bar Plot, Histogram, Counter, etc.
* Customize Panel name, legend, etc.

{% hint style="info" %}
**You can only view values stored inside snapshots.** In our example, they relate to data drift and quality. You can't see model quality metrics yet, since there is no data on it. If you add a model quality Panel, it will be empty. To populate it, add more snapshots, for example, with `ClassificationPreset()`.
{% endhint %}

Say, you want to add a new “Summary” Tab and add a couple of Panels to show:

* Inferences over time.
* The share of drifting features over time.

You can add panels both in the UI or using the Python API.

{% tabs %}
{% tab title="UI" %}
Enter the “edit” mode on the Dashboard, and use the “add Tab” and “add Panel” buttons to add a new Panel. Follow the prompts to point to a specific measurement.

* To view inferences over time, plot the value `current.number_of_rows` inside the `DatasetSummaryMetric`.
* To view the share of drifting columns, plot the value `share_of_drifted_columns` inside the `DatasetDriftMetric`.\
  Choose a Panel type - for example, LINE or BAR plot, and add your legend.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8edbc7c4923fa22c95b0de379acffc4e30f1388a%2Fadd_new_panel_2.gif?alt=media\&token=f1808273-b2c7-4b7d-9cde-1b7cf9e89b36)
{% endtab %}

{% tab title="API" %}
**Connect to a Project**. If you've edited the Dashboard in the UI since creating the Project (like adding Tabs), use \`get\_project to load the latest configuration. This ensures you won't overwrite existing Panels.

```python
project = ws.get_project("YOUR PROJECT ID HERE")
```

**Add new Panels**. Use the `add_panel` method. You can specify the Panel name, legend, plot type, destination Tab ("Summary"), etc. After implementing the changes, save the configuration with `project.save()`.

```python
project.dashboard.add_panel(
        DashboardPanelPlot(
            title="Daily inference Count",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            values=[
            	PanelValue(
                	metric_id="DatasetSummaryMetric",
                	field_path=metrics.DatasetSummaryMetric.fields.current.number_of_rows,
                	legend="count",
            	),
            ],
            plot_type=PlotType.LINE,
            size=WidgetSize.FULL,
        ),
        tab="Summary"
    )
project.dashboard.add_panel(
        DashboardPanelPlot(
            title="Share of drifting features (PSI > 0.3)",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            values=[
                PanelValue(
                	metric_id="DatasetDriftMetric",
                	field_path="share_of_drifted_columns",
                	legend="share",
                ),
            ],
            plot_type=PlotType.LINE,
            size=WidgetSize.FULL,
        ),
        tab="Summary"
)
project.save()
```

Return to the Evidently Cloud web app to view the Dashboards. Refresh the page if necessary.
{% endtab %}
{% endtabs %}

{% hint style="info" %}
**How to add and modify Panels?** Check the detailed instructions on how to [design monitoring panels](https://docs.evidentlyai.com/user-guide/monitoring/design_dashboard). You can also add text-only Panels and counters.
{% endhint %}

### 8. Monitor Test runs \[OPTIONAL]

You just created a Dashboard to track individual metric values. Another option is to run your evaluations as Tests and track their outcomes.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-113b0bd4320f52540468c761ce89a4043fe8f0a7%2Ftoy_test_dashboard-min.png?alt=media\&token=54c95fee-8f7c-4cda-8b3c-b48d92ff05bb)

To do this, use Test Suites instead of Reports. Each Test in a Test Suite checks a specific condition, e.g., “the share of missing values in this column should be less than 10%”. You can bundle many Tests together and track which passed or failed. You can combine dataset- and column-level tests.

{% hint style="info" %}
**What Tests are available?** Choose from [50+ Tests](https://docs.evidentlyai.com/reference/all-tests), use Presets and create [custom Test Suites](https://docs.evidentlyai.com/user-guide/tests-and-reports/custom-test-suite).
{% endhint %}

Let’s create a Test Suite that includes:

* **Data Drift Test Preset**. It will generate a data drift check for all columns in the dataset using the same PSI method with a 0.3 threshold.
* **Individual data quality tests**. They will check for missing values, empty rows, columns, duplicates, and constant columns. You can set test conditions using parameters like `eq` (equal) or `lte` (less than or equal). If you don't specify the conditions, Evidently will auto-generate them based on the reference data.

Here's a script that again simulates generating Test Suites for 10 days in a row:

```python
def create_tests(i: int):
    drift_tests = TestSuite(
        tests=[
            DataDriftTestPreset(stattest_threshold=0.3),
            TestShareOfMissingValues(lte=0.05),
            TestNumberOfConstantColumns(eq=0),
            TestNumberOfEmptyRows(eq=0),
            TestNumberOfEmptyColumns(eq=0),
            TestNumberOfDuplicatedColumns(eq=0)
        ],
        timestamp=datetime.datetime.now() + datetime.timedelta(days=i),
    )

    drift_tests.run(reference_data=adult_ref, current_data=adult_prod.iloc[100 * i : 100 * (i + 1), :])
    return drift_tests
```

To send Test Suites to Evidently cloud, use the `add_test_suite` method.

```python
for i in range(0, 10):
        test_suite = create_tests(i=i)
        ws.add_test_suite(project.id, test_suite)
```

To visualize the results, add a new Dashboard Tab ("Data tests") and test-specific monitoring Panels.

{% tabs %}
{% tab title="UI" %}
Enter the “edit” Dashboard mode, click the “add Tab” and “add Panel” buttons. Choose the “Test Plot” panel type, with a "detailed" option and 1D (daily) aggregation level.\
You can add:

* One Panel with all column drift checks. Choose the `TestColumnDrift` test for \`all' columns.
* One Panel with dataset-level data quality checks. Choose the `TestNumberOfConstantColumns`, `TestShareOfMissingValues`, `TestNumberOfEmptyRows`, `TestNumberOfEmptyColumns`, `TestNumberOfDuplicatedColumns` from the dropdown.
  {% endtab %}

{% tab title="API" %}
Run the following code to add two test panels.

```python
project.dashboard.add_panel(
        DashboardPanelTestSuite(
            title="Data quality tests",
            test_filters=[
                TestFilter(test_id="TestNumberOfConstantColumns", test_args={}),
                TestFilter(test_id="TestShareOfMissingValues", test_args={}),
                TestFilter(test_id="TestNumberOfEmptyRows", test_args={}),
                TestFilter(test_id="TestNumberOfEmptyColumns", test_args={}),
                TestFilter(test_id="TestNumberOfDuplicatedColumns", test_args={}),
            ],
            filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
            size=WidgetSize.FULL,
            panel_type=TestSuitePanelType.DETAILED,
            time_agg="1D",
        ),
        tab="Data Tests"
)
project.dashboard.add_panel(
        DashboardPanelTestSuite(
            title="Data drift per column in time",
            test_filters=[
                TestFilter(test_id="TestColumnDrift", test_args={}),
            ],
            filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
            size=WidgetSize.FULL,
            panel_type=TestSuitePanelType.DETAILED,
            time_agg="1D",
        ),
        tab="Data Tests"
)
project.save()
```

{% endtab %}
{% endtabs %}

You'll see Dashboards with Test results over time in the new Tab. Head to the "Test Suites" section in the left menu for individual Test Suites. This helps debug Test outcomes.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-56d3902394ec62a6972dd387a3acf8b9ff1c21ac%2Fview-tests-min.gif?alt=media)

<details>

<summary>When to use Test Suites?</summary>

You can choose between Reports and Test Suites or use both. Test Suites are useful for:

* **Monitoring multiple conditions at once**. Bundling checks in a Test Suite helps reduce alert fatigue and simplify configuration. For example, you can quickly check if all columns in a dataset are within a defined min-max range.
* **Batch testing scenarios** like comparing new vs. old models in CI/CD or validating the quality of input data batch.
* **Using Test results outside Evidently Cloud**. For instance, you can stop your pipeline if data quality tests fail.

However, Test Suites require you to define pass or fail conditions upfront. If you only want to plot metrics, you can start with Reports instead.&#x20;

Note that if you use Test Suites, you can still plot the individual values (e.g., nulls over time) in addition to the Test-specific panels.

</details>

### What's next?

To go through all the steps in more detail, refer to the complete [Monitoring User Guide](https://docs.evidentlyai.com/user-guide/monitoring/monitoring_overview). Here are some of the things you might want to explore next:

* **Customize your evaluations**. See available [Presets](https://docs.evidentlyai.com/presets), [Metrics](https://docs.evidentlyai.com/reference/all-metrics), and [Tests](https://docs.evidentlyai.com/reference/all-tests) to see other checks you can run.
* **Build your batch or real-time workflow**. For batch evaluations, you can run regular monitoring jobs - for example, using a tool like Airflow or a script to orchestrate them. If you have a live ML service, you use [Evidently collector service](https://docs.evidentlyai.com/user-guide/monitoring/collector_service) to collect incoming production data and manage the computations.
* **Add alerts**. You can enable email, Slack, or Discord [alerts](https://docs.evidentlyai.com/user-guide/monitoring/alerting) when Tests fail or specific values are out of bounds.
* **Use Tags**. You can add Metadata or Tags to your snapshots and filter monitoring Panels. For instance, build individual monitoring Panels for two model versions.

Need help? Ask in our [Discord community](https://discord.com/invite/xZjKRaNp8b).


# Tutorial - LLM Evaluation

Evaluate and test your LLM use case in 15 minutes.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Evaluating the quality of LLM outputs is essential for building a production-grade LLM application. During development, you need to compare quality with different prompts and detect regressions. Once your app is live, you need to ensure outputs are safe and accurate and understand user behavior.

Manually reviewing individual outputs doesn't scale. This tutorial shows you how to automate LLM evaluations from experiments to production.

You will learn both about the evaluation methods and the workflow to run and track them.

{% hint style="success" %}
**Want a very simple example first?** This ["Hello World"](/get-started/hello-world/oss_quickstart_llm) will take a couple minutes.
{% endhint %}

In this tutorial, you will:

* Prepare a toy chatbot dataset
* Evaluate responses using different methods:
  * Text statistics
  * Text patterns
  * Model-based evaluations
  * LLM-as-a-judge
  * Metadata analysis
* Generate visual Reports to explore evaluation results
* Get a monitoring Dashboard to track metrics over time
* Build a custom Test Suite to run conditional checks

You can run this tutorial locally, with the option to use Evidently Cloud for monitoring. You will work with a Q\&A chatbot example, but the methods will apply to other use cases, such as RAGs and agents.

**Requirements:**

* Basic Python knowledge.
* The open-source Evidently Python library.

**Optional**:

* An OpenAI API key (to use LLM-as-a-judge).
* An Evidently Cloud account (for live monitoring).

Let's get started!

To complete the tutorial, use the provided code snippets or run a sample notebook.

Jupyter notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/llm_evaluation_tutorial.ipynb>" %}

You can also follow the video version:

{% embed url="<https://youtu.be/qwn0UqXJptY>" %}

If you're having problems or getting stuck, reach out on [Discord](https://discord.com/invite/xZjKRaNp8b).

## 1. Installation and imports

Install Evidently in your Python environment:

```python
!pip install evidently[llm]
```

Run the imports. To work with toy data:

```python
import pandas as pd
import numpy as np
import requests
from datetime import datetime, timedelta
from io import BytesIO
```

To run the evals:

```python
from evidently import ColumnMapping
from evidently.report import Report
from evidently.test_suite import TestSuite
from evidently.metric_preset import TextEvals
from evidently.descriptors import *
from evidently.metrics import *
from evidently.tests import *
from evidently.features.llm_judge import BinaryClassificationPromptTemplate
```

To send results to Evidently Cloud:

```python
from evidently.ui.workspace.cloud import CloudWorkspace
```

**Optional**. To remotely manage the dashboard design in Evidently Cloud:

```python
from evidently.ui.dashboards import DashboardPanelTestSuite
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import ReportFilter
from evidently.ui.dashboards import TestFilter
from evidently.ui.dashboards import TestSuitePanelType
from evidently.renderers.html_widgets import WidgetSize
```

## 2. Prepare a dataset

We'll use a dialogue dataset that imitates a company Q\&A system where employees ask questions about HR, finance, etc. You can download the [example CSV file](https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/chat_df.csv) from source or import it using `requests`:

```python
response = requests.get("https://raw.githubusercontent.com/evidentlyai/evidently/main/examples/how_to_questions/chat_df.csv")
csv_content = BytesIO(response.content)
```

Convert it into the pandas DataFrame. Parse dates and set conversation "start\_time" as index:

```python
assistant_logs = pd.read_csv(csv_content, index_col=0, parse_dates=['start_time', 'end_time'])
assistant_logs.index = assistant_logs.start_time
assistant_logs.index.rename('index', inplace=True)
```

To get a preview:

```python
pd.set_option('display.max_colwidth', None)
assistant_logs.head(3)
```

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3ad63731e6a1736cf290af0c45f400d18017d23d%2Fllm_data_preview-min.png?alt=media)

{% hint style="info" %}
**How to collect data?**: you can use the open-source `tracely` library to collect the inputs and outputs from your LLM app. Check the [Tracing Quickstart](https://github.com/evidentlyai/docs-old/blob/main/examples/cloud_quickstart_tracing.md). You can then download the traced Dataset for evaluation.
{% endhint %}

{% hint style="success" %}
**How to pass an existing dataset?** You can import a pandas DataFrame with flexible structure. Include any text columns (e.g., inputs and responses), DateTime, and optional metadata like ID, feedback, model type, etc. If you have multi-turn conversations, parse them into a table by session or input-output pairs.
{% endhint %}

## 3. Create a Project

{% hint style="info" %}
**This step is optional**. You can also run the evaluations locally without sending results to the Cloud.
{% endhint %}

To be able to save and share results and get a live monitoring dashboard, create a Project in Evidently Cloud. Here's how to set it up:

* **Sign up**. If you do not have one yet, create a free [Evidently Cloud account](https://app.evidently.cloud/signup) and name your Organization.
* **Get an Organization ID**. Get an ID of your organization on the [organizations page](https://app.evidently.cloud/organizations).
* **Get your API token**. Click the **Key** icon in the left menu to go. Generate and save the token. ([Token page](https://app.evidently.cloud/token)).
* **Connect to Evidently Cloud**. Pass your API key to connect.

```python
ws = CloudWorkspace(token="YOUR_TOKEN", 
                    url="https://app.evidently.cloud")
```

* **Create a Project**. Create a new Project inside your Organization, adding your title and description:

```python
project = ws.create_project("My project title", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

## 4. Run evaluations

You will now learn how to apply different methods to evaluate your text data.

* **Text statistics**. Evaluate simple properties like text length.
* **Text patterns**. Detect specific words or regular patterns.
* **Model-based evals**. Use ready-made ML models to score data (e.g., by sentiment).
* **LLM-as-a-judge**. Prompt LLMs to categorize or score texts by custom criteria.
* **Similarity metrics**. Measure semantic similarity between pairs of text.

To view the evaluation results, you will generate visual Reports in your Python environment. In the following sections of the tutorial, you'll also explore other formats like conditional Test Suites and live monitoring Dashboards.

It is recommended to map the data schema to make sure it is parsed correctly.

**Create column mapping**. Identify the type of columns in your data. Pointing to a "datetime" column will also add a time index to the plots.

```python
column_mapping = ColumnMapping(
    datetime='start_time',
    datetime_features=['end_time'],
    text_features=['question', 'response'],
    categorical_features=['organization', 'model_ID', 'region', 'environment', 'feedback'],
)
```

Now, let's run evaluations!

{% hint style="info" %}
**You can skip steps**. Each example below is self-contained, so you can skip any of them or head directly to Step 6 to see the monitoring flow.
{% endhint %}

### Text statistics

Let's run a simple evaluation to understand the basic flow.

**Evaluate text length**. Generate a Report to evaluate the length of texts in the "response" column. Run this check for the first 100 rows in the `assistant_logs` dataframe:

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response",
              descriptors=[
                  TextLength(),
                  ]
              )
])

text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:100],
                      column_mapping=column_mapping)
text_evals_report
```

This calculates the number of symbols in each text and shows a summary in your notebook cell. (You can also export it in other formats - see step 5).

You can see the distribution of text length across all responses and descriptive statistics like the mean or minimal text length.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1767d7dc10d038ff87077cb9dd19ede80e05c4aa%2Fllm_tutorial_text_length-min.png?alt=media)

Click on "details" to see how the mean text length changes over time. The index comes from the `datetime` column you mapped earlier. This helps you notice any temporal patterns, such as if texts are longer or shorter during specific periods.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-27144445205e07aed689207a3b15da064c9f5e39%2Fllm_tutorial_text_length_plot-min.png?alt=media)

**Get a side-by-side comparison**. You can also generate statistics for two datasets at once. For example, compare the outputs of two different prompts or data from today against yesterday.

Pass one dataset as `reference` and another as `current`. For simplicity, let's compare the first and next 50 rows from the same dataframe:

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response",
              descriptors=[
                  TextLength(),
                  ]
              )
])

text_evals_report.run(reference_data=assistant_logs[:50],
                      current_data=assistant_logs[50:100],
                      column_mapping=column_mapping)
text_evals_report
```

You will now see the summary results for both datasets:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-f52f526b83bcd38b512a17ad50d493b00a41dbac%2Fllm_tutorial_side_by_side-min.png?alt=media)

Each evaluation that computes a score for every text in the dataset is called a `descriptor`. Descriptors can be numerical (like the `TextLength()` you just used) or categorical.

Evidently has many built-in descriptors. For example, try other simple statistics like `SentenceCount()` or `WordCount()`. We'll show more complex examples below.

{% hint style="success" %}
**List of all descriptors** See all available descriptors in the "Descriptors" section of [All Metrics](https://docs.evidentlyai.com/reference/all-metrics) table.
{% endhint %}

### Text patterns

You can use regular expressions to identify text patterns. For example, check if the responses mention competitors, named company products, include emails, or specific topical words. These descriptors return a binary score ("True" or "False") for pattern matches.

Let's check if responses contain words related to compensation (such as salary, benefits, or payroll). Pass this word list to the `IncludesWords` descriptor. This will also check for word variants.

Add an optional display name for this eval:

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response",
              descriptors=[
                  IncludesWords(
                      words_list=['salary', 'benefits', 'payroll'],
                      display_name="Mention Compensation")
            ]
        ),
        ]
)

text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:100],
                      column_mapping=column_mapping)
text_evals_report
```

Here is an example result. You can see that 10 responses out of 100 relate to the topic of compensation as defined by this word list. "Details" show occurrences in time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-33da8a022bfbee16d8df97d63a74235b1138ed19%2Fllm_tutorial_mentions_compensation-min.png?alt=media)

Such pattern evals are fast and cheap to compute at scale. You can try other descriptors like:

* `Contains(items=[])` for non-vocabulary words like competitor names or longer expressions,
* `BeginsWith(prefix="")` for specific starting sequence,
* Custom `RegEx(reg_exp=r"")`, etc.

### Model-based scoring

You can use pre-trained machine learning models to score your texts. Evidently has:

* Built-in model-based descriptors like `Sentiment`.
* Wrappers to call external models published on HuggingFace.

Let's start with a **Sentiment** check. This returns a sentiment score from -1 (very negative) to 1 (very positive).

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
            Sentiment(),
        ]
    ),
])

text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:100],
                      column_mapping=column_mapping)
text_evals_report
```

You will see the distribution of response sentiment. Most are positive or neutral, but there are a few chats with a negative sentiment.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-d0674b9ef83bc6654079a8842a59f33306a99232%2Fllm_tutorial_sentiment-min.png?alt=media)

In "details", you can look at specific times when the average sentiment of responses dipped:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c7970f2ccf3568fb6f0786c9b694d4770674a1d1%2Fllm_tutorial_sentiment_2-min.png?alt=media)

To review specific responses with sentiment below zero, you can also export the dataset with scores. We'll show this later on.

Let's first see how to use external models from HuggingFace. There are two options:

* **Pre-selected models**, like **Toxicity**. Pass the `HuggingFaceToxicityModel()` descriptor. This [model](https://huggingface.co/spaces/evaluate-measurement/toxicity) returns a predicted toxicity score between 0 to 1.
* **Custom models**, where you specify the model name and output to use. For example, let's call the `SamLowe/roberta-base-go_emotions` [model](https://huggingface.co/SamLowe/roberta-base-go_emotions) using the general `HuggingFaceModel` descriptor. This model classifies text into 28 emotions. If you pick the "neutral" label, the descriptor will return the predicted score from 0 to 1 on whether responses convey neutral emotion.

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
            HuggingFaceToxicityModel(),
            HuggingFaceModel(
                model="SamLowe/roberta-base-go_emotions",
                params={"label": "neutral"},
                display_name="Response Neutrality"),
        ]
    ),
])

text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:100],
                      column_mapping=column_mapping)
text_evals_report
```

In each case, the descriptor first downloads the model from HuggingFace to your environment and then uses it to score the data. It takes a few moments to load the model.

**How to interpret the results?** It's typical to use a predicted score above 0.5 as a "positive" label. The toxicity score is near 0 for all responses - nothing to worry about! For neutrality, most responses have predicted scores above the 0.5 threshold, but a few are below. You can review them individually.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-37a876d0d74ad86c1da6b078dbb8daa5d68c2c89%2Fllm_tutorial_neutrality-min.png?alt=media)

{% hint style="info" %}
**Choosing other models**. You can choose other models, e.g. to score texts by topic. See [docs](/user-guide/customization/huggingface_descriptor).
{% endhint %}

### LLM as a judge

For more complex or nuanced checks, you can use LLMs as a judge. This requires creating an evaluation prompt asking LLMs to assess the text by specific criteria, such as tone or conciseness.

{% hint style="info" %}
**This step is optional**. You'll need an OpenAI API key and will incur costs by running the evaluation. Skip if you don't want to use external LLMs.
{% endhint %}

**Pass the OpenAI key**. It is recommended to pass the key as an environment variable. [See Open AI docs](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety) for best practices.

```python
## import os
## os.environ["OPENAI_API_KEY"] = "YOUR KEY"
```

**Run template evals**. Let's start with built-in prompt templates.

* `DeclineLLMEval()` checks if the response contains a denial.
* `PIILLMEval()` checks if the response contains personally identifiable information. You can also ask to provide for a reasoning of the score.

To minimize API calls, we will pass only 10 data rows.

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        DeclineLLMEval(),
        PIILLMEval(include_reasoning=True), 
    ])
])

report.run(reference_data= None,
           current_data= assistant_logs[:10],
           column_mapping=column_mapping)
report 
```

**Create a custom judge**. You can also define your own LLM judge with a custom prompt. To illustrate, let's ask the LLM to judge whether the provided responses are concise and return a `Concise` or `Verbose` label with an explanation. (Or `Unknown` if not sure).

```python
custom_judge = LLMEval(
    subcolumn="category",
    template = BinaryClassificationPromptTemplate(      
        criteria = """Conciseness refers to the quality of being brief and to the point, while still providing all necessary information.
            A concise response should:
            - Provide the necessary information without unnecessary details or repetition.
            - Be brief yet comprehensive enough to address the query.
            - Use simple and direct language to convey the message effectively.
        """,
        target_category="concise",
        non_target_category="verbose",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are a judge which evaluates text.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name="Conciseness",
)
```

Include the `custom_judge` descriptor to the Report:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        custom_judge
    ])
])

report.run(reference_data= None,
           current_data= assistant_logs[:10],
           column_mapping=column_mapping)
report 
```

All our responses are concise - great! To see the individual scores, you can publish a dataframe (see Step 5), or send the results to Evidently Cloud.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-beaba1309ba323abd8c3bf1510e8b1c76d06edee%2Fllm_tutorial_conciseness.png?alt=media)

{% hint style="info" %}
**How to create your own judge**. You can create custom prompts, and optionally pass the context or reference answer alongside the response. See [docs](/user-guide/customization/llm_as_a_judge)
{% endhint %}

### Metadata summary

Our dataset also includes user upvotes and downvotes in a categorical `feedback` column. You can easily add summaries for any numerical or categorical column to the Report.

To add a summary on the “feedback” column, use `ColumnSummaryMetric()`:

```python
data_report = Report(metrics=[
   ColumnSummaryMetric(column_name="feedback"),
   ]
)

data_report.run(reference_data=None, current_data=assistant_logs[:100], column_mapping=column_mapping)
data_report
```

You will see a distribution of upvotes and downvotes.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a062ce72a292bb7644c810ec07ea294ac20862ac%2Fllm_feedback_one-min.png?alt=media)

### Semantic Similarity

You can evaluate how closely two texts are in meaning using an embedding model. This descriptor requires you to define two columns. In our example, we can compare Responses and Questions to see if the chatbot answers are semantically relevant to the question.

This descriptor converts all texts into embeddings, measures Cosine Similarity between them, and returns a score from 0 to 1:

* 0 means that texts are opposite in meaning;
* 0.5 means that texts are unrelated;
* 1 means that texts are semantically close.

To compute the Semantic Similarity:

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        SemanticSimilarity(with_column="question", 
                           display_name="Response-Question Similarity"),
    ])
])

text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:100],
                      column_mapping=column_mapping)
text_evals_report
```

In our examples, the semantic similarity always stays above 0.81, which means that answers generally relate to the question.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-981ef19fa4098f763db95cfd474ba537f7622029%2Fllm_tutorial_semantic_similarity-min.png?alt=media)

## 5. Export results

{% hint style="info" %}
**This is optional**. You can proceed without exporting the results.
{% endhint %}

You can export the evaluation results beyond viewing the visual Reports in Python. Here are some options.

**Publish a DataFrame**. Add computed scores (like semantic similarity, or LLM-based scores with an explanation) directly to your original dataset. This will let you further analyze the data, like identifying examples with the lowest scores.

```python
text_evals_report.datasets().current
```

**Python dictionary**. Get summary scores as a dictionary. Use it to export specific values for further pipeline actions:

```python
text_evals_report.as_dict()
```

**JSON**. Export summary scores as JSON:

```python
text_evals_report.json()
```

**HTML**. Save a visual HTML report as a file:

```python
text_evals_report.save_html("report.html")
```

You can also send the results to Evidently Cloud for monitoring!

## 6. Monitor results over time

In this section, you will learn how to monitor evaluations using Evidently Cloud. This allows you to:

* **Track offline experiment results**. Keep records of evaluation scores from different experiments, like comparing output quality using different prompts.
* **Run evaluations in production**. Periodically evaluate batches or samples of production data, such as hourly or daily.

Here's how you can set this up.

**Define the evaluations**. First, let's design a Report. This will specify what you want to evaluate.

Say, you want to compute summaries for metadata columns and evaluate text length, sentiment, and mentions of compensation in chatbot responses.

```python
text_evals_report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
            Sentiment(),
            TextLength(),
            IncludesWords(words_list=['salary', 'benefits', 'payroll'],
                          display_name="Mention Compensation")

        ],
    ),
    ColumnSummaryMetric(column_name="feedback"),
    ColumnSummaryMetric(column_name="region"),
    ColumnSummaryMetric(column_name="organization"),
    ColumnSummaryMetric(column_name="model_ID"),
    ColumnSummaryMetric(column_name="environment"),
])
```

You can include more complex checks like LLM-as-a-judge in the same way: just list the corresponding descriptor.

**Run the Report**. Compute the Report for the first 50 rows:

```python
text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[:50],
                      column_mapping=column_mapping)
```

**Upload the results**. Send the Report to the Evidently Cloud Project you created earlier:

```python
ws.add_report(project.id, text_evals_report)
```

**View the Report**. Go to the Project and open the Reports section using the menu on the left.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-329a5b10a9ce7ff327659efeb9a8a34029a81c34%2Fview_report-min.gif?alt=media)

A single Report gives us all the information right there. But as you run more checks, you will want to see how values change over time. Let's imitate a few consecutive runs to evaluate more batches of data.

**Imitate ongoing evaluations**. Run and send several Reports, each time taking the next 50 rows of data. For illustration, we repeat the runs. In practice, you would compute each Report after new experiments or as you get a new batch of production data to evaluate.

Run the Report for the next 50 rows of data:

```python
text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[50:100],
                      column_mapping=column_mapping)
ws.add_report(project.id, text_evals_report)
```

<details>

<summary>And a few more times!</summary>

Run 3:

```python
text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[100:150],
                      column_mapping=column_mapping)
ws.add_report(project.id, text_evals_report)
```

Run 4:

```python
text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[150:200],
                      column_mapping=column_mapping)
ws.add_report(project.id, text_evals_report)
```

Run 5:

```python
text_evals_report.run(reference_data=None,
                      current_data=assistant_logs[200:250],
                      column_mapping=column_mapping)
ws.add_report(project.id, text_evals_report)
```

</details>

Now you will have 5 Reports in the Project. Let's get a dashboard!

**Get a Monitoring Dashboard**. You can start with pre-built templates.

* Go to Project Dashboard.
* Enter the edit mode by clicking on the "Edit" button in the top right corner.
* Choose "Add Tab",
* Add a "Descriptors" Tab and then a "Columns" Tab.
* Use the "Show in Order" toggle above the dashboard to ignore the time gaps.

You will instantly get a dashboard with evaluation results over time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-90ba6bd5855c98d142a2d0be94ed14627dc558ff%2Fcreate_tabs-min.gif?alt=media)

In the "Descriptors" tab, you will see how the distributions of the text evaluation results. For example, you can notice a dip in mean Sentiment in the fourth evaluation run.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b9ac021e5442c29c246f1024014d06fb80f0c020%2Fllm_tutorial_sentiment_over_time-min.png?alt=media)

In the "Columns" tab, you can see all the metadata summaries over time. For example, you can notice that all responses in the last run were generated with gpt-3.5.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3a2ab30a76109be50d85474f87d5d85542274e83%2Fllm_tutorial_modelID_distribution-min.png?alt=media)

You can also add alerting conditions for specific values.

{% hint style="success" %}
**Monitoring Panel types**. In addition to Tabs, you can choose monitoring panels one by one. You can choose panel title, type (bar, line chart), etc. Read more on [available Panels](https://docs.evidentlyai.com/user-guide/monitoring/design_dashboard).
{% endhint %}

## 7. Run conditional tests

So far, you've used Reports to summarize evaluation outcomes. However, you often want to set specific conditions for the metric values. For example, check if all texts fall within the expected length range and review results only if something goes wrong.

This is where you can use an alternative interface called `TestSuites`. It will look like this:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-23aadf8d56b7d752876cca4486c1cff708677f29%2Fllm_tutorial_test_results-min.png?alt=media)

Test Suites work similarly to `Reports`, but instead of listing `metrics`, you define `tests` and set conditions using parameters like `gt` (greater than), `lt` (less than), `eq` (equal), etc.

**Define a Test Suite**. Let’s create a simple example:

```python
test_suite = TestSuite(tests=[
    TestColumnValueMean(column_name = Sentiment().on("response"), gte=0),
    TestColumnValueMin(column_name = TextLength().on("response"), gt=0),
    TestColumnValueMax(column_name = TextLength().on("response"), lte=2000),
    TestColumnValueMean(column_name = TextLength().on("response"), gt=500),
])
```

This test checks the following conditions:

* Average response sentiment is positive.
* Response length is always non-zero.
* Maximum response length does not exceed 2000 symbols (e.g., due to chat window constraints).
* Mean response length is above 500 symbols (e.g., this is a known pattern).

{% hint style="success" %}
**How to test set test conditions**. [Read more about Tests](https://docs.evidentlyai.com/user-guide/tests-and-reports/custom-test-suite). You can use other descriptors and tests. For example, use `TestCategoryShare` to check if the share of responses labeled "Concise" by the LLM judge is above a certain threshold. You can also automatically generate conditions from a reference dataset (e.g. expect +/- 10% of the reference values).
{% endhint %}

**Compute multiple Test Suites**. Let's simulate running 5 Test Suites sequentially, each on 50 rows of data, with timestamps spaced hourly:

```python
for i in range(5):
    test_suite.run(
        reference_data=None,
        current_data=assistant_logs.iloc[50 * i : 50 * (i + 1), :],
        column_mapping=column_mapping,
        timestamp=datetime.now() + timedelta(hours=i)
    )
    ws.add_test_suite(project.id, test_suite)
```

We use a cycle for demonstration. In production, you would run these checks sequentially.

**Add a test monitoring Panel**. Now, let's add a simple panel to display Test results over time. You can manage dashboards in the UI (like you did before) or programmatically. Let's now explore how to do it from Python.

Load the latest dashboard configuration to Python. If you skip this step, the new Test panels will override the Tabs you added earlier.

Copy the Project ID from above the dashboard:

```python
project = ws.get_project("PROJECT_ID")
```

Next, create a Test panel within the "Tests" tab to display detailed test results:

```python
project.dashboard.add_panel(
    DashboardPanelTestSuite(
        title="Test results",
        filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
        size=WidgetSize.FULL,
        panel_type=TestSuitePanelType.DETAILED,
        time_agg="1D",
    ),
    tab="Tests"
)
project.save()
```

**View the test results in time**. Go to the Evidently Cloud dashboard to see the history of all tests. You can notice that a single test failed in the last run. If you hover on the specific test, you can see that we failed the mean text length condition.

**View the individual Test Suite**. To debug, open the latest Test Suite. In "Details," you will see the distribution of text length and the current mean value, which is just slightly below the set threshold.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-85b32d8d54c257e0248590907ef4df090c2ffe5e%2Fview_test_suites-min.gif?alt=media)

When can you use these Test Suites? Here are two ideas:

* **Regression testing**. Run Test Suites whenever you change prompt or app parameters to compare new responses with references or against set criteria.
* **Continuous testing**. Run Test Suites periodically over production logs to check that the output quality stays within expectations.

You can also set up alerts to get a notification if your Tests contain failures.

{% hint style="success" %}
**What is regression testing?**. Check a separate tutorial on the [regression testing workflow](https://www.evidentlyai.com/blog/llm-testing-tutorial).
{% endhint %}

## What's next?

Here are some of the things you might want to explore next:

* **Explore other Reports**. For example, if your LLM solves a classification or retrieval task, you can evaluate classification or ranking quality. See available [Presets](https://docs.evidentlyai.com/presets), [Metrics](https://docs.evidentlyai.com/reference/all-metrics), and [Tests](https://docs.evidentlyai.com/reference/all-tests) to see other checks you can run.
* **Design the monitoring**. Read more about how to add monitoring panels, configure alerts, or send data in near real-time in the [Monitoring User Guide](https://docs.evidentlyai.com/user-guide/monitoring/monitoring_overview).

Need help? Ask in our [Discord community](https://discord.com/invite/xZjKRaNp8b).


# Self-host ML Monitoring

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In this tutorial, you'll learn how to start with Evidently ML monitoring. You will launch a locally hosted dashboard to visualize the performance of a toy model.

**Note**: If you want to start with ad hoc Reports and Tests without hosting a Monitoring UI service, go here instead: [Quickstart for Evidently Tests and Reports](/get-started/hello-world/oss_quickstart_tabular).

The tutorial is split into two parts.

**Part 1**. Estimated time: 2 minutes. You will launch a demo example.

* Install Evidently
* Launch a pre-built demo project

**Part 2**. Estimated time: 15 minutes. You will add a new project to the monitoring UI.

* Create a new workspace and project
* Imitate batch model inference
* Design a few monitoring panels
* View the dashboards

**Note**: This guide assumes you run Evidently locally.

{% hint style="info" %}
**Don't want to self-host the ML monitoring dashboard?** Check out the [Evidently Cloud tutorial](/get-started/quickstart-cloud/cloud_quickstart_tabular).
{% endhint %}

If you're having problems or getting stuck, reach out on [Discord](https://discord.com/invite/xZjKRaNp8b).

## Part 1. Pre-built demo

### 1. Create virtual environment

Optional, but highly recommended. Create a virtual environment and activate it. Run the following command in the Terminal:

```
pip install virtualenv
virtualenv venv
source venv/bin/activate
```

### 2. Install Evidently

Evidently is available as a PyPi package. Run this command to install Evidently:

```
pip install evidently
```

You can also install Evidently from Conda:

```
conda install -c conda-forge evidently
```

**Note**: ML monitoring is available starting from Evidently version 0.4.0.

### 3. Run demo project

To launch the Evidently service with the demo projects, run:

```
evidently ui --demo-projects all
```

### 4. View the project

To view the Evidently interface, go to URL <http://localhost:8000> in your web browser.

![ML monitoring](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-74cbf354caa649bce03758b2cc5fd27345b1406e%2Fevidently_ml_monitoring_main.png?alt=media\&token=0154c054-2a32-4bde-8c12-b0ae075637aa)

You will see a pre-built project that shows the toy model quality over 20 days. You can switch between tabs and view individual Reports and Test Suites. Each Report or Test Suite contains the information logged for a daily period. The monitoring dashboard parses the data from these Reports and shows how metrics change over time.

{% hint style="info" %}
**What is a Test Suite and a Report?** If you are new to Evidently, we recommend that you go through the [Quickstart for Tests and Reports](https://github.com/evidentlyai/docs-old/blob/main/examples/tutorial.md) next. This functionality helps log data and model metrics for ML monitoring.
{% endhint %}

## Part 2. Add a new project

In this section, you will create a new project as if you add a new ML model to monitor. You will imitate batch model inference to collect logs and design panels to visualize them.\
All steps are available as a Python script.

You can access it here:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/sample_notebooks/get_started_monitoring.py>" %}

**Note**: we assume you already installed Evidently at the previous step.

### 1. Open the Python script

Open and explore the `get_started_monitoring.py`.

This script does the following:

* Imports the required components
* Imports a toy dataset
* Defines the metrics to log using Evidently Reports and Test Suites
* Computes the metrics iterating over toy data
* Creates a new Evidently workspace and project
* Creates several panels to visualize the metrics

You can simply run this script without changes. The sections below are optional. They describe each step in the script and what you can change – if you are up to it!

#### 1.1. Imports

Import the required components.

```python
import datetime

from sklearn import datasets

from evidently.metrics import ColumnDriftMetric
from evidently.metrics import ColumnSummaryMetric
from evidently.metrics import DatasetDriftMetric
from evidently.metrics import DatasetMissingValuesMetric
from evidently.report import Report
from evidently.test_preset import DataDriftTestPreset
from evidently.test_suite import TestSuite
from evidently.ui.dashboards import CounterAgg
from evidently.ui.dashboards import DashboardPanelCounter
from evidently.ui.dashboards import DashboardPanelPlot
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import PlotType
from evidently.ui.dashboards import ReportFilter
from evidently.ui.remote import RemoteWorkspace
from evidently.ui.workspace import Workspace
from evidently.ui.workspace import WorkspaceBase
```

Import the data and create a `pandas.DataFrame` using the OpenML `adult` dataset.

We single out the part of the dataset `adult_ref` that we will later use as a baseline for drift detection. We use the rest `adult_cur` to imitate batch inference.

```python
adult_data = datasets.fetch_openml(name="adult", version=2, as_frame="auto")
adult = adult_data.frame

adult_ref = adult[~adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
adult_cur = adult[adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
```

This step is added for demo purposes. In practice, you should work with production data.

#### 1.2. Name the workspace and project

Assign a name to the Evidently workspace and your project. A project corresponds to an ML model you monitor. You will see this name and description in the user interface.

```python
WORKSPACE = "workspace"

YOUR_PROJECT_NAME = "New Project"
YOUR_PROJECT_DESCRIPTION = "Test project using Adult dataset."
```

Workspace defines the folder where Evidently will log data to. It will be created in the directory where you launch the script from.

#### 1.3. Define metrics to log

To capture data and model metrics, we use Evidently Reports and Test Suites. You need to define a Report and/or a Test Suite object and list the metrics or tests to include. You can pass optional parameters – such as set a drift detection method instead of the [default](/reference/data-drift-algorithm) choice.

{% hint style="info" %}
**Test Suites and Reports.** We use the same syntax to log the data as when running Evidently Reports and Tests Suites ad hoc, for example, in a Jupyter notebook. Go through the [Quickstart](https://github.com/evidentlyai/docs-old/blob/main/examples/tutorial.md) if you need a refresher.
{% endhint %}

To imitate batch inference, we run computations as if we captured data for `i` days (we will set it to 5), each time taking 100 observations. In practice, you must work with actual prediction data.

We also pass `adult_ref` as the `reference_data` to the Evidently Report. It will be used as the basis of comparison for distribution drift detection.

```python
def create_report(i: int):
    data_drift_report = Report(
        metrics=[
            DatasetDriftMetric(),
            DatasetMissingValuesMetric(),
            ColumnDriftMetric(column_name="age", stattest="wasserstein"),
            ColumnSummaryMetric(column_name="age"),
            ColumnDriftMetric(column_name="education-num", stattest="wasserstein"),
            ColumnSummaryMetric(column_name="education-num"),
        ],
        timestamp=datetime.datetime.now() + datetime.timedelta(days=i),
    )

    data_drift_report.run(reference_data=adult_ref, current_data=adult_cur.iloc[100 * i : 100 * (i + 1), :])
    return data_drift_report
```

**What you can change:**

* The complete script uses both Test Suite and Report for logging. It is not required to use both – this depends on your preference.
* You can select other Metrics or Tests to define the contents of your monitoring. Consider using other checks for data quality, integrity, or data drift. You can select individual metrics or pre-built presets. See the full list of [All Metrics](/reference/all-metrics) and [All Tests](/reference/all-tests) and their parameters.
* You can pass additional parameters to the individual Tests and Metrics.

{% hint style="info" %}
**Evaluating model quality.** For simplicity, the example works with a raw dataset. There is no model! You cannot compute model quality metrics on this data – you need to add prediction and target columns. You might also need to use [column mapping](/user-guide/input-data/column-mapping) to map your inputs.
{% endhint %}

#### 1.4. Add panels to the project

Define how you create a new project in the workspace:

```python
def create_project(workspace: WorkspaceBase):
    project = workspace.create_project(YOUR_PROJECT_NAME)
    project.description = YOUR_PROJECT_DESCRIPTION
```

Each project can include multiple panels that appear on the monitoring dashboard. A panel visualizes a particular metric or metrics captured in Reports or Test Suites over time. You can visualize metrics as counters, time series plots, bar charts, and scatter plots.

Here is an example of adding a counter metric. The complete script includes several more.

```python
    project.dashboard.add_panel(
        DashboardPanelCounter(
            title="Share of Drifted Features",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            value=PanelValue(
                metric_id="DatasetDriftMetric",
                field_path="share_of_drifted_columns",
                legend="share",
            ),
            text="share",
            agg=CounterAgg.LAST,
            size=1,
        )
    )
```

As a `metric_id`, you pass the name of the Evidently Metric that was logged as part of the Report. As a `field_path`, you select the metric result computed as part of this Metric. Since Evidently Metrics contain multiple data points, you must specify which to show on a dashboard. You can display multiple metrics on a single panel.

**What you can change**:

* You can specify the panel title and the legend visible in the interface.
* You can set the size of each panel as full-width (`size=2`, default) or half-width (`size=1`).
* You can define the aggregation function or select to show the last metric value.
* You can add panels of different types following the examples in the script.

#### 1.5. Save panels

To save changes made to a project, you must use the method save().

```python
project.save()
```

#### 1.6. Create the workspace and project

Finally, create the project, workspace, and generate the JSON `snapshots`. `Snapshot` is a JSON "version" of the `Report` or `Test Suite` that was defined earlier. It contains summaries of the captured data and model metrics. You must store the snapshots in a directory under the corresponding workspace name. This way, Evidently UI will be able to parse the metrics and visualize them on the monitoring panels.

When you execute the script, Evidently will log the snapshots with the selected metrics to the defined workspace folder, as if you captured data for 5 days. It will also create the dashboard panels.

```python
def create_demo_project(workspace: str):
    ws = Workspace.create(workspace)
    project = create_project(ws)

    for i in range(0, 5):
        report = create_report(i=i)
        ws.add_report(project.id, report)

        test_suite = create_test_suite(i=i)
        ws.add_test_suite(project.id, test_suite)

if __name__ == "__main__":
    create_demo_project("workspace")
```

If you made any edits to this script, save them.

### 2. (Optional) Delete workspace

If this is not the first run of the script, and you reuse the same project – run the command to delete a previously generated workspace:

```
cd src/evidently/ui/
rm -r workspace
```

### 3. Run the script

Run the command to generate a new example project as defined in the script above.

```
python get_started_monitoring.py
```

### 4. Run the Evidently UI service

Finally, launch the user interface that will include the defined project.

**4.1**. If you only want to include your project, run:

```
evidently ui 
```

**4.2**. If you want to see both your new project and a standard demo projects, run:

```
evidently ui --demo-projects all
```

**Note**: If you already launched a demo project previously, it will remain in the workspace. There is no need to add it the second time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-73871d486469188c097074c051c5e2755701b086%2Fview_projects-min.png?alt=media\&token=52ebf465-8a22-4a53-b455-676164576df7)

**4.3**. If you want to have your project in the specified workspace and have the UI service running at the specific port:

```
evidently ui --workspace ./workspace --port 8080
```

**4.4**. If you want to see both your project and demo projects in the specified workspace and run the UI service at the specific port:

```
evidently ui --workspace ./workspace --port 8080 --demo-projects all
```

**All flags**

To see all possible flags you can run evidently ui with the `--help` flag:

```
evidently ui --help
```

**Note**: Evidently collects anonymous telemetry about service usage. You can opt-out as described [here](/support/telemetry). We’d greatly appreciate it if you keep the telemetry on since it allows us to understand usage patterns and continue building useful free tools for the community.

### 5. View the new project

Access Evidently UI service in your browser to see the dashboards for the new project. Go to the `localhost:8000`, or a specified port if you set a different one.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2fab8874ba691ad79deb033b58b38a5b1650b6af%2Fml_monitoring_adult_example.png?alt=media\&token=a1291e6b-37c0-4b00-9815-d087bc83c69d)

## What’s next?

* **Go through the steps in more detail**

If you want to go through the steps in more details, refer to the complete [Monitoring User Guide](/user-guide/monitoring/monitoring_overview) section in the docs. A good next step is to explore how to design [monitoring panels](https://github.com/evidentlyai/docs-old/blob/main/monitoring/design_dashboard.md).

* **Build the workflow**

If you want to enable monitoring for an existing ML project, you must collect the data from your production pipelines or services, or run monitoring jobs over production logs stored in a data warehouse. The exact integration scenario depends on the model deployment type and infrastructure.

Here is a possible approach for batch monitoring. You can implement it using a workflow manager like Airflow to compute Evidently snapshots on a regular cadence.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a8c62d2f586e49888a102c8adfcbfc9139620a41%2Fmonitoring_batch_workflow_min.png?alt=media\&token=5d291882-2e9f-41eb-accf-ebef59705b04)

Alternatively, if you have a live ML service, you can run an [Evidently collector service](/user-guide/monitoring/collector_service) and send the predictions for near real-time monitoring.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a9a56351bf42412868c87d7a41e6ec74da358ecc%2Fmonitoring_collector_min.png?alt=media\&token=dd79fb6e-88d0-47d9-97bb-162599f30d40)

You can browse sample [integrations](/integrations/integrations/evidently-integrations). We’ll add more example integrations in the future.

## Join our Community!

We run a [Discord community](https://discord.gg/xZjKRaNp8b) to connect with our users and chat about ML in production topics. In case you have feedback or need help, just ask in Discord or open a GitHub issue.

Evidently is in active development, so expect things to change and evolve. Subscribe to the [user newsletter](https://www.evidentlyai.com/user-newsletter) to stay updated about the latest functionality.

And if you want to support a project, give us a star on [GitHub](https://github.com/evidentlyai/evidently)!


# LLM as a judge

How to create and evaluate an LLM judge.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In this tutorial, we'll show you how to build an evaluator for text outputs using another LLM as the judge. This lets you automatically assess the quality of your system's responses based on your custom criteria.

{% hint style="info" %}
**You can also create LLM judges using no code**. This tutorial shows an open-source workflow that you can run locally using the Evidently Python library. You can also create and run LLM judges on the platform using [no-code interface](/user-guide/evaluations/no_code_evals).
{% endhint %}

We'll explore two ways to use an LLM as a judge:

* **Reference-based**. Compare new responses against a reference. This is for regression testing workflows or whenever you have a "ground truth" or approved responses to compare against.
* **Open-ended**. Evaluate responses based on custom criteria, which helps evaluate new outputs when there's no reference available.

By the end, you'll know how to create custom LLM judges and apply them to your data. Our primary focus will be showing how to develop and tune the evaluator, which you can then apply in different contexts, like regression testing or prompt comparison.

## Tutorial scope

Here's what we'll do:

* **Create an evaluation dataset**. Create a toy Q\&A dataset with two responses to each question, and add manual labels based on the criteria we want the LLM evaluator to follow later.
* **Create and run an LLM as a judge**. Design an LLM evaluator prompt to determine whether the new response is correct compared to the reference.&#x20;
* **Evaluate the judge**. Compare the LLM judge's evaluations with manual labels to see if they meet the expectations or need tweaking.

We'll start with the reference-based evaluator, which is more complex because it requires passing two columns to the prompt. Then, we'll create a simpler judge focused on verbosity.

To complete the tutorial, you will need:

* Basic Python knowledge.&#x20;
* An OpenAI API key to use for LLM evaluator.

Use the provided code snippets or run a sample notebook.

Jupyter notebook:

{% embed url="<https://github.com/evidentlyai/community-examples/blob/main/tutorials/LLM_as_a_judge_tutorial.ipynb>" %}

Or click to [open in Colab](https://colab.research.google.com/github/evidentlyai/community-examples/blob/main/tutorials/LLM_as_a_judge_tutorial.ipynb).

We recommend running this tutorial in Jupyter Notebook or Google Colab to render rich HTML objects with summary results directly in a notebook cell.

We will work with a toy dataset, which you can replace with your production data.

## Installation and Imports

Install Evidently:

```python
!pip install evidently[llm]
```

Import the required modules:

```python
from evidently import ColumnMapping
from evidently.report import Report
from evidently.metric_preset import TextEvals
from evidently.descriptors import *
from evidently.metrics import *
from evidently.features.llm_judge import BinaryClassificationPromptTemplate
```

Pass your OpenAI key:

```python
import os
os.environ["OPENAI_API_KEY"] = "YOUR_KEY"
```

## 1. Create the Dataset

First, we'll create a toy Q\&A dataset that includes:

* **Questions**. The inputs our LLM system got.
* **Target responses**. The "approved" responses. You can curate these from previous outputs that you consider accurate.
* **New responses**. These are the responses generated by your system that we want to evaluate.

To make it more interesting, we created a synthetic dataset with 15 answers to customer support questions. We also manually labeled each new response as correct or incorrect, with brief comments explaining the decision.

Here's how you can create this dataset in one go:

<details>

<summary>Create the DataFrame</summary>

```python
data = [
    ["Hi there, how do I reset my password?",
     "To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.",
     "To change your password, select 'Forgot Password' on the login screen and follow the steps sent to your registered email address. If you don't receive the email, check your spam folder or contact support for assistance.",
     "incorrect", "adds new information (contact support)"],

    ["Where can I find my transaction history?",
     "You can view your transaction history by logging into your account and navigating to the 'Transaction History' section. Here, you can see all your past transactions. You can also filter the transactions by date or type for easier viewing.",
     "Log into your account and go to 'Transaction History' to see all your past transactions. In this section, you can view and filter your transactions by date or type. This allows you to find specific transactions quickly and easily.",
     "correct", ""],

    ["How do I add another user to my account?",
     "I am afraid it is not currently possible to add multiple users to the account. Our system supports only one user per account for security reasons. We recommend creating separate accounts for different users.",
     "To add a secondary user, go to 'Account Settings', select 'Manage Users', and enter the details of the person you want to add. You can set permissions for their access, deciding what they can and cannot do within the account.",
     "incorrect", "contradiction (incorrect answer)"],

    ["Is it possible to link multiple bank accounts?",
     "Yes, you can link multiple bank accounts by going to 'Account Settings' in the menu and selecting 'Add Bank Account'. Follow the prompts to add your bank account details. Make sure to verify each bank account by following the verification process.",
     "You can add multiple bank accounts by visiting 'Accounts' in the menu and choosing 'Add Bank Account'. Enter your bank details as prompted and complete the verification process for each account to link them successfully.",
     "incorrect", "contradiction (incorrect menu item)"],

    ["Can I use your service for cryptocurrency transactions?",
     "Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature. We are continuously working to expand our services and may include this in the future.",
     "Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature. We are continuously working to expand our services and may include this in the future.",
     "correct", ""],

    ["Hi, can I get a detailed report of my monthly transactions?",
     "Yes, you can generate a detailed monthly report of your transactions by logging into your account, going to 'Transaction History', and selecting 'Generate Report'. You can customize the report by selecting specific dates or transaction types.",
     "You can get a detailed monthly report by logging into your account, navigating to 'Transaction History', and clicking on 'Generate Report'. Customize your report by choosing the date range and types of transactions you want to include.",
     "correct", ""],

    ["I am traveling to the US. Can I use the app there?",
     "Yes, you can use the app in the US just like you do at home. Ensure you have an internet connection. You may also want to update your app to the latest version before traveling for optimal performance.",
     "The app will work in the US without any issues. Just make sure you have access to the internet. For the best experience, update your app to the latest version before you travel.",
     "correct", ""],

    ["How do I link my payment account to a new mobile number?",
     "To link a new mobile number, log in to your account, go to 'Account Settings', select 'Mobile Number', and follow the instructions to verify your new number. You will need to enter the new number and verify it via a code sent to your phone.",
     "To add a new number, navigate to the 'Account Settings' section, select 'Mobile Number' and proceed with the steps to add and confirm the new number. Enter the new mobile number and verify it using the code sent to your phone.",
     "correct", ""],

    ["Can I receive notifications for transactions in real-time?",
     "Yes, you can enable real-time notifications for transactions by going to 'Account Settings', then 'Notifications', and turning on 'Transaction Alerts'. You can choose to receive alerts via SMS, email, or push notifications on your mobile device.",
     "To receive real-time notifications for transactions, log into your account, go to 'Account Settings', select 'Notifications', and enable 'Transaction Alerts'. Choose your preferred notification method between email or push notifications.",
     "incorrect", "omits information (sms notification)"],

    ["Hey, can I set up automatic transfers to my savings account?",
     "Yes, you can set up automatic transfers by going to 'Account Settings', selecting 'Automatic Transfers', and specifying the amount and frequency. You can choose to transfer weekly, bi-weekly, or monthly. Make sure to save the settings to activate the transfers.",
     "You can arrange automatic transfers by going to 'Account Settings', choosing 'Automatic Transfers', and setting the desired amount and frequency. Don't forget to save the changes to enable the automatic transfers.",
     "incorrect", "omits information (limited frequency of transfers available)"],

    ["Hi there, how do I reset my password?",
     "To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.",
     "To change your password, select 'Forgot Password' on the login screen and follow the steps sent to your registered email address. If you don't receive the email, check your spam folder.",
     "correct", ""],

    ["How can I update my billing address?",
     "To update your billing address, log into your account, go to 'Account Settings', select 'Billing Information', and enter your new address. Make sure to save the changes once you are done.",
     "To update your billing address, log into your account, navigate to 'Account Settings', and select 'Billing Information'. Enter your new address and ensure all fields are filled out correctly. Save the changes, and you will receive a confirmation email with the updated address details.",
     "incorrect", "adds new information (confirmation email)"],

    ["How do I contact customer support?",
     "You can contact customer support by logging into your account, going to the 'Help' section, and selecting 'Contact Us'. You can choose to reach us via email, phone, or live chat for immediate assistance.",
     "To contact customer support, log into your account and go to the 'Help' section. Select 'Contact Us' and choose your preferred method: email, phone, or live chat. Our support team is available 24/7 to assist you with any issues. Additionally, you can find a FAQ section that may answer your questions without needing to contact support.",
     "incorrect", "adds new information (24/7 availability, FAQ section)"],

    ["What should I do if my card is lost or stolen?",
     "If your card is lost or stolen, immediately log into your account, go to 'Card Management', and select 'Report Lost/Stolen'. Follow the instructions to block your card and request a replacement. You can also contact our support team for assistance.",
     "If your card is lost or stolen, navigate to 'Card Management' in your account, and select 'Report Lost/Stolen'. Follow the prompts to block your card and request a replacement. Additionally, you can contact our support team for help.",
     "correct", ""],

    ["How do I enable two-factor authentication (2FA)?",
     "To enable two-factor authentication, log into your account, go to 'Security Settings', and select 'Enable 2FA'. Follow the instructions to link your account with a 2FA app like Google Authenticator. Once set up, you will need to enter a code from the app each time you log in.",
     "To enable two-factor authentication, log into your account, navigate to 'Security Settings', and choose 'Enable 2FA'. Follow the on-screen instructions to link your account with a 2FA app such as Google Authenticator. After setup, each login will require a code from the app. Additionally, you can set up backup codes in case you lose access to the 2FA app.",
     "incorrect", "adds new information (backup codes)"]
]

columns = ["question", "target_response", "new_response", "label", "comment"]

golden_dataset = pd.DataFrame(data, columns=columns)
```

</details>

To preview it:

```python
pd.set_option('display.max_colwidth', None)
golden_dataset.head(5)
```

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e4b3fead466002f32fe743d2c9a7548913e5af28%2Fllmjudge_data_preview.png?alt=media)

{% hint style="info" %}
**How do you get the data in practice?** You can pick examples from your experiments or production data, focusing on scenarios you want to evaluate. For instance, if you plan to use the LLM evaluator for regression testing, select texts that show different ways a question has been answered, both correctly and incorrectly. You can also use synthetic data.
{% endhint %}

Why start with manual labels? This process helps you:

* **Refine your criteria**. Manually labeling data helps you clarify what you want the LLM judge to detect. It also reveals edge cases so that you can craft more effective evaluator prompts.
* **Evaluate the judge's quality**. Manual labels serve as the ground truth. You can then compare the LLM's judgments with these labels to assess its accuracy.

Ultimately, an LLM judge is a small ML system, and it needs its own evals!

Here's the distribution of examples in our small dataset: we have both correct and incorrect responses.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1a1a6b87c306944eacf962da3154a37ad27ba74b%2Fllmjudge_label_dist.png?alt=media)

## 2. Correctness evaluator

Now that we have our labeled dataset, it's time to set up an LLM judge. We'll start with an evaluator that checks if responses are correct compared to the reference. The goal is to match the quality of our manual labels.

We'll use the `LLMEval` [Descriptor](/user-guide/tests-and-reports/text-descriptors) to create a custom binary evaluator. Here's how to define the prompt template for correctness:

```python
correctness_eval= LLMEval(
    subcolumn="category",
    additional_columns={"target_response": "target_response"},
    template = BinaryClassificationPromptTemplate(
        criteria = """
An ANSWER is correct when it is the same as the REFERENCE in all facts and details, even if worded differently.
The ANSWER is incorrect if it contradicts the REFERENCE, adds additional claims, omits or changes details.

REFERENCE:

=====
{target_response}
=====
        """,
        target_category="incorrect",
        non_target_category="correct",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are an expert evaluator. will be given an ANSWER and REFERENCE.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name = "Correctness",
)
```

Explanation:

* `BinaryClassificationPromptTemplate`: This template instructs the LLM to classify the input into two classes, explain its reasoning, and format everything neatly. You don't have to worry about asking for these details—they're built into the template.
* `target_category` and `non-target category`: The labels we're aiming for - "correct" and "incorrect" in our case.
* `criteria`: This is where you describe what the LLM should look for when grading the responses.
* `include_reasoning`: This asks the LLM to explain its choice.
* `additional_columns`: This allows you to include not just the primary column (the "new\_response") but also the "reference\_response" for comparison. You then add this column name placeholder to the grading criteria.

In this example, we've set up the prompt to be strict, erring on the side of labeling a correct answer as incorrect is preferable. You can write it differently. This flexibility is one of the key benefits of creating a custom judge.

{% hint style="info" %}
**What else is there?** Check the docs on [LLM judge feature](/user-guide/customization/llm_as_a_judge).
{% endhint %}

## 3. Run the evaluation

Now, let's run the evaluation. We'll apply it to the "new\_response" column in our dataset and create a report that summarizes how the LLM judged the responses.

```python
correctness_report = Report(metrics=[
    TextEvals(column_name="new_response", descriptors=[
        correctness_eval
    ])
])

correctness_report.run(reference_data=None,
           current_data=golden_dataset)
correctness_report
```

This will render an HTML report in the notebook cell. Or, use `as_dict()` for a Python dictionary output.&#x20;

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e653fd2f2e9fdca251d4e2d92b32ecf8a19c93d2%2Fllmjudge_correctness_dist.png?alt=media)

But since we're refining our LLM evaluator, we don't want just the label distribution: we want to see what the LLM got right and wrong!

{% hint style="info" %}
**Tracking the evals.** When running evaluations in production, [upload your results](/get-started/quickstart-cloud/cloud_quickstart_llm) to the Evidently Platform to store them and track them over time.
{% endhint %}

## 4. Evaluate the LLM Eval quality

This part is a bit meta: we're going to evaluate the quality of our LLM evaluator itself.

To take a look at the raw outputs:

```python
correctness_report.datasets().current
```

This will show a DataFrame with newly added scores and explanations.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3b1350562230dd38b0cc23c9a0d95b90ab1e4f3e%2Fllmjudge_correctness_examples.png?alt=media)

**Note**: your results and explanations will vary since LLMs are non-deterministic.

We can also quantify it! We'll treat this like a classification task to measure how accurately the LLM identifies incorrect responses. We'll look at metrics like precision and recall.

Let's create a DataFrame and map our data for classification: the original manual label is the target, and the LLM-provided response is the prediction.

```python
df = pd.DataFrame(correctness_report.datasets().current)

column_mapping = ColumnMapping()
column_mapping.target = 'label'
column_mapping.prediction = 'Correctness category'
column_mapping.pos_label = 'incorrect'

classification_report = Report(metrics=[
    ClassificationQualityMetric(),
    ClassificationClassBalance(),
    ClassificationConfusionMatrix(),
])

classification_report.run(reference_data=None, current_data=df, column_mapping=column_mapping)

classification_report
```

Or use `classification_report.as_dict()`.

Explanation:

* `ClassificationQualityMetric` displays precision, recall, accuracy, etc.
* `ClassificationClassBalance` shows the distribution of classes (correct vs. incorrect) in the dataset.
* `ClassificationConfusionMatrix` illustrates the types of errors.

We have one type of error each, but overall, the results are pretty good! If you want to refine the judge, you can iterate on the prompt and continue improving it.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b87e4076cdf14f6602b783ce444c6f432165724b%2Fllmjudge_confusion_matrix.png?alt=media)

## 5. Verbosity evaluator

Next, let’s create a simpler LLM judge that evaluates the verbosity of the responses. This judge will check whether the responses are concise and to the point. This only requires evaluating one column with the output.

This is perfect for production evaluations where you don’t have a reference answer to compare against.

Here's how to set up the prompt template for verbosity:

```python
verbosity_eval = LLMEval(
    subcolumn="category",
    template = BinaryClassificationPromptTemplate(
        criteria = """Conciseness refers to the quality of being brief and to the point, while still providing all necessary information.
            A concise response should:
            - Provide the necessary information without unnecessary details or repetition.
            - Be brief yet comprehensive enough to address the query.
            - Use simple and direct language to convey the message effectively.
        """,
        target_category="concise",
        non_target_category="verbose",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are a judge which evaluates text.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name = "verbosity",
)
```

Run the Report and view the summary results:&#x20;

```python
verbosity_report = Report(metrics=[
    TextEvals(column_name="new_response", descriptors=[
        verbosity_eval
    ])
])

verbosity_report.run(reference_data=None,
           current_data=golden_dataset)
verbosity_report
```

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-615caec31d3010320f86bf1c9b29cab603ef22db%2Fllmjudge_verbosity_dist.png?alt=media)

Or use `as_dict()` for a Python dictionary output.&#x20;

To access the raw results:

```python
verbosity_report.datasets().current
```

Preview:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-5f5648ed37384223d1d2d83b7fbec3ceaa10c8d9%2Fllmjudge_verbosity_examples.png?alt=media)

Don't fully agree with the results? Use these labels as a starting point, and correct the decision where you see fit - now you've got your golden dataset! Next, iterate on your judge prompt.

The LLM judge itself is just one part of your overall evaluation framework. You can now integrate this evaluator into workflows, such as testing your LLM outputs after changing a prompt.


# LLM Regression Testing

How to run regression testing for LLM outputs.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In this tutorial, we’ll show you how to do regression testing for LLM outputs. You’ll learn how to compare new and old responses after changing a prompt, model, or anything else in your system. By re-running the same inputs, you can spot any significant changes. This helps you push updates with confidence or identify issues to fix.

## Tutorial scope

Here's what we'll do:

* **Create a toy dataset**. Build a small Q\&A dataset with answers and reference responses.
* **Get new answers**. Imitate generating new answers to the same question we want to compare.
* **Create and run a Test Suite**. Compare the answers using LLM-as-a-judge to evaluate length, correctness and style match.
* **Build a monitoring Dashboard**. Get plots to track the results of Tests over time.

To complete the tutorial, you will need:

* Basic Python knowledge.&#x20;
* An OpenAI API key to use for the LLM evaluator.
* An Evidently Cloud account to track test results. If not yet, [sign up](https://www.evidentlyai.com/register) for a free account.

Use the provided code snippets or run a sample notebook.

Jupyter notebook:

{% embed url="<https://github.com/evidentlyai/community-examples/blob/main/tutorials/Regression_testing_with_debugging.ipynb>" %}

Or click to [open in Colab](https://colab.research.google.com/github/evidentlyai/community-examples/blob/main/tutorials/Regression_testing_with_debugging.ipynb).

## 1. Installation and Imports

Install Evidently:

```python
!pip install evidently[llm]
```

Import the required modules:

```python
import pandas as pd

from evidently.report import Report
from evidently.test_suite import TestSuite
from evidently.metric_preset import TextEvals
from evidently.descriptors import *
from evidently.tests import *
from evidently.metrics import *

from evidently.features.llm_judge import BinaryClassificationPromptTemplate
```

To connect to Evidently Cloud:

```python
from evidently.ui.workspace.cloud import CloudWorkspace
```

To create monitoring panels as code:

```python
from evidently.ui.dashboards import DashboardPanelPlot
from evidently.ui.dashboards import DashboardPanelTestSuite
from evidently.ui.dashboards import DashboardPanelTestSuiteCounter
from evidently.ui.dashboards import TestSuitePanelType
from evidently.ui.dashboards import ReportFilter
from evidently.ui.dashboards import PanelValue
from evidently.ui.dashboards import PlotType
from evidently.ui.dashboards import CounterAgg
from evidently.tests.base_test import TestStatus
from evidently.renderers.html_widgets import WidgetSize
```

Pass your OpenAI key:

```python
import os
os.environ["OPENAI_API_KEY"] = "YOUR KEY"
```

## 2. Create a Project

Connect to Evidently Cloud. Replace with your actual token:

```python
ws = CloudWorkspace(token="YOUR_API_TOKEN", url="https://app.evidently.cloud")
```

Create a Project:

```python
project = ws.create_project("Regression testing example", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

{% hint style="info" %}
**Need help?** Check how to find [API key](/setup/cloud_account).
{% endhint %}

## 3. Prepare the Dataset

Create a dataset with questions and reference answers. We'll later compare the new LLM responses against them:

```python
data = [
    ["Why is the sky blue?", "The sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light."],
    ["How do airplanes stay in the air?", "Airplanes stay in the air because their wings create lift by forcing air to move faster over the top of the wing than underneath, which creates lower pressure on top."],
    ["Why do we have seasons?", "We have seasons because the Earth is tilted on its axis, which causes different parts of the Earth to receive more or less sunlight throughout the year."],
    ["How do magnets work?", "Magnets work because they have a magnetic field that can attract or repel certain metals, like iron, due to the alignment of their atomic particles."],
    ["Why does the moon change shape?", "The moon changes shape, or goes through phases, because we see different portions of its illuminated half as it orbits the Earth."]
]

columns = ["question", "target_response"]

ref_data = pd.DataFrame(data, columns=columns)
```

Get a quick preview:

```python
pd.set_option('display.max_colwidth', None)
ref_data.head()
```

Here is how the data looks:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-02ce41cd8513a2b3d4382c0e6ebf15f23452e143%2Fllmregesting_text_ref.png?alt=media)

You might want to have a quick look at some data statistics to help you set conditions for Tests. Let's check the text length distribution. This will render a summary Report directly in the notebook cell.

```python
report = Report(metrics=[
    TextEvals(column_name="target_response", descriptors=[
        TextLength(),
    ]),
])

report.run(reference_data=None,
           current_data=ref_data)

report
```

If you work in a non-interactive Python environment, call `report.as_dict()` or `report.json()` instead.

Here is the distribution of text length:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1670874a16756aa7e9025fe465aa579799e3ce47%2Fllmregtesting_textlength.png?alt=media)

## 4. Get new answers

Suppose you generate new responses using your LLM after changing a prompt. We will imitate it by adding a new column with new responses to the DataFrame:

```python
data = [
    ["Why is the sky blue?",
     "The sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light.",
     "The sky appears blue because air molecules scatter the sun’s blue light more than they scatter other colors."],

    ["How do airplanes stay in the air?",
     "Airplanes stay in the air because their wings create lift by forcing air to move faster over the top of the wing than underneath, which creates lower pressure on top.",
     "Airplanes stay airborne because the shape of their wings causes air to move faster over the top than the bottom, generating lift."],

    ["Why do we have seasons?",
     "We have seasons because the Earth is tilted on its axis, which causes different parts of the Earth to receive more or less sunlight throughout the year.",
     "Seasons occur because of the tilt of the Earth’s axis, leading to varying amounts of sunlight reaching different areas as the Earth orbits the sun."],

    ["How do magnets work?",
     "Magnets work because they have a magnetic field that can attract or repel certain metals, like iron, due to the alignment of their atomic particles.",
     "Magnets generate a magnetic field, which can attract metals like iron by causing the electrons in those metals to align in a particular way, creating an attractive or repulsive force."],

    ["Why does the moon change shape?",
     "The moon changes shape, or goes through phases, because we see different portions of its illuminated half as it orbits the Earth.",
     "The moon appears to change shape as it orbits Earth, which is because we see different parts of its lit-up half at different times. The sun lights up half of the moon, but as the moon moves around the Earth, we see varying portions of that lit-up side. So, the moon's shape in the sky seems to change gradually, from a thin crescent to a full circle and back to a crescent again."]
]

columns = ["question", "target_response", "response"]

eval_data = pd.DataFrame(data, columns=columns)
```

Here is the resulting dataset with the added new column:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-5ebdc9953543ea7db5961c4842b170c6b387ad99%2Fllmregesting_text_new.png?alt=media)

{% hint style="info" %}
**How to run it in production?** In practice, replace this step with calling your LLM app to score the inputs. After you get the new responses, add them to a DataFrame. You can also use our **tracing** library to instrument your app and get traces as a tabular dataset. Check the tutorial with [tracing workflow](/tutorials-and-examples/tutorial_tracing).
{% endhint %}

## 5. Design the Test suite

To compare new answers with old ones, we need evaluation metrics. You can use deterministic or embeddings-based metrics like SemanticSimilarity. However, you often need more custom criteria. Using LLM-as-a-judge is useful for this, letting you define what to detect.

Let’s design our Tests:

* **Length check**. All new responses must be between 80 and 200 symbols.
* **Correctness**. All new responses should give the same answer without contradictions.
* **Style**. All new responses should match the style of the reference.

Text length is easy to check, but for Correctness and Style checks, we'll write our custom LLM judges.

### Correctness judge

We implement the correctness evaluator, using an Evidenty template for binary classification. We ask the LLM to classify each response as correct or incorrect based on the {target\_response} column and provide reasoning for its decision.

```python
correctness_eval= LLMEval(
    subcolumn="category",
    additional_columns={"target_response": "target_response"},
    template = BinaryClassificationPromptTemplate(
        criteria = """
An ANSWER is correct when it is the same as the REFERENCE in all facts and details, even if worded differently.
The ANSWER is incorrect if it contradicts the REFERENCE, adds additional claims, omits or changes details.

REFERENCE:

=====
{target_response}
=====
        """,
        target_category="incorrect",
        non_target_category="correct",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are an expert evaluator. will be given an ANSWER and REFERENCE.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name = "Correctness",
)
```

We recommend splitting each evaluation criterion into separate judges and using a simple grading scale, like binary classifiers, for better reliability.

{% hint style="info" %}
**Don't forget to evaluate your judge!** Each LLM evaluator is a small ML system you should tune to align with your preferences. We recommend running a couple of iterations to tune it. Check the tutorial on [creating LLM judges](/tutorials-and-examples/cookbook_llm_judge).
{% endhint %}

{% hint style="info" %}
**Docs on LLM judge**. For an explanation of each parameter, check the docs on [LLM judge functionality](/user-guide/customization/llm_as_a_judge).
{% endhint %}

### Style judge

Using a similar approach, we'll create a judge for style. We'll also add clarifications to define what we mean by a style match.

```python
style_eval= LLMEval(
    subcolumn="category",
    additional_columns={"target_response": "target_response"},
    template = BinaryClassificationPromptTemplate(
        criteria = """
An ANSWER is style-matching when it matches the REFERENCE answer in style.
The ANSWER is style-mismatched when it diverges from the REFERENCE answer in style.

Consider the following STYLE attributes:
- tone (friendly, formal, casual, sarcastic, etc.)
- sentence structure (simple, compound, complex, etc.)
- verbosity level (relative length of answers)
- and other similar attributes that may reflect difference in STYLE.

You must focus only on STYLE. Ignore any differences in contents.

=====
{target_response}
=====
        """,
        target_category="style-matching",
        non_target_category="style-mismatched",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are an expert evaluator. will be given an ANSWER and REFERENCE.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name = "Style",
)
```

### Complete Test Suite

Now, we can create a Test Suite that includes checks for correctness, style matching, and text length.

* **Choose Tests**. We select Evidently column-level tests like `TestCategoryCount` and `TestShareOfOutRangeValues`. (You can pick other Tests, like `TestColumnValueMin` or `TestColumnValueMean`).
* **Set Parameters and Conditions**. Some Tests require parameters: for example, `left` and `right` to set the allowed range for Text Length. For Test fail conditions, use parameters like `gt` (greater than), `lt` (less than), `eq` (equal), etc.
* **Set non-critical Tests**. Identify non-critical Tests, like the style match check, to trigger warnings instead of fails. This helps visually separate them on monitoring panels and set alerts only for critical failures.

We reference our two LLM judges, `style_eval` and `correctness_eval`, and apply them to the `response` column in our dataset. For text length, we use the built-in `TextLength()` descriptor for the same column.

```python
test_suite = TestSuite(tests=[
    TestCategoryCount(
        column_name=style_eval.on("response"),
        category="style-mismatched",
        eq=0,
        is_critical=False),
    TestCategoryCount(
        column_name=correctness_eval.on("response"),
        category="incorrect",
        eq=0),
    TestShareOfOutRangeValues(
        column_name=TextLength(display_name="Response Length")
            .on("response"),
        left=80,
        right=200,
        eq=0),
])
```

In this example, we expect the share of failures to be zero using the `eq=0` condition. You can adjust this, such as using lte=0.1, which means "less than 10%". This would cause the Test to fail if more than 10% of rows are out of the set length range.

Allowing some share of Tests to fail is convenient for real-world applications.

You can add additional Tests as you see fit for regular expressions, word presence, etc. and Tests for other columns in the same Test Suite.

{% hint style="info" %}
**Understand Tests**. Learn how to set [Test conditions](/user-guide/tests-and-reports/run-tests) and use [Tests with text data](/user-guide/tests-and-reports/text-descriptors). See the list of [All tests](/reference/all-tests).
{% endhint %}

{% hint style="info" %}
**Understand Descriptors**. See the list of available text Descriptors in the [All metrics](/reference/all-metrics) table.
{% endhint %}

## 6. Run the Test Suite

Now that our Test Suite is ready - let's run it!

To apply this Test Suite to the `eval_data` that we prepared earlier:

```python
test_suite.run(reference_data=None, current_data=eval_data)
```

This will compute the Test Suite: but how do you see it? You can preview the results in your Python notebook (call `test_suite`). However, we’ll now send it to Evidently Cloud along with the scored data:

```python
ws.add_test_suite(project.id, test_suite, include_data=True)
```

Including data is optional but useful for most LLM use cases since you'd want to see not just the aggregate Test results but also the raw texts to debug when Tests fail.

To view the results, navigate to the Evidently Platform. Go to the ([Home Page](https://app.evidently.cloud/)), enter your Project, and find the "Test Suites" section in the left menu. Here, you'll see the Test Suite you can explore.

You'll find both the summary Test results and the Dataset with added scores and explanations. You can zoom in on specific evaluations, such as sorting the data by Text Length or finding rows labeled as "incorrect" or "style-mismatched".

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-dc522fbca9b335cfe33342de549a2ce670e82745%2Fllmregtesting_test_1.png?alt=media)

**Note**: your explanations will vary since LLMs are non-deterministic.

{% hint style="info" %}
**Using Tags**. You can optionally attach Tags to your Test Suite to associate this specific run with some parameter, like a prompt version. Check the [docs on generating snapshots](/user-guide/evaluations/snapshots).
{% endhint %}

## 7. Test again

Let's say you made yet another change to the prompt. Our reference dataset stays the same, but we generate a new set of answers that we want to compare to this reference.

Here is the toy `eval_data_2` to imitate the result of the change.

```python
data = [
    ["Why is the sky blue?",
     "The sky is blue because molecules in the air scatter blue light from the sun more than they scatter red light.",
     "The sky looks blue because air molecules scatter the blue light from the sun more effectively than other colors."],

    ["How do airplanes stay in the air?",
     "Airplanes stay in the air because their wings create lift by forcing air to move faster over the top of the wing than underneath, which creates lower pressure on top.",
     "Airplanes fly by generating lift through the wings, which makes the air move faster above them, lowering the pressure."],

    ["Why do we have seasons?",
     "We have seasons because the Earth is tilted on its axis, which causes different parts of the Earth to receive more or less sunlight throughout the year.",
     "Seasons change because the distance between the Earth and the sun varies throughout the year."],  # This response contradicts the reference.

    ["How do magnets work?",
     "Magnets work because they have a magnetic field that can attract or repel certain metals, like iron, due to the alignment of their atomic particles.",
     "Magnets operate by creating a magnetic field, which interacts with certain metals like iron due to the specific alignment of atomic particles."],

    ["Why does the moon change shape?",
     "The moon changes shape, or goes through phases, because we see different portions of its illuminated half as it orbits the Earth.",
     "The moon's phases occur because we observe varying portions of its lit half as it moves around the Earth."]
]

columns = ["question", "target_response", "response"]

eval_data_2 = pd.DataFrame(data, columns=columns)
```

Now, we can apply the same Test Suite to this data and send it to Evidently Cloud.

```python
test_suite.run(reference_data=None, current_data=eval_data_2)
ws.add_test_suite(project.id, test_suite, include_data=True)
```

If you go and open the new Test Suite results, you can again explore the outcomes and explanations.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b345b954e26a89c79bdefa4543c971fc0ea508a6%2Fllmregtesting_test_2.png?alt=media)

## 8. Get a Dashboard

You can continue running Test Suites in this manner. As you run multiple, you may want to track Test results over time.

You can easily add this to a Dashboard, both in UI or programmatically. Let's create a couple of Panels using Dashboards as a code approach.

The following code will add:

* A counter panel to show the SUCCESS rate of the latest Test run.
* A test monitoring panel to show all Test results over time.

```python
project.dashboard.add_panel(
     DashboardPanelTestSuiteCounter(
        title="Latest Test run",
        filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
        size=WidgetSize.FULL,
        statuses=[TestStatus.SUCCESS],
        agg=CounterAgg.LAST,
    ),
    tab="Tests"
)
project.dashboard.add_panel(
    DashboardPanelTestSuite(
        title="Test results",
        filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
        size=WidgetSize.FULL,
        panel_type=TestSuitePanelType.DETAILED,
        time_agg="1min",
    ),
    tab="Tests"
)
project.save()
```

When you navigate to the UI, you will now see a Panel which shows a summary of Test results (Success, Failure, and Warning) for each Test Suite we ran. As you add more Tests to the same Project, the Panels will be automatically updated to show new Test results.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-12c25b226020a9464496a733205aae7f5e4bb0d4%2Fllmregesting_test_dashboard.png?alt=media)

If you hover over individual Test results, you will able to see the specific Test and conditions.

{% hint style="info" %}
**Using Dashboards**. You can design and add other Panel types. Check the [docs on Dashboards](/user-guide/dashboard/dashboard_overview).
{% endhint %}

**What's next?** As you design a similar Test Suite for your use case, you can integrate it with CI/CD workflows to run on every change.


# Installation

How to install the open-source Python library.

{% hint style="info" %}
**You are looking at the old Evidently documentation**. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Evidently

`Evidently` is available as a Python package. To install it using the **pip package manager**, run:

```python
pip install evidently
```

To install `evidently` using **conda installer**, run:

```sh
conda install -c conda-forge evidently
```

## Evidently LLM

To run evaluations specific to LLMs that include additional dependencies, run:

```python
pip install evidently[llm]
```

## Tracely

To use tracing based on OpenTelemetry, install the sister package **tracely**:

```sh
pip install tracely
```


# Evidently Cloud

How to set up Evidently Cloud account.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## 1. Create an Account

If not yet, [sign up for a free Evidently Cloud account](https://app.evidently.cloud/signup).

## 2. Create an Organization

After logging in, create an **Organization** and name it.

## 3. Connect from Python

You will need an access token to interact with Evidently Cloud from your Python environment.

{% hint style="info" %}
**Do I always need this?** No, only for data uploads or to run evaluations in Python. You can view data, edit dashboards, upload CSVs and run no-code evaluations without the token.
{% endhint %}

### Get a Token

Click the **Key** icon in the left menu to open the ([Token page](https://app.evidently.cloud/token)). Generate and save the token securely.

### Install Evidently

To connect to the Evidently Cloud from Python, first [install the Evidently Python library](/setup/install-evidently).

```python
pip install evidently
```

### Connect

Import the cloud workspace and pass your API token to connect:

```python
from evidently.ui.workspace.cloud import CloudWorkspace

ws = CloudWorkspace(
token="YOUR_TOKEN_HERE",
url="https://app.evidently.cloud")
```

Now, you are all set to start using Evidently Cloud! Create your first Project and choose your [next step](/get-started/quickstart-cloud).


# Self-hosting

How to self-host the open-source Evidently UI service.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In addition to using Evidently Python library, you can self-host the UI Service to get a monitoring Dashboard and organize the results of your evaluations. This is optional: you can also run evaluations and render results directly in Python or export them elsewhere.

{% hint style="info" %}
**Evidently Cloud.** Sign up for a free [Evidently Cloud](/setup/cloud_account) account to get a managed version with extra features.
{% endhint %}

{% hint style="info" %}
**Evidently Enterprise.** This page explains how to self-host the open-source platform. For the Enterprise version with extra features and support, [contact us](https://www.evidentlyai.com/get-demo). Host it in your cloud or on-premises.
{% endhint %}

To get a self-hostable Dashboard, you must:

1. Create a Workspace (local or remote) to store your data.
2. Launch the UI service.

## 1. Create a Workspace

Once you [install Evidently](/setup/install-evidently), you will need to create a `workspace`. This designates a remote or local directory where you will store the evaluation results (as JSON `snapshots` of the Evidently `Reports` or `Test Suites`). The UI Service will read the data from this source.

There are three scenarios, based on where you run the UI Service and store data.

* **Local Workspace**. Both the UI Service and data storage are local.
* **Remote Workspace**. Both the UI Service and data storage are remote.
* **Workspace with remote data storage**. You run the UI Service and store data on different servers.

### Local Workspace

In this scenario, you generate, store the snapshots and run the monitoring UI on the same machine.

Imports:

```python
from evidently.ui.workspace import Workspace
from evidently.ui.workspace import WorkspaceBase
```

To create a local Workspace and assign a name:

```python
ws = Workspace.create("evidently_ui_workspace")
```

You can pass a `path` parameter to specify the path to a local directory.

{% hint style="info" %}
**Code example**. [Self-hosting tutorial](/tutorials-and-examples/tutorial-monitoring) shows a complete Python script to create and populate a local Workspace.
{% endhint %}

### Remote Workspace

In this scenario, you send the snapshots to a remote server. You must run the Monitoring UI on the same remote server. It will directly interface with the filesystem where the snapshots are stored.

Imports:

```
from evidently.ui.remote import RemoteWorkspace
from evidently.ui.workspace import Workspace
from evidently.ui.workspace import WorkspaceBase
```

To create a remote Workspace (UI should be running at this address):

```python
workspace = RemoteWorkspace("http://localhost:8000")
```

You can pass the following parameters:

| Parameter                  | Description                                                                                  |
| -------------------------- | -------------------------------------------------------------------------------------------- |
| `self.base_url = base_url` | URL for the remote UI service.                                                               |
| `self.secret = secret`     | String with secret, None by default. Use it if access to the URL is protected by a password. |

{% hint style="info" %}
**Code example**. See the [remote service example](https://github.com/evidentlyai/evidently/tree/main/examples/service).
{% endhint %}

### Remote snapshot storage

In the examples above, you store the snapshots and run the UI on the same server. Alternatively, you can store snapshots in a remote data store (such as an S3 bucket). The Monitoring UI service will interface with the designated data store to read the snapshot data.

To connect to data stores Evidently uses `fsspec` that allows accessing data on remote file systems via a standard Python interface.

You can verify supported data stores in the Fsspec documentation: [built-in implementations](https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations) and [other implementations](https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations).

For example, to read snapshots from an S3 bucket (with MinIO running on localhost:9000), you must specify environment variables:

```
FSSPEC_S3_ENDPOINT_URL=http://localhost:9000/
FSSPEC_S3_KEY=my_key FSSPEC_S3_SECRET=my_secret
evidently ui --workspace s3://my_bucket/workspace
```

### \[DANGER] Delete Workspace

To delete a Workspace (for example, an empty or a test Workspace), run the command from the Terminal:

```
cd src/evidently/ui/
rm -r workspace
```

{% hint style="danger" %}
**You are deleting all the data**. This command will delete the snapshots stored in the folder. To maintain access to the generated snapshots, you must store them elsewhere.
{% endhint %}

## 2. Launch the UI service

To launch the Evidently UI service, you must run a command in the Terminal.

**Option 1**. If you log snapshots to a local Workspace directory, you run Evidently UI over it. Run the following command from the directory where the Workspace folder is located.

```
evidently ui
```

**Option 2**. If you have your Project in a different Workspace, specify the path:

```
evidently ui --workspace . /workspace
```

**Option 3**. If you have your Project in a specified Workspace and run the UI service at the specific port (if the default port 8000 is occupied).

```
evidently ui --workspace ./workspace --port 8080
```

To view the Evidently interface, go to URL <http://localhost:8000> or a specified port in your web browser.


# Projects


# Projects overview

Introduction to Projects in Evidently Platform.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This section describes how to use Projects. This applies if you use the Evidently web application. You can also run evaluations locally with the Evidently Python library without creating a Project.

{% hint style="success" %}
Projects are available in `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

## What is a Project?

To start working in the Evidently UI, you must create at least one evaluation or monitoring Project.

**Home Page**. You can see all your Projects on the home page of the Evidently platform:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-79a37db5082a2db316196381efe5709bda2c241e%2Fprojects-min.png?alt=media)

**Project Dashboard**. A Project helps you organize all data and evaluations for a specific use case.

Each Project has its own Dashboard and alerting rules. Inside the Project, you can also access the results of all experimental evaluations, regression testing, or ongoing online monitoring.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2375a068033618f6abee93161c20929eb582b40a%2Fproject_dashboard-min.png?alt=media)

You can organize Projects in a way that suits your workflow, for example:

* create separate Projects for the experimental and production phases of your AI use case
* create separate Projects for each ML model or individual components of an LLM-based application
* group evaluation results for different models / components within a single Project, but use Tags (such as model version, etc.) to differentiate between them.

Once you create a Project, it gets a unique ID. You can then connect via the Python API or through the UI to send data, edit the Dashboard, and manage the Project.


# Manage Projects

Set up a Project for your evaluation or monitoring use case.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Before creating a Project, you need a Workspace.

* In Evidently Cloud, your account is your Workspace. [Set up an account and connect from Python](/setup/cloud_account).
* In self-hosted deployments, a Workspace is a remote or local directory. [Create and connect to a workspace](/setup/self_hosting).

## Create a Project

You can create a Project using the Python API or directly in the user interface.

### Add a new Project - API

To create a Project inside a workspace `ws` and Organization ([see organizations](https://app.evidently.cloud/organizations)) with an `org_id`, assign a name and description, and save the changes:

```
project = ws.create_project("My test project", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()
```

In self-hosted open-source installation, you do not need to pass the Team ID. To create a Project:

```
project = ws.create_project("My test project")
project.description = "My project description"
project.save()
```

### Add a new Project - UI

Click on the “plus” sign on the home page, type your Project name and description.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-62015ecba8c22cd19bba1b4f8c6ab2a846450bc2%2Fadd_project_wide-min.png?alt=media)

After creating a Project, you can click to open a Dashboard. Since there's no data yet, it will be empty.

**Project ID**. Once you run `create_project`, you will see the Project ID. You can later use it to reference the Project. You can also copy the Project ID directly from the UI: it appears above the monitoring Dashboard.

## Manage Project

### Connect to a Project

To connect to an existing Project from Python, use the `get_project` method.

```python
project = ws.get_project("PROJECT_ID")
```

### Save changes

After making changes to the Project (such as editing description or adding monitoring Panels), always use the `save()` command:

```python
project.save()
```

### Browse Projects

You can see all available Projects on the monitoring homepage, or request a list programmatically. To get a list of all Projects in a workspace `ws`, use:

```python
ws.list_projects()
```

To find a specific Project by its name, use the `search_project` method:

```python
ws.search_project("project_name")
```

### \[DANGER] Delete Project

{% hint style="danger" %}
**You are deleting the data in a Project**. If you delete a Project, you will delete all the snapshots stored inside it.
{% endhint %}

{% tabs %}
{% tab title="API" %}
To delete the Project and all the data inside it:

```
# ws.delete_project("PROJECT ID")
```

{% endtab %}

{% tab title="UI" %}
Go to the "home page", and hover over a Project name. Click on the bin sign and confirm that you want to delete the Project.
{% endtab %}
{% endtabs %}

## Project parameters

Each Project has the following parameters.

| Parameter                                       | Description                                                                                                                                                                                                                                                                                                                                             |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name: str`                                     | Project name.                                                                                                                                                                                                                                                                                                                                           |
| `id: UUID4 = Field(default_factory=uuid.uuid4)` | Unique identifier of the Project. Assigned automatically.                                                                                                                                                                                                                                                                                               |
| `description: Optional[str] = None`             | Optional description. Visible when you browse Projects.                                                                                                                                                                                                                                                                                                 |
| `dashboard: DashboardConfig`                    | <p>Dashboard configuration that describes the composition of the monitoring Panels.<br><br><strong>Note</strong>: See <a href="/user-guide/dashboard/design_dashboard_api">Dashboard Design</a> for details. You don't need to explicitly pass <code>DashboardConfig</code> if you use the <code>.dashboard.add\_panel</code> method to add Panels.</p> |
| `date_from: Optional[datetime.datetime] = None` | <p>Start DateTime of the monitoring Dashboard. By default, Evidently shows data for all available periods based on the snapshot timestamps.<br><br>You can set a different DateTime. E.g., to refer to the last 30 days:<br><code>from datetime import datetime, timedelta</code><br><code>datetime.now() + timedelta(-30)</code></p>                   |
| `date_to: Optional[datetime.datetime] = None`   | <p>End DateTime of the monitoring Dashboard.<br>Works the same as above.</p>                                                                                                                                                                                                                                                                            |

## What’s next?

Once you create or connect to a Project, you can:

* [Send snapshots](/user-guide/evaluations/snapshots) using the `add_report` or `add_test_suite` methods.
* Configure the monitoring Dashboard in the [user interface](/user-guide/dashboard/add_dashboard_tabs) or via the [Python API](/user-guide/dashboard/design_dashboard_api).


# Tracing


# Tracing overview

Introduction to tracing.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This section covers how to instrument your AI application to collect data and send it to Evidently Platform for further evaluation and analysis.

**Note**: this feature is in active development, expect more detailed documentation soon.

{% hint style="success" %}
Tracing uses the open-source `Tracely` library.
{% endhint %}

{% hint style="info" %}
**Looking for something else?** You can also run evaluations locally on existing datasets using [Reports and Test Suites](/user-guide/tests-and-reports/introduction), and upload tabular [Datasets](/user-guide/datasets/datasets_overview) to the platform using the Python API.
{% endhint %}

## What is LLM tracing?

Tracing captures a detailed record of how your LLM application operates. It logs the system's inputs and outputs, along with intermediate steps and events, such as function calls. With this data, you can analyze and monitor your LLM system, including running evaluations on the tracing datasets.

While traces can have multiple turns, Evidently makes it easy to automatically create tabular datasets from traces for evaluation.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-4f7134458e3621bc63408694cec64cbbd26131e6%2Fqs_tracing_dataset.png?alt=media)

## What is Tracely?

**Tracely** is an open-source Python library based on OpenTelemetry that helps collect traces from LLM-powered applications.

Evidently Platform natively supports instrumentation collected with this library.

## Do I always need tracing?

No, tracing is optional on the Evidently Platform. You can also upload datasets directly, especially for batch tabular data, or run evaluations on local datasets and send the results to the platform.

However, tracing is particularly useful for tracking complex LLM chains to see all execution flows. You can use tracing during experimental runs and in production, coupled with online evaluations.


# Set up tracing

Set up LLM tracing with tracely.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

For an end-to-end example, check the [Tracing Quickstart](/get-started/quickstart-cloud/cloud_quickstart_tracing).

## Installation and Imports

Install the `tracely` package from PyPi.

```python
!pip install tracely 
```

Imports:

```python
from tracely import init_tracing
from tracely import trace_event
```

## Initialize tracing

Use `init_tracing` to enable tracely tracing. Example:

```python
init_tracing(
   address="https://app.evidently.cloud/",
   api_key=”YOUR_EVIDENTLY_TOKEN”,
   project_id="YOUR_PROJECT_ID",
   export_name="YOUR_TRACING_DATASET_NAME",
   )
```

### Tracing parameters

| **Parameter**                  | **Description**                                                                                                                                                                                                                        |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `address: Optional[str]`       | <p>The URL of the collector service where tracing data will be sent. For Evidently Cloud, set <code><https://app.evidently.cloud/></code>.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p>           |
| `exporter_type: Optional[str]` | <p>Specifies the type of exporter to use for tracing. Options are <code>grpc</code> for gRPC protocol or <code>http</code> for HTTP protocol.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p>        |
| `api_key: Optional[str]`       | <p>The authorization API key for Evidently Cloud tracing. This key authenticates your requests and is necessary for sending data to Evidently Cloud.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p> |
| `project_id: str`              | <p>The ID of your Project in Evidently Cloud.<br><strong>Required:</strong> Yes, <strong>Default:</strong> <code>None</code></p>                                                                                                       |
| `export_name: Optional[str]`   | <p>A string name assigned to the exported tracing data. All data with the same <code>export\_name</code> will be grouped into a single dataset.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p>      |
| `as_global: bool = True`       | Indicates whether to register the tracing provider globally for OpenTelemetry (`opentelemetry.trace.TracerProvider`) or use it locally within a scope. **Default:** `True`                                                             |

## Tracing a function

To trace a function call use `trace_event()` decorator.

**Example 1.** To log all arguments of the function:

```
@trace_event()
```

**Example 2.** To log only input arguments of the function:

```
@trace_event(track_args=[])
```

**Example 3.** To log only "arg1" and "arg2":

```
@trace_event(track_args=["arg1", "arg2"])
```

See the [Tracing Quickstart](/get-started/quickstart-cloud/cloud_quickstart_tracing) for an end-to-end example.

| **Parameter**                      | **Description**                                                                                                                                                                                                                                                                                               |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `span_name: Optional[str]`         | <p>The name of the span to track. This is how the event will be labeled in the trace. By giving it a name, you can identify and analyze this particular step within your tracing data.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p>                                      |
| `track_args: Optional[List[str]]`  | <p>A list of arguments to capture during tracing. If set to <code>None</code>, it captures all arguments by default. If set to <code>\[]</code>, it captures no arguments.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p>                                                  |
| `ignore_args: Optional[List[str]]` | <p>A list of arguments to ignore from tracking. For instance, if a function has sensitive information that you don’t want to log, you can list those arguments here. If set to <code>None</code>, no arguments are ignored.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>None</code></p> |
| `track_output: Optional[bool]`     | <p>Indicates whether to track the output of the function call. If set to <code>True</code>, the trace will include the function’s output, allowing you to see not just what was passed in but also what was returned.<br><strong>Required:</strong> No, <strong>Default:</strong> <code>True</code></p>       |


# Input data


# Input data overview

How to prepare the data to run Evidently Reports or Test Suites.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

To run evaluations on your datasets with the Evidently Python library, you should prepare your data in a certain way. This section covers how to do that.&#x20;

{% hint style="success" %}
This applies to `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

{% hint style="info" %}
**Looking for something else?** Check [Tracing](/user-guide/tracing/tracing_overview) to instrument your app. Check [Datasets](/user-guide/datasets/datasets_overview) to work with datasets in the user interface. To run evaluations after you prepare the data, see [Reports and Test Suites](/user-guide/tests-and-reports/introduction).
{% endhint %}

## Input data format

Evidently works with Pandas DataFrames, with some metrics also supported on [Spark](/user-guide/tests-and-reports/spark).

Your input data should be in **tabular** format. All column names must be strings. The data can include any numerical, categorical, text, DateTime, and ID columns. You can pass embeddings as numerical features.&#x20;

The structure is flexible. For example, you can pass:

* **Any tabular dataset**. You can run checks for data quality and drift for any dataset.
* **Logs of generative LLM application**. Include text inputs, outputs, and metadata.
* **ML model inferences**. You can analyze prediction logs that include model features (numerical, categorical, embeddings), predictions, and optional target values.&#x20;

To run certain evaluations, you must include specific columns. For instance, to evaluate classification quality, you need columns with predicted and actual labels. These should be named "prediction" and "target", or you’ll need to point to the columns that contain them. This process is called **Column Mapping**.

Learn more in the next section:

{% content-ref url="/pages/kFm7UkwawZ1ULAt8md8k" %}
[Column mapping](/user-guide/input-data/column-mapping)
{% endcontent-ref %}

## Reference and current data

Usually, you evaluate a single dataset, which we call the **current** dataset. In some cases, you might also use a second dataset, known as the **reference** dataset. You pass them both when running an evaluation.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-eabc7007b02ce25aabc99de56cc8db8121e87e42%2Ftwo_datasets_classification.png?alt=media\&token=89edb6a4-2d09-4a3b-a387-9b08292074b7)

When you may need two datasets:

* **Side-by-side comparison**. If you want to compare model performance or data quality over two different periods or between model versions, you can do this inside one Report. Pass one dataset as `current`, and another as `reference`.
* **Data drift detection**. To detect distribution shifts, you compare two datasets using methods like distance metrics. You always need two datasets. Use your latest production batch as `current`, and choose a `reference` dataset to compare against, such as your validation data or an earlier production batch.
* **Automatic Test generation**. If you provide a `reference` dataset, Evidently can automatically set up Test conditions, like expected min-max values for specific columns. This way, you don’t have to write each test condition manually.

If you pass two datasets, the structure of both datasets should be identical.&#x20;

## Data volume

Running computationally intensive evaluations on large datasets can take time. This depends on the specific evaluation as well as your infrastructure.

In many cases, such as for probabilistic data drift detection, it’s more efficient to work with **samples** of your data. For instance, instead of running drift detection on millions of rows, you can apply random or stratified sampling and then compare samples of your data. &#x20;

For datasets that don’t fit in memory, you can run calculations using [Spark](/user-guide/tests-and-reports/spark).


# Column mapping

How to use column mapping in Evidently.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Column mapping helps map your input data schema or specify column types. For example, to run evaluation on text data, you must specify which columns in your dataset contain texts. This allows Evidently to process the input data correctly.

You can create a `ColumnMapping` object in Python prior to generating a Report or Test Suite or map the columns visually when working in the Evidently platform.

You only need to map columns that will be used in your evaluations.

## Default mapping strategy

If the `column_mapping` is not specified or set as `None`, Evidently will use the default mapping strategy, trying to match the columns automatically.

**Column types**:

* All columns with numeric types (np.number) will be treated as Numerical.
* All columns with DateTime format (np.datetime64) will be treated as DateTime.
* All other columns will be treated as Categorical.

**Dataset structure**:

* The column named **"id"** will be treated as an ID column.
* The column named **"datetime"** will be treated as a DateTime column.
* The column named **"target"** will be treated as a target column with a true label or value.
* The column named **"prediction"** will be treated as a model prediction.

## Which columns do you need?

To run certain types of evaluations, you must include specific columns or provide a reference dataset. For example, to run text evaluations, you must have at least one column labeled as text. To run data drift checks, you always need a reference dataset.

Here are example requirements:

| Evaluation         | Feature columns | Prediction column                 | Target column                     | ID column | Datetime column | Reference dataset |
| ------------------ | --------------- | --------------------------------- | --------------------------------- | --------- | --------------- | ----------------- |
| **Text Evals**     | Required (Text) | Optional                          | Optional                          | Optional  | Optional        | Optional          |
| **Data Quality**   | Required (Any)  | Optional                          | Optional                          | Optional  | Optional        | Optional          |
| **Data Drift**     | Required (Any)  | Optional                          | Optional                          | Optional  | Optional        | Required          |
| **Target Drift**   | Optional        | Target and/or prediction required | Target and/or prediction required | Optional  | Optional        | Required          |
| **Classification** | Optional        | Required                          | Required                          | Optional  | Optional        | Optional          |
| **Regression**     | Optional        | Required                          | Required                          | Optional  | Optional        | Optional          |

{% hint style="info" %}
**It's best always to use column mapping**. Without it, Evidently will apply its own heuristics to map the input data automatically. To avoid errors, it's safer to set the column mapping manually.
{% endhint %}

## Code example

Notebook example on specifying column mapping:

{% embed url="<https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb>" %}

**Imports**. Imports to use column mapping:

```python
from evidently import ColumnMapping
```

**Basic API**. Once you create a `ColumnMapping` object, you pass it along with the data when computing the [Report or Test Suite](/user-guide/tests-and-reports/introduction). For example:

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
column_mapping.numerical_features = numerical_features
column_mapping.categorical_features = categorical_features

report = Report(metrics=[
    RegressionPreset(),
])

report.run(reference_data=ref,
           current_data=cur,
           column_mapping=column_mapping)

report
```

## Column mapping

### DateTime and ID

To map columns containing DateTime and ID:

```python
column_mapping.datetime = 'date' #'date' is the name of the column with datetime
column_mapping.id = None #there is no ID column in the dataset
```

{% hint style="info" %}
**Why map them:** A "DateTime" column serves as the index for certain plots, giving you richer visualizations in your Reports. If you have a timestamp column, it's always a good idea to map it. Mapping the "Datetime" and "ID" columns also excludes them from analyses like data drift detection, where it wouldn't add any value.
{% endhint %}

### Target and Prediction

To map columns containing Target and Prediction:

```python
column_mapping.target = 'y' 
column_mapping.prediction = 'pred' 
```

This matches regression or simple classification tasks. For more complex cases, check detailed instructions on how to map inputs for [classification](/user-guide/input-data/classification_data) and [ranking and recommendations](/user-guide/input-data/recsys_data).

{% hint style="info" %}
**Why map them:** If you have a dataset with ML inferences, it's necessary to map where Predictions and Ground Truth are to compute the quality metrics. You also need to map the Target and/or Prediction to generate the Target Drift Preset.
{% endhint %}

### Categorical and numerical columns

To split the columns into numerical and categorical types, pass them as lists:

```python
column_mapping.numerical_features = ['temp', 'atemp', 'humidity'] 
column_mapping.categorical_features = ['season', 'holiday'] 
```

{% hint style="info" %}
**Why map them:** Column types impact evaluations. For example, the data [drift algorithm](/reference/data-drift-algorithm) selects statistical tests based on column type, or `ColumnSummaryMetric` visualizations change with feature type. Manually mapping columns avoids errors, like numerical columns with few unique values being mistaken for categorical.
{% endhint %}

### Text data

To specify columns that contain text data:

```python
column_mapping.text_features = ['email_subject', 'email_body']
```

{% hint style="info" %}
**Why map them:** Always map text columns to enable text-specific evaluations. This also ensures text columns are excluded from Tests and Metrics where they don’t apply. If you don’t map text columns, they’ll be treated as categorical, potentially leading to irrelevant evaluations like raw text distribution histograms.
{% endhint %}

### Embeddings features

To specify which columns in your dataset contain embeddings, pass a dictionary where keys are embedding names and values are lists of columns.

Here is an example of how you point to the defined list of columns that contain embeddings:

```python
column_mapping = ColumnMapping()
column_mapping.embeddings = {'small_subset': embeddings_data.columns[:10]}
```

{% hint style="info" %}
**Why map them:** To apply embeddings-specific data drift detection methods.
{% endhint %}

### DateTime features

You might have temporal features in your dataset. For example, “date of the last contact.”

To map them, pass them as a list:

```python
column_mapping.datetime_features = ['last_call_date', 'join_date'] 
```

{% hint style="info" %}
**What is the difference between DateTime features and DateTime?** DateTime is a single timestamp column. It often represents the time when a data row was recorded. Use it if you want to see it as index on the plots. A DateTime feature is any time-related column in your dataset, such as input features in a ML model.
{% endhint %}

{% hint style="info" %}
**Why map them:** DateTime feature will be ignored in data drift calculation. Evidently will also calculate appropriate stats and different visualizations for DateTime features in the `ColumnSummaryMetric`.
{% endhint %}

### Task parameter for target function

It’s often important to specify whether your Target column is continuous or discrete. This impacts Data Quality, Data Drift, and Target Drift evaluations for the Target column.

To define it explicitly, specify the task parameter:

```python
column_mapping.target = 'y'
column_mapping.task = 'regression'
```

It accepts the following values:

* `regression`
* `classification`
* `recsys` (for ranking and recommenders)

**Default**: If you don't specify the task, Evidently will use a simple strategy: if the target has a numeric type and the number of unique values > 5: task == ‘regression.’ In all other cases, the task == ‘classification’.

{% hint style="info" %}
**Why map it:** Classes encoded as numbers may sometimes look like continuous targets. Explicitly specifying the target type ensures the right visualizations and statistical tests for the target (prediction) are selected.
{% endhint %}


# Data for Classification

How to define the data schema for classification.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

To evaluate classification model performance, you must correctly map the input data schema.

## Code example

{% embed url="<https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_use_column_mapping.ipynb>" %}

## Column Mapping

To evaluate the classification performance, you need both true labels and prediction. Depending on the classification type (e.g., binary, multi-class, probabilistic), you have different options of how to pass the predictions.

### Multiclass classification

#### Option 1

Target: encoded labels, Preds: encoded labels + Optional\[target\_names].

| target | prediction |
| ------ | ---------- |
| 1      | 1          |
| 0      | 2          |
| …      | …          |
| 2      | 2          |

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
column_mapping.target_names = ['Setosa', 'Versicolour', 'Virginica']
```

If you pass the target names, they will appear on the visualizations.

You can also pass the target names as a dictionary:

```python
column_mapping.target_names = {'0':'Setosa', '1':'Versicolor', '2':'Virginica'}
```

or

```python
column_mapping.target_names = {0:'Setosa', 1:'Versicolor', 2:'Virginica'} 
```

#### Option 2

Target: labels, Preds: labels.

| target        | prediction    |
| ------------- | ------------- |
| ‘Versicolour’ | ‘Versicolour’ |
| ‘Setosa’      | ‘Virginica’   |
| …             | …             |
| ‘Virginica’   | ‘Virginica’   |

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
```

### Multiclass probabilistic classification

Target: labels, Preds: columns named after labels.

| target      | ‘Versicolour’ | ‘Setosa’ | ‘Virginica’ |
| ----------- | ------------- | -------- | ----------- |
| ‘Setosa’    | 0.98          | 0.01     | 0.01        |
| ‘Virginica’ | 0.5           | 0.2      | 0.3         |
| …           | …             |          |             |
| ‘Virginica’ | 0.2           | 0.7      | 0.1         |

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = ['Setosa', 'Versicolour', 'Virginica']

```

Naming the columns after the labels is a requirement. You cannot pass a custom list.

### Binary classification

#### Option 1

Target: encoded labels, Preds: encoded labels + pos\_label + Optional\[target\_names]

| target | prediction |
| ------ | ---------- |
| 1      | 1          |
| 0      | 1          |
| …      | …          |
| 1      | 0          |

By default, Evidently expects the positive class to be labeled as ‘1’. If you have a different label, specify it explicitly.

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
column_mapping.target_names = ['churn', 'not_churn']
column_mapping.pos_label = 0

```

If you pass the target names, they will appear on the visualizations.

#### Option 2

Target: labels, Preds: labels + pos\_label

| target       | prediction   |
| ------------ | ------------ |
| ‘churn’      | ‘churn’      |
| ‘not\_churn’ | ‘churn’      |
| …            | …            |
| ‘churn’      | ‘not\_churn’ |

Passing the name of the positive class is a requirement in this case.

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
column_mapping.pos_label = 'churn'

```

### Binary probabilistic classification

#### Option 1

Target: labels, Preds: columns named after labels + pos\_label

| target       | ‘churn’ | ‘not\_churn’ |
| ------------ | ------- | ------------ |
| ‘churn’      | 0.9     | 0.1          |
| ‘churn’      | 0.7     | 0.3          |
| …            | …       |              |
| ‘not\_churn’ | 0.5     | 0.5          |

Passing the name of the positive class is a requirement in this case.

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = ['churn', 'not_churn']
column_mapping.pos_label = 'churn'

```

#### Option 2

Target: labels, Preds: a column named like one of the labels + pos\_label

| target       | ‘not\_churn’ |
| ------------ | ------------ |
| ‘churn’      | 0.5          |
| ‘not\_churn’ | 0.1          |
| …            | …            |
| ‘churn’      | 0.9          |

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'not_churn'
column_mapping.pos_label = 'churn'

```

Both naming the column after one of the labels and passing the name of the positive class are requirements.

#### Option 3

Target: encoded labels, Preds: one column with any name + pos\_label

| target | prediction |
| ------ | ---------- |
| 1      | 0.5        |
| 1      | 0.1        |
| …      | …          |
| 0      | 0.9        |

```python
column_mapping = ColumnMapping()

column_mapping.target = 'target'
column_mapping.prediction = 'prediction'
column_mapping.pos_label = 1
column_mapping.target_names = ['churn', 'not_churn']

```

If you pass the target names, they will appear on the visualizations.


# Data for Recommendations

How to define the data schema for ranking and recommendations.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

To evaluate data from recommender systems, you must correctly map the input data schema. You can also pass an optional additional dataset with training data.

**Note**: this mapping will also apply to search and retrieval systems. Treat "user\_id" as "query\_id".

## Code example

Notebook example on using column mapping and additional data for recommender systems:

{% embed url="<https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_run_recsys_metrics.ipynb>" %}

## Column mapping

You must define column mapping to run evaluations for recommender or ranking systems on your `current` and (optional) `reference` data. Column mapping helps point to the columns with user ID, item ID, prediction, and target.

To evaluate the quality of a ranking or a recommendation system, you must pass:

* The score or rank generated by the system as the prediction.
* The relevance labels as the target (e.g., this could be an interaction result like user click, assigned relevance label, etc.)

Here are the examples of the expected data inputs.

If the model prediction is a score (expected by default):

| user\_id | item\_id | prediction (score) | target (relevance) |
| -------- | -------- | ------------------ | ------------------ |
| user\_1  | item\_1  | 1.95               | 0                  |
| user\_1  | item\_2  | 0.8                | 1                  |
| user\_1  | item\_3  | 0.05               | 0                  |

If the model prediction is a rank:

| user\_id | item\_id | prediction (rank) | target (relevance) |
| -------- | -------- | ----------------- | ------------------ |
| user\_1  | item\_1  | 1                 | 0                  |
| user\_1  | item\_2  | 2                 | 1                  |
| user\_1  | item\_3  | 3                 | 0                  |

The **target** column with the interaction result or relevance label can contain either:

* a binary label (where `1` is a positive outcome)
* any true labels or scores (any positive values, where a higher value corresponds to a better match or a more valuable user action).

You might need to add additional details about your dataset via column mapping:

* `recommendations_type`: `score` (default) or `rank`. Helps specify whether the prediction column contains ranking or predicted score.
* `user_id`: helps specify the column that contains user IDs.
* `item_id`: helps specify the column that contains ranked items.

## Additional data

Some metrics like novelty or popularity bias require training data, which has a different structure from production data. To pass it, use the `additional_data` object. You can pass your training data as `current_train_data` and (optional) `reference_train_data`.

Example:

```python
report = Report(metrics=[
   UserBiasMetric(column_name='age'),
])
report.run(reference_data=ref, current_data=cur, column_mapping=column_mapping, additional_data={'current_train_data': train})
report
```

### Requirements:

* The additional training dataset should have the following structure:

| user | item | target |
| ---- | ---- | ------ |
| id1  | id1  | 1      |
| id2  | id9  | 1      |
| id3  | id2  | 1      |
| id3  | id1  | 1      |
| id4  | id6  | 1      |

* The names of the columns with `user_id` and `item_id` should match the corresponding columns in the current (and optional reference) data.
* The name of the column with the interaction result should match the name of the `target` column in the current (and optional reference) data.
* If you use metrics that refer to specific columns (such as `UserBiasMetric` metric), these columns must also be present in the training dataset.
* You can pass a single training dataset or two datasets (in case your reference and current dataset have different training data).

{% hint style="info" %}

#### What is the difference between training and reference data?

The reference dataset can belong to a previous production period or a different model you compare against. The training dataset is used to train the model. Their structure usually differs:

* Production data typically includes a list of all recommended items, where some of them earn a positive interaction result. It also contains negative examples (ignored recommendations) and data about model prediction (predicted rank or score).
* Training data typically contains a history of positive actions, such as user viewing history, page reads, or upvotes. Since it only includes the interaction results, it lacks negative examples (e.g., ignored recommendations) and column with the model output (predicted rank or score).
  {% endhint %}


# Load data to pandas

At the moment Evidently works with datasets in **Pandas DataFrame** **format** **only**. These datasets should fit into memory to be processed correctly.

In this tutorial you will see how to load and sample data from other data sources to Pandas DataFrame for further analysis with Evidently.

### Tensorflow Datasets

Tensorflow supports conversion from Tensorflow Dataset to Pandas DataFrame with `as_dataframe` method.

For bigger datasets that do not fit into memory use `take` for sampling before conversion.\
Check that the dataset is **shuffled** to obtain a representative sample.

```python
import tensorflow_datasets as tfds

MAXIMUM_DATASET_SIZE = 10000 # set up the maximum number of lines in your sample

# tensorflow_ds is a shuffled Tensorflow Dataset
pandas_df = tfds.as_dataframe(tensorflow_ds.take(MAXIMUM_DATASET_SIZE))
```

> Note that `as_dataframe` method loads everything **in memory**,\
> make sure to run it on a sample from your dataset to control for its size

### Pytorch Datapipes

To sample data from Pytorch Datapipes shuffle it first with `shuffle()` and take the first batch of the chosen size.\
This sample can be converted to Pandas DataFrame

See example with AG News dataset:

```python
import pandas as pd
from torchdata.datapipes.iter import HttpReader

MAXIMUM_DATASET_SIZE = 10000 # set up the maximum number of lines in your sample

# Load data to Pytorch Datapipe
URL = "https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/train.csv"
ag_news_train = HttpReader([URL]).parse_csv().map(lambda t: (int(t[0]), " ".join(t[1:])))

# Shuffle and sample data
batches = ag_news_train.shuffle().batch(MAXIMUM_DATASET_SIZE)
sample = next(iter(batches))

# Load sampled data to Pandas DataFrame
pandas_df = pd.DataFrame({'text': [el[1] for el in sample],
                         'label': [el[0] for el in sample]})
```

Note that resulting Pandas DataFrame schema is arbitrary, just make sure to specify text and target columns with[column\_mapping](https://github.com/evidentlyai/docs-old/blob/main/book/tests-and-reports/column-mapping.md\)) later

### PySpark DataFrames

PySpark supports conversion to Pandas DataFrame with `toPandas()` method.

For bigger DataFrames that do not fit into memory use `sample` for sampling before conversion.

```python
fraction = 0.5 # set the fraction of original DataFrame to be sampled

# df_spark is a PySpark DataFrame
df_pandas = df_spark.sample(withReplacement=False, fraction=fraction, seed=None).toPandas()
```

You can ensure that sampling provides the same result each run by passing a fixed `seed` value to `sample` method

### Files in a directory

If your data is organized in separate files for each text with folder names corresponding to class labels, like so:

```
main_directory/
...class_a/
......a_text_file_1.txt
......a_text_file_2.txt
...class_b/
......b_text_file_1.txt
......b_text_file_2.txt
```

use the following steps to sample data preserving the balance of classes:

```python
import os, random
import pandas as pd

DIRECTORY_NAME = '/main_directory/' # define data source directory
MAXIMUM_DATASET_SIZE = 10000 # set up the maximum number of lines in your sample

# find the names of classes
classes_names = [class_name for class_name in os.listdir(DIRECTORY_NAME) \
                if os.path.isdir(os.path.join(DIRECTORY_NAME, class_name))]

# determine classes sizes
classes_sizes_dict = {class_name: len(os.listdir(os.path.join(DIRECTORY_NAME,
                                                             class_name))) \
                     for class_name in classes_names}
total_size = sum(classes_sizes_dict.values())

# sample objects from classes in the correct proportion
texts = []
labels = []

for class_name in classes_names:
 sample_size = int((classes_sizes_dict[class_name] / total_size) * MAXIMUM_DATASET_SIZE)
 random_files_names = random.sample(os.listdir(os.path.join(DIRECTORY_NAME, class_name)),
                                    sample_size)
 for file_name in random_files_names:
   with open(os.path.join(DIRECTORY_NAME, class_name, file_name), 'r') as f:
         text = f.read()
   texts.append(text)
   labels.append(class_name)

# load sampled data to Pandas DataFrame
df_pandas = pd.DataFrame({'text': texts, 'label': labels})
```


# Tests and Reports


# Reports and Tests Overview

Introduction to Reports and Test Suites.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

{% hint style="success" %}
This applies to `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

You can use the Evidently Python library as a standalone open-source tool or as part of the Evidently Platform to track and store evaluation results over time.

This section of the documentation explains the core evaluation API of the Evidently library. This page, specifically, gives a conceptual overview of the components. For detailed API guides, check the links below:

**Reports**. Run evaluations for your dataset:

{% content-ref url="/pages/qaWBxdftzYnLxMOka9lW" %}
[Get a Report](/user-guide/tests-and-reports/get-reports)
{% endcontent-ref %}

**Test Suites**. Run Tests that check defined conditions and return pass/fail results:

{% content-ref url="/pages/PTEM697jz4F7npQz8ZUi" %}
[Run a Test Suite](/user-guide/tests-and-reports/run-tests)
{% endcontent-ref %}

**Descriptors**. Run row-level evaluations for text data:

{% content-ref url="/pages/AWeLDPMNchCgoye8pB74" %}
[Evaluate Text Data](/user-guide/tests-and-reports/text-descriptors)
{% endcontent-ref %}

{% hint style="info" %}
**Looking for something else?** To check **what** you can evaluate, browse [Presets](/presets/all-presets), [Metrics](/reference/all-metrics) and [Tests](/reference/all-tests). To understand workflows on Evidently Platform, check the [evaluation workflow](/user-guide/evaluations/evals_overview) or section on [live monitoring](/user-guide/monitoring/monitoring_overview).
{% endhint %}

## Report

A Report computes various Metrics to assess data, ML, or LLM quality. You specify the Metrics you want to include, run the Report on your dataset, and get a summary of the results.

Metrics can help answer various questions, such as:

* **Gen AI**: Are the texts concise, professional, and true to the source context?
* **Classification**: How many predictions are correct? What is the recall/precision?
* **Data Quality**: Are there missing values or duplicates in the dataset?
* **Recommender Systems**: Are recommendations diverse?
* **Regression**: Are you over- or under-predicting?
* **Data Drift**: Is the data your AI system receives different from before?

You can start with **Metric Presets** which are pre-built combinations of Metrics, designed to evaluate specific aspects of a model, dataset, or AI system's performance.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3a9e8034a127297a8e5db990e6414107cfa282a2%2Freports-min.png?alt=media)

You can think of Preset as Report templates that help you get started quickly. For example, there are presets for checking Data Drift, assessing Data Quality, or evaluating Regression Performance.

Here is an example of a Data Drift Report that checks for shifts in data distribution:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-29538ba6d1b155fc1c1e6d35f771d6dd7afcfe85%2Foverview_drift_report-min.png?alt=media)

You can also create a custom Report by listing Metrics you’d like to include one by one. Evidently comes with built-in checks, so you can start quickly without building from scratch. You can also add custom evaluations.

An Evidently Metric isn't just a single value. It's best to think of it as an “evaluation unit” that includes both the computation results (which can include multiple values) and its visual representation.

For example, when you run a `ColumnSummaryMetric`, you'll receive a range of descriptive statistics for the column and a distribution histogram. For monitoring, you can later pull any of these individual values (min, max, mean, etc.) in time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-95838e7b5ca27328bf9249a0e5354c8c97cd2e48%2Foverview_num_metric_summary_example-min.png?alt=media)

This Metric contents will vary based on the column type. For a categorical variable, you will receive a different set of statistics:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c30e6cddafc52ad1b3e07adb520150e4fcaaf8eb%2Foverview_cat_metric_summary_example.png?alt=media)

You can also pass two datasets to the Metric to get a side-by-side comparison.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0abcd70ac63cff100436d1062bf7df11a8ca73fa%2Foverview_cat_metric_compare_example.png?alt=media)

In addition to column-level Metrics like `ColumnSummaryMetric`, you can compute many checks on the Dataset level. For example, `ClassificationQualityByClass` will summarize the metrics like F-score, precision and recall by class.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-4ad2b38d6db61b96474c4b4b2b6630eec99429e7%2Foverview_class_metric_example-min.png?alt=media)

The `DatasetCorrelationsMetric` shows the correlations between all columns in the dataset on a heatmap.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0528e987dc9e3f1f620c57e5a26f26f2b3d621e7%2Foverview_correlation_metric_example-min.png?alt=media)

There are lots of other checks! See all available Metrics and Presets:

{% content-ref url="/pages/9Oow6J1hxIAkXHKJhMC6" %}
[All metrics](/reference/all-metrics)
{% endcontent-ref %}

You can get Report results in various formats:

* **Interactive visualizations**. Each Metric has a visual representation like shown above. You can view plots directly in environments like Jupyter Notebook or Colab, or save as an HTML file.
* **Raw scores**. You can export as a JSON, Python dictionary, or Pandas dataframe. This includes the calculated values and optional details like histogram bins.
* **Evidently Snapshot**. If you send the evaluation results to Evidently Cloud, it will be saved as a JSON `snapshot`. This format allows you to recreate the original visual Report.

When to use Reports:

* **Visual analysis, experiments and debugging**. Use Reports as a standalone tool for ad hoc evaluations, data exploration, debugging, and comparing models or prompts.
* **Metric computation layer for monitoring**. Integrate Reports into your pipeline to log and track model and data performance over time by computing consecutive Reports for different data batches or samples.
* **AI quality reporting**. Add text comments and create Model Cards or document data and AI quality to share with the team and stakeholders.

How to generate Reports:

{% content-ref url="/pages/qaWBxdftzYnLxMOka9lW" %}
[Get a Report](/user-guide/tests-and-reports/get-reports)
{% endcontent-ref %}

## Test Suite

A Test Suite is a collection of Tests that verify if specific conditions are met within a given dataset.

Think of a Test Suite as a more structured way to use Metrics you’d find in a Report, but with added rules. Each Test pairs a Metric with a condition. It calculates a value, checks it against your rule, and then tells you if it passed or failed.

For example, you might set up a Test to check that:

* **Gen AI**: The share of responses labeled “incorrect” by the LLM judge is under 5%.
* **Classification**: Precision and recall are within +/-10% from the reference quality.
* **Data Quality**: The share of missing values in the dataset is under 1%.
* **Recommender Systems**: The hit rate is over 20%.
* **Regression**: The model quality is better than the quality of the dummy model.
* **Data Drift**: The share of drifting features in the dataset is less than 30%.

Here is an example of a classification Test Suites. The same interface works for all types of checks.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0b58824b2803733084c522a060a16a19eef99960%2Foverview_test_suite_example-min.png?alt=media)

There are also **Test Presets** — pre-built Test Suites designed for specific use cases. They act as templates to help you get started quickly. Presets automatically generate multiple Tests, and derive Test conditions from the reference dataset or heuristics.

For example, the `DataQualityTestPreset` automatically checks for issues like missing values or duplicates, while the `RegressionTestPreset` focuses on evaluating regression model performance.

For each Test in a suite, you'll receive one of the following outcomes:

* **Pass**: the condition was met.
* **Fail**: the condition wasn’t met.
* **Warning**: the condition wasn’t met, but the check was labeled as non-critical.
* **Error**: something went wrong with the Test itself, such as a failure in execution.

If your Test Suite includes many Tests, you can easily navigate the output by status or specific columns. Clicking an individual Test provides a supporting visual to help debug the outcome.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-bd2a381274da54f382fb8970bbace6a2897350bd%2Foverview_test_example-min.png?alt=media)

You can create a custom Test Suite with individual Tests. Just like with Metrics, you can run Tests:

* on specific columns, like checking if mean value is within a stable range (`TestColumnValueMean`)
* on the dataset as a whole, like checking how many rows are empty (`TestNumberOfEmptyRows`).

You can set custom conditions for each Test, or let Evidently assist.

See all available Tests:

{% content-ref url="/pages/WX5hwVJgs7YiY4Dx1i2L" %}
[All tests](/reference/all-tests)
{% endcontent-ref %}

Similar to Reports, you can view Test results in Python environments like Jupyter notebook, export as HTML, JSON, a Python dictionary or a Pandas DataFrame, and save a JSON snapshot to the Evidently platform.

When to use Tests:

* **Automated checks in pipelines**. Tests are ideal for integrating into data and ML pipelines. You can perform data validation or evaluation with Test Suites directly as a pipeline step, such as after you pull fresh data. You can add conditional actions — like triggering an alert or halting the pipeline if a Test fails. Evidently Tests easily integrate with orchestrator tools like Airflow.
* **For continuous testing for production systems**. You can run Tests on your production data to ensure ongoing quality. Structuring your evaluations as Tests can help reduce alert fatigue and simplify configuration when evaluating multiple conditions at once. For example, you can quickly verify that all columns in the input data are within a defined min-max range.
* **Any batch testing scenarios**. Tests are great for any automated scenario where you need to evaluate batches of data. For example, you can use them in CI/CD pipelines to assess model quality after retraining or to perform regression testing on a golden dataset after updating a prompt.

How to run Test Suites:

{% content-ref url="/pages/PTEM697jz4F7npQz8ZUi" %}
[Run a Test Suite](/user-guide/tests-and-reports/run-tests)
{% endcontent-ref %}

## Descriptors

When evaluating text data, you'll need **Descriptors**. A Descriptor is a row-level score that assesses a specific quality or dimension of the text.

Say, you want to analyze the length, sentiment, or semantic similarity of your LLM-generated answers compared to the reference. Descriptors let you evaluate each text individually:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8bbe8a33495fa10d2cd047c61730d32f6d65cd1e%2Fdescriptors_export.png?alt=media)

This differs from many ML or data quality metrics that provide a single value for a column or entire dataset (like accuracy). Descriptors give scores at the per-row level, which is useful for debugging—such as finding the shortest responses.

Evidently computes these extra values and adds them to your evaluated dataset. Descriptors can be:

* **Numerical**. Stats like text length or sentiment score.
* **Categorical**. Labels like “correct” or “incorrect,” or “true” and “false” for pattern matches.
* **Strings of text**. For example, reasoning generated by an LLM to explain a score.

You can create Descriptors that use two columns at once, such as measuring the semantic similarity of one text column to another.

Once you’ve chosen the Descriptors, you can include them in a Report or Test Suite using column-level Metrics or Tests. This works similarly to other Evidently checks.

Under the hood, instead of computing a Metric or Test directly for an existing numerical or categorical column, Evidently computes a Metric or Test **for a Descriptor** that’s generated simultaneously for a text column in your dataset.

For instance, you can compute a `ColumnSummaryMetric` for Text Length descriptor of your text column that contains "Generated summary”. This will give you an overview of the text length across all inputs:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1c4574a94a142bc58f9820b7a0d74fd3483a4a44%2Foverview_descriptor_metric_example.png?alt=media)

You can alternatively run a Test that explicitly checks if all text lengths are within a specific defined range:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e107b093527b430e12e6f36ff896d1407aa0afef%2Foverview_descriptor_test_example-min.png?alt=media)

You can access these scores by exporting a Pandas DataFrame or via the Evidently Platform.

How to generate Descriptors:

{% content-ref url="/pages/AWeLDPMNchCgoye8pB74" %}
[Evaluate Text Data](/user-guide/tests-and-reports/text-descriptors)
{% endcontent-ref %}

## Test Suites or Reports?

**Reports** and **Test Suites** are complementary. You can choose one option or use both.

**Reports** are best for debugging, exploratory, and ad hoc analytics. They focus on interactive visualizations and don’t require setting conditions upfront. Use them to run evaluations during experiments and compare datasets, models, or prompts.

**Test Suites** are best for automation, like CI/CD checks. Use them when you can set up conditions upfront (or derive them from the reference dataset). Tests force you to think through what you expect from your data and systems, and you can run them at scale, only reacting to failure alerts.

You can also use both Reports and Test Suites together. For example, run Tests for automated checks and if they fail, use Reports for visual debugging.


# Get a Report

How to generate Reports using Evidently Python library.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Code examples

Check the sample notebooks in [Examples](/tutorials-and-examples/examples).

## Imports

After [installing Evidently](/setup/install-evidently), import the `Report` component and the necessary `metric_presets` or `metrics` you plan to use:

```python
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset
from evidently.metrics import *
```

## How it works

Here is the general flow:

* **Input data**. Prepare data as a Pandas DataFrame. This will be your `current` data to run evaluations for. For some checks, you may need a second `reference` dataset. Check the [input data requirements](/user-guide/input-data/data-requirements).
* **Schema mapping**. Define your data schema using [Column Mapping](/user-guide/input-data/column-mapping). Optional, but highly recommended.
* **Define the Report**. Create a `Report` object and list the selected `metrics`.
* **Run the Report**. Run the Report on your `current_data`. If applicable, pass the `reference_data` and `column_mapping`.
* **Get the results**. View the Report in Jupyter notebook, export the metrics, or [upload](/user-guide/evaluations/snapshots) to Evidently Platform.

You can use Metric Presets, which are pre-built Reports that work out of the box, or create a custom Report selecting Metrics one by one.

## Metric Presets

To generate a Report using Metric Preset, simply include the selected Metric Preset in the `metrics` list.

**Example 1**. To generate the Data Quality Report for a single dataset and get the visual output in Jupyter notebook or Colab:

```python
data_quality_report = Report(metrics=[
    DataQualityPreset()
])

data_quality_report.run(current_data=my_dataset,
                        reference_data=None,
                        column_mapping=None)
data_quality_report
```

If nothing else is specified, the Report will run with the default parameters for all columns in the dataset.

{% hint style="info" %}
**Available Presets**. There are other Presets: for example, `DataDriftPreset`, `RegressionPreset` and `ClassificationPreset`. Check the list of [Presets](/presets/all-presets) to understand individual, and all available [Metrics](/reference/all-metrics).
{% endhint %}

**Example 2**. You can include multiple Presets in a Report. To combine Data Drift and Data Quality and run them over two datasets, including a reference dataset necessary for data drift evaluation:

```python
drift_report = Report(metrics=[
     DataDriftPreset(),
     DataQualityPreset()
])
 
drift_report.run(reference_data=my_ref_dataset,
                 current_data=my_cur_dataset)
drift_report
```

It will display the combined Report in Jupyter notebook or Colab.

{% hint style="info" %}
**Raw data in visuals**. Visuals in the Reports are aggregated. This reduces load time and Report size for larger datasets, even with millions of rows. If you work with small datasets or samples, you can [generate plots with raw data](/user-guide/customization/report-data-aggregation).
{% endhint %}

**Example 3**. To export the values computed inside the Report, export it as a Python dictionary.

```python
drift_report.as_dict()
```

{% hint style="info" %}
**There are more output formats!**. You can also export Report results in formats like HTML, JSON, dataframe, and more. Refer to the [Output Formats](/user-guide/tests-and-reports/output_formats) for details.
{% endhint %}

**Example 4**. You can customize some of the Metrics inside the Preset. For example, set a custom decision threshold (instead of default 0.5) when computing classification quality metrics:

```python
dataset_report = Report(metrics=[
    ClassificationPreset(probas_threshold=0.7),
])
```

**Example 5**. You can pass a list of columns to the Preset, so column-specific Metrics are generated only for those columns, not the entire dataset.

```python
drift_report = Report(metrics=[
    DataDriftPreset(columns=["age", "position"]),
])
```

{% hint style="info" %}
Refer to the [All metrics](/reference/all-metrics) table to see defaults and available parameters that you can pass for each Preset.
{% endhint %}

## Get a custom Report

While Presets are a great starting point, you may want to customize the Report by choosing Metrics or adjusting their parameters even more. To do this, create a custom Report.

### 1. Choose metrics

First, define which Metrics you want to include in your custom Report. Metrics can be either dataset-level or column-level.

{% hint style="info" %}
**Available Metrics**: See the [All metrics](/reference/all-metrics) table. For a preview, check [Example notebooks](/tutorials-and-examples/examples).
{% endhint %}

{% hint style="info" %}
**Row-level evals**: To generate row-level scores for text data, check [Text Descriptors](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/text-descriptors.md.md).
{% endhint %}

**Dataset-level metrics**. Some Metrics evaluate the entire dataset. For example, a Metric that checks for data drift across the whole dataset or calculates accuracy.

To create a custom Report with dataset-level metrics, create a `Report` object and list the `metrics`:

```python
data_drift_dataset_report = Report(metrics=[
    DatasetDriftMetric(),
    DataseSummaryMetric(),  
])
```

**Column-level Metrics**. Some Metrics focus on individual columns, like evaluating distribution drift or summarizing specific columns. To include column-level Metrics, pass the name of the column to each such Metric:

```python
data_drift_column_report = Report(metrics=[
    ColumnSummaryMetric(column_name="age"),
    ColumnDriftMetric(column_name="age"),   
])
```

{% hint style="info" %}
**Generating multiple column-level Metrics**: You can use a helper function to easily generate multiple column-level Metrics for a list of columns. See the page on [Metric Generator](/user-guide/tests-and-reports/test-metric-generator).
{% endhint %}

**Combining Metrics and Presets**. You can mix Metrics Presets and individual Metrics in the same Report, and also combine column-level and dataset-level Metrics.

```python
my_report = Report(metrics=[
    DataQualityPreset(),
    DatasetDriftMetric(),
    ColumnDriftMetric(column_name="age"),
])
```

### 2. Set metric parameters

Metrics can have optional or required parameters. For example, the data drift detection algorithm selects a method automatically, but you can override this by specifying your preferred method (Optional). To calculate the number of values matching a regular expression, you must always define this expression (Required).

**Example 1**. How to specify a regular expression (required parameter):

```python
data_integrity_column_report = Report(metrics=[
    ColumnRegExpMetric(column_name="education", reg_exp=r".*-.*", top=5),
    ColumnRegExpMetric(column_name="relationship", reg_exp=r".*child.*")
])

data_integrity_column_report.run(reference_data=adult_ref, current_data=adult_cur)
data_integrity_column_report
```

**Example 2**. How to specify a custom Data Drift test (optional parameter).

```python
data_drift_column_report = Report(metrics=[
    ColumnDriftMetric('age'),
    ColumnDriftMetric('age', stattest='psi'),
])
data_drift_column_report.run(reference_data=adult_ref, current_data=adult_cur)

data_drift_column_report
```

{% hint style="info" %}
**Reference**: The available parameters for each Metric are listed in the [All metrics](/reference/all-metrics) table.
{% endhint %}


# Run a Test Suite

How to run Test Suites using Evidently Python library.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Code examples

Check the [sample notebooks](/tutorials-and-examples/examples) for examples of how to generate Test Suites.

## Imports

After [installing Evidently](/setup/install-evidently), import the `TestSuite` component and the necessary `test_presets` or `tests` you plan to use:

```python
from evidently.test_suite import TestSuite
from evidently.test_preset import DataQualityTestPreset, DataStabilityTestPreset
from evidently.tests import *
```

## How it works

Here is the general flow.

* **Input data**. Prepare data as a Pandas DataFrame. This will be your `current` data to test. You may also pass a `reference` dataset to generate Test conditions from this reference or run data distribution Tests. Check the [input data requirements](/user-guide/input-data/data-requirements).
* **Schema mapping**. Define your data schema using [Column Mapping](/user-guide/input-data/column-mapping). Optional, but highly recommended.
* **Define the Test Suite**. Create a `TestSuite` object and pass the selected `tests`.
* **Set the parameters**. Optionally, specify Test conditions and mark certain Tests as non-critical.
* **Run the Test Suite**. Execute the Test Suite on your `current_data`. If applicable, pass the `reference_data` and `column_mapping`.
* **Get the results**. View the results in Jupyter notebook, export the summary, or send to the Evidently Platform.

You can use Test Presets or create your Test Suite.

## Test Presets

Test Presets are pre-built Test Suites that generate Tests for a specific aspect of the data or model performance.

Evidently also automatically generates Test conditions in two ways:

* **Based on the reference dataset**. If you provide a `reference`, Evidently derives conditions from it. For example, the `TestShareOfOutRangeValues` will fail if over 10% of `current` values fall outside the min-max range seen in the reference. 10% is an encoded heuristic.
* **Based on heuristics**. Without a reference, Evidently uses heuristics. For example, `TestAccuracyScore()` fails if the model performs worse than a dummy model created by Evidently. Data quality Tests like `TestNumberOfEmptyRows()` or `TestNumberOfMissingValues()` assume both should be zero.

{% hint style="info" %}
**Reference**: Check the default Test conditions in the [All tests](/reference/all-tests) table.
{% endhint %}

**Example 1**. To apply the `DataQualityTestPreset` to a single `curr` dataset, with conditions generated based on heuristics:

```python
data_quality = TestSuite(tests=[
    DataQualityTestPreset(),
])

data_quality.run(reference_data=None, 
                   current_data=curr)
```

{% hint style="info" %}
**Available Test Presets**. There are others: for example, `DataStabilityTestPreset`, `DataDriftTestPreset` or `RegressionTestPreset`. See all [Presets](/presets/all-presets). For interactive preview, check [example notebooks](/tutorials-and-examples/examples).
{% endhint %}

To get the visual report with Test results, call the object in Jupyter notebook or Colab:

```python
data_quality
```

To get the Test results summary, generate a Python dictionary:

```python
data_quality.as_dict()
```

{% hint style="info" %}
**There are more output formats!** You can also export the results in formats like HTML, JSON, dataframe, and more. Refer to the [Output Formats](/user-guide/tests-and-reports/output_formats) for details.
{% endhint %}

**Example 2**. To apply the `DataStabilityTestPreset`, with conditions generated from reference, pass the `reference_data`:

```python
data_stability = TestSuite(tests=[
    DataStabilityTestPreset(),
])

data_stability.run(reference_data=ref, 
                   current_data=curr)
```

**Example 3**. To apply the `NoTargetPerformanceTestPreset` with additional parameters:

```python
no_target_performance = TestSuite(tests=[
    NoTargetPerformanceTestPreset(columns=['education-num', 'hours-per-week'], 
                                  cat_stattest=ks, 
                                  cat_statest_threshold=0.05),
])
```

By selecting specific columns for the Preset, you reduce the number of generated column-level Tests. When you specify the data drift detection method and threshold, it will override the defaults.

{% hint style="info" %}
Refer to the [All tests](/reference/all-tests) table to see available parameters and defaults for each Test and Test Preset.
{% endhint %}

## Custom Test Suite

You can use Presets as a starting point, but eventually, you'll want to design a Test Suite to pick specific Tests and set conditions more precisely. Here’s how:

* **Choose individual Tests**. Select the Tests you want to include in your Test Suite.
* **Pass Test parameters**. Set custom parameters for applicable Tests. (Optional).
* **Set custom conditions**. Define when Tests should pass or fail. (Optional).
* **Mark Test criticality**. Mark non-critical Tests to give a Warning instead of Fail. (Optional).

### 1. Choose tests

First, decide which Tests to include. Tests can be either dataset-level or column-level.

{% hint style="info" %}
**Reference**: see [All tests](/reference/all-tests) table. To see interactive examples, refer to the [Example notebooks](/tutorials-and-examples/examples).
{% endhint %}

{% hint style="info" %}
**Row-level evaluations**: To Test row-level scores for text data, read more about [Text Descriptors](/user-guide/tests-and-reports/text-descriptors).
{% endhint %}

**Dataset-level Tests**. Some Tests apply to the entire dataset, such as checking the share of drifting features or accuracy. To add them to a Test Suite, create a `TestSuite` object and list the `tests` one by one:

```python
data_drift_suite = TestSuite(tests=[
    TestShareOfDriftedColumns(),
    TestNumberOfEmptyRows(),
])
```

**Column-level Tests**. Some Tests focus on individual columns, like checking if a specific column's values stay within a range. To include column-level Tests, pass the name of the column to each Test:

```python
feature_suite = TestSuite(tests=[
    TestColumnShareOfMissingValues(column_name='hours-per-week'),
    TestColumnDrift(column_name='education'),
    TestMeanInNSigmas(column_name='hours-per-week')
])
```

{% hint style="info" %}
**Generating many column-level Tests**: To simplify listing many Tests at once, use the [generator helper function](/user-guide/tests-and-reports/test-metric-generator).
{% endhint %}

**Combining Tests**. You can combine column-level and dataset-level Tests in a single Test Suite. You can also include Presets and individual Tests together.

```python
my_data_quality_tests = TestSuite(tests=[
    DataQualityTestPreset(),
    TestColumnAllConstantValues(column_name='education'),
    TestNumberOfDriftedColumns()
])
```

### 2. Set Test parameters

Tests can have optional or required parameters.

**Example 1**. To test a quantile value, you need to specify the quantile (Required parameter):

```python
column_tests_suite = TestSuite(tests=[
    TestColumnQuantile(column_name='mean perimeter', quantile=0.25),
])
```

**Example 2:** To override the default drift detection method, pass the chosen statistical method (Optional), or modify the Mean Value Test to use 3 sigmas:

```python
dataset_suite = TestSuite(tests=[
    TestShareOfDriftedColumns(stattest=psi),
    TestMeanInNSigmas(column_name='hours-per-week', n_sigmas=3),
])
```

**Example 3:** To change the decision threshold for probabilistic classification to 0.8:

```python
model_tests = TestSuite(tests=[
    TestPrecisionScore(probas_threshold=0.8),
    TestRecallScore(probas_threshold=0.8)
])
```

{% hint style="info" %}
**Reference**: you can browse available Test parameters and defaults in the [All tests](/reference/all-tests) table.
{% endhint %}

### 3. Set Test conditions

You can set up your Test conditions in two ways:

* **Automatic**. If you don’t specify individual conditions, the defaults (reference or heuristic-based) will apply, just like in Test Presets.
* **Manual**. You can define when exactly a Test should pass or fail. For example, set a lower boundary for the expected model precision. If the condition is violated, the Test fails.

You can mix both approaches in the same Test Suite, where some Tests run with defaults and others with custom conditions.

Use the following parameters to set Test conditions:

| Condition      | Explanation                                                      | Usage Example                                |
| -------------- | ---------------------------------------------------------------- | -------------------------------------------- |
| `eq: val`      | <p>equal<br><code>test\_result == val</code></p>                 | `TestColumnValueMin("col", eq=5)`            |
| `not_eq: val`  | <p>not equal<br><code>test\_result != val</code></p>             | `TestColumnValueMin("col", not_eq=0)`        |
| `gt: val`      | <p>greater than<br><code>test\_result > val</code></p>           | `TestColumnValueMin("col", gt=5)`            |
| `gte: val`     | <p>greater than or equal<br><code>test\_result >= val</code></p> | `TestColumnValueMin("col", gte=5)`           |
| `lt: val`      | <p>less than<br><code>test\_result < val</code></p>              | `TestColumnValueMin("col", lt=5)`            |
| `lte: val`     | <p>less than or equal<br><code>test\_result <= val</code></p>    | `TestColumnValueMin("col", lte=5)`           |
| `is_in: list`  | `test_result ==` one of the values                               | `TestColumnValueMin("col", is_in=[3,5,7])`   |
| `not_in: list` | `test_result !=` any of the values                               | `TestColumnValueMin("col", not_in=[-1,0,1])` |

**Example 1**. To Test that no values are out of range, and less than (`lt`) 20% of values are missing:

```python
feature_level_tests = TestSuite(tests=[
TestShareOfOutRangeValues(column_name='hours-per-week', eq=0),
TestColumnShareOfMissingValues(column_name='education', lt=0.2),
])
```

**Example 2**. You can specify both the Test condition and parameters together.

In the example above, Evidently automatically derives the feature range from the reference. You can also manually set the range (e.g., between 2 and 10). The Test fails if any value is out of this range:

```python
feature_level_tests = TestSuite(tests=[
TestShareOfOutRangeValues(column_name='hours-per-week', left=2, right=10, eq=0),
])
```

**Example 3**. To Test that the precision and recall is over 90%, with a set decision for the classification model:

```python
model_tests = TestSuite(tests=[
    TestPrecisionScore(probas_threshold=0.8, gt=0.9),
    TestRecallScore(probas_threshold=0.8, gt=0.9)
])
```

#### Custom conditions with Approx

If you want to set an upper and/or lower limit to the value, you can use **approx** instead of calculating the value itself. You can set the relative or absolute range.

```python
approx(value, relative=None, absolute=None)
```

To use `approx`, first import this component:

```python
from evidently.tests.utils import approx
```

**Example 1**. Here is how you can set the upper boundary as 5+10%:

```python
lte=approx(5, relative=0.1)
```

**Example 2**. Here is how you can set the boundary as 5 +/-10%:

```python
eq=approx(5, relative=0.1)
```

### 4. Set Test criticality

By default, all Tests will return a `Fail` if the Test condition is not fulfilled. If you want to get a `Warning` instead, use the `is_critical` parameter and set it to `False`. Example:

```python
data_integrity_column_tests = TestSuite(tests=[
    TestColumnAllConstantValues(column_name='education', is_critical=False),
    TestColumnAllUniqueValues(column_name='education', is_critical=False),
])
```

Notebook example on setting Test criticality:

{% embed url="<https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_specify_test_criticality.ipynb>" %}


# Evaluate Text Data

How to run evaluations for text data with Descriptors.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

A Descriptor is a row-level score evaluating a specific characteristic of a text data. A simple example is text length.

Descriptors range from regular expressions and text statistics to ML- and LLM-based checks. For example, you can calculate the semantic similarity between two texts, or ask LLM to label responses as "relevant" or "not relevant".

You can use Descriptors in two ways:

* **In a Report**. This helps visualize and summarize the scores, like show text length across all texts.
* **In a Test Suite**. This checks if conditions are met, like if all texts are within a certain length (True/False).

{% content-ref url="/pages/mmkPjxYbEMNfDGWhKMk6" %}
[Reports and Tests Overview](/user-guide/tests-and-reports/introduction)
{% endcontent-ref %}

## Code examples

Using descriptors to evaluate LLM outputs using `TextEvals` Preset:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_evaluate_llm_with_text_descriptors.ipynb>" %}

Using descriptors with tabular Metrics and Tests:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_apply_table_metrics_and_tests_to_text_descriptors.ipynb>" %}

## Imports

After [installing Evidently](/setup/install-evidently), import the selected descriptors and the relevant components based on whether you want to generate Reports or run Tests.

```python
from evidently import ColumnMapping
from evidently.report import Report
from evidently.test_suite import TestSuite
from evidently.metric_preset import TextEvals
from evidently.metrics import ColumnSummaryMetric, ColumnDriftMetric 
from evidently.tests import TestColumnValueMin, TestColumnValueMean, TestCategoryShare, TestShareOfOutRangeValues
from evidently.descriptors import Contains, TextLength, Sentiment
```

**Note**. For some Descriptors that use vocabulary-based checks (like `IncludesWords` or `OOV` for out-of-vocabulary words), you may need to download `nltk` dictionaries:

```python
nltk.download('words')
nltk.download('wordnet')
nltk.download('omw-1.4')
nltk.download('vader_lexicon')
```

## How it works

Here is the general flow to run an evaluation:

* **Input data**. Prepare the data as a Pandas DataFrame. Include at least one text column. This will be your `current_data` to run evals on. Optionally, prepare the `reference` dataset.
* **Schema mapping**. Define your data schema using [Column Mapping](/user-guide/input-data/column-mapping). Optional, but highly recommended.
* **Define the Report or Test Suite**. Create a `Report` or a `TestSuite` object with the selected checks.
* **Run the Report**. Run the Report on your `current_data`, passing the `column_mapping`. Optionally, pass the `reference_data`.
* **Get the summary results**. Get a visual Report in Jupyter notebook, export the metrics, or upload it to Evidently Platform.
* **Get the scored datasets**. To see row-level scores, export the Pandas DataFrame with added descriptors. (Or view this on Evidently Platform).

{% hint style="info" %}
**Available Descriptors**. See Descriptors in the [All Metrics](/reference/all-metrics) page.
{% endhint %}

{% hint style="info" %}
**Reports and Test Suites**. For basic API, read how to [run Reports](/user-guide/tests-and-reports/get-reports) and [Test Suites](/user-guide/tests-and-reports/run-tests).
{% endhint %}

### Text Evals

For most cases, we recommend using the `TextEvals` Preset. It provides an easy way to create a Report that summarizes Descriptor values for a specific column.

**Basic example**. To evaluate the Sentiment and Text Length in symbols for the `response` column:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        Sentiment(),
        TextLength(),
    ]),
])
```

Run the Report on your DataFrame `df`:

```python
report.run(reference_data=None, 
           current_data=df)
```

You can access the Report just like usual, and export the results as HTML, JSON, a Python dictionary, etc. To view the interactive Report directly in Jupyter Notebook or Colab:

```python
report 
```

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e3f5b445e733759c0a06933ff8edb6767ed3af2e%2Fllm_report_preview-min.gif?alt=media)

You can add the Descriptors to your original dataset. To view the DataFrame:

```
report.datasets().current
```

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8bbe8a33495fa10d2cd047c61730d32f6d65cd1e%2Fdescriptors_export.png?alt=media)

To create a DataFrame:

```
df_with_scores = pd.DataFrame(report.datasets().current)
```

{% hint style="info" %}
**How to get the outputs**. Check the details on all available [Output Formats](/user-guide/tests-and-reports/output_formats).
{% endhint %}

**Display name**. It’s a good idea to add a `display_name` to each Descriptor. This name shows up in visualizations and column headers. It’s especially handy if you’re using checks like regular expressions with word lists, where the auto-generated title could get very long.

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        Sentiment(display_name="Response sentiment"),
        TextLength(display_name="Response length"),
    ]),
])
```

**Evaluations for multiple columns**. If you want to evaluate several columns, like "response" and "question", just list multiple Presets in the same Report and include the Descriptors you need for each one.

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        Sentiment(),
        TextLength(),
    ]),
    TextEvals(column_name="question", descriptors=[
        Sentiment(),
        TextLength(),
    ])
])
```

**Descriptor parameters**. Some Descriptors have required parameters. For example, if you’re testing for competitor mentions using the `Contains` Descriptor, you must include the names in the `items` list:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        Contains(display_name="Competitor Mentions", 
                items=["AcmeCorp", "YetAnotherCorp"]),
    ]),
])
```

**Multi-column descriptors**. Some Descriptors like `SemanticSimilarity` require a second column. Pass it as a parameter:

```python
report = Report(metrics=[
    TextEvals(column_name="question", descriptors=[
        SemanticSimilarity(with_column="response")
    ]),
])
```

Some Descriptors, like custom LLM judges, might require a more complex setup, but you can still include them in the Report just like any other Descriptor.

{% hint style="info" %}
**Reference**. To see the Descriptor parameters, check the [All Metrics](/reference/all-metrics) page.
{% endhint %}

{% hint style="info" %}
**LLM-as-a-judge**. For a detailed guide on setting up LLM-based evals, check the guide to [LLM as a jugde](/user-guide/customization/llm_as_a_judge).
{% endhint %}

{% hint style="info" %}
**Custom descriptors**. You can implement descriptors as Python functions. Check the [guide on custom descriptors](/user-guide/customization/add-custom-descriptor).
{% endhint %}

### Using Metrics

The `TextEvals` Preset works by generating a `ColumnSummaryMetric` for each Descriptor you calculate. You can achieve the same results by explicitly creating this Metric for each Descriptor:

```python
report = Report(metrics=[
    ColumnSummaryMetric(TextLength().on("response")),
    ColumnSummaryMetric(Sentiment().on("response")),
])
```

For two-column descriptor like `SemanticSimilarity()`, pass both columns as a list:

```python
report = Report(metrics=[
    ColumnSummaryMetric(column_name=SemanticSimilarity(with_column="question").on("response"))
])
```

**Text Descriptor Drift detection**. Sometimes, you might want to use a different Metric, like `ColumnDriftMetric`. Here is how to do this:

```python
report = Report(metrics=[
   ColumnDriftMetric(column_name = TextLength().on("response")),
])
```

In this case, you’ll need to pass both `reference` and `current` datasets. The Metric will compare the distribution of "response" Text Length in the two datasets and return a drift score.

You can use other column-level Metrics this way:

```python
report = Report(metrics=[
    ColumnSummaryMetric(TextLength().on("response")),
    ColumnDriftMetric(TextLength().on("response")),
    ColumnCorrelationsMetric(TextLength().on("response")),
    ColumnDistributionMetric(TextLength().on("response")),
    ColumnValueRangeMetric(TextLength().on("response"), left=0, right=20)
])
```

However, in most cases, it's better to first generate a DataFrame with the scores through `TextEvals`. You can then run evaluations on the new dataset by referencing the newly added column directly.

### Run Tests

You can also run Tests with text Descriptors to verify set conditions and return a Pass or Fail result.

**Example 1**. To test that the average response sentiment is greater or equal (`gte`) to 0, and that the maximum text length is less than or equal (`lte`) to 200 characters:

```python
test_suite = TestSuite(tests=[
    TestColumnValueMean(column_name = Sentiment().on("response"), gte=0),
    TestColumnValueMax(column_name = TextLength().on("response"), lte=200),
])
```

**Example 2**. To test that the number of responses mentioning competitors is zero:

```python
test_suite = TestSuite(tests=[
    TestCategoryCount(
        column_name=Contains(
            items=["AcmeCorp", "YetAnotherCorp"],
            display_name="Competitor Mentions").
        on("new_response"),
        category=True,
        eq=0),
])
```

**Example 3**. To test that Semantic similarity between two columns is greater or equal to 0.9:

```python
test_suite = TestSuite(tests=[
    TestColumnValueMin(
        column_name=SemanticSimilarity(
        display_name="Response Similarity",
        with_column="target_response").
        on("new_response"),
        gte=0.9),
])
```

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-f381178bad9607b6b3a083305be2567c08a89604%2Ftest_descriptor_example.png?alt=media)

**Available Tests**. You can use any column-level Tests with Descriptors. Here are a few particularly useful:

For numerical Descriptors:

```python
test_suite = TestSuite(tests=[
    TestValueRange(column_name = TextLength().on("response")),
    TestNumberOfOutRangeValues(column_name = TextLength().on("response")),
    TestShareOfOutRangeValues(column_name = TextLength().on("response")),
    TestColumnValueMin(column_name = TextLength().on("response")),
    TestColumnValueMax(column_name = TextLength().on("response")),
    TestColumnValueMean(column_name = TextLength().on("response")), 
    TestColumnValueMedian(column_name = TextLength().on("response")),
])
```

In these examples, the Test conditions come from the `reference` dataset. You can also pass custom ones.

{% hint style="info" %}
**Test conditions.** See the list of [All tests](/reference/all-tests) with defaults. Learn how to set [custom Test conditions](/user-guide/tests-and-reports/run-tests).
{% endhint %}

For categorical Descriptors, use `TestCategoryCount` or `TestCategoryShare` Tests.

For example, to test if the share of responses that contain travel-related words is less than or equal to 20%:

```python
test_suite = TestSuite(tests=[
    TestCategoryShare(
        column_name=IncludesWords(words_list=['hotel', 'flight'], display_name="Travel Mentions").
        on("new_response"),
        category=True,
        lte=0.2),
])
```


# Output formats

How to export the results of evaluations.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

You can view or export results from Evidently Reports or Test Suites in multiple formats.

## View in Jupyter notebook

You can directly render the visual summary of evaluation results in interactive Python environments like Jupyter notebook or Colab.

After running the Report, simply call the resulting Python object:

```python
drift_report
```

This will render the HTML object directly in the notebook cell.

## HTML

You can also save this interactive visual report as an HTML file to open in a browser:

```python
drift_report.save_html(“file.html”)
```

This option is useful for sharing Reports with others or if you're working in a Python environment that doesn’t display interactive visuals.

## JSON

You can get the results of the calculation as a JSON. It is useful for storing and exporting results elsewhere.

To view the JSON in Python:

```python
drift_report.json()
```

To save the JSON as a separate file:

```python
drift_report.save_json("file.json")
```

## Python dictionary

You can get the output as a Python dictionary. This format is convenient for automated evaluations in data or ML pipelines, allowing you to transform the output or extract specific values.

To get the dictionary:

```python
drift_report.as_dict()
```

{% hint style="info" %}
**Inlcude/exclude**. Check how to [manage verbosity](/user-guide/customization/json-dict-output) of `json` or `as_dict` output.
{% endhint %}

## Scored DataFrame

If you generated text Descriptors during your evaluation, you can retrieve a DataFrame with all generated descriptors added to each row of your original input data.

```python
text_evals_report.datasets().current
```

This returns the complete original dataset with new scores.

## Evidently snapshot

You can save the output of a Report or Test Suite as an Evidently JSON `snapshot`.

{% hint style="info" %}
**How is a JSON snapshot different from `json()`?** A snapshot contains all supplementary and render data. This lets you restore the output in any Evidently format (like HTML) without accessing the initial raw data.
{% endhint %}

This is a rich JSON format used for storing the evaluation results on Evidently platform. When you save Reports or Test Suites to the platform, a snapshot is generated automatically. However, you can also generate and save a snapshot explicitly.

To save the Report as a snapshot:

```python
drift_report.save('snapshot.json')
```

To load the snapshot back, use the “load” function.

```
loaded_report = Report.load('snapshot.json')
```

After you load the snapshot back, you can again view it in Python or export it to other formats.

{% hint style="info" %}
**Generating snaphots**. Check how to [get snapshots](/user-guide/evaluations/snapshots) and upload the evaluation results to the Evidently Platform.
{% endhint %}

## DataFrame with a Report summary

**Note**: this export option is only supported for Reports, and not Test Suites.

You can get the Report results in a tabular format as a DataFrame.

To export results for a specific Metric:

```python
drift_report.as_dataframe("DataDriftTable")
```

To export results for the entire Report, which returns a dictionary of DataFrames:

```python
drift_report.as_dataframe()
```

This will return all relevant values that are computed inside the Metric as the metric result.


# Generate multiple Tests or Metrics

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Sometimes you need to generate multiple column-level Tests or Metrics.

To simplify it, you can:

* Pass a list of parameters or columns to a chosen Test or Metric
* Use test/metric generator helper functions

## List comprehension

You can pass a list of parameters/conditions or columns. It works the same for Tests and Metrics.

**Example 1**. Pass the list of quantile values to run multiple Tests for the same column.

```python
suite = TestSuite(tests=[
   TestColumnQuantile(column_name="education-num", quantile=quantile) for quantile in [0.5, 0.9, 0.99]
])

suite.run(current_data=current_data, reference_data=reference_data)
suite
```

**Example 2**. Apply the same Test with a defined custom condition for all columns in the list:

```python
suite = TestSuite(
    tests=[
        TestColumnValueMin(column_name=column_name, gt=0) 
        for column_name in ["age", "fnlwgt", "education-num"]
    ]
)

suite.run(current_data=current_data, reference_data=reference_data)
suite
```

## Column test generator

You can also use the `generate_column_tests` function to create multiple Tests.

**Example 1.** Generate the same Test for all the columns in the dataset. It will use defaults if you do not specify the test condition.

```python
suite = TestSuite(
    tests=[generate_column_tests(TestColumnShareOfMissingValues)]
)
```

You can also pass a custom Test condition:

```python
suite = TestSuite(
    tests=[
        generate_column_tests(
            TestColumnShareOfMissingValues, 
            columns="all", 
            parameters={"lt": 0.5}
        )
    ]
)
```

**Example 2.** You can generate Tests for different subsets of columns. Here is how you generate tests only for **numerical columns**:

```python
suite = TestSuite(
    tests=[
        generate_column_tests(
            TestColumnValueMin, 
            columns="num"
        )
    ]
)
```

Here is how you generate tests only for **categorical columns**:

```python
suite = TestSuite(
    tests=[
        generate_column_tests(
            TestColumnShareOfMissingValues, 
            columns="cat", 
            parameters={"lt": 0.1}
        )
    ]
)
```

You can also generate Tests with a certain condition for a **defined column list**:

```python
suite = TestSuite(
    tests=[
        generate_column_tests(
            TestColumnValueMin, 
            columns=["age", "fnlwgt", "education-num"],
            parameters={"gt": 0}
        )
    ]
)
```

### Column parameter

You can use the parameter `columns` to define a list of columns to which you apply the tests. If it is a list, just use it as a list of the columns. If `columns` is a string, it can take the following values:

* `"all"` - apply tests/metrics for all columns, including target/prediction columns.
* `"num"` - for numerical features, as provided by column mapping or defined automatically
* `"cat"` - for categorical features, as provided by column mapping or defined automatically
* `"features"` - for all features, excluding the target/prediction columns.
* `"none"` - the same as "all."

## Column metric generator

It works the same way for metrics. In this case, you should use `generate_column_metrics` function.

To generate multiple metrics for all the columns in the list with a custom parameter.

```python
metric_generator_report = Report(
    metrics=[
        generate_column_metrics(
            ColumnValueRangeMetric,
            columns=['mean radius', 'mean texture', 'mean perimeter'],
            parameters={"left": 5, "right": 25}
        )
    ]
)
metric_generator_report.run(current_data=bcancer_ref, reference_data=bcancer_cur)
metric_generator_report
```


# Run Evidently on Spark

How to run calculations on Spark.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

You can run distributed computation using Spark if you work with large datasets.

## Supported metrics

Currently, the following Tests, Metrics and Presets are supported:

* `ColumnDriftMetric()`
* `DataDriftTable()`
* `DatasetDriftMetric()`
* `DataDriftPreset()`
* `TestColumnDrift()`
* `TestShareOfDriftedColumns()`
* `TestNumberOfDriftedColumns()`
* `DataDriftTestPreset()`

For drift calculation, the following methods are supported:

* `chisquare`
* `jensen shannon`
* `psi`
* `wasserstein`

The following data types are supported:

* `numerical_features`
* `categorical_features`

## Code example

You can refer to an example How-to-notebook showing how to use Evidently on Spark:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_run_calculations_on_spark.ipynb>" %}

## Run Evidently with Spark

To run Evidently on a Spark DataFrame, you need to specify the corresponding engine in the `run()` method for the Report calculation:

To import `SparkEngine` from Evidently, use the following command:

```
from evidently.spark.engine import SparkEngine
```

Pass the `SparkEngine` to the `run` method when you create the Report:

```
spark_report_table = Report(metrics=[
    DataDriftTable()
])
spark_report_table.run(reference_data=reference, current_data=current, engine=SparkEngine)

spark_report_table.show()  # OR spark_report_table.show()
```


# Evaluations


# Evaluations overview

How to run evaluation workflows in Evidently Platform.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This documentation section covers how to run evals, whether you're using code or a no-code interface. It also shows you how to explore, compare, and track evaluation results in the Evidently Platform UI.

{% hint style="info" %}
**Looking for something else?** For details on the Python evaluation workflow, check the [Reports and Test Suites](/user-guide/tests-and-reports/introduction). For online evaluations, check [Monitoring](/user-guide/monitoring/monitoring_overview). To check **what** you can evaluate, browse [Presets](/presets/all-presets), [Metrics](/reference/all-metrics) and [Tests](/reference/all-tests).
{% endhint %}

## When you need evals

You need evals at different stages of your AI product development:

| When You Need Evals                                                                                                                                              | Example Questions                                                                                                                                                                                                           |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Ad-hoc analysis**. You may run evaluations on the fly whenever you need to troubleshoot an issue or understand a specific trend.                               | <ul><li>What are the predictions with the highest error?</li><li>How often do particular topics come up?</li><li>How does my system perform on adversarial inputs?</li></ul>                                                |
| **Experimenting**. During development, you may test different parameters, models, or prompts. Evaluations help you compare outcomes and iterate with confidence. | <ul><li>Does the classification precision and recall improve with iterations?</li><li>Which prompt delivers more accurate answers?</li><li>Does switching from GPT to Claude enhance the quality of retrieval?</li></ul>    |
| **Regression Testing**. When you update a model or make a fix, you need to evaluate its quality on new or previous inputs, often as part of CI/CD pipelines.     | <ul><li>Does changing the prompt lead to different answers to previous user queries?</li><li>What is the quality of my ML model after retraining it on new data?</li></ul>                                                  |
| **Online evaluations and monitoring**. Once in production, you can automatically assess the live quality of your AI system using incoming data.                  | <ul><li>Is the environment (input features) changing? Are predictions shifting?</li><li>How professional and concise were the chatbot’s answers today?</li><li>Is version A or B performing better in production?</li></ul> |

Evidently supports all these workflows.

## Evaluation workflow

You perform evaluations by generating [Reports or Test Suites](/user-guide/tests-and-reports/introduction) using one of these methods:

* Local evaluations using the Python library
* No-code evaluations directly in the UI

### Evals in Python

{% hint style="success" %}
Supported in: `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

This is perfect for development, CI/CD workflows, or custom evaluation pipelines. Once you run an eval in Python on your dataset, you upload the results to the Evidently Platform.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-29f96cf57cca6885ee22f193a202da485f88390a%2Fevals_flow_python.png?alt=media)

You get to choose what you upload:

* **Results only (Default)**. You can upload just the evaluation outcomes as JSON snapshots. They include column summaries, metric distributions, and test results. For tabular data, this typically provides the necessary insights while keeping data private.
* **Results and Dataset**. Alternatively, you can include the dataset along with the evaluation results. This is useful for text data evaluations or whenever you want detailed row-level analysis, like investigating specific completions with low scores.

{% hint style="info" %}
**What’s a snapshot?** This is a rich JSON version of the Report or a Test Suite with the results of an individual evaluation run that you upload to the platform.
{% endhint %}

See how to run local evals in detail:

{% content-ref url="/pages/0DSrUUOdBwyB3NsyVNPf" %}
[Generate snapshots](/user-guide/evaluations/snapshots)
{% endcontent-ref %}

### No-code evals

{% hint style="success" %}
Supported in: `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

With no-code evaluations, you work directly in the user interface. This is great for non-technical users or when you prefer to run evaluations on Evidently infrastructure.

Here's what you can do:

* **Evaluate uploaded datasets**. Run evaluations on collected [traces](/user-guide/tracing/tracing_overview) (if you've instrumented your LLM application) or on [Datasets](/user-guide/datasets/datasets_overview) you previously uploaded.
* **Upload CSV data**. Use a drag-and-drop interface to upload CSV files and run evaluations entirely on the Platform.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2daadfe4e61ae0daaad37b202ee55317eee674fd%2Fevals_flow_nocode.png?alt=media)

Once you run the evaluation using a no-code flow, you create the same Report or Test Suite that you would generate using Python.

How to run No-code evals:

{% content-ref url="/pages/M2sZy688jBFycTHdbFgD" %}
[Run no code evals](/user-guide/evaluations/no_code_evals)
{% endcontent-ref %}

The rest of the workflow is the same. After you run your evals with any method, you can access the results in the UI, and go to the Explore view for further analysis.&#x20;

## Evaluation results&#x20;

{% hint style="success" %}
Supported in: Supported in: `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

The result of each evaluation is either a Report (when it's just a summary of metrics) or a Test Suite (when it also includes pass/fail results on set conditions).

**Browse the results**. To access them, enter your Project and navigate to the "Reports" or "Test Suites" section in the left menu. Here, you can view all your evaluation artifacts and browse them by Tags, time, or metadata. You can also download them as HTML or JSON.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-4e12a9f57e9dfeee381743efa9708d4eaf80b2ee%2Fbrowse_reports-min.png?alt=media)

To see and compare the evaluation results, click on "Explore" next to the individual Report or Test Suite.&#x20;

**Explore view**. You'll get the Report or Test Suite and, if available, the dataset linked to the evaluation.

{% hint style="success" %}
Supported in: `Evidently Cloud` and `Evidently Enterprise`. In Evidently OSS, you can can access only the Report.
{% endhint %}

* To view the Report only, click on the "dataset" sign at the top to hide the dataset.
* To see results from a specific evaluation within the Report, use the dropdown menu to select the Metric.
* To compare Reports side by side, click on "duplicate snapshot" (this will keep the current Metric in view), and then select a different Report for comparison.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3c786cced77562d7baffa100636ce6e30297e62a%2Fexplore_view-min.png?alt=media)

**Dashboard**. As you run multiple evaluations, you can build a Dashboard to visualize results over time.

{% hint style="success" %}
Supported in: Supported in: `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

The Dashboard aggregates data from various Reports or Test Suites within a Project, allowing you to track progress, see performance improvements, and monitor how tests perform over time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2375a068033618f6abee93161c20929eb582b40a%2Fproject_dashboard-min.png?alt=media)

How to create a Dashboard:

{% content-ref url="/pages/AoVr9uW2K4ASJFRnsT2V" %}
[Dashboard overview](/user-guide/dashboard/dashboard_overview)
{% endcontent-ref %}


# Generate snapshots

How to run evals in Python and upload to Evidently Platform.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This page walks you through how to run evaluations locally in Python and send the results to Evidently Platform. This applies both to on-the-fly evaluations during experiments and to those you run automatically during batch monitoring or regression testing.

Once you upload the evaluation results as JSON `snapshots`, you can explore, compare, and track them on the Evidently Platform.

## What is snapshot?

A `snapshot` is a JSON summary containing evaluation results. It captures the data and AI system performance for the specific dataset or data batch you evaluated. Snapshots can include metrics, test results, column summaries, and additional render data. You choose what to include when running your evaluation.

The snapshot functionality is based on Evidently [Reports and Test Suites](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/overview.md). Put simply, a snapshot is a JSON "version" of an Evidently Report or Test Suite.

When you run individual evaluations, you can explore and compare their results. As you send multiple snapshots to a Project, you can also use a Dashboard to track results over time. This helps you monitor metric changes across experiments or track evaluations on production data.

You can optionally include the Dataset together with the evaluation results you upload.

## How it works

Here is the general workflow.

**1. Create or connect to a** [**Project**](/user-guide/projects/add_project) in your Workspace where you want to send the snapshots. This will organize all your evaluations in one place.

```python
project = ws.get_project("PROJECT_ID")
```

**2. Prepare the data**. You run each evaluation on a Dataset.

You can prep your input data locally as a Pandas DataFrame or first upload it to the Evidently Platform and call it from there.

{% hint style="info" %}
**Working with data**. Check how to prepare your [input data](/user-guide/input-data/data-requirements) or upload and manage [Datasets](/user-guide/datasets/datasets_overview).
{% endhint %}

**3. Define the snapshot compostion**. Define what you want to evaluate.

* Create a `Report` or `Test Suite` object.
* Pass the chosen `metrics` or `tests`.
* Optionally, pass custom parameters for Metric calculations and/or Test conditions.

{% hint style="info" %}
**Reports and Tests**. Check how to get [Reports](/user-guide/tests-and-reports/get-reports), run [Test Suites](/user-guide/tests-and-reports/run-tests) or generate [Text Descriptors](/user-guide/tests-and-reports/text-descriptors).
{% endhint %}

**4. Run the Report or Test Suite**. Execute the evaluation on your dataset.

* Pass the `current` dataset you want to evaluate or profile.
* Optional (but highly recommended): pass the `column_mapping` to define the data schema.
* Optional (required for data distribution checks): pass the `reference` dataset.
* Optional: add a `tags` or `metadata` to identify this specific evaluation run.
* Optional: and a custom `timestamp` to the current run.

**5. Send the snapshot**.

After you compute the Report or Test Suite, use the `add_report` or `add_test_suite` methods to send them to a corresponding Project in your workspace.

## Examples

### Send snapshots

**Report**. To create and send a Report with data summaries for a single dataset `batch1` to the workspace `ws`:

```python
data_report = Report(
      metrics=[
          DataQualityPreset(),
      ],
   )
data_report.run(reference_data=None, current_data=batch1)
ws.add_report(project.id, data_report)
```

**Test Suite**. To create and send a Test Suite with data drift checks, passing current and reference data:

```python
drift_checks = TestSuite(tests=[
  DataDriftTestPreset(),
])
drift_checks.run(reference_data=reference_batch, current_data=batch1)
ws.add_test_suite(project.id, drift_checks)
```

**Send a snapshot**. The `add_report` or `add_test_suite` methods generate snapshots automatically. If you already have a snapshot (e.g., you previously saved it), you can load it load and send it to your Project:

```python
ws.add_snapshot(project.id, snapshot.load("data_drift_snapshot.json"))
```

{% hint style="info" %}
**Snapshot size**. A single upload to Evidently Cloud should not exceed 50MB (Free plan) or 500MB (Pro plan). This limitation applies to the size of the JSON, not the dataset itself. Example: a data drift report for 50 columns and 10,000 rows of current and reference data results in a snapshot of approximately 1MB. (For 100 columns x 10,000 rows: \~ 3.5MB; for 100 columns x 100,000 rows: \~ 9MB). The size varies depending on the metrics or tests used.
{% endhint %}

### Add dataset

When you upload a Report or Test Suite, you can optionally include the Dataset you evaluated, together with added Descriptors (if any). This helps with row-level debugging and analysis.

Use the `include_data` parameters (defaults to False):

```python
ws.add_report(project.id, data_report, include_data=True)
```

### Add timestamp

Each `snapshot` is associated with a single timestamp. By default, Evidently will assign the `datetime.now()` using the Report/Test Suite computation time based on the user time zone.

You can also add your own timestamp. Pass it to the run method when you compute the snapshot:

```python
drift_checks.run(
    reference_data=reference_batch,
    current_data=batch1,
    timestamp=datetime.now()
)
```

Since you can assign arbitrary timestamps, you can log snapshots asynchronously or with a delay (for example, after you receive ground truth) and assign it to the specific period.

### Add tags and metadata

You can include `tags` and `metadata` in snapshots. This is useful for search and data filtering. By adding **Tags**, you can then visualize data only from a specific subset of your snapshots on a monitoring Panel.

Examples of when to use tags include:

* You want to identify a specific evaluation run or group of experiments by model version or test scenario.
* You are logging data on production/shadow, champion/challenger, or A/B model versions.
* You compute snapshots with different reference datasets (for example, to compare distribution drift week-by-week and month-by-month).
* You have data for multiple models of the same type inside a Project.
* You capture snapshots for multiple segments in your data.
* You want to tag individual Reports, e.g., a datasheet card, a model card, etc.

**Custom tags**. Pass any custom Tags as a list:

```python
data_drift_report = Report(
	metrics=[
	DatasetSummaryMetric().
	],
	tags=["groupA", "shadow"],
)
```

**Custom metadata**. Pass metadata as a Python dictionary in key:value pairs:

```python
data_drift_report = Report(
	metrics=[
	DatasetSummaryMetric(),
	],
	metadata = {
	"deployment": "shadow",
	"status": "production",
	}
)
```

**Default metadata**. Use built-in metadata fields `model_id`, `reference_id`, `batch_size`, `dataset_id`:

```python
data_drift_report = Report(
	metrics=[
	DatasetSummaryMetric(),
	],
	model_id=model_id,
	reference_id=reference_id,
	batch_size=batch_size,
	dataset_id=dataset_id,
)
```

**Add Tags to existing Reports.**. You can add Tags to a previously generated Report or Test Suite:

```python
data_summary_report.tags=["training_data"]
```

## Delete snapshots

To delete snapshots in the Workspace `ws`, pass the Project ID and snapshot ID. You can see the snapshot ID on the Report or Test Suite page.

```python
ws.delete_snapshot(project_id, snapshot_id)
```

## What's next?

Once you've sent data to the Project, you can [add monitoring Panels and Tabs](/user-guide/dashboard/design_dashboard).


# Run no code evals

How to evaluate your data in a no-code interface.

{% hint style="info" %}
**You are looking at the old Evidently documentation**. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

The platform supports several evaluations directly from the user interface.

| Name                   | Type       | Description                                                   |
| ---------------------- | ---------- | ------------------------------------------------------------- |
| Text Evals             | Report     | Analyze text data, from regular expressions to LLM judges.    |
| Data Quality           | Report     | Get descriptive statistics and distributions for all columns. |
| Classification Quality | Report     | Evaluate the quality of a classification model.               |
| Regression Quality     | Report     | Evaluate the quality of a regression model.                   |
| Data Quality Tests     | Test Suite | Automatically check for missing values, duplicates, etc.      |

Before you start, pick a dataset to evaluate. For example, this could be a CSV file containing inputs and outputs of your AI system, like chatbot logs.

## 1. Create a Project

On the main page of Evidently Platform, click “Create new Project.” Give your Project a title and description to help organize your evaluations.

{% hint style="info" %}
**What is a Project?** It's a way to organize your evals and datasets together. Check the [Project](/user-guide/projects/add_project) docs.
{% endhint %}

## 2. Start an evaluation

Inside the Project, go to the “Evaluate” menu item on the left. Let's start!

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-4b7fd94d222153ec790c437fd73ff4d22c98a152%2Fnocode_start_eval-min.png?alt=media)

**(Optional) Add Open AI key.** Add a token in the “Secrets” menu section if you plan to use an LLM for evaluations. You can proceed without it, using other types of evals.

## 3. Pass the Dataset

There are two options:

* **Upload a CSV**. Drag and drop your Dataset.
* **Use an existing Dataset**. Select a Dataset you previously uploaded to the platform or one collected through [Tracing](/user-guide/tracing/tracing_overview).

Your dataset can have any structure, but certain evaluations require specific columns:

* **Text evaluations**: make sure there’s at least one text column.
* **Regression and classification**: include columns with predictions and ground truth.
* **Data quality**: any dataset with categorical, numerical or text columns.

{% hint style="info" %}
**What are Datasets?** Learn how to manage and upload [Datasets](/user-guide/datasets/datasets_overview) to the platform.
{% endhint %}

## 4. Column Mapping

Map your data schema. You don't have to fill in all the options: choose what applies. For example, for text evals, point to the columns containing text.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0140c01cb4df98cde091dc441d3c797111e0a714%2Fnocode_column_mapping-min.png?alt=media)

If you have a timestamp, map it as a "DateTime." You will get additional plots with score changes in time.

{% hint style="info" %}
**What is Column mapping?** Understand how to map your dataset schema in the [Column Mapping](/user-guide/input-data/column-mapping) docs section.
{% endhint %}

## 5. Configure the evaluation

Some evaluations (Classification, Regression, and Data Quality) run as is. Once you passed the data, hit "Run" to get your Report.

{% hint style="info" %}
**What is the Raw Data option?** It keeps raw data points in plots like error time series. This is relevant for small datasets: generally leave it off.
{% endhint %}

Text Evals require a bit more setup. You must choose which column to evaluate and how. You can choose from the following methods:

* **Model-based**: use built-in machine learning models, like sentiment analysis.
* **Regular expressions**: check for specific words or patterns.
* **Text stats**: measure stats like the number of symbols or sentences.
* **LLM-based**: use external LLMs to evaluate your text data.

Select specific checks one by one:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3b5197321281b9528fd3d583d858d1dc0012e780%2Fnocode_choose_evals-min.png?alt=media)

Each evaluation result is called a **Descriptor**. No matter the method, you’ll get a label or score for every evaluated text. Some, like “Sentiment,” work instantly, while others may need setup.

{% hint style="info" %}
**What other evaluators are there?** Check the list of Descriptors on the [All Metrics](/reference/all-metrics) page.
{% endhint %}

Here are few examples of Descriptors and how to configure them:

### Words presence

**Include Words**. This Descriptor checks for listed words and returns "True" or "False."

Set up these parameters:

* Add a list of words.
* Choose whether to check for “any” or “all” of the words present.
* Set the **lemmatize** parameter to check for inflected and variant words automatically.
* Give your check a name so you can easily find it in your results.

Example setup:![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-33db81282f0082a631fbc36a41778d8e9cdc095b%2Fnocode_includes_words-min.png?alt=media)

### Semantic Similarity

**Semantic Similarity**. This descriptor converts texts to embeddings and calculates Cosine Similarity between your evaluated column and another column. It scores from 0 to 1 (0: completely different, 0.5: unrelated, 1: identical). It's useful for checking if responses are semantically similar to a question or reference.

Select the column to compare against: ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3a9309a6e9dc7262b96ad056583e18116a541300%2Fnocode_semantic_similarity-min.png?alt=media)

### LLM as a judge

**Custom LLM evaluator**. If you've added your token, use LLM-based evals (built-in or custom) to send your texts to LLMs for grading or scoring. You can choose a specific LLM model from the provider.

For example, you can create a custom evaluator to classify texts as “cheerful” or “neutral.” Fill in the parameters, and Evidently will generate the evaluation prompt:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2d65c9a488964a96b9a4e5724d683547e5d35af9%2Fnocode_llm_judge-min.png?alt=media)

For a binary classification template, you can configure:

* **Criteria**: define custom criteria in free text to clarify the classification task.
* **Target/Non-target Category**: labels you want to use.
* **Uncertain Category**: how the model should respond when it can’t decide.
* **Reasoning**: choose to include explanation (Recommended).
* **Category** and/or **Score**: have the LLM respond with the category (Recommended) or score.
* **Visualize as**: when both Category and Score are computed, choose which to display in the Report.

{% hint style="info" %}
**What other evaluators are there?** Check the list of Descriptors on the [All Metrics](/reference/all-metrics) page.
{% endhint %}

To add evaluations for another column in the same Report, click “Add Preset,” select “Text Evals,” and follow the same steps for the new column. You can include evals for multiple columns at once.

## 6. Run the evaluation

Before generating the Report, you can add a Tag to identify or reference the Report later. It's optional.

Click “Run”, and the calculation will start! It may take some time to process, especially on a large dataset.

## 7. View the results

Once your evaluation is complete, you can view the results. You can either

* view the Dataset with the added scores, or
* use the **Explore** view to see the Dataset with a summary Report (recommended).

If you have many Metrics, the Explore page will initially show the long Report. Use the selector at the top to focus on a specific Metric.

Here’s what the Explore page might look like with the results from the Cheerfulness evaluator for the “Answer” column:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2acc277ccfbb8d716920107b1bf4427abb9120bc%2Fnocode_judge_result-min.png?alt=media)


# Monitoring


# Monitoring overview

How to set up online evaluations and monitoring.

{% hint style="info" %}
**You are looking at the old Evidently documentation**. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

AI quality monitoring automatically evaluates your AI application’s inputs and outputs. This helps you spot and fix issues while keeping an up-to-date view of your system behavior.

{% hint style="info" %}
**New to AI quality evaluations?** Start with individual evaluations first. Read more on [evaluations](/user-guide/evaluations/evals_overview).
{% endhint %}

Evidently offers several ways to set up monitoring. Consider the following:

* **Type of AI application**. Do you work with simple tabular data or need to capture complex LLM interactions?
* **Batch or real-time**. Does your AI system process data in batches, or does it generate predictions live?
* **Evaluation frequency**. How often do you need to check the system's performance? Is real-time monitoring necessary, or can you evaluate at intervals like hourly or daily?
* **Where to store prediction logs**. Do you want to store raw data (inferences or traces) together with monitoring metrics, or would you prefer to manage them separately?

Considering these factors will help you choose the best monitoring setup. Here are three recommended architectures you can implement with Evidently.

## Tracing with scheduled evals

**Best for**: LLM-powered applications.

{% hint style="success" %}
Supported in: `Evidently Cloud` and `Evidently Enterprise`. Scheduled evaluations are in beta on Evidently Cloud. Contact our team to try it.
{% endhint %}

How it works:

* **Instrument your app**. Use the `Tracely` library (based on OpenTelemetry) to capture all relevant data from your application, including inputs, outputs, tool calls, and intermediate steps.
* **Store raw data**. Evidently Platform stores all raw data, providing a complete record of activity.
* **Schedule evaluations**. Set up evaluations to run automatically at scheduled times. This will generate Reports or run Tests directly on the Evidently Platform.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-069f8efe9d66afc133f5d7b91c4bb2dcc054d87c%2Fmonitoring_flow_tracing.png?alt=media)

You can also manually run evaluations anytime to assess individual predictions.

Benefits of this approach:

* **Solves the data capture**. You collect complex traces and all production data in one place, making it easier to manage and analyze.
* **Easy to re-run evals**. With raw traces stored on the platform, you can easily re-run evaluations or add new metrics whenever needed.
* **No-code**. Once your trace instrumentation is set up, you can manage everything from the UI.

**What’s next?** To get started, [instrument your app with Tracely](/user-guide/tracing/tracing_overview) and set up [scheduled evaluations](/user-guide/monitoring/scheduled_evals).

{% content-ref url="/pages/QLgNQWmllpcjTIwGdCfJ" %}
[Set up tracing](/user-guide/tracing/set_up_tracing)
{% endcontent-ref %}

## Batch monitoring jobs

**Best for**: batch ML pipelines, regression testing, and near real-time ML systems that don’t need instant evaluations.

{% hint style="success" %}
Supported in: `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

How it works:

* **Build your evaluation pipeline**. Create a pipeline in your infrastructure to run monitoring jobs. This can be a Python script, cron job, or orchestrated with a tool like Airflow. Run it at regular intervals (e.g., hourly, daily) or trigger it when new data or labels arrive.
* **Run metric calculations**. Implement the evaluation step in the pipeline using the Evidently Python library. Select the evals, and compute JSON `snapshots` that will summarize data, metrics, and test results.
* **Store and visualize snapshots**. Store and monitor results in Evidently Cloud, or in a designated self-hosted workspace.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-75071bd49566a9c9edbb7ce43f3f59c11e52c16d%2Fmonitoring_flow_batch.png?alt=media)

Benefits of this approach:

* **Decouples log storage and monitoring metrics**. Evidently generates snapshots with data summaries and test results. It does not store raw data or model predictions unless you choose to. This protects data privacy and avoids duplicating logs if they’re already stored elsewhere, like for retraining.
* **Full control over the evaluation pipeline**. You decide when evaluations happen. This setup is great for batch ML models, where you can easily add monitoring as another step in your existing pipeline. For online inference, you can log your predictions to a database and set up separate monitoring jobs to query data at intervals.
* **Fits most ML evaluation scenarios**. Many evaluations, like data drift detection, naturally work in batches since you need to collect a set of new data points before running them. Model quality checks often only happen when new labeled data arrives, which can be delayed. Analyzing prediction or user behavior shifts is also usually more meaningful when done at intervals rather than recalculating after every single event.

**What’s next?** Understand the batch workflow in more detail:

{% content-ref url="/pages/1P7kJmgId6oLt38tgcus" %}
[Batch monitoring](/user-guide/monitoring/batch_monitoring)
{% endcontent-ref %}

## Near real-time with collector

**Best for**: near real-time integration with an ML prediction service.

{% hint style="success" %}
Supported in: `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

Near real-time monitoring is for scenarios where you need frequent insights into your AI system's performance. In this setup, data from your ML service is sent directly to a monitoring system, where metrics are calculated on the fly and visualized in an online dashboard.

In Evidently, this works through the Evidently collector service that you deploy on your infrastructure.

How it works:

* **Deploy Evidently collector service**. Install the Evidently collector and configure it to run evaluations at set intervals with chosen Evidently Metrics and Tests.
* **Send inferences**. Post data (inputs and predictions) from your ML prediction service to the collector. The Evidently Collector manages data batching, computes Reports or Test Suites based on the configuration that you set, and sends them to the Evidently Cloud or designated self-hosted workspace.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-9be20ba2e1357d32db3b4e215b1ab5ba5be02579%2Fmonitoring_flow_collector.png?alt=media)

The benefits of the approach is that you do not need to write your own evaluation pipelines, and it is suited for frequent calculations.

What’s next? Read how to set up the collector:

{% content-ref url="/pages/YeaJmw4dvlNWfHwigGRl" %}
[Collector service](/user-guide/monitoring/collector_service)
{% endcontent-ref %}


# Batch monitoring

How to run batch monitoring workflows.

{% hint style="info" %}
**You are looking at the old Evidently documentation**. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

For an overview of monitoring architectures, read the [introduction](/user-guide/monitoring/monitoring_overview).

In a batch monitoring workflow, you regularly run evaluation jobs and log the results to the Evidently Platform for ongoing monitoring.

## Code example

Check the tutorials:

{% content-ref url="/pages/nMUnnneEGgQj0eQHDh9z" %}
[Self-host ML Monitoring](/tutorials-and-examples/tutorial-monitoring)
{% endcontent-ref %}

{% content-ref url="/pages/7XC3sQCPEUuWxfy7PhCK" %}
[Tutorial - LLM Evaluation](/tutorials-and-examples/tutorial-llm)
{% endcontent-ref %}

## How it works

Here’s a general overview of how batch monitoring works:

**1. Create a monitoring** [**Project**](/user-guide/projects/add_project). This will organize all your data and evaluation results in one place.

**2. Configure evaluations**. Decide which [Reports and Test Suites](/user-guide/tests-and-reports/introduction) you want to run at regular intervals. These could focus on data quality, data drift, model performance, or LLM output quality.

**3. Design monitoring jobs**. Set up jobs to run evaluations at specific points in your prediction pipeline or by querying your prediction database. These jobs will generate JSON snapshots to upload to the platform.

Essentially, you're setting up a routine process to run the same evaluations you might run manually during experimentation. You can run these jobs via a Python script or manage them using workflow tools like Airflow.

Learn how to generate and send snapshots:

{% content-ref url="/pages/0DSrUUOdBwyB3NsyVNPf" %}
[Generate snapshots](/user-guide/evaluations/snapshots)
{% endcontent-ref %}

{% hint style="info" %}
**Example**: in a daily scoring pipeline, you could add validation steps to check input data quality and prediction drift with the `DataStabilityTestPreset` and `DataDriftTestPreset`. Another job could compute model quality metrics once the true labels are available, using something like `BinaryClassificationTestPreset`.
{% endhint %}

**4. Run evaluations and upload results**. Once you generate the Reports or Test Suites, upload them to your Project on the Evidently Platform. It's common to send snapshots at regular intervals, such as hourly or daily, though you can also upload them with a delay if needed.

Use **Tags** and **timestamps** to organize your evaluation results.

Optionally, you can also upload the **Dataset** along with the snapshot to store it on the platform.

**5. Configure the Dashboard**. Set up a Dashboard to monitor key metrics over time.

You can use pre-built dashboards like "Column Tabs," "Data Drift," or "Data Quality," or create custom ones to track specific results. The original Reports or Test Suites will still be available in your Project, making it easy to dig into the details if something goes wrong. For example, if you notice an increase in drifting features, you can jump into the source Report to see column distribution summaries and troubleshoot the issue.

Learn more about the monitoring Dashboard:

{% content-ref url="/pages/AoVr9uW2K4ASJFRnsT2V" %}
[Dashboard overview](/user-guide/dashboard/dashboard_overview)
{% endcontent-ref %}

**5. Set Alerts**. Configure alerts to notify you when Tests fail, or if specific metrics go out of bounds.

Learn more about alerts:

{% content-ref url="/pages/8XOQWHjHeQNik65tNYLI" %}
[Send alerts](/user-guide/monitoring/alerting)
{% endcontent-ref %}

## Special cases

### Delayed ground truth

In many ML applications, ground truth data is delayed. For example, you might generate demand forecasts, but the actual sales values to assess the model’s quality arrive after a week.

To handle this, you can split your monitoring steps:

* **Check inputs**. When you get a new batch of data, capture a summary snapshot and check the data quality (missing values, duplicates, features out of range, etc.)
* **Evaluate prediction drift**. After scoring, check for prediction drift to detect shifts in model behavior.
* **Assess model quality**. Once you receive the true labels, compute model quality metrics (e.g., mean error, RMSE for regression, or accuracy, precision, recall for classification) and add them to the same monitoring project.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a8c62d2f586e49888a102c8adfcbfc9139620a41%2Fmonitoring_batch_workflow_min.png?alt=media\&token=5d291882-2e9f-41eb-accf-ebef59705b04)

Evidently lets you backdate snapshots, so you can log them retroactively. For example, once you receive labeled data and evaluate model quality, you can create a snapshot with the correct timestamp to match the prediction time. This helps ensure your Dashboard accurately reflects the quality for that specific period.

You can also delete and re-upload snapshots if necessary.

### A/B testing

If you have two model versions in production with split traffic (like champion/challenger models) or a shadow model running alongside, you can monitor the quality of both models simultaneously.

Attach **Tags** when generating snapshots to differentiate between the models (versions, prompts, etc.). This way, you can later visualize the performance on a Dashboard using data from snapshots with specific Tags, effectively tracking the quality of Model A versus Model B side by side.

### Multiple models in a Project

If you're working with several related models or segments, you can log data for all of them within the same Project.

Evidently lets you log multiple snapshots of the same type for the same period, so you can freely design your workflow. Use **Tags** to distinguish between different models or segments.

For example, you could track the performance of multiple demand forecasting models for different locations by logging all snapshots to the Project with location-specific Tags. Then, you can add separate monitoring Panels on your Dashboard for each area and see how the model performs in each one.

### In-pipeline testing

You can directly use the results of Evidently evaluations within your infrastructure in addition to sending the data to the platform.

For instance, if you've integrated data checks into your prediction pipeline, you might want to halt the pipeline if data quality or data drift tests fail, or if the retrained model's quality doesn't meet a certain threshold.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-adb55e3ac3c2840ffbc846a7c31b4ff1b82a1be7%2Ftest_suite_lifecycle-min.png?alt=media\&token=ae289a0d-c719-4b28-8ae3-2b340f88a32d)

In this scenario, after running an Evidently Report or Test Suite, you can do both:

* Send snapshots to Evidently Platform for tracking the outcomes.
* Run an in-pipeline action. You can export the results of Evidently evaluation as a [Python dictionary or a JSON](/user-guide/tests-and-reports/output_formats) and use the output (e.g., the success or failure of a Test Suite) to trigger actions within the pipeline. This enables you to react immediately while still recording results on the platform.


# Collector service

How to send data in near real-time using the Collector service.

{% hint style="info" %}
**You are looking at the old Evidently documentation**. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Overview

In this scenario, you deploy an **Evidently Collector** service for near real-time monitoring.

Evidently Collector is a service that allows you to collect online events into batches, create `Reports` or `TestSuites` over batches of data, and save them as `snapshots` to your Workspace.

You will need to POST the predictions from the ML service to the Evidently Collector service. You can POST data on every prediction or batch them. The Evidently collector service will perform asynchronous computation of monitoring snapshots based on the provided configuration.

You can also pass the path to the optional reference dataset.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a9a56351bf42412868c87d7a41e6ec74da358ecc%2Fmonitoring_collector_min.png?alt=media\&token=dd79fb6e-88d0-47d9-97bb-162599f30d40)

If you receive delayed ground truth, you can later compute and log the model quality to the same Project. You can run it as a separate process or a batch job.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-7ed96210ff5f0e6694c950ea0eabbf8a7f5c0c7a%2Fmonitoring_collector_delayed_labels_min.png?alt=media\&token=76ccc4cd-194a-46dc-8868-2f0204d76c18)

## Code example

Refer to this example:

{% embed url="<https://github.com/evidentlyai/evidently/tree/main/examples/integrations/collector_service>" %}

## Collector configuration

Before sending events, you must configure the collector and start the service.

You can choose either of the two options:

* Create configuration via code, save it to a JSON file, and run the service using it.
* Run the service first and create configuration via API.

The collector service can simultaneously run multiple “collectors” that compute and save snapshots to different Workspaces or Projects. Each one is represented by a `CollectorConfig` object.

### `CollectorConfig` Object

You can configure the following parameters:

| Parameter         | Type               | Description                                                                                                                              |
| ----------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `trigger`         | `CollectorTrigger` | Defines when to create a new snapshot from the current batch.                                                                            |
| `report_config`   | `ReportConfig`     | Configures the contents of the snapshot: `Report` or `TestSuite` computed for each batch of data.                                        |
| `reference_path`  | Optional\[str]     | Local path to a *.parquet* file with the reference dataset.                                                                              |
| `cache_reference` | bool               | Defines whether to cache reference data or re-read it each time.                                                                         |
| `api_url`         | str                | URL where the Evidently UI Service runs and snapshots will be saved to. For Evidently Cloud, use `api_url="https://app.evidently.cloud"` |
| `api_secret`      | Optional\[str]     | Evidently UI Service secrets.                                                                                                            |
| `project_id`      | str                | ID of the project to save snapshots to.                                                                                                  |

You can create a `ReportConfig` object from `Report` or `TestSuite` objects. You must run them first so that all `Metrics` and `Tests` are collected (including when you use Presets or Test/Metric generators).

```python
report = Report(...) 
report.run(...) 
report_config = ReportConfig.from_report(report) 

# or 

test_suite = TestSuite(...) 
test_suite.run(...) 
report_config = ReportConfig.from_test_suite(test_suite)
```

### CollectorTrigger

Currently, there are two options available:

* `IntervalTrigger`: triggers the snapshot calculation at set intervals (in seconds).
* `RowsCountTrigger`: triggers the snapshot calculation when a specific row count is reached.

**Note**: we are also working on `CronTrigger` and other triggers. Would you like to see additional scenarios? Please open a GitHub issue with your suggestions.

### Setup via file

You can define the configuration and save it as a JSON file. Example:

```python
config = CollectorServiceConfig(collectors={
        "main": CollectorConfig(
            trigger=IntervalTrigger(interval=60 * 60),
            report_config=ReportConfig.from_report(report),
            reference_path="reference_data.parquet",
            project_id="834ec9a0-ee58-4e64-816b-c593b0b6c45c",
            api_url="http://localhost:8000"
        )
    })

config.save("collector_config.json")
```

Then, run the following command:

```bash
evidently collector --config-path collector_config.json
```

### Setup via API

First, run the collector service:

```bash
evidently collector
```

Then, use the `CollectorClient` to add a new collector config:

```python
config = CollectorConfig(
        trigger=IntervalTrigger(interval=60 * 60),
        report_config=ReportConfig.from_report(report),
        reference_path="reference_data.parquet",
        project_id="834ec9a0-ee58-4e64-816b-c593b0b6c45c",
        api_url="<http://localhost:8000>"
    )
```

### Update reference via API

To specify the path to the reference dataset:

```python
reference: pd.DataFrame = ...
client = CollectorClient("<http://localhost:8001>")
client.set_reference("main", reference)
```

### Send events via API

To send events from your ML service:

```python
client = CollectorClient("http://localhost:8001")

events: pd.DataFrame = ...
client.send_data("main", events)
```

### Send events via curl

To send data with `curl`:

```bash
curl 

POST '.../<collector config id>/data'

headers {'evidently-secret': '...', 'Content-Type': 'application/json'}

body '{"column1": {"0": 7.0, "1": 5.0}, "column2": {"0": "a", "1": "b"}}'
```

Example:

```bash
curl -d '{"column1": {"0": 7.0, "1": 5.0}, "column2": {"0": "a", "1": "b"}}' -H 'Content-Type: application/json' http://0.0.0.0:8001/default/data
```

This is how it looks in the Terminal.

Sending data:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-bad2ed0f8a8a7526b7f20b554d42a03619c8ccba%2Fcollector_data-min.png?alt=media\&token=667a2a34-9e32-466d-9825-ff1981bda357)

The data is received by the collector service:

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-6fe5754ca27ad6b221093ca8781768b0fe732999%2Fcollector_send-min.png?alt=media\&token=65411ca4-6b71-42a3-bf88-fb94acffabf3)


# Scheduled evaluations

Scheduled evaluations are in beta. Detailed docs are coming soon. Contact our team to try it first!


# Send alerts

Get notifications in Slack or email.

{% hint style="success" %}
Built-in alerting is a Pro feature available in the Evidently Cloud and Enterprise.
{% endhint %}

To enable alerts, open the Project and navigate to the "Alerts" section in the left menu. You must set:

* A notification channel.
* An alert condition.

## Notification channels

You can choose between the following options:

* **Email**. Add email addresses to send alerts to.
* **Slack**. Add a Slack webhook.
* **Discord**. Add a Discord webhook.

## Alert conditions

### Failed tests

If you use Test Suites, you can tie alerting to the failed Tests in a Test Suite. Toggle this option on the Alerts page. Evidently will set an alert to the defined channel if any of the Tests fail.

{% hint style="info" %}
**How to avoid alert fatigue?** When you create a Test Suite, you can [mark certain conditions as Warnings](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/custom-test-suite.md) using the `is_critical` parameter. Set is `False` for non-critical checks to avoid triggering alerts.
{% endhint %}

### Custom conditions

You can also set alerts on individual Metric values for both Reports and Test Suites. For example, you can generate Alerts when the share of drifting features is above a certain threshold.

Click on the plus sign below the “Add new Metric alert” and follow the prompts to set an alert condition.

![](https://github.com/evidentlyai/docs-old/blob/main/.gitbook/assets/cloud/alerts.png)


# Dashboard


# Dashboard overview

Introduction to Dashboards.

{% hint style="success" %}
Supported in: `Evidently OSS`, `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## What is a Dashboard?

Each Project has its Dashboard. A Dashboard lets you evaluation results over time, providing a clear view of the quality of your AI application and data.

When you create a new Project, the Dashboard starts empty. To populate it, run [evaluations](/user-guide/evaluations/evals_overview) or set up [monitoring](/user-guide/monitoring/monitoring_overview). Once you have data, you can configure the Dashboard to show the values you want to see.

You can use the Dashboard to monitor live data in production or to keep track of results from batch experiments and tests. The "Show in order" toggle lets you switch between two views:

* **Time series**. Displays data with actual time intervals, ideal for live monitoring.
* **Sequential**. Shows results in order with equal spacing, perfect for experiments.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-74cbf354caa649bce03758b2cc5fd27345b1406e%2Fevidently_ml_monitoring_main.png?alt=media\&token=0154c054-2a32-4bde-8c12-b0ae075637aa)

All Panels within the same view reflect the date range set by the time range filter. You can also zoom in on any time series visualizations for deeper analysis.

## What is a Panel?

A Dashboard consists of \*\*Panels((, each visualizing specific values or test results. Panels can be counters, line or bar plots, and more.

{% content-ref url="/pages/WNsTEJdiGN5MsuLTN6MV" %}
[Panel types](/user-guide/dashboard/design_dashboard)
{% endcontent-ref %}

You can customize your Dashboard by adding Panels through the Python API using dashboard-as-code.

In Evidently Cloud and Enterprise, you have additional options:

* Add Panels directly from the UI
* Use multiple Tabs within the same Dashboard
* Start with pre-built Tabs as templates

{% content-ref url="/pages/NC6E5mDNUtuqLQmJRaXR" %}
[Pre-built Tabs](/user-guide/dashboard/add_dashboard_tabs)
{% endcontent-ref %}

## What is the data source?

Panels pull data from `snapshots`, which are Reports or Test Suites you've generated and saved to a Project.

Each Test Suite and Report contains a wealth of information and visuals. To add a Panel to the Dashboard, you must choose a specific **value** you'd like to plot and select other parameters, such as the Panel type and title.

{% content-ref url="/pages/9gvks8hk88ZhwtpTpSCX" %}
[Adding Panels](/user-guide/dashboard/design_dashboard_api)
{% endcontent-ref %}

For example, if your Reports include the `ColumnSummaryMetric`, you can visualize values like mean, max, min, etc. within your Panels. This method works for all other Metrics. If you're running Tests, say `TestColumnValueMin`, you can also display the Test result (pass or fail).

You can also use Tags, which you should add to Reports or Test Suites during generation. Tags allow you to filter and visualize data from specific subsets of snapshots when creating a Panel.


# Pre-built Tabs

Get a pre-built monitoring Dashboard using templates.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Pre-built Tabs

{% hint style="success" %}
Dashboard templates is a Pro feature available in the Evidently Cloud and Enterprise.
{% endhint %}

Template Tabs include a preset combination of monitoring Panels, so you don't have to add them one by one.

To use a template:

* Enter the "Edit" mode by clicking on the top right corner of the Dashboard.
* Click on the "Add tab" button.
* Choose a template Tab in the dropdown.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-08545232dd208adddbb1437d9e16b44d25d97466%2Fqs_add_data_quality_tab_2.gif?alt=media)

Optionally, give a custom name to the Tab.

## Available Tabs

You have the following options:

| Tab Template | Description                                                                                                         | Data source                                                                                                                                                    |
| ------------ | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Descriptors  | Shows the results of text evaluations over time.                                                                    | `TextEvalPreset()`, or individual `ColumnSummaryMetric()` Metrics or Tests that use Descriptors.                                                               |
| Columns      | Plots column distributions over time for categorical and numerical columns.                                         | `DataQualityPreset()` or `ColumnSummaryMetric()` for individual columns.                                                                                       |
| Data Quality | Shows dataset quality metrics (e.g., missing values, duplicates, etc.) over time and results of Data Quality Tests. | For the Metric Panels: `DataQualityPreset()` or `DatasetSummaryMetric()`. For the Test Panel: any individual Tests from Data Quality or Data Integrity groups. |
| Data Drift   | Shows the share of drifting features over time and the results of Column Drift Tests.                               | For the Metric Panel: `DataDriftPreset()` or `DataDriftTestPreset()`. For the Test Panel: `DataDriftTestPreset()` or individual `TestColumnDrift()` Tests.     |

## What’s next?

* Understand available [monitoring Panels types](/user-guide/dashboard/design_dashboard).
* See how to [customize your Dashboard](/user-guide/dashboard/design_dashboard_api).


# Panel types

Overview of the available monitoring Panel types.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## What is a monitoring Panel?

A monitoring Panel is an individual plot or counter on the Monitoring Dashboard.&#x20;

You can add multiple Panels and organize them by **Tabs**. You can choose from Metric, Distribution, and Test Panels, and customize titles and legends.

When adding a Panel, you point to the **source Metric or Test** and the value (`field_path`) inside it. Evidently will pull selected value(s) from all snapshots in the Projects and add them to the Panel.&#x20;

You can use **Tags** to filter data from specific snapshots. For example, you can plot the accuracy of Model A and Model B next to each other. To achieve this, add relevant Tags when creating a snapshot.

{% hint style="info" %}
**How to add Panels**. This page explains the Panel types. Check the next section on [adding Panels](/user-guide/dashboard/design_dashboard_api).
{% endhint %}

## Metric Panels

Metric Panels (`DashboardPanel`) show individual values from inside the snapshots.

For example, if you capture Data Quality Reports (include mean, max, min, etc., for each column) or Data Drift Reports (include the share of drifting columns), you can plot any of these values over time.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-80aaf1c36c1da5c051b97d1cbbe48e59fe4bbda1%2Fmetric_panels.png?alt=media)

Panel time resolution depends on snapshot frequency. For instance, if you log Data Drift Reports daily, you can plot the `share_of_drifting_features` with daily granularity. You can also open the source Report to see feature distributions on a specific day.

You can create metric Panels from either Reports or Test Suites.&#x20;

### Counter

Class `DashboardPanelCounter`.

| Panel Type                                                                       | Example                                                                                                                                                                                                                                                             |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Shows a value with supporting text or text alone (perfect for dashboard titles). | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-b953ddc5974aea66a47c29dd7af09d4ab8b89984%2Fpanel_counter_example-min.png?alt=media\&token=e581c2ca-26c9-4775-a870-427f874da7a4) |

### Plot

Class `DashboardPanelPlot`. Shows individual values as bar, line, scatter plot, or histogram.

| Panel Type                                                                                  | Example                                                                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Line plot: `PlotType.LINE`. Shows values over time.                                         | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-6c040bebe575c2ab0732028399f371134ef48d9f%2Fpanel_line_plot_example.png?alt=media)                                                |
| Scatter plot: `PlotType.SCATTER`. Shows values over time.                                   | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-5fecec2e2510831645b6cbcdf53ffcb912214a30%2Fpanel_scatter_plot_example.png?alt=media\&token=8aa985a9-60b9-49fd-a99f-4bd67c8bbe47) |
| Bar plot: `PlotType.BAR`. Shows values over time.                                           | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-df99c25ab4ac5369d09182b9c4c42e14560a4dcf%2Fpanel_bar_plot_example.png?alt=media\&token=be2be3aa-345d-4d3a-b1b4-d27489c08bf7)     |
| Histogram: `PlotType.HISTOGRAM`. Shows the frequency of individual values across snapshots. | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a82b2bd43ee5a4b46c4843f30a8944cb04bbc430%2Fpanel_hist_example.png?alt=media\&token=2d97c001-d9bc-436a-b0a5-50bd000f3bfc)         |

## Test Panels

Test Panels show the Test results.

For example, if you run the same Test Suite repeatedly, you can visualize the pass/fail outcomes or result counts. You choose which Test results to include.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e75e663fc00faf201d95062a6e8cab403511bd43%2Ftest_panels.png?alt=media)

To improve the readability of Test plots in case of frequent runs (e.g., multiple runs per day), you can additionally aggregate Test results on a plot (e.g., inside 1 Day).

Test Panels only work with Test Suites. You cannot create it from Reports.&#x20;

### Test counter

Class `DashboardPanelTestSuiteCounter`

| Panel Type                                     | Example                                                                                                                                                                                                                                                               |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Shows a counter of Tests with selected status. | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-3d3b4184b50aae38372185b2cd57bfdf10d133b0%2Fpanel_tests_counter_example.png?alt=media\&token=6b4181c1-2cbb-42c7-8669-c17da431b637) |

### Test plot

Class `DashboardPanelTestSuite`.

| Panel Type                                                                                            | Example                                                                                                                                                                                                                                                                        |
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Detailed plot: `TestSuitePanelType.DETAILED`. Individual Test results are visible.                    | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2cbc22a7a80529d8a91cb15af8a283a8ad69338b%2Fpanel_tests_detailed_hover_example.png?alt=media\&token=dacbfc90-5db8-4fc2-9902-a55d45b4597c)   |
| Aggregated plot: `TestSuitePanelType.AGGREGATE`. Only the total number of Tests by status is visible. | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-7d582caa3d6ea08c475bc703e3167e8fbaf902e5%2Fpanel_tests_aggregated_hover_example.png?alt=media\&token=687f65bd-cf34-4a37-8a00-52023cc11879) |

## Distribution Panel

Class `DashboardPanelDistribution`. Shows a distribution of values over time. For example, if you capture Data Quality or Data Drift Reports that include histograms for categorical values, you can plot how the frequency of categories changes.

You can create distribution plots from either Reports or Test Suites.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-10d1fe9852aae004b8b06b2ca02406d43c182485%2Fdistribution_panels.png?alt=media)

| Panel Type                                                                                  | Example                                                                                                                                                                                                                |
| ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Stacked bar chart: `HistBarMode.STACK`. Shows absolute counts.                              | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-ba60ee6cd3afd2f45b88e50ea0422070da1b485e%2Fpanel_dist_stacked_2-min.png?alt=media) |
| Grouped bar chart: `HistBarMode.GROUP`. Shows absolute counts.                              | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8e759a67280271b1dc8702486fdb826fa20aa609%2Fpanel_dist_group_2-min.png?alt=media)   |
| Overlay bar chart: `HistBarMode.OVERLAY`. Shows absolute counts. Values overlay each other. | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-6727eb42d02fd1c91a80f987a779143f6cc0aa03%2Fpanel_dist_overlay-min.png?alt=media)   |
| Stacked bar chart: `HistBarMode.RELATIVE`. Shows relative frequency (percentage).           | ![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-1ab2f6dede8657e14fccf1a8956297e3d9ae9fcb%2Fpanel_dist_relative-min.png?alt=media)  |

{% hint style="info" %}
**What is the difference between a Distribution Panel and a Histogram Plot?** A Histogram shows the distribution of the selected values from all snapshots. Each source snapshot contains a **single value** (e.g., a "number of drifting features"). A Distribution Panel shows how a distribution changes over time. Each source snapshot contains a **histogram** (e.g. frequency of different categories).
{% endhint %}

## What's next?

How to add [monitoring Panels and Tabs](/user-guide/dashboard/design_dashboard_api).


# Adding Panels

Design your Dashboard with custom Panels.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

We recommend starting with [pre-built Tabs](/user-guide/dashboard/add_dashboard_tabs) for a quick start.

## Code example

To see end-to-end code examples with custom Panels, check:

* [Evidently Cloud Tutorial](/tutorials-and-examples/tutorial-cloud)
* [Self-hosting ML Monitoring](/tutorials-and-examples/tutorial-monitoring).

You can also explore the [source code](https://github.com/evidentlyai/evidently/tree/d43d33017a0fc4a69f9ff72581fe3f34b4abd45e/src/evidently/ui/demo_projects) for the open-source [live demo dashboards](https://demo.evidentlyai.com/).

## Adding Panels

You can add monitoring Panels using the Python API or the Evidently Cloud user interface.

Here is the general flow:

* Define the **Panel type**: Counter, Plot, Distribution, Test Counter, or Test Plot. (See [Panel types](/user-guide/dashboard/design_dashboard)).
* Specify panel **title** and **size**.
* Add optional **Tags** to filter data. Without Tags, the Panel will use data from all Project snapshots.
* Select Panel **parameters**, e.g., aggregation level.
* Define the **Panel value(s)** to show:
  * For Test Panels, specify `test_id`.
  * For Metric and Distribution Panels, specify `metric_id` and `field_path`.
* If applicable, pass `test_args` or `metric_args` to identify the exact value when they repeat in a snapshot. For instance, to plot the mean value of a given column, pass the column name as an argument.

This page explains each step in detail.

### Add a new Panel

You can add monitoring Panels using the Python API, or directly in the user interface (Evidently Cloud or Enterprise).

{% tabs %}
{% tab title="UI" %}
Enter "edit" mode on the Dashboard (top right corner) and click "add Panel." Follow the steps to create a Panel. You can Preview the Panel before publishing.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8edbc7c4923fa22c95b0de379acffc4e30f1388a%2Fadd_new_panel_2.gif?alt=media\&token=f1808273-b2c7-4b7d-9cde-1b7cf9e89b36)

Some tips:

* Use the "Show description" toggle to get help on specific steps.
* You can identify the `field_path` in two ways. Use the "Manual mode" toggle to switch.
  * Default mode displays popular values from existing Project snapshots.
  * Manual mode mirrors the Python API. You can select any value, even if it's not yet in the Project. Note that the Panel may be empty until you add the snapshot.
    {% endtab %}

{% tab title="API - Metric Panel" %}
**Connect to a Project**. Load the latest dashboard configuration into your Python environment.

```python
project = ws.get_project("YOUR PROJECT ID HERE")
```

**Add a new Panel**. Use the `add_panel` method and pass the parameters. You can add multiple Panels: they will appear in the listed order. Save the configuration with `project.save()`.

```python
project.dashboard.add_panel(
        DashboardPanelPlot(
            title="Share of drifting features",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            values=[
                PanelValue(
                	metric_id="DatasetDriftMetric",
                	field_path="share_of_drifted_columns",
                	legend="share",
                ),
            ],
            plot_type=PlotType.LINE,
            size=WidgetSize.FULL,
        ),
        tab="Summary"
)
project.save()
```

Go back to the web app to see the Dashboard. Refresh the page if needed.
{% endtab %}

{% tab title="API - Test Panel" %}
**Connect to a Project**. Load the latest dashboard configuration into your Python environment.

```python
project = ws.get_project("YOUR PROJECT ID HERE")
```

**Add a new Test Panel**. Use the `add_panel` method, set `include_test_suites=True` and pass the parameters. You can add multiple Panels: they will appear in the listed order. Save the configuration with `project.save()`.

```python
project.dashboard.add_panel(
        DashboardPanelTestSuite(
            title="Data quality tests",
            test_filters=[
                TestFilter(test_id="TestNumberOfConstantColumns", test_args={}),
                TestFilter(test_id="TestShareOfMissingValues", test_args={}),
                TestFilter(test_id="TestNumberOfEmptyRows", test_args={}),
            ],
            filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
            size=WidgetSize.FULL,
            panel_type=TestSuitePanelType.DETAILED,
            time_agg="1D",
        ),
        tab="Data Tests"
)
project.save()
```

Go back to the web app to see the Dashboard. Refresh the page if needed.
{% endtab %}
{% endtabs %}

### Add a new Tab

{% hint style="success" %}
Multiple tabs is a Pro feature available in the Evidently Cloud.
{% endhint %}

By default, you add all Panels to a single Dashboard view. You can create multiple Tabs to organize them.

{% tabs %}
{% tab title="UI" %}
Enter the "edit" mode on the Dashboard (top right corner) and click "add Tab". To create a custom Tab, choose an “empty” tab and give it a name.

Proceed with adding Panels to this Tab as usual.
{% endtab %}

{% tab title="API" %}
**Connect to a Project**. Load the latest Dashboard configuration into your Python environment.

```python
project = ws.get_project("YOUR PROJECT ID HERE")
```

**Create a new Tab**. Add a Tab, give it a name and save the configuration. You can later reference this Tab when adding Panels to the Project.

```python
project.dashboard.create_tab("My tab")
project.save()
```

**Add a new Panel to a Tab**. Alternatively, use the `add_panel` method as usual and specify the name of the destination Tab (e.g., "Summary").

```python
project.dashboard.add_panel(
        DashboardPanelPlot(
            title="Share of drifting features",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            values=[
                PanelValue(
                	metric_id="DatasetDriftMetric",
                	field_path="share_of_drifted_columns",
                	legend="share",
                ),
            ],
            plot_type=PlotType.LINE,
            size=WidgetSize.FULL,
        ),
        tab="Summary"
)
project.save()
```

If no "Summary" Tab exists, you will create both a new Tab and Panel. If it already exists, a new Panel will appear below others in this Tab.
{% endtab %}
{% endtabs %}

### Delete Tabs or Panels

To delete all the existing monitoring Panels using the Python API:

```
project.dashboard.panels = []

project.save()
```

**Note**: This does not delete the snapshots; it only deletes the Panel configuration.

To delete the Tabs or Panels in the UI, use the “Edit” mode and click the “Delete” sign on the corresponding Panel or Tab.

## Panel parameters

{% hint style="success" %}
**Panel types**. To preview all Panel types, check the previous [docs section](/user-guide/dashboard/design_dashboard). This page details the parameters and API.
{% endhint %}

Class `DashboardPanel` is a base class. Its parameters apply to all Panel types.

| Parameter                                                                                                                                                                          | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `title: str`                                                                                                                                                                       | Panel name visible at the header.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| <p><code>filter: ReportFilter</code><br><br><code>metadata\_values: Dict\[str, str]</code><br><code>tag\_values: List\[str]</code><br><code>include\_test\_suites=False</code></p> | <p>Filters define a subset of snapshots from which to display the values.</p><ul><li>To select a group of snapshots as a data source, pass <code>metadata\_values</code> or <code>tag\_values</code>. You must add these Tags when logging Reports or Test Suites. (See <a href="https://github.com/evidentlyai/docs-old/blob/main/dashboard/snapshots.md#add-tags">docs</a>).</li><li>To include Test Suites data, set <code>include\_test\_suites</code> as <code>True</code> (default: <code>False</code>).</li></ul> |
| <p><code>size: WidgetSize = WidgetSize.FULL</code><br><br><strong>Available</strong>: <code>WidgetSize.FULL</code>, <code>WidgetSize.HALF</code></p>                               | Sets the Panel size to half-width or full-sized (Default).                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

See usage examples below together with panel-specific parameters.

### Counter

`DashboardPanelCounter` shows a value count or works as a text-only Panel.

| Parameter                                                                                                                        | Description                                                                                                                                                                                                                      |
| -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `value: Optional[PanelValue] = None`                                                                                             | <p>Specifies the value to display. If empty, you get a text-only panel.<br><br><em>Refer to the Panel Value section below for examples.</em></p>                                                                                 |
| `text: Optional[str] = None`                                                                                                     | Supporting text to display on the Counter.                                                                                                                                                                                       |
| <p><code>agg: CounterAgg</code><br><br><strong>Available:</strong><br><code>SUM</code>, <code>LAST</code>, <code>NONE</code></p> | <p>Data aggregation options:<br><code>SUM</code>: Calculates the value sum (from all snapshots or filtered by Tag).<br><code>LAST</code>: Displays the last available value.<br><code>NONE</code>: Reserved for text panels.</p> |

See examples:

{% tabs %}
{% tab title="Text Panel" %}
**Text Panel**. To create a Panel with the Dashboard title only:

```python
project.dashboard.add_panel(
    DashboardPanelCounter(
        filter=ReportFilter(metadata_values={}, tag_values=[]),
        agg=CounterAgg.NONE,
        title="Bike Rental Demand Forecast",
        )
    )
```

{% endtab %}

{% tab title="Value sum" %}
**Panel with a sum of values**. To create a Panel that sums up the number of rows over time:

```python
project.dashboard.add_panel(
    DashboardPanelCounter(
        title="Number of Rows",
        filter=ReportFilter(metadata_values={}, tag_values=[]),
        value=PanelValue(
            metric_id="DatasetSummaryMetric",
            field_path=DatasetSummaryMetric.fields.current.number_of_rows,
            legend="count",
        ),
        text="count",
        agg=CounterAgg.SUM,
        size=WidgetSize.HALF,
    )
)
```

{% endtab %}
{% endtabs %}

### Plot

`DashboardPanelPlot` shows individual values over time.

Line Plot example:![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-6c040bebe575c2ab0732028399f371134ef48d9f%2Fpanel_line_plot_example.png?alt=media)

| Parameter                                                                                                                                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `values: List[PanelValue]`                                                                                                                                    | <p>Specifies the value(s) to display in the Plot.<br><br>The field path must point to the individual <strong>MetricResult</strong> (e.g., not a dictionary or a histogram).<br><br>If you pass multiple values, they will appear together, e.g., as separate lines on a Line plot, bars on a Bar Chart, or points on a Scatter Plot.<br><br><em>Refer to the Panel Value section below for examples.</em></p> |
| <p><code>plot\_type: PlotType</code><br><br><strong>Available:</strong> <code>SCATTER</code>, <code>BAR</code>, <code>LINE</code>, <code>HISTOGRAM</code></p> | Specifies the plot type: scatter, bar, line, or histogram.                                                                                                                                                                                                                                                                                                                                                    |

See examples:

{% tabs %}
{% tab title="Single value" %}
**Single value on a Plot**. To plot MAPE over time in a line plot:

```python
project.dashboard.add_panel(
    DashboardPanelPlot(
        title="MAPE",
        filter=ReportFilter(metadata_values={}, tag_values=[]),
        values=[
        PanelValue(
            metric_id="RegressionQualityMetric",
            field_path=metrics.RegressionQualityMetric.fields.current.mean_abs_perc_error,
            legend="MAPE",
        ),
    ],
    plot_type=PlotType.LINE,
    size=WidgetSize.HALF,
    )
)
```

{% endtab %}

{% tab title="Multiple values" %}
**Multiple values on a Plot**. To plot MAPE and reference MAPE on the same plot:

```python
project.dashboard.add_panel(
   DashboardPanelPlot(
       title="MAPE",
       filter=ReportFilter(metadata_values={}, tag_values=[]),
       values=[
       PanelValue(
           metric_id="RegressionQualityMetric",
           field_path=metrics.RegressionQualityMetric.fields.current.mean_abs_perc_error,
           legend="MAPE",
       ),
       PanelValue(
           metric_id="RegressionQualityMetric",
           field_path=metrics.RegressionQualityMetric.fields.reference.mean_abs_perc_error,
           legend="Reference MAPE",
       ),
   ],
   plot_type=PlotType.LINE,
   size=WidgetSize.HALF,
   )
)
```

{% endtab %}
{% endtabs %}

### Distribution

`DashboardPanelDistribution` shows changes in the distribution over time.

Stack distribution Panel example:![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-ba60ee6cd3afd2f45b88e50ea0422070da1b485e%2Fpanel_dist_stacked_2-min.png?alt=media)

| Parameter                                                                                                                                                       | Description                                                                                                                                                                                    |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `value: PanelValue`                                                                                                                                             | <p>Specifies the distribution to display on the Panel.<br><br>The <code>field\_path</code> must point to a histogram.<br><br><em>Refer to the Panel Value section below for examples.</em></p> |
| <p><code>barmode: HistBarMode</code><br><br><strong>Available:</strong> <code>STACK</code>, <code>GROUP</code>, <code>OVERLAY</code>, <code>RELATIVE</code></p> | Specifies the distribution plot type: stacked, grouped, overlay or relative.                                                                                                                   |

**Example**. To plot the distribution of the "education" column over time using STACK plot:

```python
p.dashboard.add_panel(
        DashboardPanelDistribution(
            title="Column Distribution: current",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            value=PanelValue(
                field_path=ColumnDistributionMetric.fields.current,
                metric_id="ColumnDistributionMetric",
                metric_args={"column_name.name": "education"},
            ),
            barmode = HistBarMode.STACK
        )
    )
```

### Test Counter

`DashboardPanelTestSuiteCounter` shows a counter with Test results.

| Parameter                                                                                                                                                                                                                                          | Description                                                                                                                                                                                                                                                                                                    |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><code>test\_filters: List\[TestFilter]=\[]</code><br><code>test\_id: test\_id</code><br><code>test\_arg: List\[str]</code></p>                                                                                                                  | <p>Test filters select specific Test(s). Without a filter, the Panel considers the results of all Tests.<br><br>You must reference a <code>test\_id</code> even if you used a Preset. You can check the Tests included in each Preset <a href="https://docs.evidentlyai.com/reference/all-tests">here</a>.</p> |
| <p><code>statuses: List\[statuses]</code><br><br><strong>Available</strong>:<br><code>TestStatus.ERROR</code>, <code>TestStatus.FAIL</code>, <code>TestStatus.SUCCESS</code>, <code>TestStatus.WARNING</code>, <code>TestStatus.SKIPPED</code></p> | Status filters select Tests with specific outcomes. (E.g., choose the FAIL status to display a counter for failed Tests). Without a filter, the Panel considers Tests with any status.                                                                                                                         |
| <p><br><code>agg: CounterAgg</code><br><br><strong>Available</strong>:<br><code>SUM</code>, <code>LAST</code></p>                                                                                                                                  | <p>Data aggregation options:<br><code>SUM</code>: Calculates the sum of Test results from all snapshots (or filtered by Tags).<br><code>LAST</code>: Displays the last available Test result.</p>                                                                                                              |

See examples.

{% tabs %}
{% tab title="Show latest" %}
**Last Test**. To display the result of the latest Test in the Project.

```python
project.dashboard.add_panel(
    DashboardPanelTestSuiteCounter(
        title="Success of last",
        agg=CounterAgg.LAST
    )
)
```

{% endtab %}

{% tab title="Filter by Test and Status" %}
**Filter by Test ID and Status**. To display the number of failed Tests and errors for a specific Test (Number of unique values in the column "age").

```python
project.dashboard.add_panel(
    DashboardPanelTestSuiteCounter(
        title="Success of 1",
        test_filters=[TestFilter(test_id="TestNumberOfUniqueValues", test_args={"column_name.name": "1"})],
        statuses=[TestStatus.ERROR, TestStatus.FAIL]
    )
)
```

{% endtab %}
{% endtabs %}

### Test Plot

`DashboardPanelTestSuite` shows Test results over time.

Detailed Test panel example:![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2cbc22a7a80529d8a91cb15af8a283a8ad69338b%2Fpanel_tests_detailed_hover_example.png?alt=media\&token=dacbfc90-5db8-4fc2-9902-a55d45b4597c)

| Parameter                                                                                                                                                                                                                                                                                | Description                                                                                                                                                                                                                                                                        |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><code>test\_filters: List\[TestFilter]=\[]</code><br><code>test\_id: test\_id</code><br><code>test\_arg: List\[str]</code></p>                                                                                                                                                        | <p>Test filters select specific Test(s). Without a filter, the Panel shows the results of all Tests.<br><br>You must reference a <code>test\_id</code> even if you used a Preset. Check the <a href="https://docs.evidentlyai.com/reference/all-tests">Preset composition</a>.</p> |
| <p><code>statuses: List\[statuses]</code><br><br><strong>Available</strong>:<br><code>TestStatus.ERROR</code>, <code>TestStatus.FAIL</code>, <code>TestStatus.SUCCESS</code>, <code>TestStatus.WARNING</code>, <code>TestStatus.SKIPPED</code></p>                                       | Status filters select Tests with specific outcomes. Without a filter, the Panel shows all Test statuses.                                                                                                                                                                           |
| <p><code>panel\_type=TestSuitePanelType</code><br><br><strong>Available</strong>:<br><code>TestSuitePanelType.DETAILED</code><br><code>TestSuitePanelType.AGGREGATE</code></p>                                                                                                           | Defines the Panel type. **Detailed** shows individual Test results. **Aggregate** (default) shows the total number of Tests by status.                                                                                                                                             |
| <p><br><code>time\_agg: Optional\[str] = None</code><br><br><strong>Available</strong>:<br><code>1H</code>, <code>1D</code>, <code>1W</code>, <code>1M</code> (see <a href="https://pandas.pydata.org/docs/user_guide/timeseries.html#timeseries-period-aliases">period aliases</a>)</p> | Groups all Test results in a period (e.g., 1 DAY).                                                                                                                                                                                                                                 |

{% tabs %}
{% tab title="Detailed Tests" %}
**Detailed Tests**. To show the results of all individual Tests, with daily level aggregation.

```python
project.dashboard.add_panel(
    DashboardPanelTestSuite(
        title="All tests: detailed",
        filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
        size=WidgetSize.HALF,
        panel_type=TestSuitePanelType.DETAILED,
        time_agg="1D",
    )
)
```

{% endtab %}

{% tab title="Aggregated by Status" %}
**Aggregated by Status**. To show the total number of failed Tests (status filter), with daily level aggregation.

```python
project.dashboard.add_panel(
    DashboardPanelTestSuite(
        title="All tests: aggregated",
        filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
        size=WidgetSize.HALF,
        time_agg="1D",
        statuses=[TestStatus.ERROR, TestStatus.FAIL],
    )
)
```

{% endtab %}

{% tab title="Filtered by Test ID" %}
**Filtered by Test ID**. To show all results for a specified list of Tests (on constant columns, missing values, empty rows) with daily-level aggregation.

```python
project.dashboard.add_panel(
        DashboardPanelTestSuite(
            title="Data quality tests",
            test_filters=[
                TestFilter(test_id="TestNumberOfConstantColumns", test_args={}),
                TestFilter(test_id="TestShareOfMissingValues", test_args={}),
                TestFilter(test_id="TestNumberOfEmptyRows", test_args={}),
            ],
            filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
            size=WidgetSize.FULL,
            panel_type=TestSuitePanelType.DETAILED,
            time_agg="1D",
        ),
        tab="Data Tests"
)
```

{% endtab %}

{% tab title="Filtered by Args" %}
**Filtered by Test ID and Test Args**. To show the results of individual column-level Tests with daily aggregation, you must use both `test_id` and `test_arg` (column name):

```python
project.dashboard.add_panel(
    DashboardPanelTestSuite(
        title="Column Drift tests for key features: detailed",
        test_filters=[
            TestFilter(test_id="TestColumnDrift", test_args={"column_name.name": "hours-per-week"}),
            TestFilter(test_id="TestColumnDrift", test_args={"column_name.name": "capital-gain"}),
        ],
        filter=ReportFilter(metadata_values={}, tag_values=[], include_test_suites=True),
        size=WidgetSize.HALF,
        panel_type=TestSuitePanelType.DETAILED,
        time_agg="1D",
    )
)
```

{% endtab %}
{% endtabs %}

## Panel Value

To define the value to show on a Metric Panel (Counter, Distribution, or Plot), you must pass the `PanelValue`. This includes source `metric_id`, `field_path` and `metric_args`.

| Parameter                | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metric_id`              | <p>The ID corresponds to the Evidently <code>metric</code> in a snapshot.<br><br>Note that if you used a Metric Preset, you must still reference a <code>metric\_id</code>. Check the Metric <a href="https://docs.evidentlyai.com/reference/all-metrics">Preset composition</a>.<br><br>If you used a Test Suite but want to plot individual values from it on a Metric Panel, you must also reference the <code>metric\_id</code> that the Test relies on.</p> |
| `field_path`             | The path to the computed Result inside the Metric. You can provide a complete field path or a `field_name`. For Counter and Plot, the `field_path` must point to a single value. For the Distribution Panel, the `field_path` must point to a histogram.                                                                                                                                                                                                         |
| `metric_args` (optional) | Use additional arguments (e.g., column name, text descriptor, drift detection method) to identify the exact value when it repeats inside the same snapshot.                                                                                                                                                                                                                                                                                                      |
| `legend` (optional)      | Value legend to show on the Plot.                                                                                                                                                                                                                                                                                                                                                                                                                                |

See examples to specify the `field_path`:

{% tabs %}
{% tab title="Exact field" %}
**Exact field name**. To include the `share_of_drifted_columns` available inside the `DatasetDriftMetric()`:

```python
value=PanelValue(
    metric_id="DatasetDriftMetric",
    field_path="share_of_drifted_columns",
    legend="share",
)
```

In this example, you pass the exact name of the field.
{% endtab %}

{% tab title="Complete field path" %}
**Complete field path**. To include the `current.share_of_missing_values` available inside the `DatasetMissingValueMetric()`:

```python
value=PanelValue(
    metric_id="DatasetMissingValuesMetric",
    field_path=DatasetMissingValuesMetric.fields.current.share_of_missing_values,
    legend="Missing Values Share",
)
```

In this example, you pass the complete field path inside the source Metric.
{% endtab %}
{% endtabs %}

See examples using different `metric_args`:

{% tabs %}
{% tab title="Column names" %}
**Column names as arguments**. To show the mean values of target and prediction on a line plot.

```python
values=[
    PanelValue(
        metric_id="ColumnSummaryMetric",
        field_path="current_characteristics.mean",
        metric_args={"column_name.name": "cnt"},
        legend="Target (daily mean)",
    ),
    PanelValue(
        metric_id="ColumnSummaryMetric",
        field_path="current_characteristics.mean",
        metric_args={"column_name.name": "prediction"},
        legend="Prediction (daily mean)",
    ),
]
```

{% endtab %}

{% tab title="Descriptors" %}
**Descriptors as arguments**. To specify the text descriptor (share of out-of-vocabulary words) using `metric_args`:

```python
values=[
    PanelValue(
        metric_id="ColumnSummaryMetric",
        metric_args={"column_name": descriptors.OOV(display_name="OOV").for_column("Review_Text")},
        field_path="current_characteristics.mean",
        legend="OOV % (mean)",
    ),
]
```

{% endtab %}

{% tab title="Metric parameters" %}
**Metric parameters as arguments**. To specify the `euclidean` drift detection method (when results from multiple methods are logged inside a snapshot) using `metric_args`:

```python
values=[
    PanelValue(
        metric_id="EmbeddingsDriftMetric", 
        metric_args={"drift_method.dist": "euclidean"}
    )
}
```

{% endtab %}
{% endtabs %}

#### How to find the field path?

Let's take an example of `DataDriftPreset()`. It contains two Metrics: `DatasetDriftMetric()` and `DataDriftTable()`. (Check the [Preset ccomposition](https://docs.evidentlyai.com/reference/all-metrics).

You can point to any of them as a `metric_id`, depending on what you’d like to plot.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e3813264a9b6935d5747cc85520da64095898f07%2Ffield_path.png?alt=media)

Most Metrics contain multiple measurements inside (MetricResults) and some render data. To point to the specific value, use the `field path`.

To find available fields in the chosen Metric, you can explore the contents of the individual snapshot or use automated suggestions in UI or Python.

{% tabs %}
{% tab title="Open the snapshot" %}
Each snapshot is a JSON file. You can download or open it in Python to see the available fields.

Alternatively, you can [generate a Report](/user-guide/tests-and-reports/get-reports) with the selected Metrics on any test data. Get the output as a Python dictionary using `as_dict()` and explore the keys with field names.

Here is a partial example of the contents of `DatasetDriftMetric()`:

```python
'number_of_columns': 15,
'number_of_drifted_columns': 5,
'share_of_drifted_columns': 0.3333333333333333,
'dataset_drift': False,
```

Once you identify the value you’d like to plot (e.g., `number_of_drifted_columns`), pass it as the `field_path` to the `PanelValue` parameter. Include the `DatasetDriftMetric` as the `metric_id`.

Other Metrics and Tests follow the same logic.
{% endtab %}

{% tab title="Python autocomplete" %}
You can use autocomplete in interactive Python environments (like Jupyter notebook or Colab) to see available fields inside a specific Metric. They appear as you start typing the `.fields.` path for a specific Metric.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-999cd3adb1ead66de0ed15efb357aea929633925%2Fmetric_fields_autocomplete-min.png?alt=media\&token=1ffc6585-f941-42dd-b400-2395e980a843)

**Note**: some types of values (e.g. mean, sum, max, min) will not be visible using this method. This is because they match the names of the standard Python fields.
{% endtab %}

{% tab title="Suggestions in UI" %}
When working in the Evidently Cloud, you can see available fields in the drop-down menu as you add a new Panel.
{% endtab %}
{% endtabs %}

Note that some data inside the snapshots cannot currently be plotted on a monitoring Dashboard (for example, render data or dictionaries). You can only plot values that exist as individual data points or histograms.


# Datasets


# Datasets overview

Introduction to Datasets in Evidently Platform.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

This section covers how work with Datasets in Evidently Platform.

{% hint style="success" %}
Supported in: `Evidently Cloud` and `Evidently Enterprise`.
{% endhint %}

Datasets are key for evaluating and monitoring AI products.

They consist of data collected from your application, which you can analyze manually or set up for automated checks. You can bring in existing datasets or capture data from your live production systems or test environments.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e0108e7420c1368dce9fac5f9ef8ecde4815f30e%2Fdataset-min.png?alt=media)

It's optional: you can also run all evals locally and only upload summary results.

## Uploading Datasets

You can add datasets to the platform in several ways:

* **Direct upload**. Upload a dataset through the user interface using a CSV file. For automation, use the Python API to [upload datasets](/user-guide/datasets/datasets_api) programmatically. Once uploaded, you can start analyzing the data on the platform.
* **Created from Traces**. When you use the [Tracing](/user-guide/tracing/tracing_overview), Evidently automatically creates a structured tabular dataset from traces. You can also curate subsets of this data for focused analysis.
* **Included with** [**Reports or Test Suites**](/user-guide/evaluations/snapshots). When running a local evaluation in Python, attach the Dataset associated with a specific Report or Test Suite. This allows you to store raw data alongside summary evaluation results. It’s optional.

{% content-ref url="/pages/DtVdS9rYXlvx7X5DkAkS" %}
[Work with Datasets](/user-guide/datasets/datasets_api)
{% endcontent-ref %}

## When do you need Datasets?

Evidently provides flexibility in how you structure datasets, supporting various workflows.&#x20;

Common scenarios for Datasets include:

* **Collecting ML inference logs or LLM traces**. You can gather raw logs from your production or experimental runs, which capture system inputs, outputs, and any other relevant metadata. Once you’ve uploaded these logs, you can view the data and run evaluations. This serves as a source of truth on the performance of your AI system.
* **Debugging evaluation results**. After running an evaluation, Evidently generates a new version of your Dataset that includes extra columns. For instance, if you scored model responses by sentiment, you’d get a new column with those sentiment scores. This helps you identify specific failed rows and debug quality issues efficiently.
* **Organizing evaluation datasets**. You can upload any dataset for storage and easy access. This could include curated input test scenarios, golden datasets with approved reference outputs, or newly labeled datasets. You can access these datasets programmatically for regression testing or evaluations or build your workflow directly within the UI.


# Work with Datasets

Working with Datasets on Evidently Platform.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

To access or upload your Datasets, navigate to the [Datasets page](https://app.evidently.cloud/datasets) in the user interface.

You will be able to view all Datasets: created from traces, uploaded directly to the platform, or generated as a result of an evaluation.

## User interface

### Upload a CSV file

Once you go to the Datasets page, you can upload any existing dataset as a CSV file directly there. Click on "Add dataset".

When you upload the Dataset, you must also add a [column mapping](/user-guide/input-data/column-mapping). This allows Evidently to understand the meaning of specific columns and prepare your Dataset for future evaluations.

## Python API

To work with Datasets programmatically from Python, you must first [connect to Workspace](/setup/cloud_account).

### Upload the Dataset

Prepare your dataset as a Pandas DataFrame. To upload a dataframe `df` to the specified Project in workspace `ws`, use the `add_dataset` method:

```python
ws.add_dataset(
    df,
    name = "dataset_name",
    project_id = project.id, 
    description = "Optional description")
```

You must always specify the "dataset\_name" you want to see in the UI. The description is optional.

To get a Project ID, grab it from the existing Project page or create a new Project first. (How to [work with Projects](https://github.com/evidentlyai/docs-old/blob/main/project/projects_overview.md).)

### Download Dataset

You can also download the Dataset from Evidently platform to your local environment. For example, if you store the test dataset on the platform and want to pull it into your local evaluation script.

Use the `load_dataset` method:

```python
downloaded_df = ws.load_dataset(dataset_id = "YOUR_DATASET_ID") 
```

### Include Dataset

You can also include Datasets when you upload Reports or Test Suites to the platform. This way, after running an evaluation locally you simultaneously upload the evaluation result and the Dataset it was generated for, with added scores if applicable.

Use `include_data` parameter (False by default):

```python
ws.add_report(project.id, data_report, include_data=True)
```

This is optional. Check the docs on [generating snapshots](/user-guide/evaluations/snapshots) for details.


# Customization


# Data drift parameters

How to set custom data drift conditions and thresholds for tabular and text data.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate Reports or Test Suites with default parameters.
* You know how to pass custom parameters for Reports or Test Suites.
* You know how to use Column Mapping to set the input data type.

## Default

All Presets, Tests, and Metrics that include data or target (prediction) drift evaluation use the default [Data Drift algorithm](/reference/data-drift-algorithm). It automatically selects an appropriate drift detection method based on the feature type and volume.

You can override the defaults by passing a custom parameter to the chosen Test, Metric, or Preset. You can define the drift detection method, the threshold, or both.

## Code example

You can refer to an example How-to-notebook showing how to pass custom drift parameters:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb>" %}

## Examples

To set a custom drift method and threshold on the **column level**:

```python
ColumnDriftMetric(column_name='feature1', stattest='wasserstein', stattest_threshold=0.2) 
```

If you have a Preset, Test or Metric that checks for drift in **multiple columns** at the same time, you can set a custom drift method for all columns, all numerical/categorical columns, or for each column individually.

Here is how you set the drift detection method for all categorical columns:

```python
DataDriftPreset(cat_stattest='ks', cat_statest_threshold=0.05)
```

To set a custom condition for the **dataset drift** (share of drifting columns in the dataset) in the relevant Metrics or Presets:

```python
DatasetDriftMetric(drift_share=0.7)
```

Note that this works slightly differently for Tests. To set a custom condition for the **dataset drift** when you run a relevant **Test**, you should set a condition for the share of drifted features using standard `lt` and `gt` parameters:

```python
TestShareOfDriftedColumns(lt=0.5)
```

When you set drift threshold for `ColumnDriftTest()`, you should use `stattest_threshold` and other parameters the same way as it works in Metrics (not `lt` and `gt`).

## Tabular drift detection

The following methods and parameters apply to **tabular** data (as parsed automatically or specified as numerical or categorical columns in the column mapping).

### Drift parameters - Tabular

The following drift detection parameters are available in the `DataDriftTable()`, `DatasetDriftMetric()`, `ColumnDriftMetric()`, related Tests, and Presets that contain them.

| Parameter                                                                                  | Description                                                                                                                                                                                                               |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stattest`                                                                                 | Defines the drift detection method for a given column (if a single column is tested), or all columns in the dataset (if multiple columns are tested).                                                                     |
| `stattest_threshold`                                                                       | <p>Sets the drift threshold in a given column or all columns.<br>The threshold meaning varies based on the drift detection method, e.g., it can be the value of a distance metric or a p-value of a statistical test.</p> |
| `drift_share`                                                                              | Defines the share of drifting columns as a condition for Dataset Drift in `DatasetDriftMetric` or inside a Preset.                                                                                                        |
| <p><code>cat\_stattest</code><br><code>cat\_stattest\_threshold</code></p>                 | Sets the drift method and/or threshold for all categorical columns in the dataset.                                                                                                                                        |
| <p><code>num\_stattest</code><br><code>num\_stattest\_threshold</code></p>                 | Sets the drift method and/or threshold for all numerical columns in the dataset.                                                                                                                                          |
| <p><code>per\_column\_stattest</code><br><code>per\_column\_stattest\_threshold</code></p> | Sets the drift method and/or threshold for the listed columns (accepts a dictionary).                                                                                                                                     |

{% hint style="info" %}
**How to check available parameters.** You can verify which parameters are available for a specific test, metric, or preset in the [All tests](/reference/all-tests) or [All metrics](/reference/all-metrics) tables or consult the [API reference](https://github.com/evidentlyai/docs-old/blob/main/customization/\[../reference/api-reference]\(https:/docs.evidentlyai.com/reference/api-reference\)/README.md)
{% endhint %}

### Drift detection methods - Tabular

To use the following drift detection methods, pass them using the `stattest` parameter.

| StatTest                                                         | Applicable to                                                                                                                      | Drift score                                                                                                                               |
| ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| <p><code>ks</code><br>Kolmogorov–Smirnov (K-S) test</p>          | <p>tabular data<br>only numerical<br><br><strong>Default method for numerical data, if <= 1000 objects</strong></p>                | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>chisquare</code><br>Chi-Square test</p>                 | <p>tabular data<br>only categorical<br><br><strong>Default method for categorical with > 2 labels, if <= 1000 objects</strong></p> | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>z</code><br>Z-test</p>                                  | <p>tabular data<br>only categorical<br><br><strong>Default method for binary data, if <= 1000 objects</strong></p>                 | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>wasserstein</code><br>Wasserstein distance (normed)</p> | <p>tabular data<br>only numerical<br><br><strong>Default method for numerical data, if > 1000 objects</strong></p>                 | <p>returns <code>distance</code><br>drift detected when <code>distance</code> >= <code>threshold</code><br>default threshold: 0.1</p>     |
| <p><code>kl\_div</code><br>Kullback-Leibler divergence</p>       | <p>tabular data<br>numerical and categorical</p>                                                                                   | <p>returns <code>divergence</code><br>drift detected when <code>divergence</code> >= <code>threshold</code><br>default threshold: 0.1</p> |
| <p><code>psi</code><br>Population Stability Index (PSI)</p>      | <p>tabular data<br>numerical and categorical</p>                                                                                   | <p>returns <code>psi\_value</code><br>drift detected when <code>psi\_value</code> >= <code>threshold</code><br>default threshold: 0.1</p> |
| <p><code>jensenshannon</code><br>Jensen-Shannon distance</p>     | <p>tabular data<br>numerical and categorical<br><br><strong>Default method for categorical, if > 1000 objects</strong></p>         | <p>returns <code>distance</code><br>drift detected when <code>distance</code> >= <code>threshold</code><br>default threshold: 0.1</p>     |
| <p><code>anderson</code><br>Anderson-Darling test</p>            | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>fisher\_exact</code><br>Fisher's Exact test</p>         | <p>tabular data<br>only categorical</p>                                                                                            | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>cramer\_von\_mises</code><br>Cramer-Von-Mises test</p>  | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>g-test</code><br>G-test</p>                             | <p>tabular data<br>only categorical</p>                                                                                            | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>hellinger</code><br>Hellinger Distance (normed)</p>     | <p>tabular data<br>numerical and categorical</p>                                                                                   | <p>returns <code>distance</code><br>drift detected when <code>distance</code> >= <code>threshold</code><br>default threshold: 0.1</p>     |
| <p><code>mannw</code><br>Mann-Whitney U-rank test</p>            | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>ed</code><br>Energy distance</p>                        | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>distance</code><br>drift detected when <code>distance</code> >= <code>threshold</code><br>default threshold: 0.1</p>     |
| <p><code>es</code><br>Epps-Singleton tes</p>                     | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>t\_test</code><br>T-Test</p>                            | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>empirical\_mmd</code><br>Empirical-MMD</p>              | <p>tabular data<br>only numerical</p>                                                                                              | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |
| <p><code>TVD</code><br>Total-Variation-Distance</p>              | <p>tabular data<br>only categorical</p>                                                                                            | <p>returns <code>p\_value</code><br>drift detected when <code>p\_value</code> < <code>threshold</code><br>default threshold: 0.05</p>     |

## Text drift detection

Text drift detection applies to columns with **raw text data**, as specified in column mapping.

{% hint style="info" %}
**Embedding drift detection.** If you work with embeddings, you can use [Embeddings Drift Detection methods](/user-guide/customization/embeddings-drift-parameters).
{% endhint %}

### Drift parameters - Text

The following text drift detection parameters are available in the `DataDriftTable()`, `DatasetDriftMetric()`, `ColumnDriftMetric()`, related Tests and Presets that contain them.

| Parameter                 | Description                                                                                                                                        |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `stattest`                | Defines the drift detection method for a given column that contains text data, or for all columns in the dataset if all columns contain text data. |
| `stattest_threshold`      | Sets the threshold as a drift detection parameter.                                                                                                 |
| `text_stattest`           | Defines the drift detection method for all text columns in the dataset.                                                                            |
| `text_stattest_threshold` | Sets the threshold as a drift detection parameter.                                                                                                 |

### Drift detection methods - Text

To use the following text drift detection methods, pass them using the `stattest` parameter.

| StatTest                                                                                                                      | Description                                                                                                                                                                                        | Drift score                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><code>perc\_text\_content\_drift</code><br>Text content drift (domain classifier, with statistical hypothesis testing)</p> | <p>Applies only to text data. Trains a classifier model to distinguish between text in “current” and “reference” datasets.<br><br><strong>Default for text data when <= 1000 objects.</strong></p> | <ul><li>returns <code>roc\_auc</code> of the classifier as a <code>drift\_score</code></li><li>drift detected when <code>roc\_auc</code> > possible ROC AUC of the random classifier at a set percentile</li><li><code>threshold</code> sets the percentile of the possible ROC AUC values of the random classifier to compare against</li><li>default threshold: 0.95 (95th percentile)</li><li><code>roc\_auc</code> values can be 0 to 1 (typically 0.5 to 1); a higher value means more confident drift detection</li></ul> |
| <p><code>abs\_text\_content\_drift</code><br>Text content drift (domain classifier)</p>                                       | <p>Applies only to text data. Trains a classifier model to distinguish between text in “current” and “reference” datasets.<br><br><strong>Default for text data when > 1000 objects.</strong></p>  | <ul><li>returns <code>roc\_auc</code> of the classifier as a <code>drift\_score</code></li><li>drift detected when <code>roc\_auc</code> > <code>threshold</code></li><li><code>threshold</code> sets the ROC AUC threshold</li><li>default threshold: 0.55</li><li><code>roc\_auc</code> values can be 0 to 1 (typically 0.5 to 1); a higher value means more confident drift detection</li></ul>                                                                                                                              |

### Text descriptors drift

You can also check for distribution drift in text descriptors (such as text length, etc.)

To use this method, call a separate `TextDescriptorsDriftMetric()`. You can pass any of the tabular drift detection methods as a parameter.

```python
report = Report(metrics=[
    TextDescriptorsDriftMetric("Review_Text"),
])

report.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)
report
```


# Embeddings drift parameters

How to customize data drift detection for embeddings.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate Reports or Test Suites with default parameters.
* You know how to pass custom parameters for Reports or Test Suites.
* You know how to use Column Mapping to map embeddings in the input data.

## Code example

You can refer to an example How-to-notebook showing how to pass parameters for different embeddings drift detection methods:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_calculate_embeddings_drift.ipynb>" %}

## Default

When you calculate embeddings drift, Evidently automatically applies the default drift detection method (“model”).

In Reports:

```python
report = Report(metrics=[
    EmbeddingsDriftMetric('small_subset')
])
```

In Test Suites:

```python
tests = TestSuite(tests=[
    TestEmbeddingsDrift(embeddings_name='small_subset')
])
```

It works the same inside presets, like `DataDriftPreset`.

## Embedding parameters - Metrics and Tests

You can override the defaults by passing a custom `drift_method` parameter to the relevant Metric or Test. You can define the embeddings drift detection method, the threshold, or both.

Pass the `drift_method` parameter:

```python
from evidently.metrics.data_drift.embedding_drift_methods import model
report = Report(metrics = [
    EmbeddingsDriftMetric('small_subset', 
                          drift_method = model()
                         )
])
```

## Embedding parameters - Presets

When you use `NoTargetPerformanceTestPreset`, `DataDriftTestPreset` or `DataDriftPreset` you can specify which subsets of columns with embeddings to include using `embeddings`, and the drift detection method using `embeddings_drift_method`.

By default, the Presets will include all columns mapped as containing embeddings in `column_mapping`.

To exclude columns with embeddings:

```python
embeddings = []
```

To specify which sets of columns to include (with the default drift detection method):

```python
embeddings = [‘set1’, ‘set2’]
```

To specify which sets of columns to include, and specify the method:

```python
embeddings = [‘set1’, ‘set2’]
embeddings_drift_method = {‘set1’: model(), ‘set2’: ratio())}
```

## Embedding drift detection methods

Currently 4 embeddings drift detection methods are available.

| Embeddings drift detection method                    | Description and default                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><code>drift\_method=model</code><br>(Default)</p> | <ul><li>A binary classifier model to distinguish between embeddings in “current” and “reference” distributions.</li><li>Returns <strong>ROC AUC</strong> as a <code>drift\_score</code>.</li><li>Drift detected when <code>drift\_score</code> > <code>threshold</code> or when <code>drift\_score</code> > ROC AUC of the random classifier at a set <code>quantile\_probability</code>.</li><li>Default threshold: 0.55 (ROC AUC).</li><li>Default quantile\_probability: 0.95. Applies when bootstrap is True; default True if <= 1000 objects.</li></ul>                                                                                                         |
| `drift_method=ratio`                                 | <ul><li>Computes the distribution drift between individual embedding components using any of the tabular numerical drift detection methods available in Evidently.</li><li>Default tabular drift detection method: Wasserstein distance, with the 0.1 threshold.</li><li>Returns the <strong>share of drifted embeddings</strong> as <code>drift\_score</code>.</li><li>Drift detected when <code>drift\_score</code> > <code>threshold</code></li><li>Default threshold: 0.2 (share of drifted embedding components).</li></ul>                                                                                                                                     |
| `drift_method=distance`                              | <ul><li>Computes the distance between average embeddings in “current” and “reference” datasets using a specified distance metric (euclidean, cosine, cityblock, chebyshev). Default: <code>euclidean</code>.</li><li>Returns the <strong>distance metric value</strong> as <code>drift\_score</code>.</li><li>Drift detected when drift\_score > threshold or when <code>drift\_score</code> > obtained distance in reference at a set <code>quantile\_probability</code>.</li><li>Default threshold: 0.2 (relevant for Euclidean distance).</li><li>Default quantile\_probability: 0.95. Applies when bootstrap is True; default True if <= 1000 objects.</li></ul> |
| `drift_method=mmd`                                   | <ul><li>Computes the Maximum Mean Discrepancy (MMD)</li><li>Returns the <strong>MMD value</strong> as a <code>drift\_score</code></li><li>Drift detected when <code>drift\_score</code> > <code>threshold</code> or when <code>drift\_score</code> > obtained MMD values in reference at a set <code>quantile\_probability</code>.</li><li>Default threshold: 0.015 (MMD).</li><li>Default quantile\_probability: 0.95. Applies when bootstrap is True; default True if <= 1000 objects.</li></ul>                                                                                                                                                                   |

If you specify an embedding drift detection method but do not pass additional parameters, defaults will apply.

You can also specify parameters for any chosen method. Since the methods are different, each has a different set of parameters. Note that you should pass the parameters **directly to the chosen drift detection method**, not to the Metric.

### Model-based (“model”)

```python
report = Report(metrics = [
    EmbeddingsDriftMetric('small_subset', 
                          drift_method = model(
                              threshold = 0.55,
                              bootstrap = None,
                              quantile_probability = 0.05,
                              pca_components = None,
                          )
                         )
])
```

| Parameter                         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `threshold`                       | <p>Sets the threshold for drift detection (ROC AUC). Drift is detected when <code>drift\_score</code> > <code>threshold</code>.<br>Applies when <code>bootstrap</code> != True.<br><br><strong>Default: 0.55</strong>.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `bootstrap` (optional)            | <p>Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing.<br>If applied, the ROC AUC of the classifier is compared to the ROC AUC of the random classifier at a set percentile. The calculation is repeated 1000 times with randomly assigned target class probabilities. This produces a distribution of random roc\_auc scores with a mean of 0,5. We then take the 95th percentile (default) of this distribution and compare it to the ROC-AUC score of the classifier. If the classifier score is higher, data drift is detected.<br><br><strong>Default: True if <= 1000 objects, False if > 1000 objects.</strong></p> |
| `quantile_probability` (optional) | <p>Sets the percentile of the possible ROC AUC values of the random classifier to compare against.<br>This applies when bootstrap is True.<br><br><strong>Default: 0.95</strong></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `pca_components` (optional)       | <p>The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of <code>pca\_components</code>.<br><br><strong>Default: None.</strong></p>                                                                                                                                                                                                                                                                                                                                                                                                                                      |

### Maximum mean discrepancy (“mmd”)

```python
report = Report(metrics = [
    EmbeddingsDriftMetric('small_subset', 
                          drift_method = mmd(
                              threshold = 0.015,
                              bootstrap = None,
                              quantile_probability = 0.05,
                              pca_components = None,
                          )
                         )
])
```

| Parameter                         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `threshold`                       | <p>Sets the threshold value of MMD for drift detection. Drift is detected when <code>drift\_score</code> > <code>threshold</code>.<br>Applies when <code>bootstrap</code> != True.<br><br><strong>Default: 0.015 (MMD)</strong>.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `bootstrap` (optional)            | <p>Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing.<br><br>If applied, the value of MMD between reference and current (<em>mmd\_0</em>) is tested against possible MMD values in reference. We randomly split the reference data into two parts and compute MMD values (<em>mmd\_i</em>) between them. The calculation is repeated 100 times. This produces a distribution of MMD values obtained for a reference dataset. We then take the 95th percentile (default) of this distribution and compare it to the MMD between reference and current datasets. If the <em>mmd\_0</em> > <em>mmd\_95</em>, data drift is detected.<br><br><strong>Default: True if <= 1000 objects, False if > 1000 objects.</strong></p> |
| `quantile_probability` (optional) | <p>Sets the percentile of the possible MMD values in reference to compare against.<br>Applies when <code>bootstrap</code> == True.<br><br><strong>Default: 0.95.</strong></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `pca_components` (optional)       | <p>The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of <code>pca\_components</code>.<br><br><strong>Default: None.</strong></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

### Share of drifted embedding components (“ratio”)

```python
report = Report(metrics = [
    EmbeddingsDriftMetric('small_subset', 
                          drift_method = ratio(
                              component_stattest = 'wasserstein',
                              component_stattest_threshold = 0.1,
                              threshold = 0.2,
                              pca_components = None,
                          )
                         )
])
```

| Parameter                                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `component_stattest` (optional)           | <p>Sets the tabular drift detection method (any of the tabular drift detection methods for numerical features available in Evidently).<br><br><strong>Default: Wasserstein</strong></p>                                                                                                                                                                                                                                                                                       |
| `component_stattest_threshold` (optional) | <p>Sets the threshold for drift detection for individual embedding components. Drift is detected when <code>drift\_score</code> > <code>component\_stattest\_threshold</code> in case of distance/divergence metrics where the threshold is the metric value or <code>drift\_score</code> < <code>component\_stattest\_threshold</code> in case of statistical tests where the threshold is the p-value.<br><br><strong>Default: 0.1</strong> (relevant for Wasserstein).</p> |
| `threshold` (optional)                    | <p>Sets the threshold (share of drifted embedding components) for drift detection for the overall dataset.<br><br><strong>Default: 0.2</strong></p>                                                                                                                                                                                                                                                                                                                           |
| `pca_components` (optional)               | <p>The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of <code>pca\_components</code>.<br><br><strong>Default: None</strong>.</p>                                                                                                                                                                                                                                                |

### Distance-based methods (“distance”)

```python
report = Report(metrics = [
    EmbeddingsDriftMetric('small_subset', 
                          drift_method = distance(
                              dist = 'euclidean', #"euclidean", "cosine", "cityblock" or "chebyshev"
                              threshold = 0.2,
                              pca_components = None,
                              bootstrap = None,
                              quantile_probability = 0.05
                          )
                         )
])
```

| Parameter                                                                                                                                                                       | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><code>dist</code> (optional)<br><br>Available:<br><code>euclidean</code><br><code>cosine</code><br><code>cityblock</code> (manhattan distance)<br><code>chebyshev</code></p> | <p>Sets the distance metric for drift detection.<br><br><strong>Default: Euclidean distance</strong></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `threshold` (optional)                                                                                                                                                          | <p>Sets the threshold for drift detection. Drift is detected when <code>drift\_score</code> > <code>threshold</code>.<br>Applies when bootstrap != True<br><br><strong>Default: 0.2</strong> (relevant for euclidean distance)</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `bootstrap` (optional)                                                                                                                                                          | <p>Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing.<br><br>If applied, the distance between reference and current is tested against possible distance values in reference. We randomly split the reference data into two parts and compute the distance between them. The calculation is repeated 100 times. This produces a distribution of distance values obtained for a reference dataset. We then take the 95th percentile (default) of this distribution and compare it to the distance between reference and current datasets. If the distance between the reference and current is higher than the 95th percentile of the distance obtained for the reference dataset, the drift is detected.<br><br><strong>Default: True if <= 1000 objects, False if > 1000 objects</strong>.</p> |
| `quantile_probability` (optional)                                                                                                                                               | <p>Sets the percentile of the possible distance values in reference to compare against.<br>Applies when <code>bootstrap</code> == True.<br><br><strong>Default: 0.95</strong>.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `pca_components` (optional)                                                                                                                                                     | <p>The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of <code>pca\_components</code>.<br><br><strong>Default: None</strong>.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |


# Feature importance in data drift

How to show feature importance in Data Drift evaluations.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

You can add feature importances to the dataset-level data drift Tests and Metrics:

* `DataDriftTable`
* `TestShareOfDriftedColumns`

## Code example

Notebook example on showing feature importance:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_add_feature_importances_to_drift.ipynb>" %}

## Compute feature importances

By default, the feature importance column is not shown. To display them, you must set the `feature_importance` parameter as `True`.

```python
report = Report(metrics = [
    DataDriftTable(feature_importance=True)
])
```

If you do not specify anything else, Evidently will train a random forest model using the provided dataset and derive the feature importances.

**Notes**:

* This is only possible if your dataset contains the `target` column.
* If you have both `current` and `reference` datasets, two different models will be trained. You will have two columns with feature importance: one for `reference` and one for `current` data.
* If your dataset also contains the `prediction` column, you should clearly label it using Column Mapping to avoid it being treated as a feature.

### Pass your own importances

You can also pass the list of feature importances derived during the model training process. This is a recommended option.

In this case, pass it as a list using the `additional_data` parameter when running the Report.

```python
report = Report(metrics = [
    DataDriftTable(feature_importance=True)
])
report.run(reference_data=reference,
           current_data=current.loc['2011-01-29 00:00:00':'2011-02-07 23:00:00'],
           column_mapping=column_mapping,
           additional_data = {'current_feature_importance':
              dict(map(lambda i,j : (i,j), numerical_features + categorical_features, regressor.feature_importances_))
            }
           )
```

You can pass the `current_feature_importance` – a single column will appear in this case. You can also optionally pass `reference_feature_importance`.


# Text evals with LLM-as-judge

How to use external LLMs to score text data.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate Reports or Test Suites for text data using Descriptors.
* You know how to pass custom parameters for Reports or Test Suites.
* You know how to specify text data in column mapping.

You can use external LLMs to score your text data. This method lets you evaluate texts by any custom criteria you define in a prompt.

The LLM “judge” will return a numerical score or a category for each text in a column. It works like any other Evidently `descriptor`: you can view and analyze scores, run conditional Tests, and monitor evaluation results in time.

Evidently currently supports scoring data using Open AI LLMs (more LLMs coming soon). Use the `LLMEval` descriptor to create an evaluator with any custom criteria, or choose any of the built-in evaluators (like detection of Denials, Personally identifiable information, etc.).

## LLM Eval

### Code example

Refer to a How-to example:

{% embed url="<https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_use_llm_judge_template.ipynb>" %}

{% hint style="info" %}
**OpenAI key.** Add the token as the environment variable: [see docs](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety). You will incur costs when running this eval.
{% endhint %}

### Built-in evaluators

You can use built-in evaluators that include pre-written prompts for specific criteria. These descriptors default to returning a binary category label with reasoning and using `gpt-4o-mini` model from OpenAI.

**Imports**. Import the `LLMEval` and built-in evaluators you want to use:

```python
from evidently.descriptors import LLMEval, NegativityLLMEval, PIILLMEval, DeclineLLMEval, BiasLLMEval, ToxicityLLMEval, ContextQualityLLMEval
```

**Get a Report**. To create a Report, simply list them like any other descriptor:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        NegativityLLMEval(),
        PIILLMEval(),
        DeclineLLMEval()
    ])
])
```

**Parametrize evaluators**. You can switch the output format from `category` to `score` (0 to 1) or exclude the reasoning:

```python
report = Report(metrics=[
    TextEvals(column_name="question", descriptors=[
        NegativityLLMEval(include_category=False),   
        PIILLMEval(include_reasoning=False), 
        DeclineLLMEval(include_score=True)
    ])
])
```

**Run descriptors over two columns**. An evaluator that assesses if the context contains enough information to answer the question requires both columns. Run the evaluation over the `context` column and pass the name of the column containing the `question` as a parameter.

```python
report = Report(metrics=[
    TextEvals(column_name="context", descriptors=[
        ContextQualityLLMEval(question="question"),
    ])
])
```

{% hint style="info" %}
**Which descriptors are there?** See the list of available built-in descriptors in the [All Metrics](/reference/all-metrics) page.
{% endhint %}

### Custom LLM judge

You can also create a custom LLM evaluator using the provided templates. You specify the parameters and evaluation criteria, and Evidently will generate the complete evaluation prompt to send to the LLM together with the evaluation data.

**Imports**. To import the template for the Binary Classification evaluator prompt:

```python
from evidently.features.llm_judge import BinaryClassificationPromptTemplate
```

**Fill in the template**. Include the definition of your `criteria`, names of categories, etc. For example, to define the prompt for "conciseness" evaluation:

```python
custom_judge = LLMEval(
    subcolumn="category",
    template = BinaryClassificationPromptTemplate(      
        criteria = """Conciseness refers to the quality of being brief and to the point, while still providing all necessary information.
            A concise response should:
            - Provide the necessary information without unnecessary details or repetition.
            - Be brief yet comprehensive enough to address the query.
            - Use simple and direct language to convey the message effectively.
        """,
        target_category="concise",
        non_target_category="verbose",
        uncertainty="unknown",
        include_reasoning=True,
        pre_messages=[("system", "You are a judge which evaluates text.")],
        ),
    provider = "openai",
    model = "gpt-4o-mini",
    display_name="Conciseness",
)
```

See the explanation of each parameter below.

You do not need to explicitly ask the LLM to classify your input into two classes, ask for reasoning, or format it specially. This is already part of the template.

**Using text from multiple columns**. You can use this template to run evals that use data from multiple columns.

For example, you can evaluate the output in the `response` column, simultaneously including data from the `context` or `question` column. This applies to scenarios like classifying the relevance of the response in relation to the question or its factuality based on context, etc.

Pass the names of the `additional_columns` in your dataset and reference the `{column}` when you write the `criteria`. When you run the eval, Evidently will insert the contents of each text in the corresponding column in the evaluation prompt.

```python
multi_column_judge = LLMEval(
        subcolumn="category",
        additional_columns={"question": "question"},
        template=BinaryClassificationPromptTemplate(
            criteria=""""Relevance" refers to the response directly addresses the question and effectively meets the user's intent.  
Relevant answer is an answer that directly addresses the question and effectively meets the user's intent.

=====
{question}
=====
            """,
            target_category="Relevant",
            non_target_category="Irrelevant",
            include_reasoning=True,
            pre_messages=[("system",
                           "You are an expert evaluator assessing the quality of a Q&A system. Your goal is to determine if the provided answer is relevant to the question based on the criteria below.")],
        ),
        provider="openai",
        model="gpt-4o-mini",
        display_name="Relevancy"
    )
```

You do not need to explicitly include the name of your primary column in the evaluation prompt. Since you include it as `column_name` in the `TextEvals` preset, it will be automatically passed to the template.

```
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        multi_column_judge
    ])
])
```

### Parameters

#### LLMEval Parameters

| Parameter   | Description                                                                                  |
| ----------- | -------------------------------------------------------------------------------------------- |
| `subcolumn` | Specifies the type of descriptor. Available values: `category`, `score`.                     |
| `template`  | Forces a specific template for evaluation. Available: `BinaryClassificationPromptTemplate`.  |
| `provider`  | The provider of the LLM to be used for evaluation. Available: `openai`.                      |
| `model`     | Specifies the model used for evaluation within the provider, e.g., `gpt-3.5-turbo-instruct`. |

#### BinaryClassificationPromptTemplate

| Parameter             | Description                                                                                                                                                                                                                   |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `criteria`            | Free-form text defining evaluation criteria.                                                                                                                                                                                  |
| `target_category`     | Name of the desired or positive category.                                                                                                                                                                                     |
| `non_target_category` | Name of the undesired or negative category.                                                                                                                                                                                   |
| `uncertainty`         | Category to return when the provided information is not sufficient to make a clear determination. Available: `unknown` (Default), `target`, `non_target`.                                                                     |
| `include_reasoning`   | Specifies whether to include reasoning in the classification. Available: `True` (Default), `False`. It will be included with the result.                                                                                      |
| `pre_messages`        | List of system messages that set context or instructions before the evaluation task. For example, you can explain the evaluator role ("you are an expert..") or context ("your goal is to grade the work of an intern..")     |
| `additional_columns`  | A dictionary of additional columns present in your dataset to include in the evaluation prompt. Use it to map the column name to the placeholder name you reference in the `criteria`. For example: `({"mycol": "question"}`. |

## OpenAIPrompting

There is an earlier implementation of this approach with `OpenAIPrompting` descriptor. See the documentation below.

<details>

<summary>OpenAIPrompting Descriptor</summary>

To import the Descriptor:

```python
from evidently.descriptors import OpenAIPrompting
```

Define a prompt. This is a simplified example:

```python
pii_prompt = """
Please identify whether the below text contains personally identifiable information, such as name, address, date of birth, or other.
Text: REPLACE 
Use the following categories for PII identification:
1 if text contains PII
0 if text does not contain PII
0 if the provided data is not sufficient to make a clear determination
Return only one category.
"""
```

The prompt has a REPLACE placeholder that will be filled with the texts you want to evaluate. Evidently will take the content of each row in the selected column, insert into the placeholder position in a prompt and pass it to the LLM for scoring.

To compute the score for the column `response` and get a summary Report:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        OpenAIPrompting(
            prompt=pii_prompt,
            prompt_replace_string="REPLACE",
            model="gpt-3.5-turbo-instruct",
            feature_type="cat",
            display_name="PII for response (by gpt3.5)"
            ),       
    ])
])
```

You can do the same for Test Suites.

### Descriptor parameters

* **`prompt: str`**
  * The text of the evaluation prompt that will be sent to the LLM.
  * Include at least one placeholder string.
* **`prompt_replace_string: str`**
  * A placeholder string within the prompt that will be replaced by the evaluated text.
  * The default string name is "REPLACE".
* **`feature_type: str`**
  * The type of Descriptor the prompt will return.
  * Available types: `num` (numerical) or `cat` (categorical).
  * This affects the statistics and default visualizations.
* **`context_replace_string: str`**
  * An optional placeholder string within the prompt that will be replaced by the additional context.
  * The default string name is "CONTEXT".
* **`context: Optional[str]`**
  * Additional context that will be added to the evaluation prompt, which **does not change** between evaluations.
  * Examples: a reference document, a set of positive and negative examples, etc.
  * Pass this context as a string.
  * You cannot use `context` and `context_column` simultaneously.
* **`context_column: Optional[str]`**
  * Additional context that will be added to the evaluation prompt, which is **specific to each row**.
  * Examples: a chunk of text retrieved from reference documents for a specific query.
  * Point to the column that contains the context.
  * You cannot use `context` and `context_column` simultaneously.
* **`model: str`**
  * The name of the OpenAI model to be used for the LLM prompting, e.g., `gpt-3.5-turbo-instruct`.
* **`openai_params: Optional[dict]`**
  * A dictionary with additional parameters for the OpenAI API call.
  * Examples: temperature, max tokens, etc.
  * Use parameters that OpenAI API accepts for a specific model.
* **`possible_values: Optional[List[str]]`**
  * A list of possible values that the LLM can return.
  * This helps validate the output from the LLM and ensure it matches the expected categories.
  * If the validation does not pass, you will get `None` as a response label.
* **`display_name: Optional[str]`**
  * A display name visible in Reports and as a column name in tabular export.
  * Use it to name your Descriptor.

</details>


# Text evals with HuggingFace

How to use models available on HuggingFace as text Descriptors.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate Reports or Test Suites for text data using Descriptors.
* You know how to pass custom parameters for Reports or Test Suites.
* You know to specify text data in column mapping.

You can use an external machine learning model to score text data. This method lets you evaluate texts based on any criteria from the source model, e.g. classify it into a set number of labels.

The model you use must return a numerical score or a category for each text in a column. You will then be able to view scores, analyze their distribution or run conditional tests through the usual Descriptor interface.

Evidently supports using HuggingFace models: use the general `HuggingFaceModel()` descriptor to select models on your own or simplified interfaces like `HuggingFaceToxicityModel()`.

## Code example

You can refer to an end-to-end example with different Descriptors:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_evaluate_llm_with_text_descriptors.ipynb>" %}

To import the Descriptor:

```python
from evidently.descriptors import HuggingFaceModel, HuggingFaceToxicityModel
```

To get a Report with a Toxicity score for the `response` column:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        HuggingFaceToxicityModel(toxic_label="hate"),
    ])
])
```

To get a Report with with several different scores using the general `HuggingFaceModel()` descriptor:

```python
report = Report(metrics=[
    TextEvals(column_name="response", descriptors=[
        HuggingFaceModel(model="DaNLP/da-electra-hatespeech-detection", display_name="Response Toxicity"),
        HuggingFaceModel(model="SamLowe/roberta-base-go_emotions", params={"label": "disappointment"}, 
                         display_name="Disappointments in Response"), 
        HuggingFaceModel(model="SamLowe/roberta-base-go_emotions", params={"label": "optimism"}, 
                         display_name="Optimism in Response"),     
    ])
])
```

You can do the same for Test Suites.

{% hint style="info" %}
**Which descriptors are there?** See the list of available built-in descriptors in the [All Metrics](/reference/all-metrics) page.
{% endhint %}

## Sample models

Here are some example models you can call using the `HuggingFaceModel()` descriptor.

| Model                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Parameters                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>Emotion classification</strong><br><code>SamLowe/roberta-base-go\_emotions</code></p><ul><li>Scores texts by 28 emotions.</li><li>Returns the predicted probability for the chosen emotion label.</li><li>Scale: 0 to 1.</li><li><code>toxic\_label="hate"</code> (default)</li></ul><p><strong>Example use</strong>:<br><code>HuggingFaceModel(model="SamLowe/roberta-base-go\_emotions", params={"label": "disappointment"})</code><br><br><strong>Source</strong>: <a href="https://huggingface.co/SamLowe/roberta-base-go_emotions">HuggingFace Model</a></p>                                                                                                                                                                                                                                                                                                | <p><strong>Required</strong>:</p><ul><li><code>params={"label":"label"}</code></li></ul><p><strong>Available labels</strong>:</p><ul><li>admiration</li><li>amusement</li><li>anger</li><li>annoyance</li><li>approval</li><li>caring</li><li>confusion</li><li>curiosity</li><li>desire</li><li>disappointment</li><li>disapproval</li><li>disgust</li><li>embarrassment</li><li>excitement</li><li>fear</li><li>gratitude</li><li>grief</li><li>joy</li><li>love</li><li>nervousness</li><li>optimism</li><li>pride</li><li>realization</li><li>relief</li><li>remorse</li><li>sadness</li><li>surprise</li><li>neutral</li></ul><p><strong>Optional</strong>:</p><ul><li><code>display\_name="display name"</code></li></ul> |
| <p><strong>Toxicity detection</strong><br><code>facebook/roberta-hate-speech-dynabench-r4-target</code></p><ul><li>Detects hate speech.</li><li>Returns predicted probability for the “hate” label.</li><li>Scale: 0 to 1.</li></ul><p><strong>Example use</strong>:<br><code>HuggingFaceModel(model="facebook/roberta-hate-speech-dynabench-r4-target", display\_name="Toxicity")</code><br><br><strong>Source</strong>: <a href="https://huggingface.co/facebook/roberta-hate-speech-dynabench-r4-target">HuggingFace Model</a></p>                                                                                                                                                                                                                                                                                                                                       | <p><strong>Optional</strong>:</p><ul><li><code>toxic\_label="hate"</code> (default)</li><li><code>display\_name="display name"</code></li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| <p><strong>Zero-shot classification</strong><br><code>MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli</code></p><ul><li>A natural language inference model.</li><li>Use it for zero-shot classification by user-provided topics.</li><li>List candidate topics as <code>labels</code>. You can provide one or several topics.</li><li>You can set a classification threshold: if the predicted probability is below, an "unknown" label will be assigned.</li><li>Returns a label.</li></ul><p><strong>Example use</strong>:<br><code>HuggingFaceModel(model="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli", params={"labels": \["HR", "finance"], "threshold":0.5}, display\_name="Topic")</code><br><br><strong>Source</strong>: <a href="https://huggingface.co/MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli">HuggingFace Model</a></p> | <p><strong>Required</strong>:</p><ul><li><code>params={"labels": \["label"]}</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>params={"score\_threshold": 0.7}</code> (default: 0.5)</li><li><code>display\_name="display name"</code></li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| <p><strong>GPT-2 text detection</strong><br><code>openai-community/roberta-base-openai-detector</code></p><ul><li>Predicts if a text is Real or Fake (generated by a GPT-2 model).</li><li>You can set a classification threshold: if the predicted probability is below, an "unknown" label will be assigned.</li><li>Note that it is not usable as a detector for more advanced models like ChatGPT.</li><li>Returns a label.</li></ul><p><strong>Example use</strong>:<br><code>HuggingFaceModel(model="openai-community/roberta-base-openai-detector", params={"score\_threshold": 0.7})</code><br><br><strong>Source</strong>: <a href="https://huggingface.co/openai-community/roberta-base-openai-detector">HuggingFace Model</a></p>                                                                                                                                | <p><strong>Optional</strong>:</p><ul><li><code>params={"score\_threshold": 0.7}</code> (default: 0.5)</li><li><code>display\_name="display name"</code></li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

This list is not exhaustive, and the Descriptor may support other models published on Hugging Face. The implemented interface generally works for models that:

* Output a single number (e.g., predicted score for a label) or a label, **not** an array of values.
* Can process raw text input directly.
* Name labels using `label` or `labels` fields.
* Use methods named `predict` or `predict_proba` for scoring.

However, since each model is implemented differently, we cannot provide a complete list of models with a compatible interface. We suggest testing the implementation on your own using trial and error. If you discover useful models, feel free to share them with the community in Discord. You can also open an issue on GitHub to request support for a specific model.


# Add a custom text descriptor

How to add custom text descriptors.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

You can implement custom row-level evaluations for text data that you will later use just like any other descriptor across Metrics and Tests. You can implement descriptors that use a single column or two columns.

Note that if you want to use LLM-based evaluations, you can write custom prompts using [LLM judge templates](/user-guide/customization/llm_as_a_judge).

## Code example

Refer to a How-to example:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_use_llm_judge_template.ipynb>" %}

## Custom descriptors

Imports:

```python
from evidently.descriptors import CustomColumnEval, CustomPairColumnEval
```

### Single column descriptor

You can create a custom descriptor that will take a single column from your dataset and run a certain evaluation for each row.

**Implement your evaluation as a Python function**. It will take a pandas Series as input and return a transformed Series.

Here, the `is_empty_string_callable` function takes a column of strings and returns an "EMPTY" or "NON EMPTY" outcome for each.

```python
def is_empty_string_callable(val1):
    return pd.Series(["EMPTY" if val == "" else "NON EMPTY" for val in val1], index=val1.index)
```

**Create a custom descriptor**. Create an example of `CustomColumnEval` class to wrap the evaluation logic into an object that you can later use to process specific dataset input.

```python
empty_string = CustomColumnEval(
    func=is_empty_string_callable,
    feature_type="cat",
    display_name="Empty response"
)
```

Where:

* `func: Callable[[pd.Series], pd.Series]` is a function that returns a transformed pandas Series.
* `display_name: str` is the new descriptor's name that will appear in Reports and Test Suites.
* `feature_type` is the type of descriptor that the function returns (`cat` for categorical, `num` for numerical)

**Apply the new descriptor**. To create a Report with a new Descriptor, pass it as a `column_name` to the `ColumnSummaryMetric`. This will compute the new descriptor for all rows in the specified column and summarize its distribution:

```python
report = Report(metrics=[
    ColumnSummaryMetric(column_name=empty_string.on("response")),
])
```

Run the Report on your `df` dataframe as usual:

```python
report.run(reference_data=None, 
           current_data=df)
```

### Double column descriptor

You can create a custom descriptor that will take two columns from your dataset and will run a certain evaluation for each row. (For example, for pairwise evaluators).

**Implement your evaluation as a Python function**. Here, the `exact_match_callable` function takes two columns and checks whether each pair of values is the same, returning "MATCH" if they are equal and "MISMATCH" if they are not.

```python
def exact_match_callable(val1, val2):
    return pd.Series(["MATCH" if val else "MISMATCH" for val in val1 == val2])
```

**Create a custom descriptor**. Create an example of the `CustomPairColumnEval` class to wrap the evaluation logic into an object that you can later use to process two named columns in a dataset.

```python
exact_match =  CustomPairColumnEval(
    func=exact_match_callable,
    first_column="response",
    second_column="question",
    feature_type="cat",
    display_name="Exact match between response and question"
)
```

Where:

* `func: Callable[[pd.Series, pd.Series], pd.Series]` is a function that returns a transformed pandas Series after evaluating two columns.
* `first_column: str` is the name of the first column to be passed into the function.
* `second_column: str` is the name of the second column to be passed into the function.
* `display_name: str` is the new descriptor's name that will appear in Reports and Test Suites.
* `feature_type` is the type of descriptor that the function returns (`cat` for categorical, `num` for numerical).

**Apply the new descriptor**. To create a Report with a new Descriptor, pass it as a `column_name` to the ColumnSummaryMetric. This will compute the new descriptor for all rows in the dataset and summarize its distribution:

```python
report = Report(metrics=[
    ColumnSummaryMetric(column_name=exact_match.as_column())
])
```

Run the Report on your `df` dataframe as usual:

```python
report.run(reference_data=None, 
           current_data=df)
```


# Add a custom drift method

How to implement a new drift detection method.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to set custom drift methods and which methods are available in the library.

If you do not find a suitable drift detection method, you can implement a custom function.

## Code example

Notebook example with custom data drift function example:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb>" %}

## Custom StatTest function requirements:

The StatTest function should match `(reference_data: pd.Series, current_data: pd.Series, threshold: float) -> Tuple[float, bool]` signature:

* `reference_data: pd.Series` - reference data series
* `current_data: pd.Series` - current data series to compare
* `feature_type: str` - feature type
* `threshold: float` - Stat Test threshold for drift detection

Returns:

* `score: float` - Stat Test score (actual value)
* `drift_detected: bool` - indicates is drift detected with given threshold

### Example:

```python
from typing import Tuple

import numpy as np
import pandas as pd
from scipy.stats import anderson_ksamp


def anderson_stat_test(reference_data: pd.Series, current_data: pd.Series, _feature_type: str, threshold: float) -> Tuple[float, bool]:
  p_value = anderson_ksamp(np.array([reference_data, current_data]))[2]
  return p_value, p_value < threshold
```

## StatTest meta information (StatTest class):

To use the StatTest function, we recommended writing a specific instance of the StatTest class for that function:

To create the instance of the `StatTest` class, you need:

* `name: str` - a short name used to reference the Stat Test from the options (the StatTest should be registered globally)
* `display_name: str` - a long name displayed in the Dashboard and Profile
* `func: Callable` - a StatTest function
* `allowed_feature_types: List[str]` - the list of allowed feature types to which this function can be applied (available values: `cat`, `num`)

### Example:

```python
from evidently.calculations.stattests import StatTest


def example_stat_test(reference_data, current_data, feature_type, threshold):
  return 0.1, False


example_stat_test = StatTest(
  name="example_test",
  display_name="Example Stat Test (score)",
  func=example_stat_test,
  allowed_feature_types=["cat"],
)
```


# Add a custom Metric or Test

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

There are two ways to add a custom Metric or Test to Evidently:

* Add it as a Python function (Recommended).
* Implement a custom metric with custom Plotly render.

Implementing a new Metric or Test means that you implement a completely custom column- or dataset-level evaluation.

There are other ways to customize your evaluations that do not require creating Metrics or Tests from scratch:

* Add a custom descriptor for row-level evaluations. Read on [adding custom text descriptors](/user-guide/customization/add-custom-descriptor).
* Write a custom LLM-based evaluator using templates. Read on [designing LLM judges](/user-guide/customization/llm_as_a_judge).
* Add a custom data drift detection method, re-using the existing Data Drift metric render. Read on [drift method customization](/user-guide/customization/add-custom-drift-method) option.

## 1. Add a new Metric or Test as a Python function. (Recommended).

You can implement any custom Metric or Test as a Python function. The visual render in the Report will default to a simple counter.

This is a recommended path to add custom Metrics. Using this method, you can send Reports with custom Metrics to Evidently Cloud (or view them in the self-hosted Monitoring UI).

Example notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_build_metric_over_python_function.ipynb>" %}

## 2. Implement a new Metric and Test from scratch.

You can also implement a new Metric or Test from scratch, defining both the calculation method and the optional visualization.

This is suitable if you want to access the new Metric (or Test) only in a Report or a Test Suite. However, you won't be able to view such custom Metrics in the Monitoring user interface or Evidently Cloud since the code that defines the custom render won't be accessible from the UI service.

**Note**: this is advanced functionality that expects you to be able to work with the codebase independently. To implement the visualization, you should be familiar with Plotly.\
You can use the metrics and tests in the codebase as an example.

This example Jupyter notebook shows how to implement a simple metric and test and include them in a custom Report and Test Suite:

{% embed url="<https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_make_custom_metric_and_test.ipynb>" %}

## Open a GitHub issue

If you want to suggest adding specific metrics and tests that are currently not covered to the core library, you can also [open a GitHub issue](https://github.com/evidentlyai/evidently/issues) with the feature request or a proposal of the new Metric you'd like to contribute.


# Customize JSON output

How to modify what is included in the Evidently JSON or Python dictionary output.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate Evidently Reports and Test Suites and get the output as JSON or Python dictionary

## Code example

Notebook example on JSON / Python dictionary customization:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_customize_json_output.ipynb>" %}

**Note**: to export the Evidently Metric output in the tabular format, you can use the `as_dataframe()` option instead of JSON or Python dictionary export.

## Default

Default JSON and Python dictionary output includes the values of Metrics and Test results but does not include visualizations.

Using the following options, you can customize the JSON/Python dictionary output for Reports and Test Suites.

## Include render

If you want to include some of the information about the visualizations computed by Evidently (for example, distribution bins), you can specify it:

```python
report.json(include_render=True)
```

This way, you will receive a verbose output with additional info on visualizations.

This is convenient if you want to integrate Evidently output with other visualization tools.

## Include and exclude components

You can also flexibly define which specific output to include in JSON or Python dictionary.

To specify which metrics, tests, or components to include, use `include`:

```python
report.as_dict(include={
    "DataDriftTable": {
        "drift_by_columns":{
            "target":{
                "column_name", "column_type", "drift_score"
            }}}})
```

This is convenient if you want to get a minimalistic output that is easy to parse.

To understand which components exist, generate the complete example output, and look at the keys.

Use `exclude` to specify which metrics, tests, or components to omit. You can combine using include and exclude, for example:

```python
report.as_dict(
    include={"DatasetDriftMetric": {"share_of_drifted_columns"}},
    exclude={"DataDriftTable":{"drift_by_columns"}}
)
```

If you want to include all fields for a dictionary, use `True`.

```python
report.as_dict(include={"DataDriftTable":{"drift_by_columns":{"target":True}}})
```

To apply filters to column-based results, use `AllDict`.

```python
report.as_dict(include={
    "DataDriftTable": {
        "drift_by_columns":AllDict({
                "column_name", "column_type", "drift_score"
            })}},
              exclude={"DataDriftTable": {"drift_by_columns":AllDict({
                 "column_type"
            })}})
```

This way, you can flexibly define the exact output format. Sometimes using exclude/include together might be more convenient than simply listing the components.


# Show raw data in Reports

How to change data aggregation in plots.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate Reports with default parameters.
* You know how to pass custom parameters for Reports or Metrics.

## Code example

You can refer to an example How-to-notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_use_aggregated_data_option.ipynb>" %}

## Default

Evidently Reports include visualizations, such as plotting values over time, which are aggregated by default. This keeps Reports size manageable, even with millions of evaluated rows.

For example, you can create a custom Report:

```python
report = Report(metrics=[
    RegressionPredictedVsActualScatter(),
    RegressionPredictedVsActualPlot()
])
report.run(reference_data=housing_ref, current_data=housing_cur)
report
```

Here is how the Scatter Plot in this Report will look:

![RegressionPredictedVsActualScatter()](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-481fbae53d0214f0f3c937f2d080a30c17465cbb%2Fmetric_regression_predvsactual_scatter_agg-min.png?alt=media\&token=180fdb4d-ea7a-448e-8a74-a40c310ce9bf)

{% hint style="info" %}
**This does not affect Test Suites.** All visualizations in Test Suites are already aggregated.
{% endhint %}

## Non-aggregated plots for Reports

If you prefer to see raw data plots (individual prediction points), you can enable this option. This will store raw data points inside the Report.

To see non-aggregated plots, set the `raw_data` parameter as `True` in the render options.

You can set it on the Report level:

```python
report = Report(
    metrics=[
      RegressionPredictedVsActualScatter(),
      RegressionPredictedVsActualPlot()
    ],
    options={"render": {"raw_data": True}}
  )
report.run(reference_data=housing_ref, current_data=housing_cur)
report
```

All plots in the Report will be non-aggregated. Here is how the Scatter Plot in this Report will look:

![RegressionPredictedVsActualScatter()](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-97cff96c5315089b0550d1a5b435456573b1c380%2Fmetric_regression_predvsactual_scatter_non_agg-min.png?alt=media\&token=3c9c159d-a21d-46bd-b3d2-9a9d1fb29e20)

{% hint style="info" %}
**Consider the data size.** We recommend setting this option for smaller datasets or when you apply sampling. With non-aggregated plots, the HTML will contain all the data on individual data points. For large datasets this will result in a very large Report and can make the plots unreadable.&#x20;
{% endhint %}

{% hint style="info" %}
**Raw data is not available on Spark.** If you run the computations using Spark, the raw data option is not available.
{% endhint %}

## Non-aggregated plots for Metrics

If you want to generate non-aggregated plots only for some visualizations, you can pass the option to the chosen Metrics:

```python
report = Report(
    metrics=[
      RegressionPredictedVsActualScatter(options={"render": {"raw_data": True}}),
      RegressionPredictedVsActualPlot()
    ],
  )
report.run(reference_data=housing_ref, current_data=housing_cur)
report
```


# Add text comments to Reports

How to add a widget with custom text.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

**Pre-requisites**:

* You know how to generate custom Reports using individual Metrics.

## Code example

How-to notebook:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_add_a_text_comment_to_the_report.ipynb>" %}

## What you can do

You can add a widget that contains any custom text to the Evidently Report. Here is how this can look:

![Text Comment()](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-39c6c39c92c7721a0055b2d01b37b5fa85f1e922%2Fmetric_comment-min.png?alt=media\&token=dcd140c1-0b8d-4961-b21a-4ffd477e09fb)

You can include multiple text widgets in a single Report.

## Using “Comment” Metric

To add a text widget, you must first define the contents of the comment. You can use markdown to format the text.

An example of adding "model\_description" comment:

```python
model_description = """
 # Model Description
 This is a demand forecasting model.


 ## Intended use
 * Weekly sales planning
 * Weekly capacity planning
"""
```

When creating the Report, include the `Comment` Metric and reference the earlier defined text.

Example:

```python
report = Report(metrics=[
   Comment(model_description),
   ColumnDistributionMetric('TOTAL_ORDERS')
])


report.run(current_data=raw_data, reference_data=None)
report
```


# Change color schema

You can modify the colors in the Reports and Tests.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

## Code example

You can refer to an example How-to-notebook showing how to customize the color schema in your reports and test suites:

{% embed url="<https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_customize_color_schema.ipynb>" %}

## Options for Color Scheme

By default, Evidently widgets use the red-grey color scheme. However, you can either define individual colors or take advantage of preconfigured color schemes.

### Change individual colors

To change the colors in the widgets, you can create an object `ColorOptions` from the `evidently.options.color_scheme`, replace the values you need, and use it in the options list when you create a report or test suite.

```python
from evidently.options import ColorOptions

color_scheme = ColorOptions(
    color_scheme.primary_color = "#5a86ad"
    color_scheme.fill_color = "#fff4f2"
    color_scheme.zero_line_color = "#016795"
    color_scheme.current_data_color = "#c292a1" 
    color_scheme.reference_data_color = "#017b92"
)
```

To define values for the colors, you can use CSS and Plotly compatible strings. For example:

* color names: "blue", "orange", "green"
* RGB values: #fff4f2, #ee00aa
  \
  and so on.

Here is the list of all color scheme options with the type and meaning of each:

| A Variable in the ColorOptions object | Option type      | Option description                                                                                                                                                               |
| ------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| primary\_color                        | string           | A basic color for data visualization. Used by default for all bars and lines in widgets with one dataset. Used as the default for the current data in widgets with two datasets. |
| secondary\_color                      | string           | A basic color to visualize the second dataset in the widgets with two datasets. For example, the reference data.                                                                 |
| current\_data\_color                  | string           | A color for the current data. By default, the primary color is used.                                                                                                             |
| reference\_data\_color                | string           | A color for the reference data. By default, the secondary color is used.                                                                                                         |
| color\_sequence                       | array of strings | A set of colors to draw a number of lines in one graph. For example, in the Data Quality dashboard.                                                                              |
| fill\_color                           | string           | A fill color for areas in line graphs.                                                                                                                                           |
| zero\_line\_color                     | string           | A color for the base, zero line in line graphs.                                                                                                                                  |
| non\_visible\_color                   | string           | A color for technical, not visible dots or points for better scalability.                                                                                                        |
| underestimation\_color                | string           | A color for the "underestimation" line in the Regression Performance dashboard.                                                                                                  |
| overestimation\_color                 | string           | A color for the "overestimation" line in the Regression Performance dashboard.                                                                                                   |
| majority\_color                       | string           | A color for the majority line in the Regression Performance dashboard.                                                                                                           |
| vertical\_lines                       | string           | A color for vertical lines.                                                                                                                                                      |
| heatmap                               | string           | Colors for heatmaps.                                                                                                                                                             |

***

### Use existing color schemes

Evidently also provides some sensible alternative default schemas that have been pre-selected for your convenience:

* 'Solarised'
* 'Karachi Sunrise'
* 'Berlin Autumn'
* 'Nightowl'

To use them, simply import them directly and pass them into your `Report` or `TestSuite` options as follows (taking the Berlin Autumn scheme as an example):

```python
from evidently.options import BERLIN_AUTUMN_COLOR_OPTIONS
```

### Customize color in Reports

For example, here is how the Data Drift Report looks without customizing the color:

![Data Drift](https://github.com/evidentlyai/docs-old/blob/main/images/01_data_drift.png)

Either pass the above defined `color_scheme` to `options` of your `Report`:

```python
from evidently.report import Report
from evidently.metric_preset.data_drift import DataDriftPreset

# import the data as usual...
data_drift_report = Report(
    metrics=[DataDriftPreset()], 
    options=[color_scheme]
)

data_drift_report.run(iris_ref, iris_cur)
```

Here is an example of the report with the modified color scheme:
![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2b33653dd41464624657f054592a4364f136891c%2Fcustomization_color_scheme_example.png?alt=media\&token=ce1b1491-bc00-4b1b-9d62-74878a377bf5)

Or pass one of the preconfigured schemes to your `Report`:

```python
data_drift_report = Report(
    metrics=[DataDriftPreset()], 
    options=[BERLIN_AUTUMN_COLOR_OPTIONS]
)

data_drift_report.run(iris_ref, iris_cur)

data_drift_report.save_html("output.html")
```

### Customize color in Test Suites

Either pass the above defined `color_scheme` to `options` of your `TestSuite`:

```python
from evidently.test_preset import DataDriftTestPreset

# import the data as usual...
data_drift_dataset_tests = TestSuite(
    tests=[DataDriftTestPreset()], 
    options=[color_scheme]
)

data_drift_dataset_tests.run(reference_data=iris_ref, current_data=iris_cur)
data_drift_dataset_tests
```

Or pass one of the preconfigured schemes to your `TestSuite`:

```python
data_drift_dataset_tests = TestSuite(
    tests=[DataDriftTestPreset()], 
    options=[BERLIN_AUTUMN_COLOR_OPTIONS]
)

data_drift_dataset_tests.run(reference_data=iris_ref, current_data=iris_cur)
data_drift_dataset_tests
```


# How-to guides

How-to guides.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer docs version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

These example notebooks and how-to guides show how to solve specific tasks. You can also browse on [GitHub](https://github.com/evidentlyai/evidently/tree/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions).

| Topic            | Question                                                                                   | Guide or example                                                                                                                                                                                                                                                                                    |
| ---------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Input data       | How to load data from different sources to pandas.Dataframes?                              | <ul><li><a href="/user-guide/input-data/load-data-to-pandas">Guide</a></li></ul>                                                                                                                                                                                                                    |
| Input data       | How to use column mapping?                                                                 | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_use_column_mapping.ipynb">Jupyter notebook</a></li></ul>                                                                                                   |
| Test and Reports | How to generate multiple Tests or Metrics quickly? (Test and Metric Generators).           | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_apply_tests_and_metrics_generators.ipynb">Jupyter notebook</a></li></ul>                                                                                   |
| Test and Reports | How to calculate drift in embeddings?                                                      | <ul><li><a href="https://github.com/elenasamuylova/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_calculate_embeddings_drift.ipynb">Jupyter notebook</a></li></ul>                                                                                        |
| Test and Reports | How to run evaluations on raw text data? (Tests, Metrics and Presets that work with Text). | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_run_calculations_over_text_data.ipynb">Jupyter notebook</a></li></ul>                                                                                      |
| Test and Reports | How to use text descriptors in text-specific Metrics?                                      | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_use_text_descriptors_in_text_specific_metrics.ipynb">Jupyter notebook</a></li></ul>                                                                        |
| Test and Reports | How to use text descriptors in tabular Metrics and Tests?                                  | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_apply_table_metrics_and_tests_to_text_descriptors.ipynb">Jupyter notebook</a></li></ul>                                                                    |
| Test and Reports | How to calculate metrics for ranking and recommender systems?                              | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_run_recsys_metrics.ipynb">Jupyter notebook</a></li></ul>                                                                                                   |
| Test and Reports | How to import text data, convert to embeddings and detect drift?                           | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_run_drift_report_for_text_data.ipynb">Jupyter notebook</a></li></ul>                                                                                       |
| Test and Reports | How to set Test criticality as "warning"?                                                  | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_specify_test_criticality.ipynb">Jupyter notebook</a></li></ul>                                                                                             |
| Test and Reports | How to save and load Test Suites and Reports as JSON "snapshots"?                          | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_use_snapshots.ipynb">Jupyter notebook</a></li></ul>                                                                                                        |
| Test and Reports | How to export Report results to a dataframe?                                               | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_get_report_as_a_dataframe.ipynb">Jupyter notebook</a></li></ul>                                                                                            |
| Test and Reports | How to run calculations on Spark?                                                          | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_run_calculations_on_spark.ipynb">Jupyter notebook</a></li></ul>                                                                                            |
| Customization    | How to assign a particular method for Data Drift detection?                                | <ul><li><a href="/user-guide/customization/options-for-statistical-tests">Guide</a></li><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_specify_stattest_for_a_testsuite.ipynb">Jupyter notebook</a></li></ul> |
| Customization    | How to define a custom list of Missing Values?                                             | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_run_tests_with_different_missing_values.ipynb">Jupyter notebook</a></li></ul>                                                                              |
| Customization    | How to add a custom Metric or Test from scratch)?                                          | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_make_custom_metric_and_test.ipynb">Jupyter notebook</a></li></ul>                                                                                          |
| Customization    | How to add a custom Metric as a Python function?                                           | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_build_metric_over_python_function.ipynb">Jupyter notebook</a></li></ul>                                                                                    |
| Visual render    | How to add text comments to the Reports?                                                   | <ul><li><a href="/user-guide/customization/text-comments">Guide</a></li><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_add_a_text_comment_to_the_report.ipynb">Jupyter notebook</a></li></ul>                 |
| Visual render    | How to specify a color scheme in Reports and Test Suites?                                  | <ul><li><a href="/user-guide/customization/options-for-color-schema">Guide</a></li><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_customize_color_schema.ipynb">Jupyter notebook</a></li></ul>                |
| Visual render    | How to get non-aggregated visuals in Reports?                                              | <ul><li><a href="/user-guide/customization/report-data-aggregation">Guide</a></li><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_use_aggregated_data_option.ipynb">Jupyter notebook</a></li></ul>             |
| Outputs          | How to customize JSON output?                                                              | <ul><li><a href="/user-guide/customization/json-dict-output">Guide</a></li><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_customize_json_output.ipynb">Jupyter notebook</a></li></ul>                         |
| Outputs          | How to get Report or Test Suite output in csv?                                             | <ul><li><a href="https://github.com/evidentlyai/evidently/blob/ad71e132d59ac3a84fce6cf27bd50b12b10d9137/examples/how_to_questions/how_to_get_report_results_in_csv.ipynb">Jupyter notebook</a></li></ul>                                                                                            |


# All tests

List of all Tests and Test Presets available in Evidently.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

<details>

<summary>How to use this page</summary>

This is a reference page. You can return here:

* To discover **available Tests** and choose which to include in a custom Test suite.
* To understand which **parameters** you can change for a specific Test or Preset.
* To verify which tests are included in a **Test Preset**.

You can use the menu on the right to navigate the sections.

## How to read the tables

* **Name**: the name of the Test or Test preset.
* **Description**: plain text explanation. For Tests, we specify whether it applies to the whole dataset or individual columns.
* **Parameters**: available configurations.
  * Required parameters are necessary for calculations, e.g. a column name for a column-level test.
  * Optional parameters modify how the underlying metric is calculated, e.g. which statistical test or correlation method is used.
  * [*Test condition parameters*](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/custom-test-suite.md#custom-conditions) set the conditions (e.g. equal, not equal, greater than, etc.) that define the expectations from the Test output. If the condition is violated, the test returns a fail. They apply to most of the Tests, and are optional.
* **Default Test condition**: they apply if you do not set a custom сondition.
  * With reference: the Test conditions that apply when you pass a reference dataset and Evidently can derive conditions from it. (E.g. expect +/- 10% from reference).
  * No reference: the Test conditions that apply if you do not provide the reference. They are based on heuristics.

**Test visualizations**. Each Test also includes a default render. If you want to see the visualization, navigate to the [example notebooks](/tutorials-and-examples/examples).

</details>

{% hint style="info" %}
We are doing our best to maintain this page up to date. In case of discrepancies, consult the [API reference](https://docs.evidentlyai.com/reference/api-reference) or the "All tests" notebook in the [Examples](/tutorials-and-examples/examples) section. If you notice an error, please send us a pull request to update the documentation!
{% endhint %}

## Test Presets

Default conditions for each Test in the Preset match the Test's defaults. You can see them in the tables below. The listed Preset parameters apply to the relevant individual Tests inside the Preset.

<details>

<summary>NoTargetPerformance Test Preset</summary>

Preset name: `NoTargetPerformanceTestPreset()`

**Composition**:

* `TestShareOfDriftedColumns()`
* `TestColumnDrift(column_name=prediction)`
* `TestColumnShareOfMissingValues()` for `all` or `сolumns` if provided
* `TestShareOfOutRangeValues()` for all numerical or specified `columns`
* `TestShareOfOutListValues()` for all categorical or specified `columns`
* `TestMeanInNSigmas()` for all numerical or specified `columns`

**Optional parameters**:

* `columns`
* `stattest`
* `cat_stattest`
* `num_stattest`
* `per_column_stattest`
* `text_stattest`
* `stattest_threshold`
* `cat_stattest_threshold`
* `num_stattest_threshold`
* `per_column_stattest_threshold`
* `text_stattest_threshold`
* `embeddings`
* `embeddings_drift_method`
* `drift_share`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests), [embeddings drift parameters](/user-guide/customization/embeddings-drift-parameters).

</details>

<details>

<summary>Data Stability Test Preset</summary>

Preset name: `DataStabilityTestPreset()`

**Composition**:

* `TestNumberOfRows()`
* `TestNumberOfColumns()`
* `TestColumnsType()`
* `TestColumnShareOfMissingValues()` for all or specified `columns`
* `TestShareOfOutRangeValues()` for all numerical or specified `columns`
* `TestShareOfOutListValues()` for all categorical or specified `columns`
* `TestMeanInNSigmas()` for all numerical or specified `columns`

**Optional parameters**:

* `columns`

</details>

<details>

<summary>Data Quality Test Preset</summary>

Preset name: `DataQualityTestPreset()`

**Composition**:

* `TestColumnShareOfMissingValues()` for all or specified `columns`
* `TestMostCommonValueShare()` for all or specified `columns`
* `TestNumberOfConstantColumns()`
* `TestNumberOfDuplicatedColumns()`
* `TestNumberOfDuplicatedRows()`

**Optional parameters**:

* `columns`

</details>

<details>

<summary>Data Drift Test Preset</summary>

Preset name: `DataDriftTestPreset()`

**Composition**:

* `TestShareOfDriftedColumns()`
* `TestColumnDrift()` for all or specified `columns`

**Optional parameters**:

* `columns`
* `stattest`
* `cat_stattest`
* `num_stattest`
* `per_column_stattest`
* `text_stattest`
* `stattest_threshold`
* `cat_stattest_threshold`
* `num_stattest_threshold`
* `per_column_stattest_threshold`
* `text_stattest_threshold`
* `embeddings`
* `embeddings_drift_method`
* `drift_share`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests), [embeddings drift parameters](/user-guide/customization/embeddings-drift-parameters).

</details>

<details>

<summary>Regression Test Preset</summary>

Preset name: `RegressionTestPreset()`

**Composition**:

* `TestValueMeanError()`
* `TestValueMAE()`
* `TestValueRMSE()`
* `TestValueMAPE()`

**Optional parameters**:\
N/A

</details>

<details>

<summary>Multiclass Classification Test Preset</summary>

Preset name: `MulticlassClassificationTestPreset()`

**Composition**:

* `TestAccuracyScore()`
* `TestF1Score()`
* `TestPrecisionByClass()`
* `TestRecallByClass()`
* `TestColumnDrift(column_name=target)`
* `TestNumberOfRows()`
* `TestLogLoss()` - if probabilistic classification
* `TestRocAuc()` - if probabilistic classification

**Optional parameters** for target drift:

* `stattest`
* `stattest_threshold`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests)

</details>

<details>

<summary>Binary Classification (Top K) Test Preset</summary>

Preset name: `BinaryClassificationTopKTestPreset()`

**Composition**:

* `TestAccuracyScore(k=k)`
* `TestPrecisionScore(k=k)`
* `TestRecallScore(k=k)`
* `TestF1Score(k=k)`
* `TestColumnDrift(column_name=target)`
* `TestRocAuc()`
* `TestLogLoss()`

**Required parameters**:

* `k`

**Optional parameters**:

* `stattest`
* `stattest_threshold`
* `probas_threshold`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests)

</details>

<details>

<summary>Binary Classification Test Preset</summary>

Preset name: `BinaryClassificationTestPreset()`

**Composition**:

* `TestColumnDrift(column_name=target)`
* `TestPrecisionScore()`
* `TestRecallScore()`
* `TestF1Score()`
* `TestAccuracyScore()`
* `TestRocAuc()` - if probabilistic classification

**Optional parameters**:

* `stattest`
* `stattest_threshold`
* `probas_threshold`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests)

</details>

<details>

<summary>RecSys (Recommender Systems) Test Preset</summary>

Preset name: `RecsysTestPreset()`

**Composition**:

* `TestPrecisionTopK()`
* `TestRecallTopK()`
* `TestMAPK()`
* `TestNDCGK()`
* `TestHitRateK()`

**Required parameters:**

* `k`

**Optional parameters:**

* `min_rel_score: Optional[int]`
* `no_feedback_users: bool`

</details>

{% hint style="info" %}
**How to set custom Test conditions?** Use parameters (e.g. equal, not equal, greater than, etc.) to set [Test Conditions](https://github.com/evidentlyai/docs-old/blob/main/tests-and-reports/custom-test-suite.md#custom-conditions).
{% endhint %}

## Data Quality

### Data Integrity

| Test name                                                                 | Description                                                                                                                                                  | Parameters                                                                                                                                                                                                                                | Default test condition                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestNumberOfRows()**                                                    | <p>Dataset-level.<br><br>Tests the number of rows against the reference or a defined condition.</p>                                                          | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects +/-10% or >30.<br><br><strong>With reference</strong>: the test fails if the number of rows differs by over 10% from the reference.<br><br><strong>No reference</strong>: the test fails if the number of rows is <= 30.</p>                                                  |
| **TestNumberOfColumns()**                                                 | <p>Dataset-level.<br><br>Tests the number of columns against the reference or a defined condition.</p>                                                       | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects the same or non-zero.<br><br><strong>With reference</strong>: the test fails if the number of columns differs from the reference.<br><br><strong>No reference</strong>: the test fails if the number of columns is 0.</p>                                                     |
| **TestNumberOfConstantColumns()**                                         | <p>Dataset-level.<br><br>Tests the number of columns with all constant values against reference or a defined condition.</p>                                  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects =< or none.<br><br><strong>With reference</strong>: the test fails if the number of constant columns is higher than in the reference.<br><br><strong>No reference</strong>: the test fails if there is at least one constant column.</p>                                      |
| **TestNumberOfEmptyRows()**                                               | <p>Dataset-level.<br><br>Tests the number of empty rows against reference or a defined condition.</p>                                                        | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects +/- 10% or none.<br><br><strong>With reference</strong>: the test fails if the share of empty rows is over 10% higher or lower than in the reference.<br><br><strong>No reference</strong>: the test fails if there is at least one empty row.</p>                            |
| **TestNumberOfEmptyColumns()**                                            | <p>Dataset-level.<br><br>Tests the number of empty columns against reference or a defined condition.</p>                                                     | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects =< or none.<br><br><strong>With reference</strong>: the test fails if the number of empty columns is higher than in the reference.<br><br><strong>No reference</strong>: the test fails if there is at least one empty column.</p>                                            |
| **TestNumberOfDuplicatedRows()**                                          | <p>Dataset-level.<br><br>Tests the number of duplicate rows against reference or a defined condition.</p>                                                    | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects +/- 10% or none.<br><br><strong>With reference</strong>: the test fails if the share of duplicate rows is over 10% higher or lower than in the reference.<br><br><strong>No reference</strong>: the test fails if there is at least one duplicate row.</p>                    |
| **TestNumberOfDuplicatedColumns()**                                       | <p>Dataset-level.<br><br>Tests the number of duplicate columns against reference or a defined condition.</p>                                                 | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                  | <p>Expects =< or none.<br><br><strong>With reference</strong>: the test fails if the number of duplicate columns is higher than in the reference.<br><br><strong>No reference</strong>: the test fails if there is at least one duplicate column.</p>                                    |
| **TestConflictTarget()**                                                  | <p>Dataset-level.<br><br>Tests if there are conflicts in the target (instances where a different label is assigned for an identical input).</p>              | N/A                                                                                                                                                                                                                                       | Expects no conflicts in the target (with or without reference).                                                                                                                                                                                                                          |
| **TestConflictPrediction()**                                              | <p>Dataset-level.<br><br>Tests if there are conflicts in the prediction (instances where a different prediction is made for an identical input).</p>         | N/A                                                                                                                                                                                                                                       | Expects no conflicts in the target (with or without reference).                                                                                                                                                                                                                          |
| **TestColumnsType()**                                                     | <p>Dataset-level.<br><br>Tests the types of all columns against the reference.</p>                                                                           | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br><code>columns\_type: dict</code><br><br><strong>Test conditions</strong>:<br>N/A</p>                                                                            | <p>Expects types to match.<br><br><strong>With reference</strong>: the test fails if at least one column type does not match.<br><strong>No reference</strong>: N/A</p>                                                                                                                  |
| **TestColumnAllConstantValues**(column\_name='name')                      | <p>Column-level.<br><br>Tests if all the values in a given column are constant.</p>                                                                          | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:<br>N/A</p>                                                                      | <p>Expects non-constant.<br><br>The test fails if all values in a given column are constant.</p>                                                                                                                                                                                         |
| **TestColumnAllUniqueValues**(column\_name='name')                        | <p>Column-level.<br><br>Tests if all the values in a given column are unique.</p>                                                                            | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:<br>N/A</p>                                                                      | <p>Expects all unique (e.g., IDs).<br><br>The test fails if at least one value in a given column is not unique.</p>                                                                                                                                                                      |
| <p><strong>TestNumberOfUniqueValues</strong>(column\_name='name')<br></p> | <p>Column-level.<br><br>Tests the number of unique values in a given column against reference or a defined condition.</p>                                    | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                               | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the share of unique values is different by more than 10%.<br><br><strong>No reference</strong>: N/A</p>                                                                                                     |
| **TestUniqueValuesShare**(column\_name='name')                            | <p>Column-level.<br><br>Tests the share of unique values in a given column against reference or a defined condition.</p>                                     | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                               | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the share of unique values is different by more than 10%.<br><br><strong>No reference</strong>: N/A</p>                                                                                                     |
| **TestMostCommonValueShare**(column\_name='name')                         | <p>Column-level.<br><br>Tests the share of the most common value in a given column against reference or a defined condition.</p>                             | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                               | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the share of the most common value is different by more than 10% from the reference.<br><br><strong>No reference</strong>: the test fails if the share of the most common value is >= 80%.</p>              |
| **TestColumnRegExp**(column\_name='name, reg\_exp='^\[0..9]')             | <p>Column-level.<br><br>Tests the number of values in a column that do not match a defined regular expression, against reference or a defined condition.</p> | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li><li><code>reg\_exp</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p><strong>With reference</strong>: the test fails if the share of values that match a regular expression is over 10% higher or lower than in the reference.<br><br><strong>No reference</strong>: the test fails if at least one of the values does not match a regular expression.</p> |

### Missing Values

**Defaults for Missing Values**. The metrics that calculate the number or share of missing values detect four types of the values by default: Pandas nulls (None, NAN, etc.), "" (empty string), Numpy "-inf" value, Numpy "inf" value. You can also pass a custom missing values as a parameter and specify if you want to replace the default list. Example:

```python
TestNumberOfMissingValues(missing_values=["", 0, "n/a", -9999, None], replace=True)
```

| Test name                                                         | Description                                                                                                                                                                                                                   | Parameters                                                                                                                                                                                                                                                                                             | Default test condition                                                                                                                                                                                                                                                       |
| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestNumberOfMissingValues()**                                   | <p>Dataset-level.<br><br>Tests the number of missing values in the dataset against the reference or a defined condition.</p>                                                                                                  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                    | <p>Expects up to +10% or 0.<br><br><strong>With reference</strong>: the test fails if the share of missing values is over 10% higher than in reference.<br><br><strong>No reference</strong>: the test fails if the dataset contains missing values.</p>                     |
| **TestShareOfMissingValues()**                                    | <p>Dataset-level.<br><br>Tests the share of missing values in the dataset against the reference or a defined condition.</p>                                                                                                   | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                    | <p>Expects up to +10% or 0.<br><br><strong>With reference</strong>: the test fails if the share of missing values is over 10% higher than in reference.<br><br><strong>No reference</strong>: the test fails if the dataset contains missing values.</p>                     |
| **TestNumberOfColumnsWithMissingValues()**                        | <p>Dataset-level.<br><br>Tests the number of columns that contain missing values in the dataset against the reference or a defined condition.</p>                                                                             | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                    | <p>Expects <= or 0.<br><br><strong>With reference</strong>: the test fails if the number of columns with missing values is higher than in reference.<br><strong>No reference</strong>: the test fails if the dataset contains columns with missing values.</p>               |
| **TestShareOfColumnsWithMissingValues()**                         | <p>Dataset-level.<br><br>Tests the share of columns that contain missing values in the dataset against the reference or a defined condition.</p>                                                                              | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                    | <p>Expects <= or 0.<br><br><strong>With reference</strong>: the test fails if the share of columns with missing values is higher than in reference.<br><br><strong>No reference</strong>: the test fails if the dataset contains columns with missing values.</p>            |
| **TestNumberOfRowsWithMissingValues()**                           | <p>Dataset-level.<br><br>Tests the number of rows that contain missing values against the reference or a defined condition.</p>                                                                                               | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>                                     | <p>Expects up to +10% or 0.<br><br><strong>With reference</strong>: the test fails if the share of rows with missing values is over 10% higher than in reference.<br><br><strong>No reference</strong>: the test fails if the dataset contains rows with missing values.</p> |
| **TestShareOfRowsWithMissingValues()**                            | <p>Dataset-level.<br><br>Tests the share of rows that contain missing values against the reference or a defined condition.</p>                                                                                                | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>                                     | <p>Expects up to +10% or 0.<br><br><strong>With reference</strong>: the test fails if the share of rows with missing values is over 10% higher than in reference.<br><br><strong>No reference</strong>: the test fails if the dataset contains rows with missing values.</p> |
| **TestNumberOfDifferentMissingValues()**                          | <p>Dataset-level.<br><br>Tests the number of differently encoded missing values in the dataset against the reference or a defined condition. Detects 4 types of missing values by default and/or values from a user list.</p> | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>missing\_values: list \<br>replace: bool = True</code>(default = default list)</li></ul><p><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>                                  | <p>Expects <= or none.<br><br><strong>With reference</strong>: the test fails if the current dataset has more types of missing values.<br><br><strong>No reference</strong>: the test fails if the current dataset contains missing values.</p>                              |
| **TestColumnNumberOfMissingValues**(column\_name='name')          | <p>Column-level.<br><br>Tests the number of missing values in a given column against the reference or a defined condition.</p>                                                                                                | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>  | <p>Expects up to 10% or none.<br><br><strong>With reference</strong>: the test fails if the share of missing values in a column is over 10% higher than in reference.<br><br><strong>No reference</strong>: the test fails if the column contains missing values.</p>        |
| **TestColumnShareOfMissingValues**(column\_name='name')           | <p>Column-level.<br><br>Tests the share of missing values in a given column against the reference or a defined condition.</p>                                                                                                 | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>  | <p>Expects up to 10% or none.<br><br><strong>With reference</strong>: the test fails if the share of missing values in a column is over 10% higher than in reference.<br><strong>No reference</strong>: the test fails if the column contains missing values.</p>            |
| **TestColumnNumberOfDifferentMissingValues**(column\_name='name') | <p>Column-level.<br><br>Tests the number of differently encoded missing values in the column against reference or a defined condition. Detects 4 types of missing values by default and/or values from a user list.</p>       | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = default list)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects <= or none.<br><br><strong>With reference</strong>: the test fails if the current column has more types of missing values.<br><br><strong>No reference</strong>: The test fails if the column contains missing values.</p>                                        |

### Correlations

| Test name                                | Description                                                                                                                                                             | Parameters                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Default test conditions                                                                                                                                                                                                                                                                                                                               |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestTargetPredictionCorrelation()**    | <p>Dataset-level.<br><br>Tests the strength of correlation between the target and prediction.</p>                                                                       | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>method</code> (default = <code>pearson</code>, available = <code>pearson</code>, <code>spearman</code>, <code>kendall</code>, <code>cramer\_v</code>)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                                                                                                                  | <p>Expects +/- 0.25 in correlation strength, or > 0.<br><br><strong>With reference</strong>: the test fails if there is a 0.25+ change in the correlation strength between target and prediction.<br><br><strong>No reference</strong>: the test fails if the correlation between target and prediction <=0</p>                                       |
| **TestHighlyCorrelatedColumns()**        | <p>Dataset-level.<br><br>Tests the strongest correlation between a pair of features, against reference or a defined condition.<br></p>                                  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>method</code> (default = <code>pearson</code>, available = <code>pearson</code>, <code>spearman</code>, <code>kendall</code>, <code>cramer\_v</code>)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                                                                                                                  | <p>Expects +/- 10% in max correlation strength, or < 0.9.<br><br><strong>With reference</strong>: the test fails if there is a 10%+ change in the correlation strength for the most correlated feature pair.<br><br><strong>No reference</strong>: the test fails if there is at least one pair of features with the correlation >= 0.9</p>           |
| **TestTargetFeaturesCorrelations()**     | <p>Dataset-level.<br><br>Tests if any of the features is highly correlated with the target.<br>Example use: to detect target leak.</p>                                  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li>'<code>method</code> (default = <code>pearson</code>, available = <code>pearson</code>, <code>spearman</code>, <code>kendall</code>, <code>cramer\_v</code>)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                                                                                                                 | <p>Expects +/- 10% in max correlation strength, or < 0.9.<br><br><strong>With reference</strong>: the test fails if there is a 10%+ change in the correlation strength for the feature most correlated with the target.<br><br><strong>No reference</strong>: the test fails if at least one feature is correlated with the target >= 0.9</p>         |
| **TestPredictionFeaturesCorrelations()** | <p>Dataset-level.<br><br>Tests if any of the features is highly correlated with the prediction<br>Example use: to detect when predictions rely on a single feature.</p> | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>method</code> (default = <code>pearson</code>, available = <code>pearson</code>, <code>spearman</code>, <code>kendall</code>, <code>cramer\_v</code>)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                                                                                                                  | <p>Expects +/- 10% in max correlation strength, or < 0.9.<br><br><strong>With reference</strong>: the test fails if there is a 10%+ change in the correlation strength for the feature most correlated with the prediction.<br><br><strong>No reference</strong>: the test fails if at least one feature is correlated with the prediction >= 0.9</p> |
| **TestCorrelationChanges()**             | <p>Dataset-level.<br><br>Tests the number of correlation violations (significant change in the correlation strength between any two columns).</p>                       | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>method</code> (default = <code>pearson</code>, available = <code>pearson</code>, <code>spearman</code>, <code>kendall</code>, <code>cramer\_v</code>)</li><li><code>corr\_diff</code> (default = 0.25)</li><li><code>column\_name</code>(checks for correlation changes only between a chosen column and other columns in the dataset)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects none.<br><br><strong>With reference</strong>: the test fails if at least 1 correlation violation is detected.<br><br><strong>No reference</strong>: N/A</p>                                                                                                                                                                                |

### Column Values

| Test name                                                                         | Description                                                                                                                                                                  | Parameters                                                                                                                                                                                                                                                        | Default test conditions                                                                                                                                                                                                      |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestColumnValueMin**(column\_name='num-column')                                 | <p>Column-level.<br><br>Tests the minimum value of a given numerical column against reference or a defined condition.</p>                                                    | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong> N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                          | <p>Expects not lower.<br><br><strong>With reference</strong>: the test fails if the minimum value is lower than in the reference.<br><br><strong>No reference</strong>: N/A</p>                                              |
| **TestColumnValueMax**(column\_name='num-column')                                 | <p>Column-level.<br><br>Tests the maximum value of a given numerical column against reference or a defined condition.</p>                                                    | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong> N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                          | <p>Expects not higher.<br><br><strong>With reference</strong>: the test fails if the maximum value is higher than in the reference.<br><br><strong>No reference</strong>: N/A</p>                                            |
| **TestColumnValueMean**(column\_name='num-column')                                | <p>Column-level.<br><br>Tests the mean value of a given numerical column against reference or a defined condition.</p>                                                       | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                       | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the mean value is different by more than 10%.<br><br><strong>No reference</strong>: N/A</p>                                                     |
| **TestColumnValueMedian**(column\_name='num-column')                              | <p>Column-level.<br><br>Tests the median value of a given numerical column against reference or a defined condition.</p>                                                     | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                       | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the median value is different by more than 10%.<br><br><strong>No reference</strong>: N/A</p>                                                   |
| <p><strong>TestColumnValueStd</strong>(column\_name='num-column')<br></p>         | <p>Column-level.<br><br>Tests the standard deviation of a given numerical column against reference or a defined condition.</p>                                               | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                       | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the standard deviation is different by more than 10%.<br><br><strong>No reference</strong>: N/A</p>                                             |
| **TestColumnQuantile**(column\_name='num\_column', quantile=0.25)                 | <p>Column-level.<br><br>Computes a quantile value and compares it to the reference or against a defined condition.</p>                                                       | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li><li><code>quantile</code></li></ul><p><strong>Optional:</strong> N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                            | <p>Expects +/-10%.<br><br><strong>With reference</strong>: the test fails if the quantile value is over 10% higher or lower.<br><br><strong>No reference</strong>: N/A</p>                                                   |
| **TestMeanInNSigmas**(column\_name='num-column')                                  | <p>Column-level.<br><br>Tests if the mean value in a given numerical column is within the expected range , defined in standard deviations. This test requires reference.</p> | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>n\_sigmas</code></li></ul>                                                                                                             | <p>Expects +/- 2 std dev.<br><br><strong>With reference</strong>: the test fails if the current mean value is out of the +/- 2 std dev interval from the reference mean value.<br><br><strong>No reference</strong>: N/A</p> |
| **TestValueRange**(column\_name='num\_column')                                    | <p>Column-level.<br><br>Tests if a numerical column contains values out of the min-max range.</p>                                                                            | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>left</code></li><li><code>right</code></li></ul><p><strong>Test conditions</strong>:<br>N/A</p>                                        | <p>Expects all values to be in range.<br><br><strong>With reference</strong>: the test fails if the column contains values out of the min-max range as seen in the reference.<br><br><strong>No reference</strong>: N/A</p>  |
| **TestShareOfOutRangeValues**(column\_name='num\_column')                         | <p>Column-level.<br><br>Tests the share of values out of the min-max range against reference or a defined condition.</p>                                                     | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>left</code></li><li><code>right</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> |                                                                                                                                                                                                                              |
| **TestNumberOfOutRangeValues**(column\_name='num\_column')                        | <p>Column-level.<br><br>Tests the number of values out of the min-max range against reference or a defined condition.</p>                                                    | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>left</code></li><li><code>right</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> |                                                                                                                                                                                                                              |
| **TestCategoryShare**(column\_name='education', category='Some-college', lt=0.5)) | <p>Column-level.<br><br>Tests if the number of objects belonging to a defined category (or having a defined numerical value) is within the threshold.</p>                    | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li><li><code>category</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                         | <p>Expects the category to be present.<br><br>The test fails if the category is not present.</p>                                                                                                                             |
| **TestCategoryCount**(column\_name='education', category='Some-college', lt=0.5)) | <p>Column-level.<br><br>Tests if the share of objects belonging to a defined category (or having a defined numerical value) is within the threshold.</p>                     | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li><li><code>category</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                         | <p>Expects the category to be present.<br><br>The test fails if the category is not present.</p>                                                                                                                             |
| **TestValueList**(column\_name='cat\_column')                                     | <p>Column-level.<br><br>Tests if a categorical column contains values out of the list.</p>                                                                                   | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>values: List\[str]</code></li></ul><p><strong>Test conditions</strong>:<br>N/A</p>                                                     | <p>Expects all values to be in the list.<br><br><strong>With reference</strong>: the test fails if the column contains values out of the list (as seen in reference).<br><br><strong>No reference</strong>: N/A</p>          |
| **TestNumberOfOutListValues**(column\_name='cat\_column')                         | <p>Column-level.<br><br>Tests the number of values in a given column that are out of list, against reference or a defined condition.</p>                                     | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>values: List\[str]</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>              | <p>Expects all values to be in the list.<br><br><strong>With reference</strong>: the test fails if the column contains values out of the list (as seen in reference).<br><br><strong>No reference</strong>: N/A</p>          |
| **TestShareOfOutListValues**(column\_name='cat\_column')                          | <p>Column-level.<br><br>Tests the share of values in a given column that are out of list against reference or a defined condition.</p>                                       | <p><strong>Required</strong>:</p><ul><li><code>column\_name</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>values: List\[str]</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>              | <p>Expects all values to be in the list.<br><br><strong>With reference</strong>: the test fails if the column contains values out of the list (as seen in reference).<br><strong>No reference</strong>: N/A</p>              |

## Data Drift

By default, all data drift tests use the Evidently [drift detection logic](/reference/data-drift-algorithm) that selects a different statistical test or metric based on feature type and volume. You always need a reference dataset. To modify the logic or select a different test, you should set [data drift parameters](/user-guide/customization/options-for-statistical-tests).

| Test name                                                 | Description                                                                                                                                                                          | Parameters                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Default test conditions                                                                                                                                                            |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestNumberOfDriftedColumns()**                          | <p>Dataset-level.<br><br>Compares the distribution of each column in the current dataset to the reference and tests the number of drifting features against a defined condition.</p> | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>сolumns</code></li><li><code>stattest</code>(default=automated selection)</li><li><code>cat\_stattest</code></li><li><code>num\_stattest</code></li><li><code>per\_column\_stattest</code></li><li><code>stattest\_threshold</code>(default=test default)</li><li><code>cat\_stattest\_threshold</code></li><li><code>num\_stattest\_threshold</code></li><li><code>per\_column\_stattest\_threshold</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects =< ⅓ features to drift.<br><br><strong>With reference:</strong> If > 1/3 of features drifted, the test fails.<br><br><strong>No reference:</strong> N/A</p>             |
| **TestShareOfDriftedColumns()**                           | <p>Dataset-level.<br><br>Compares the distribution of each column in the current dataset to the reference and tests the share of drifting features against a defined condition.</p>  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>сolumns</code></li><li><code>stattest</code>(default=automated selection)</li><li><code>cat\_stattest</code></li><li><code>num\_stattest</code></li><li><code>per\_column\_stattest</code></li><li><code>stattest\_threshold</code>(default=test default)</li><li><code>cat\_stattest\_threshold</code></li><li><code>num\_stattest\_threshold</code></li><li><code>per\_column\_stattest\_threshold</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects =< ⅓ features to drift.<br><br><strong>With reference:</strong> If > 1/3 of features drifted, the test fails.<br><br><strong>No reference:</strong> N/A</p>             |
| **TestColumnDrift(column\_name='name')**                  | <p>Column-level.<br><br>Tests if there is a distribution shift in a given column compared to the reference.</p>                                                                      | <p><strong>Required</strong>:</p><ul><li>column\_name</li></ul><p><strong>Optional</strong>:</p><ul><li><code>stattest</code>(default=automated selection)</li><li><code>stattest\_threshold</code>(default=test default)</li></ul>                                                                                                                                                                                                                                                                                                                                                                 | <p>Expects no drift.<br><br><strong>With reference:</strong> the test fails if the distribution drift is detected in a given column.<br><br><strong>No reference:</strong> N/A</p> |
| **TestEmbeddingsDrift(embeddings\_name='small\_subset')** | <p>Column-level.<br><br>Tests if there is drift in embeddings compared to reference.</p>                                                                                             | <p><strong>Required</strong>:</p><ul><li><code>embeddings\_name</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>drift\_method</code>(default=model)</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                        | <p>Expects no drift.<br><br><strong>With reference:</strong> the test fails if the drift is detected in a given subset of columns.<br><br><strong>No reference:</strong> N/A</p>   |

## Regression

**Defaults for Regression tests**: if there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model that predicts the optimal constant (varies by the metric). You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.

| Test name                                        | Description                                                                                                                                        | Parameters                                                                                                                                                               | Default test conditions                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <p><strong>TestValueMAE()</strong><br></p>       | <p>Dataset-level.<br><br>Computes the Mean Absolute Error (MAE) and compares it to the reference or against a defined condition.</p>               | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% or better than a dummy model.<br><br><strong>With reference</strong>: if MAE is higher or lower by over 10%, the test fails.<br><br><strong>No reference</strong>: the test fails if the MAE value is higher than the MAE of the dummy model that predicts the optimal constant (median of the target value).</p>                                                        |
| **TestValueRMSE()**                              | Dataset-level. Computes the Root Mean Square Error (RMSE) and compares it to the reference or against a defined condition.                         | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>  | <p>Expects +/-10% or better than a dummy model.<br><br><strong>With reference</strong>: if RMSE is higher or lower by over 10%, the test fails.<br><br><strong>No reference</strong>: the test fails if the RMSE value is higher than the RMSE of the dummy model that predicts the optimal constant (mean of the target value).</p>                                                       |
| <p><strong>TestValueMeanError()</strong><br></p> | <p>Dataset-level.<br><br>Computes the Mean Error (ME) and tests if it is near zero or compares it against a defined condition.</p>                 | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong></p><ul><li><em>standard parameters</em></li></ul>  | <p>Expects the Mean Error to be near zero.<br><br><strong>With/without reference</strong>: the test fails if the Mean Error is skewed and the condition is violated.<br>Condition: eq = approx(absolute=0.1\*error\_std)<br>error\_std = (curr\_true - curr\_preds).std()</p>                                                                                                              |
| **TestValueMAPE()**                              | <p>Dataset-level.<br><br>Computes the Mean Absolute Percentage Error (MAPE) and compares it to the reference or against a defined condition.</p>   | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% or better than a dummy model.<br><br><strong>With reference</strong>: if MAPE is higher or lower by over 10%, the test fails.<br><br><strong>No reference</strong>: the test fails if the MAPE value is higher than the MAPE of the dummy model that predicts the optimal constant (weighted median of the target value).</p>                                            |
| **TestValueAbsMaxError()**                       | <p>Dataset-level.<br><br>Computes the absolute maximum error and compares it to the reference or against a defined condition.</p>                  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% or better than a dummy model.<br><br><strong>With reference</strong>: if the absolute maximum error is higher or lower by over 10%, the test fails.<br><br><strong>No reference</strong>: the test fails if the absolute maximum error is higher than the absolute maximum error of the dummy model that predicts the optimal constant (median of the target value).</p> |
| **TestValueR2Score()**                           | <p>Dataset-level.<br><br>Computes the R2 Score (coefficient of determination) and compares it to the reference or against a defined condition.</p> | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% or > 0.<br><br><strong>With reference</strong>: if R2 is higher or lower by over 10%, the test fails.<br><br><strong>No reference</strong>: the test fails if the R2 value is =< 0.</p>                                                                                                                                                                                  |

## Classification

You can apply the tests for non-probabilistic, probabilistic classification, and ranking. The underlying metrics will be calculated slightly differently depending on the provided inputs: only labels, probabilities, decision threshold, and/or K (to compute, e.g., precision\@K).

**Defaults for Classification tests**. If there is no reference data or defined conditions, Evidently will compare the model performance to a dummy model. It is based on a set of heuristics to verify that the quality is better than random. You can also pass the reference dataset and run the test with default conditions, or define custom test conditions.

| Test name                                | Description                                                                                                                                                        | Parameters                                                                                                                                                                                                                                                                                                                                                              | Default test conditions                                                                                                                                                                                                                                                                                                       |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestAccuracyScore()**                  | <p>Dataset-level.<br><br>Computes the Accuracy and compares it to the reference or against a defined condition.</p>                                                | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>threshold\_probas</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                              | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: if the Accuracy is over 20% higher or lower, the test fails.<br><br><strong>No reference</strong>: if the Accuracy is lower than the Accuracy of the dummy model, the test fails.</p>                                                 |
| **TestPrecisionScore()**                 | <p>Dataset-level.<br><br>Computes the Precision and compares it to the reference or against a defined condition.</p>                                               | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>threshold\_probas</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                              | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: if the Precision is over 20% higher or lower, the test fails.<br><br><strong>No reference</strong>: if the Precision is lower than the Precision of the dummy mode, the test fails.</p>                                               |
| **TestRecallScore()**                    | <p>Dataset-level.<br><br>Computes the Recall and compares it to the reference or against a defined condition.</p>                                                  | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>threshold\_probas</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                              | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: if the Recall is over 20% higher or lower, the test fails.<br><br><strong>No reference</strong>: if the Recall is lower than the Recall of the dummy model, the test fails.</p>                                                       |
| **TestF1Score()**                        | <p>Dataset-level.<br><br>Computes the F1 score and compares it to the reference or against a defined condition.</p>                                                | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional</strong>:</p><ul><li><code>threshold\_probas</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                              | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: if the F1 is over 20% higher or lower, the test fails.<br><br><strong>No reference</strong>: if the F1 is lower than the F1 of the dummy model, the test fails.</p>                                                                   |
| **TestPrecisionByClass**(label='classN') | <p>Dataset-level.<br><br>Computes the Precision for the specified class and compares it to the reference or against a defined condition.</p>                       | <p><strong>Required</strong>:</p><ul><li><code>label</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: if the Precision is over 20% higher or lower, the test fails.<br><br><strong>No reference</strong>: if the Precision is lower than the Precision of the dummy model, the test fails.</p>                                              |
| **TestRecallByClass**(label='classN')    | <p>Dataset-level.<br><br>Computes the Recall for the specified class and compares it to the reference or against a defined condition.</p>                          | <p><strong>Required</strong>:</p><ul><li><code>label</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: if the Recall is over 20% higher or lower, the test fails.<br><br><strong>No reference</strong>: if the Recall is lower than the Recall of the dummy model, the test fails.</p>                                                       |
| **TestF1ByClass**(label='classN')        | <p>Dataset-level.<br><br>Computes the F1 for the specified class and compares it to the reference or against a defined constraint.</p>                             | <p><strong>Required</strong>:</p><ul><li><code>label</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: the test fails if the F1 is over 20% higher or lower.<br><br><strong>No reference</strong>: the test fails if the F1 is lower than the F1 of the dummy model.</p>                                                                     |
| **TestTPR()**                            | <p>Dataset-level.<br><br>Computes the True Positive Rate and compares it to the reference or against a defined condition.</p>                                      | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                             | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: the test fails if the TPR is over 20% higher or lower.<br><br><strong>No reference</strong>: the test fails if the TPR is lower than the TPR of the dummy model.</p>                                                                  |
| **TestTNR()**                            | <p>Dataset-level.<br><br>Computes the True Negative Rate and compares it to the reference or against a defined condition.</p>                                      | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                             | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: the test fails if the TNR is over 20% higher or lower.<br><br><strong>No reference</strong>: the test fails if the TNR is lower than the TNR of the dummy model.</p>                                                                  |
| **TestFPR()**                            | <p>Dataset-level.<br><br>Computes the False Positive Rate and compares it to the reference or against a defined condition.</p>                                     | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                             | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: the test fails if the FPR is over 20% higher or lower.<br><br><strong>No reference</strong>: the test fails if the FPR is higher than the FPR of the dummy model.</p>                                                                 |
| **TestFNR()**                            | <p>Dataset-level.<br><br>Computes the False Negative Rate and compares it to the reference or against a defined condition.</p>                                     | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                             | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: the test fails if the FNR is over 20% higher or lower.<br><br><strong>No reference</strong>: the test fails if the FNR is higher than the FNR of the dummy model.</p>                                                                 |
| **TestRocAuc()**                         | <p>Dataset-level.<br><br>Applies to probabilistic classification.<br><br>Computes the ROC AUC and compares it to the reference or against a defined condition.</p> | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                                                                                                                                | <p>Expects +/-20% or > 0.5<br><br><strong>With reference</strong>: the test fails if the ROC AUC is over 20% higher or lower than in the reference.<br><br><strong>No reference</strong>: the test fails if ROC AUC is <= 0.5.</p>                                                                                            |
| **TestLogLoss()**                        | <p>Dataset-level.<br><br>Applies to probabilistic classification.<br><br>Computes the LogLoss and compares it to the reference or against a defined condition.</p> | <p><strong>Required</strong>:<br>N/A<br><br><strong>Optional:</strong><br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                                                                                                                                | <p>Expects +/-20% or better than a dummy model.<br><br><strong>With reference</strong>: the test fails if the LogLoss is over 20% higher or lower than in the reference.<br><br><strong>No reference</strong>: the test fails if LogLoss is higher than the LogLoss of the dummy model (equals 0.5 for a constant model).</p> |

## Ranking and Recommendations

Check individual metric descriptions [here](/reference/all-metrics/ranking-metrics).

Optional shared parameters:

* `no_feedback_users: bool = False`. Specifies whether to include the users who did not select any of the items, when computing the quality metrics. Default: False.
* `min_rel_score: Optional[int] = None`. Specifies the minimum relevance score to consider relevant when calculating the quality metrics for non-binary targets (e.g., if a target is a rating or a custom score).

| Test name                                               | Description                                                                                                                                                                                                       | Parameters                                                                                                                                                                                                                                                                      | Default test conditions                                                                                                                                                                                                                       |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TestPrecisionTopK(k=k)**                              | <p>Dataset-level.<br><br>Computes the Precision at the top K and compares it to the reference or against a defined condition.</p>                                                                                 | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Precision at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if precision > 0.</p>                        |
| **TestRecallTopK(k=k)**                                 | <p>Dataset-level.<br><br>Computes the Recall at the top K and compares it to the reference or against a defined condition.</p>                                                                                    | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Recall at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if recall > 0.</p>                              |
| **TestFBetaTopK(k=k)**                                  | <p>Dataset-level.<br><br>Computes the F-beta score at the top K and compares it to the reference or against a defined condition.</p>                                                                              | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the F-beta score at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if F-beta > 0.</p>                        |
| **TestHitRateK(k=k)**                                   | <p>Dataset-level.<br><br>Computes the Hit Rate at the top K recommendations and compares it to the reference or against a defined condition.</p>                                                                  | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Hit Rate at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if Hit Rate > 0.</p>                          |
| **TestMAPK(k=k)**                                       | <p>Dataset-level.<br><br>Computes the Mean Average Precision at the top K and compares it to the reference or against a defined condition.</p>                                                                    | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the MAP at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if MAP > 0.</p>                                    |
| **TestMRRK(k=k)**                                       | <p>Dataset-level.<br><br>Computes the Mean Reciprocal Rank at the top K and compares it to the reference or against a defined condition.</p>                                                                      | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the MRR at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if MRR > 0.</p>                                    |
| **TestNDCGK(k=k)**                                      | <p>Dataset-level.<br><br>Computes the Normalized Discounted Cumulative Gain at the top K and compares it to the reference or against a defined condition.</p>                                                     | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul> | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Normalized Discounted Cumulative Gain at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if NDCG > 0.</p> |
| **TestNovelty(k=k)**                                    | <p>Dataset-level.<br><br>Computes the Novelty at the top K recommendations and compares it to the reference or against a defined condition.<br><br>Requires a training dataset.</p>                               | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Novelty at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if novelty > 0.</p>                            |
| **TestPersonalization(k=k)**                            | <p>Dataset-level.<br><br>Computes the Personalization at the top K recommendations and compares it to the reference or against a defined condition.</p>                                                           | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Personalization at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if personalization > 0.</p>            |
| **TestSerendipity**(k=k, item\_features=item\_features) | <p>Dataset-level.<br><br>Computes the Serendipity at the top K recommendations considering item features and compares it to the reference or against a defined condition.<br><br>Requires a training dataset.</p> | <p><strong>Required</strong>:</p><ul><li><code>k</code></li><li><code>item\_features</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>min\_rel\_score</code></li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>      | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Serendipity at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if serendipity > 0.</p>                    |
| **TestDiversity**(k=k, item\_features=item\_features)   | <p>Dataset-level.<br><br>Computes the Diversity at the top K recommendations considering item features and compares it to the reference or against a defined condition.</p>                                       | <p><strong>Required</strong>:</p><ul><li><code>k</code></li><li><code>item\_features</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                            | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Diversity at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if diversity > 0.</p>                        |
| **TestARP(k=k)**                                        | <p>Dataset-level.<br><br>Computes the Average Recommendation Popularity at the top K recommendations and compares it to the reference or against a defined condition.<br><br>Requires a training dataset.</p>     | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>normalize\_arp</code> (default: False)</li></ul><p><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                          | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the ARP at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if ARP > 0.</p>                                    |
| **TestGiniIndex(k=k)**                                  | <p>Dataset-level.<br><br>Computes the Gini Index at the top K recommendations and compares it to the reference or against a defined condition.<br><br>Requires a training dataset.</p>                            | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Gini Index at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if Gini Index < 1.</p>                      |
| **TestCoverage(k=k)**                                   | <p>Dataset-level.<br><br>Computes the Coverage at the top K recommendations and compares it to the reference or against a defined condition.<br><br>Requires a training dataset.</p>                              | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:<br>N/A<br><br><strong>Test conditions</strong>:</p><ul><li><em>standard parameters</em></li></ul>                                                                                | <p>Expects +/-10% from reference.<br><br><strong>With reference</strong>: if the Coverage at the top K is over 10% higher or lower, the test fails.<br><br><strong>No reference</strong>: Tests if Coverage > 0.</p>                          |


# All metrics

List of Metrics, Descriptors and Metric Presets available in Evidently.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

<details>

<summary>How to use this page</summary>

This is a reference page. It shows all the available Metrics, Descriptors and Presets.

You can use the menu on the right to navigate the sections. We organize the Metrics by logical groups. Note that these groups do **not** match the Presets with a similar name. For example, there are more Data Quality Metrics than included in the `DataQualityPreset`.

## How to read the tables

* **Name**: the name of the Metric.
* **Description**: plain text explanation. For Metrics, we also specify whether it applies to the whole dataset or individual columns.
* **Parameters**: required and optional parameters for the Metric or Preset. We also specify the defaults that apply if you do not pass a custom parameter.

**Metric visualizations**. Each Metric includes a default render. To see the visualization, navigate to the [example notebooks](https://github.com/evidentlyai/docs-old/blob/main/get-started/examples.md) and run the notebook with all Metrics or Metric Presets.

</details>

{% hint style="info" %}
We are doing our best to maintain this page up to date. In case of discrepancies, check the "All metrics" notebook in [examples](/tutorials-and-examples/examples). If you notice an error, please send us a pull request with an update!
{% endhint %}

## Metric Presets

**Defaults**: Presets use the default parameters for each Metric. You can see them in the tables below.

<details>

<summary>Data Quality Preset</summary>

`DataQualityPreset` captures column and dataset summaries. Input columns are required. Prediction and target are optional.

**Composition**:

* `DatasetSummaryMetric()`
* `ColumnSummaryMetric()` for `all` or specified `сolumns`
* `DatasetMissingValuesMetric()`

**Optional parameters**:

* `columns`

</details>

<details>

<summary>Data Drift Preset</summary>

`DataDriftPreset` evaluates the data distribution drift in all individual columns, and share of drifting columns in the dataset. Input columns are required.

**Composition**:

* `DataDriftTable()` for all or specified `columns`
* `DatasetDriftMetric()` for all or specified `columns`

**Optional parameters**:

* `columns`
* `stattest`
* `cat_stattest`
* `num_stattest`
* `per_column_stattest`
* `text_stattest`
* `stattest_threshold`
* `cat_stattest_threshold`
* `num_stattest_threshold`
* `per_column_stattest_threshold`
* `text_stattest_threshold`
* `embeddings`
* `embeddings_drift_method`
* `drift_share`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests), [embeddings drift parameters](/user-guide/customization/embeddings-drift-parameters).

</details>

<details>

<summary>Target Drift Preset</summary>

`TargetDriftPreset` evaluates the prediction or target drift. Target and/or prediction is required. Input features are optional.

**Composition**:

* `ColumnDriftMetric()` for `target` and/or `prediction` columns
* `ColumnCorrelationsMetric()` for `target` and/or `prediction` columns
* `TargetByFeaturesTable()` for all or specified `columns`
* `ColumnValuePlot()` for `target` and/or `prediction` columns - if the task is `regression`

**Optional parameters**:

* `columns`
* `stattest`
* `cat_stattest`
* `num_stattest`
* `per_column_stattest`
* `stattest_threshold`
* `cat_stattest_threshold`
* `num_stattest_threshold`
* `per_column_stattest_threshold`

How to set [data drift parameters](/user-guide/customization/options-for-statistical-tests).

</details>

<details>

<summary>Regression Preset</summary>

`RegressionPreset` evaluates the quality of a regression model. Prediction and target are required. Input features are optional.

**Composition**:

* `RegressionQualityMetric()`
* `RegressionPredictedVsActualScatter()`
* `RegressionPredictedVsActualPlot()`
* `RegressionErrorPlot()`
* `RegressionAbsPercentageErrorPlot()`
* `RegressionErrorDistribution()`
* `RegressionErrorNormality()`
* `RegressionTopErrorMetric()`
* `RegressionErrorBiasTable()` for all or specified `columns`

**Optional parameters**:

* `columns`

</details>

<details>

<summary>Classification Preset</summary>

`ClassificationPreset` evaluates the quality of a classification model. Prediction and target are required. Input features are optional.

**Composition**:

* `ClassificationQualityMetric()`
* `ClassificationClassBalance()`
* `ClassificationConfusionMatrix()`
* `ClassificationQualityByClass()`
* `ClassificationClassSeparationPlot()` - if probabilistic classification
* `ClassificationProbDistribution()`- if probabilistic classification
* `ClassificationRocCurve()` - if probabilistic classification
* `ClassificationPRCurve()` - if probabilistic classification
* `ClassificationPRTable()` - if probabilistic classification
* `ClassificationQualityByFeatureTable()` for all or specified `columns`

**Optional parameters**:

* `columns`
* `probas_threshold`

</details>

<details>

<summary>Text Evals</summary>

`TextEvals()` provides a simplified interface to list `Descriptors` for a given text column. It it returns a summary of evaluation results.

**Composition**:

* `ColumnSummaryMetric()` for text descriptors for the specified text column:
  * `Sentiment()`
  * `SentenceCount()`
  * `OOV()`
  * `TextLength()`
  * `NonLetterCharacterPercentage()`

**Required parameters**:

* `column_name`

**Optional parameters**:

* `descriptors` list

</details>

<details>

<summary>RecSys (Recommender System) Preset</summary>

`RecsysPreset` evaluates the quality of the recommender system. Recommendations and true relevance scores are required. For some metrics, training data and item features are required.

**Composition**:

* `PrecisionTopKMetric()`
* `RecallTopKMetric()`
* `FBetaTopKMetric()`
* `MAPKMetric()`
* `NDCGKMetric()`
* `MRRKMetric()`
* `HitRateKMetric()`
* `PersonalizationMetric()`
* `PopularityBias()`
* `RecCasesTable()`
* `ScoreDistribution()`
* `DiversityMetric()`
* `SerendipityMetric()`
* `NoveltyMetric()`
* `ItemBiasMetric()` (pass column as a parameter)
* `UserBiasMetric()`(pass column as a parameter)

**Required parameter**:

* `k`

**Optional parameters**:

* `min_rel_score: Optional[int]`
* `no_feedback_users: bool`
* `normalize_arp: bool`
* `user_ids: Optional[List[Union[int, str]]]`
* `display_features: Optional[List[str]]`
* `item_features: Optional[List[str]]`
* `user_bias_columns: Optional[List[str]]`
* `item_bias_columns: Optional[List[str]]`

</details>

## Data Quality

| Metric                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Parameters                                                                                                                                                                                                                                                                                                           |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>DatasetSummaryMetric()</strong><br><br>Dataset-level.<br><br>Calculates descriptive dataset statistics, including:</p><ul><li>Number of columns by type</li><li>Number of rows</li><li>Missing values</li><li>Empty columns</li><li>Constant and almost constant columns</li><li>Duplicated and almost duplicated columns</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                  | <p><strong>Required</strong>:<br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>missing\_values = \[], replace = True/False</code> (see default types below)</li><li><code>almost\_constant\_threshold</code> (default = 0.95)</li><li><code>almost\_duplicated\_threshold</code> (default = 0.95)</li></ul> |
| <p><strong>DatasetMissingValuesMetric()</strong><br><br>Dataset-level.<br><br>Calculates the number and share of missing values in the dataset.<br><br>Displays the number of missing values per column.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | <p><strong>Required</strong>:<br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = four types of missing values, see above)</li></ul>                                                                                                             |
| <p><strong>DatasetCorrelationsMetric()</strong><br><br>Dataset-level.<br><br>Calculates the correlations between all columns in the dataset. Uses: Pearson, Spearman, Kendall, Cramer\_V.<br><br>Visualizes the heatmap.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                                    |
| <p><strong>ColumnSummaryMetric()</strong><br><br>Column-level.<br><br>Calculates various descriptive statistics for numerical, categorical, text or DateTime columns, including:</p><ul><li>Count</li><li>Min, max, mean (for numerical)</li><li>Standard deviation (for numerical)</li><li>Quantiles - 25%, 50%, 75% (for numerical)</li><li>Unique value share</li><li>Most common value share</li><li>Missing value share</li><li>New and missing categories (for categorical)</li><li>Last and first date (for DateTime)</li><li>Length, OOV% and Non-letter % (for text)</li></ul><p><br>Plots the distribution histogram. If DateTime is provided, also plots the distribution over time. If Target is provided, also plots the relation with Target.</p> | <p><strong>Required</strong>:<br><code>column\_name</code><br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                              |
| <p><strong>ColumnMissingValuesMetric()</strong><br><br>Column-level.<br><br>Calculates the number and share of missing values in the column.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | <p><strong>Required</strong>:<br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>missing\_values = \[], replace = True/False</code> (default = four types of missing values, see below)</li></ul>                                                                                                             |
| <p><strong>ColumnRegExpMetric()</strong><br><br>Column-level.<br><br>Calculates the number and share of the values that do not match a defined regular expression.<br><br>Example use: <code>ColumnRegExpMetric(column\_name="status", reg\_exp=r".*child.*")</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | <p><strong>Required:</strong></p><ul><li><code>column\_name</code></li><li><code>reg\_exp</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>top</code> (the number of the most mismatched columns to return, default = 10)</li></ul>                                                                    |
| <p><strong>ColumnDistributionMetric()</strong><br><br>Column-level.<br><br>Plots the distribution histogram and returns bin positions and values for the given column.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | <p><strong>Required:</strong><br><code>column\_name</code><br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                              |
| <p><strong>ColumnValuePlot()</strong><br><br>Column-level.<br><br>Plots the values in time.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | <p><strong>Required:</strong><br><code>column\_name</code><br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                              |
| <p><strong>ColumnQuantileMetric()</strong><br><br>Column-level.<br><br>Calculates the defined quantile value and plots the distribution for the given numerical column.<br><br>Example use: <code>ColumnQuantileMetric(column\_name="name", quantile=0.75)</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | <p><strong>Required:</strong></p><ul><li><code>column\_name</code></li><li><code>quantile</code></li></ul><p><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                   |
| <p><strong>ColumnCorrelationsMetric()</strong><br><br>Column-level.<br><br>Calculates the correlations between the defined column and all the other columns in the dataset.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | <p><strong>Required:</strong><br><code>column\_name</code><br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                              |
| <p><strong>ColumnValueListMetric()</strong><br><br>Column-level.<br><br>Calculates the number of values in the list / out of the list / not found in a given column. The value list should be specified.<br><br>Example use: <code>ColumnValueListMetric(column\_name="city", values=\["London", "Paris"])</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                           | <p><strong>Required:</strong></p><ul><li><code>column\_name</code></li><li><code>values</code></li></ul><p><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                     |
| <p><strong>ColumnValueRangeMetric()</strong><br><br>Column-level.<br><br>Calculates the number and share of values in the specified range / out of range in a given column. Plots the distributions.<br><br>Example use: <code>ColumnValueRangeMetric(column\_name="age", left=10, right=20)</code></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                         | <p><strong>Required:</strong></p><ul><li><code>column\_name</code></li><li><code>left</code></li><li><code>right</code></li></ul>                                                                                                                                                                                    |
| <p><strong>ConflictPredictionMetric()</strong><br><br>Dataset-level.<br><br>Calculates the number of instances where the model returns a different output for an identical input. Can be a signal of low-quality model or data errors.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                                    |
| <p><strong>ConflictTargetMetric()</strong><br><br>Dataset-level.<br><br>Calculates the number of instances where there is a different target value or label for an identical input. Can be a signal of a labeling or data error.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                                    |

**Defaults for Missing Values**. The metrics that calculate the number or share of missing values detect four types of missing values by default: Pandas nulls (None, NAN, etc.), "" (empty string), Numpy "-inf" value, Numpy "inf" value. You can also pass custom missing values as a parameter and specify if you want to replace the default list. Example:

```python
DatasetMissingValuesMetric(missing_values=["", 0, "n/a", -9999, None], replace=True)
```

## Text Evals

Text Evals only apply to text columns. To compute a Descriptor for a single text column, use a `TextEvals` Preset. Read [docs](/user-guide/tests-and-reports/text-descriptors).

You can also explicitly specify the Evidently Metric (e.g., `ColumnSummaryMetric`) to visualize the descriptor, or pick a [Test](/reference/all-tests) (e.g., `TestColumnValueMin`) to run validations.

### Descriptors: Text Patterns

Check for regular expression matches.

| Descriptor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Parameters                                                                                                                                                                                                                                                                  |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>RegExp()</strong></p><ul><li>Matches text against any specified regular expression.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>RegExp(reg\_exp=r"^I")</code></p>                                                                                                                                                                                                                                                                                               | <p><strong>Required:</strong><br><code>reg\_exp</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                    |
| <p><strong>BeginsWith()</strong></p><ul><li>Checks if the text begins with a specified combination.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>BeginsWith(prefix="How")</code></p>                                                                                                                                                                                                                                                                                        | <p><strong>Required:</strong><br><code>prefix</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>case\_sensitive = True</code> or <code>False</code></li></ul>                                                                    |
| <p><strong>EndsWith()</strong></p><ul><li>Checks if the text ends with a specified combination.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>EndsWith(suffix="Thank you.")</code></p>                                                                                                                                                                                                                                                                                       | <p><strong>Required:</strong><br><code>suffix</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>case\_sensitive = True</code> or <code>False</code></li></ul>                                                                    |
| <p><strong>Contains()</strong></p><ul><li>Checks if the text contains any or all specified items (e.g. competitor names, etc.)</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>Contains(items=\["medical leave"])</code></p>                                                                                                                                                                                                                                                   | <p><strong>Required:</strong><br><code>items: List\[str]</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'any'</code> or <code>'all'</code></li><li><code>case\_sensitive = True</code> or <code>False</code></li></ul> |
| <p><strong>DoesNotContain()</strong></p><ul><li>Checks if the text does not contain any or all specified items.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>DoesNotContain(items=\["as a large language model"]</code></p>                                                                                                                                                                                                                                                 | <p><strong>Required:</strong><br><code>items: List\[str]</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'all'</code> or <code>'any'</code></li><li><code>case\_sensitive = True</code> or <code>False</code></li></ul> |
| <p><strong>IncludesWords()</strong></p><ul><li>Checks if the text includes <strong>any</strong> (default) or <strong>all</strong> specified words.</li><li>Considers only vocabulary words (from NLTK vocabulary).</li><li>By default, considers inflected and variant forms of the same word.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>IncludesWords(words\_list=\['booking', 'hotel', 'flight']</code></p>                                                            | <p><strong>Required:</strong><br><code>words\_list: List\[str]</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'any'</code> or <code>'all'</code></li><li><code>lemmatize = True</code> or <code>False</code></li></ul> |
| <p><strong>ExcludesWords()</strong></p><ul><li>Checks if the text excludes all specified words.</li><li>Considers only vocabulary words (from NLTK vocabulary).</li><li>By default, considers inflected and variant forms of the same word.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>ExcludesWords(words\_list=\['buy', 'sell', 'bet']</code></p>                                                                                                                       | <p><strong>Required:</strong><br><code>words\_list: List\[str]</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'all'</code> or <code>'any'</code></li><li><code>lemmatize = True</code> or <code>False</code></li></ul> |
| <p><strong>ItemMatch()</strong></p><ul><li>Checks whether the text contains <strong>any</strong> (default) or <strong>all</strong> specified items that are specific to each row (represented as tuples)</li><li>Returns True/False for each row.</li></ul><p>Example use:<br><code>ItemMatch(with\_column="expected")</code></p>                                                                                                                                                                            | <p><strong>Required:</strong><br><code>with\_column: str</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'all'</code> or <code>'any'</code></li><li><code>case\_sensitive = True</code> or <code>False</code></li></ul> |
| <p><strong>ItemNoMatch()</strong></p><ul><li>Checks whether the text excludes <strong>any</strong> (default) or <strong>all</strong> specified items that are specific to each row (represented as tuples)</li><li>Returns True/False for each row.</li></ul><p>Example use:<br><code>ItemMatch(with\_column="forbidden")</code></p>                                                                                                                                                                         | <p><strong>Required:</strong><br><code>with\_column: str</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'all'</code> or <code>'any'</code></li><li><code>case\_sensitive = True</code> or <code>False</code></li></ul> |
| <p><strong>WordMatch()</strong></p><ul><li>Checks whether the text includes <strong>any</strong> (default) or <strong>all</strong> specified words for each row (represented as tuples).</li><li>Considers only vocabulary words (from NLTK vocabulary).</li><li>By default, considers inflected and variant forms of the same word.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>WordMatch(with\_column="expected")</code></p>                                             | <p><strong>Required:</strong><br><code>with\_column: str</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'any'</code> or <code>'all'</code></li><li><code>lemmatize = True</code> or <code>False</code></li></ul>       |
| <p><strong>WordNoMatch()</strong></p><ul><li>Checks whether the text excludes <strong>any</strong> (default) or <strong>all</strong> specified words for each row (represented as tuples).</li><li>Considers only vocabulary words (from NLTK vocabulary).</li><li>By default, considers inflected and variant forms of the same word.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>WordMatch(with\_column="forbidden")</code></p>                                          | <p><strong>Required:</strong><br><code>with\_column: str</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>mode = 'any'</code> or <code>'all'</code></li><li><code>lemmatize = True</code> or <code>False</code></li></ul>       |
| <p><strong>ExactMatch()</strong></p><ul><li>Checks if the text matches between two columns.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>ExactMatch(with\_column='reference')</code></p>                                                                                                                                                                                                                                                                                    | <p><strong>Required:</strong><br><code>with\_column: str</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                           |
| <p><strong>IsValidJSON()</strong></p><ul><li>Checks if the text in a specified column is a valid JSON.</li><li>Returns True/False for every input.</li></ul>                                                                                                                                                                                                                                                                                                                                                 | <p><strong>Required:</strong> n/a<br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                                             |
| <p><strong>JSONSchemaMatch()</strong></p><ul><li>Checks if the text contains a JSON object matching the expected\_schema.</li><li>Supports exact (<strong>exact=True</strong>) or minimal (<strong>exact=False</strong>) matching, with optional strict type validation (<strong>validate\_types=True</strong>).</li><li>Returns True/False for each row.</li></ul><p>Example use:<br><code>JSONSchemaMatch(expected\_schema={"name": str, "age": int}, exact\_match=False, validate\_types=True)</code></p> | <p><strong>Required:</strong><br><code>expected\_schema: Dict\[str, type]</code><br><br><strong>Optional:</strong></p><ul><li><code>exact\_match = True</code> or <code>False</code></li><li><code>validate\_types = True</code> or <code>False</code></li></ul>            |
| <p><strong>JSONMatch()</strong></p><ul><li>Compares two columns of a dataframe and checks whether the two objects in each row of the dataframe are matching JSONs or not.</li><li>Returns True/False for every input.</li></ul><p>Example use:<br><code>JSONMatch(with\_column="column\_2")</code></p>                                                                                                                                                                                                       | <p><strong>Required:</strong><br><code>with\_column : str</code><br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                          |
| <p><strong>ContainsLink()</strong></p><ul><li>Checks if the text contains at least one valid URL.</li><li>Returns True/False for each row.</li></ul>                                                                                                                                                                                                                                                                                                                                                         | <p><strong>Required:</strong> n/a<br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                                             |
| <p><strong>IsValidPython()</strong></p><ul><li>Checks if the text is valid Python code without syntax errors.</li><li>Returns True/False for every input.</li></ul>                                                                                                                                                                                                                                                                                                                                          | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                                      |
| <p><strong>IsValidSQL()</strong></p><ul><li>Checks if the text in a specified column is a valid SQL query without executing the query.</li><li>Returns True/False for every input.</li></ul>                                                                                                                                                                                                                                                                                                                 | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                                      |

### Descriptors: Text stats

Computes descriptive text statistics.

| Descriptor                                                                                                                                                                          | Parameters                                                                                                                                                            |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>TextLength()</strong></p><ul><li>Measures the length of the text in symbols.</li><li>Returns an absolute number.</li></ul>                                               | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                |
| <p><strong>OOV()</strong></p><ul><li>Calculates the percentage of out-of-vocabulary words based on imported NLTK vocabulary.</li><li>Return a score on a scale: 0 to 100.</li></ul> | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li><li><code>ignore\_words: Tuple = ()</code></li></ul> |
| <p><strong>NonLetterCharacterPercentage()</strong></p><ul><li>Calculates the percentage of non-letter characters.</li><li>Return a score on a scale: 0 to 100.</li></ul>            | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                |
| <p><strong>SentenceCount()</strong></p><ul><li>Counts the number of sentences in the text.</li><li>Returns an absolute number.</li></ul>                                            | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                |
| <p><strong>WordCount()</strong></p><ul><li>Counts the number of words in the text.</li><li>Returns an absolute number.</li></ul>                                                    | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                |

### Descriptors: LLM-based

Use external LLMs with an evaluation prompt to score text data. (Also known as LLM-as-a-judge method).

| Descriptor                                                                                                                                                                                                                                   | Parameters                                                                                                                                                       |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>LLMEval()</strong><br><br>Scores the text using the user-defined criteria, automatically formatted in a templated evaluation prompt.</p>                                                                                          | See [docs](/user-guide/customization/llm_as_a_judge) for examples and parameters.                                                                                |
| <p><strong>DeclineLLMEval()</strong><br><br>Detects texts containing a refusal or a rejection to do something. Returns a label (DECLINE or OK) or score.</p>                                                                                 | See [docs](/user-guide/customization/llm_as_a_judge) for parameters.                                                                                             |
| <p><strong>PIILLMEval()</strong><br><br>Detects texts containing PII (Personally Identifiable Information). Returns a label (PII or OK) or score.</p>                                                                                        | See [docs](/user-guide/customization/llm_as_a_judge) for parameters.                                                                                             |
| <p><strong>NegativityLLMEval()</strong><br><br>Detects negative texts (containing critical or pessimistic tone). Returns a label (NEGATIVE or POSITIVE) or score.</p>                                                                        | See [docs](/user-guide/customization/llm_as_a_judge) for parameters.                                                                                             |
| <p><strong>BiasLLMEval()</strong><br><br>Detects biased texts (containing prejudice for or against a person or group). Returns a label (BIAS or OK) or score.</p>                                                                            | See [docs](/user-guide/customization/llm_as_a_judge) for parameters.                                                                                             |
| <p><strong>ToxicityLLMEval()</strong><br><br>Detects toxic texts (containing harmful, offensive, or derogatory language). Returns a label (TOXICITY or OK) or score.</p>                                                                     | See [docs](/user-guide/customization/llm_as_a_judge) for parameters.                                                                                             |
| <p><strong>ContextQualityLLMEval()</strong><br><br>Evaluates if CONTEXT is VALID (has sufficient information to answer the QUESTION) or INVALID (has missing or contradictory information). Returns a label (VALID or INVALID) or score.</p> | Run the descriptor over the `context` column and pass the `question` column as a parameter. See [docs](/user-guide/customization/llm_as_a_judge) for parameters. |

### Descriptors: Model-based

Use pre-trained machine learning models for evaluation.

| Descriptor                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Parameters                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>Semantic Similarity()</strong></p><ul><li>Calculates pairwise semantic similarity between columns.</li><li>Generates text embeddings using a <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">transformer model</a>.</li><li>Calculates Cosine Similarity between each pair of texts.</li><li>Return a score on a scale: 0 to 1. (0: different, 0.5: unrelated, 1: identical).</li></ul><p>Example use:<br><code>SemanticSimilarity(with\_column="response")</code></p> | <p><strong>Required:</strong></p><ul><li><code>with\_column</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                                                                                                                                                                         |
| <p><strong>Sentiment()</strong></p><ul><li>Analyzes the sentiment of the text using a word-based model.</li><li>Return a score on a scale: -1 (negative) to 1 positive).</li></ul>                                                                                                                                                                                                                                                                                                                        | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>display\_name</code></li></ul>                                                                                                                                                                                                                                                                                                                            |
| <p><strong>HuggingFaceModel()</strong><br><br>Scores the text using the user-selected HuggingFace model.</p>                                                                                                                                                                                                                                                                                                                                                                                              | See [docs](/user-guide/customization/huggingface_descriptor) with some example models (classification by topic, emotion, etc.)                                                                                                                                                                                                                                                                                                                    |
| <p><strong>HuggingFaceToxicityModel()</strong></p><ul><li>Detects hate speech using <a href="https://huggingface.co/facebook/roberta-hate-speech-dynabench-r4-target">HuggingFace Model</a>.</li><li>Returns predicted probability for the “hate” label.</li><li>Scale: 0 to 1.</li></ul>                                                                                                                                                                                                                 | <p><strong>Optional</strong>:</p><ul><li><code>toxic\_label="hate"</code> (default)</li><li><code>display\_name</code></li></ul>                                                                                                                                                                                                                                                                                                                  |
| <p><strong>BERTScore()</strong></p><ul><li>Calculates similarity between two text columns based on token embeddings from a pre-trained BERT model.</li><li>Returns <a href="https://arxiv.org/pdf/1904.09675">BERTScore</a> (F1 Score) based on cosine similarity between token embeddings.</li></ul>                                                                                                                                                                                                     | <p><strong>Required:</strong></p><ul><li><code>with\_column</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>model</code>:Name of the pre-trained BERT model to use (default: <code>"bert-base-uncased"</code>).</li><li><code>tfidf\_weighted</code>: Boolean indicating if embeddings should be weighted with inverse document frequency (IDF) scores (default: <code>False</code>).</li><li><code>display\_name</code></li></ul> |

## Data Drift

**Defaults for Data Drift**. By default, all data drift metrics use the Evidently [drift detection logic](/reference/data-drift-algorithm) that selects a drift detection method based on feature type and volume. You always need a reference dataset.

To modify the logic or select a different test, you should set [data drift parameters](/user-guide/customization/options-for-statistical-tests) or [embeddings drift parameters](/user-guide/customization/embeddings-drift-parameters). You can choose from 20+ drift detection methods and optionally pass [feature importances](/user-guide/customization/feature-importance).

| Metric                                                                                                                                                                                                                                                                             | Parameters                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>DatasetDriftMetric()</strong></p><ul><li>Dataset-level.</li><li>Calculates the number and share of drifted features in the dataset.</li><li>Each feature is tested for drift individually using the default algorithm, unless a custom approach is specified.</li></ul> | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>сolumns</code> (default=all)</li><li><code>drift\_share</code>(default for dataset drift = 0.5)</li><li><code>stattest</code></li><li><code>cat\_stattest</code></li><li><code>num\_stattest</code></li><li><code>per\_column\_stattest</code></li><li><code>stattest\_threshold</code></li><li><code>cat\_stattest\_threshold</code></li><li><code>num\_stattest\_threshold</code></li><li><code>per\_column\_stattest\_threshold</code></li></ul><p><a href="/user-guide/customization/options-for-statistical-tests">How to set data drift parameters</a>.</p>                 |
| <p><strong>DataDriftTable()</strong></p><ul><li>Dataset-level.</li><li>Calculates data drift for all or selected columns.</li><li>Returns drift detection results for each column.</li><li>Visualizes distributions for all columns in a table.</li></ul>                          | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>сolumns</code></li><li><code>stattest</code></li><li><code>cat\_stattest</code></li><li><code>num\_stattest</code></li><li><code>per\_column\_stattest</code></li><li><code>stattest\_threshold</code></li><li><code>cat\_stattest\_threshold</code></li><li><code>num\_stattest\_threshold</code></li><li><code>per\_column\_stattest\_threshold</code></li></ul><p><a href="/user-guide/customization/options-for-statistical-tests">How to set data drift parameters</a>, <a href="/user-guide/customization/embeddings-drift-parameters">embeddings drift parameters</a>.</p> |
| <p><strong>ColumnDriftMetric()</strong></p><ul><li>Column-level.</li><li>Calculates data drift for a defined column (tabular or text).</li><li>Visualizes distributions.</li></ul>                                                                                                 | <p><strong>Required:</strong></p><ul><li><code>column\_name</code></li></ul><p><br><strong>Optional:</strong></p><ul><li><code>stattest</code></li><li><code>stattest\_threshold</code></li></ul><p><a href="/user-guide/customization/options-for-statistical-tests">How to set data drift parameters</a></p>                                                                                                                                                                                                                                                                                                                                                            |
| <p><strong>EmbeddingsDriftMetric()</strong></p><ul><li>Column-level.</li><li>Calculates data drift for embeddings.</li><li>Requires embedding column mapping.</li></ul>                                                                                                            | <p><strong>Required:</strong></p><ul><li><code>embeddings\_name</code></li></ul><p><br><strong>Optional:</strong></p><ul><li><code>drift\_method</code></li></ul><p><a href="/user-guide/customization/embeddings-drift-parameters">How to set embeddings drift parameters</a>.</p>                                                                                                                                                                                                                                                                                                                                                                                       |

## Classification

The metrics work both for probabilistic and non-probabilistic classification. All metrics are dataset-level. All metrics require column mapping of target and prediction.

| Metric                                                                                                                                                                                                                                                                                                                                                                                                                                                        | Parameters                                                                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>ClassificationDummyMetric()</strong><br><br>Calculates the quality of the dummy model built on the same data. This can serve as a baseline.</p>                                                                                                                                                                                                                                                                                                    | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationQualityMetric()</strong><br><br>Calculates various classification performance metrics, including:</p><ul><li>Accuracy</li><li>Precision</li><li>Recall</li><li>F-1 score</li><li>TPR (True Positive Rate)</li><li>TNR (True Negative Rate)</li><li>FPR (False Positive Rate)</li><li>FNR (False Negative Rate)</li><li>ROC AUC Score (for probabilistic classification)</li><li>LogLoss (for probabilistic classification)</li></ul> | <p><strong>Required:</strong>:<br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code> (default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul> |
| <p><strong>ClassificationClassBalance()</strong><br><br>Calculates the number of objects for each label. Plots the histogram.</p>                                                                                                                                                                                                                                                                                                                             | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationConfusionMatrix()</strong><br><br>Calculates the TPR, TNR, FPR, FNR, and plots the confusion matrix.</p>                                                                                                                                                                                                                                                                                                                             | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul>   |
| <p><strong>ClassificationQualityByClass()</strong><br><br>Calculates the classification quality metrics for each class. Plots the matrix.</p>                                                                                                                                                                                                                                                                                                                 | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>probas\_threshold</code>(default for classification = None; default for probabilistic classification = 0.5)</li><li><code>k</code> (default = None)</li></ul>   |
| <p><strong>ClassificationClassSeparationPlot()</strong><br><br>Visualization of the predicted probabilities by class. Applicable for probabilistic classification only.</p>                                                                                                                                                                                                                                                                                   | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationProbDistribution()</strong><br><br>Visualization of the probability distribution by class. Applicable for probabilistic classification only.</p>                                                                                                                                                                                                                                                                                     | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationRocCurve()</strong><br><br>Plots ROC Curve. Applicable for probabilistic classification only.</p>                                                                                                                                                                                                                                                                                                                                    | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationPRCurve()</strong><br><br>Plots Precision-Recall Curve. Applicable for probabilistic classification only.</p>                                                                                                                                                                                                                                                                                                                        | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationPRTable()</strong><br><br>Calculates the Precision-Recall table that shows model quality at a different decision threshold.</p>                                                                                                                                                                                                                                                                                                      | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                       |
| <p><strong>ClassificationQualityByFeatureTable()</strong><br><br>Plots the relationship between feature values and model quality.</p>                                                                                                                                                                                                                                                                                                                         | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>columns</code>(default = all categorical and numerical columns)</li></ul>                                                                                       |

## Regression

All metrics are dataset-level. All metrics require column mapping of target and prediction.

| Metric                                                                                                                                                                                                                                                                                                                                                             | Parameters                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <p><strong>RegressionDummyMetric()</strong><br><br>Calculates the quality of the dummy model built on the same data. This can serve as a baseline.</p>                                                                                                                                                                                                             | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionQualityMetric()</strong><br><br>Calculates various regression performance metrics, including:</p><ul><li>RMSE</li><li>Mean error (+ standard deviation)</li><li>MAE(+ standard deviation)</li><li>MAPE (+ standard deviation)</li><li>Max absolute error</li></ul>                                                                            | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionPredictedVsActualScatter()</strong><br><br>Visualizes predicted vs actual values in a scatter plot.</p>                                                                                                                                                                                                                                       | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionPredictedVsActualPlot()</strong><br><br>Visualizes predicted vs. actual values in a line plot.</p>                                                                                                                                                                                                                                            | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionErrorPlot()</strong><br><br>Visualizes the model error (predicted - actual) in a line plot.</p>                                                                                                                                                                                                                                               | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionAbsPercentageErrorPlot()</strong><br><br>Visualizes the absolute percentage error in a line plot.</p>                                                                                                                                                                                                                                         | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionErrorDistribution()</strong><br><br>Visualizes the distribution of the model error in a histogram.</p>                                                                                                                                                                                                                                        | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionErrorNormality()</strong><br><br>Visualizes the quantile-quantile plot (Q-Q plot) to estimate value normality.</p>                                                                                                                                                                                                                            | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong><br>n/a</p>                                                                                                                                                                                                                            |
| <p><strong>RegressionTopErrorMetric()</strong><br><br>Calculates the regression performance metrics for different groups:</p><ul><li>top-X% of predictions with overestimation</li><li>top-X% of predictions with underestimation</li><li>Majority(the rest)</li></ul><p><br>Visualizes the group division on a scatter plot with predicted vs. actual values.</p> | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>top\_error</code> (default=0.05; the metrics are calculated for top-5% predictions with overestimation and underestimation).</li></ul>                                                                               |
| <p><strong>RegressionErrorBiasTable()</strong><br><br>Plots the relationship between feature values and model quality per group (for top-X% error groups, as above).</p>                                                                                                                                                                                           | <p><strong>Required:</strong><br>n/a<br><br><strong>Optional:</strong></p><ul><li><code>columns</code>(default = all categorical and numerical columns)</li><li><code>top\_error</code> (default=0.05; the metrics are calculated for top-5% predictions with overestimation and underestimation).</li></ul> |

## Ranking and Recommendations

All metrics are dataset-level. Check individual metric descriptions [here](/reference/all-metrics/ranking-metrics). All metrics require recommendations column mapping.

Optional shared parameters for multiple metrics:

* `no_feedback_users: bool = False`. Specifies whether to include the users who did not select any of the items, when computing the quality metric. Default: False.
* `min_rel_score: Optional[int] = None`. Specifies the minimum relevance score to consider relevant when calculating the quality metrics for non-binary targets (e.g., if a target is a rating or a custom score).

| Metric                                                                                                                                                                                                                                                                        | Parameters                                                                                                                                                                                                                                             |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <p><strong>RecallTopKMetric()</strong><br><br>Calculates the recall at <code>k</code>.</p>                                                                                                                                                                                    | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul>                                                              |
| <p><strong>PrecisionTopKMetric()</strong><br><br>Calculates the precision at <code>k</code>.</p>                                                                                                                                                                              | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul>                                                              |
| <p><strong>FBetaTopKMetric()</strong><br><br>Calculates the F-measure at <code>k</code>.</p>                                                                                                                                                                                  | <p><strong>Required</strong>:</p><ul><li><code>beta</code>(default = 1)</li><li><code>k</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul>                       |
| <p><strong>MAPKMetric()</strong><br><br>Calculates the Mean Average Precision (MAP) at <code>k</code>.</p>                                                                                                                                                                    | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul>                                                              |
| <p><strong>MARKMetric()</strong><br><br>Calculates the Mean Average Recall (MAR) at <code>k</code>.</p>                                                                                                                                                                       | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul>                                                              |
| <p><strong>NDCGKMetric()</strong><br><br>Calculates the Normalized Discounted Cumulative Gain at <code>k</code>.</p>                                                                                                                                                          | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional:</strong></p><ul><li><code>no\_feedback\_users</code></li><li><code>min\_rel\_score</code></li></ul>                                                              |
| <p><strong>MRRKMetric()</strong><br><br>Calculates the Mean Reciprocal Rank (MRR) at <code>k</code>.</p>                                                                                                                                                                      | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>min\_rel\_score</code></li><li><code>no\_feedback\_users</code></li></ul>                                                              |
| <p><strong>HitRateKMetric()</strong><br><br>Calculates the hit rate at <code>k</code>: a share of users for which at least one relevant item is included in the K.</p>                                                                                                        | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>min\_rel\_score</code></li><li><code>no\_feedback\_users</code></li></ul>                                                              |
| <p><strong>DiversityMetric()</strong><br><br>Calculates intra-list Diversity at <code>k</code>: diversity of recommendations shown to each user in top-K recommendations, averaged by all users.</p>                                                                          | <p><strong>Required</strong>:</p><ul><li><code>k</code></li><li><code>item\_features: List</code></li></ul><p><strong>Optional:</strong></p><ul><li>-</li></ul>                                                                                        |
| <p><strong>NoveltyMetric()</strong><br><br>Calculates novelty at <code>k</code>: novelty of recommendations shown to each user in top-K recommendations, averaged by all users.<br><br>Requires a training dataset.</p>                                                       | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li>-</li></ul>                                                                                                                                  |
| <p><strong>SerendipityMetric()</strong><br><br>Calculates serendipity at <code>k</code>: how unusual the relevant recommendations are in top-K, averaged by all users.<br><br>Requires a training dataset.</p>                                                                | <p><strong>Required</strong>:</p><ul><li><code>k</code></li><li><code>item\_features: List</code></li></ul><p><strong>Optional</strong>:</p><ul><li><code>min\_rel\_score</code></li></ul>                                                             |
| <p><strong>PersonalizationMetric()</strong><br><br>Measures the average uniqueness of each user's top-K recommendations.<br><br></p>                                                                                                                                          | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li>-</li></ul>                                                                                                                                  |
| <p><strong>PopularityBias()</strong><br><br>Evaluates the popularity bias in recommendations by computing ARP (average recommendation popularity), Gini index, and coverage.<br><br>Requires a training dataset.</p>                                                          | <p><strong>Required</strong>:</p><ul><li><code>K</code></li><li><code>normalize\_arp (default: False)</code> - whether to normalize ARP calculation by the most popular item in training</li></ul><p><strong>Optional</strong>:</p><ul><li>-</li></ul> |
| <p><strong>ItemBiasMetric()</strong><br><br>Visualizes the distribution of recommendations by a chosen dimension (column), сomparative to its distribution in the training set.<br><br>Requires a training dataset.</p>                                                       | <p><strong>Required</strong>:</p><ul><li><code>k</code></li><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:</p><ul><li>-</li></ul>                                                                                                |
| <p><strong>UserBiasMetric()</strong><br><br>Visualizes the distribution of the chosen category (e.g. user characteristic), comparative to its distribution in the training dataset.<br><br>Requires a training dataset.</p>                                                   | <p><strong>Required</strong>:</p><ul><li><code>k</code></li><li><code>column\_name</code></li></ul><p><strong>Optional</strong>:</p><ul><li>-</li></ul>                                                                                                |
| <p><strong>ScoreDistribution()</strong><br><br>Computes the predicted score entropy. Visualizes the distribution of the scores at <code>k</code> (and all scores, if available).<br><br>Applies only when the <code>recommendations\_type</code> is a <code>score</code>.</p> | <p><strong>Required</strong>:</p><ul><li><code>k</code></li></ul><p><strong>Optional</strong>:</p><ul><li>-</li></ul>                                                                                                                                  |
| <p><strong>RecCasesTable()</strong><br><br>Shows the list of recommendations for specific user IDs (or 5 random if not specified).</p>                                                                                                                                        | <p><strong>Required</strong>:</p><ul><li>-</li></ul><p><strong>Optional</strong>:</p><ul><li><code>display\_features: List</code></li><li><code>user\_ids: List</code></li><li><code>train\_item\_num: int</code></li></ul>                            |


# Ranking metrics

Open-source metrics for ranking and recommendations.

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

The following metrics can be used for ranking, retrieval and recommender systems.

## Recall

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-fce5066cc508d5006a132ed0358c077e0bd80b26%2Fmetric_recall-min.png?alt=media\&token=8cfe3b6e-cef3-4e70-bebf-5d0196c2b882)

**Evidently Metric**: `RecallTopKMetric`.

Recall at K reflects the ability of the recommender or ranking system to retrieve all relevant items within the top K results.

**Implemented method:**

* **Compute recall at K by user**. Compute the recall at K for each individual user (or query), by measuring the share of all relevant items in the dataset that appear in the top K results.

$$
\text{Recall at } K = \frac{\text{Number of relevant items in } K}{\text{Total number of relevant items}}
$$

* **Compute overall recall**. Average the results across all users (queries) in the dataset.

**Range**: 0 to 1.

**Interpretation**: a higher recall at K indicates that the model can retrieve a higher proportion of relevant items, which is generally desirable.

**Notes**: if the total number of relevant items is greater than K, it's impossible to recall all of them within the top K results (making 100% recall impossible).

## Precision

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-fe91166a149c323f5e04d39bf1d76a65da4f1bdb%2Fmetric_precision-min.png?alt=media\&token=f13cbf7b-144c-4226-8024-c831547870d0)

**Evidently Metric**: `PrecisionTopKMetric`.

Precision at K reflects the ability of the system to suggest items that are truly relevant to the users’ preferences or queries.

**Implemented method:**

* **Compute precision at K by user**. Compute the precision at K for each user (or query) by measuring the share of the relevant results within the top K.

$$
\text{Precision at } K = \frac{\text{Number of relevant items in } K}{\text{Total number of items in }K}
$$

* **Compute overall precision**. Average the results across all users (queries) in the dataset.

**Range**: 0 to 1.

**Interpretation**: a higher precision at K indicates that a larger proportion of the top results are relevant, which is generally desirable.

## F Beta

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-7ba0b64765c96d0f4753c1755f1044b2ba517694%2Fmetric_fbeta-min.png?alt=media\&token=bb17ed6b-0346-491f-8481-8c97de87686a)

**Evidently Metric**: `FBetaTopKMetric`.

The F Beta score at K combines precision and recall into a single value, providing a balanced measure of a recommendation or ranking system's performance.

$$
F\_{\beta} = \frac{(1 + \beta^2) \times \text{Precision at K} \times \text{Recall at K}}{(\beta^2 \times \text{Precision at K}) + \text{Recall at K}} \
$$

`Beta` is a parameter that determines the weight assigned to recall relative to precision. `Beta` > 1 gives more weight to recall, while `beta` < 1 favors precision.

If `Beta` = 1 (default), it is a traditional F1 score that provides a harmonic mean of precision and recall at K. It provides a balanced estimation, considering both false positives (items recommended that are not relevant) and false negatives (relevant items not recommended).

**Range**: 0 to 1.

**Interpretation**: Higher F Beta at K values indicate better overall performance.

## Mean average precision (MAP)

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-0a0f5880a402305a91ab1c271504aefd60db7954%2Fmetric_map-min.png?alt=media\&token=4a9e1d21-8a9d-4365-977b-6d6c2baadfcd)

**Evidently Metric**: `MAPKMetric`.

MAP (Mean Average Precision) at K assesses the ability of the recommender or retrieval system to suggest relevant items in the top-K results, while placing more relevant items at the top.

Compared to precision at K, MAP at K is rank-aware. It penalizes the system for placing relevant items lower in the list, even if the total number of relevant items at K is the same.

**Implemented method:**

* **Compute Average Precision (AP) at K by user**. The Average Precision at K is computed for each user (or query) as an average of precision values at each relevant item position within the top K. To do that, we sum up precision at all values of K when the item is relevant (e.g., Precision @1, Precision\@2..), and divide it by the total number of relevant items in K.

$$
\text{AP\@K} = \frac{1}{N} \sum\_{k=1}^{K} Precision(k) \times rel(k)
$$

Where *N* is the total number of relevant items at K, and *rel(k)* is equal to 1 if the item is relevant, and is 0 otherwise.

Example: if K = 10, and items in positions 1, 2, and 10 are relevant, the formula will look as:

$$
AP\@10 = \frac{Precision\@1+Precision\@2+Precision\@10}{3}
$$

* **Compute Mean Average Precision (MAP) at K**. Average the results across all users (or queries) in the dataset.

$$
\text{MAP\@K} = \frac{1}{U} \sum\_{u=1}^{U} \text{AP\@K}\_u
$$

Where *U* is the total number of users or queries in the dataset, and *AP* is the average precision for a given list.

**Range**: 0 to 1.

**Interpretation**: Higher MAP at K values indicates a better ability of the system to place relevant items high in the list.

## Mean average recall (MAR)

**Evidently Metric**: `MARKMetric`.

MAR (Mean Average Recall) at K assesses the ability of a recommendation system to retrieve all relevant items within the top-K results, averaged by all relevant positions.

**Implemented method:**

* **Compute the average recall at K by user**. Compute and average the recall at each relevant position within the top K for every user (or query). To do that, we sum up the recall at all values of K when the item is relevant (e.g. Recall @1, Recall\@2..), and divide it by the total number of relevant recommendations in K.

$$
\text{AR\@K} = \frac{1}{N} \sum\_{k=1}^{K} Recall(k) \times rel(k)
$$

Example: if K = 10, and items in positions 1, 2, and 10 are relevant, the formula will look as:

$$
\text{AR\@10} = \frac{Recall\@1+Recall\@2+Recall\@10}{3}
$$

* **Compute mean average recall at K**. Average the results across all users (or queries).

$$
\text{MAR\@K} = \frac{1}{U} \sum\_{u=1}^{U} \text{AR\@K}\_u
$$

Where *U* is the total number of users or queries in the dataset, and *AR* is the average recall for a given list.

**Range**: 0 to 1.

**Interpretation**: Higher MAR at K values indicates a better ability of the system to retrieve relevant items across all users or queries.

## Normalized Discounted Cumulative Gain (NDCG)

**Evidently Metric**: `NDCGKMetric`.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-d8f6c823bfb4802d4554792cd681f9eb39636a74%2Fmetric_ndcg-min.png?alt=media\&token=66dda0f4-e34b-471b-adbb-6aa599c79cdd)

NDCG (Normalized Discounted Cumulative Gain) at K reflects the ranking quality, comparing it to an ideal order where all relevant items for each user (or query) are placed at the top of the list.

**Implemented method**:

* **Provide the item relevance score**. You can assign a relevance score for each item in each top-K list for user or query. Depending on the model type, it can be a binary outcome (1 is relevant, 0 is not) or a score.
* **Compute the discounted cumulative gain (DCG)** at K by the user or query. DCG at K measures the quality of the ranking (= total relevance) for a list of top-K items. We add a logarithmic discount to account for diminishing returns from each following item being lower on the list. To get the resulting DCG, you can compute a weighted sum of the relevance scores for all items from the top of the list to K with an applied discount.

$$
\text{DCG\@K} = \sum\_{i=1}^{K} \frac{rel\_i}{\log\_2(i + 1)}
$$

Where *Rel(i)* is the relevance score of the item at rank *i*.

* **Compute the normalized DCG (NDCG)**. To normalize the metric, we divide the resulting DCG by the ideal DCG (IDCG) at K. Ideal DCG at K represents the maximum achievable DCG when the items are perfectly ranked in descending order of relevance.

$$
\text{NDCG\@K} = \frac{DCG\@K}{IDCG\@K}
$$

This way, it is possible to compare NDCG values across different use cases. The resulting NDCG values for all users or queries are averaged to measure the overall performance of a model.

**Range**: 0 to 1, where 1 indicates perfect ranking.

**Interpretation**: Higher NDCG at K indicates a better ability of the system to place more relevant items higher up in the ranking.

## Hit Rate

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2cef216a2ea59b5e1c5ae0ca94e3fc86cb998575%2Fmetric_hitrate-min.png?alt=media\&token=eaddd61f-17d0-4301-8dd6-8e325fa1f4cb)

**Evidently Metric**: `HitRateKMetric`.

Hit Rate at K calculates the share of users or queries for which at least one relevant item is included in the K.

**Implemented method**:

* **Compute “hit” for each user**. For each user or query, we evaluate if any of the top-K recommended items is relevant. It is a binary metric equal to 1 if any relevant item is included in K, or 0 otherwise.
* **Compute average hit rate**. The average of this metric is calculated across all users or queries.

**Range**: 0 to 1, where 1 indicates that each user / query gets at least one relevant recommendation / retrieval.

**Interpretation**: A higher Hit Rate indicates that a higher share of users / queries have relevant items in their lists.

**Note**: the Hit Rate will typically increase for higher values of K (since there is a higher chance that a relevant item will be recommended in a longer list).

## Mean Reciprocal Rank (MRR)

**Evidently Metric**: `MRRKMetric`

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2ba9ea0f5500b674c2f31fa430b460f7c4adf279%2Fmetric_mrr-min.png?alt=media\&token=43faafeb-d7c5-49d2-b9c7-c79e915be82a)

Mean Reciprocal Rank (MRR) measures the ranking quality considering the position of the first relevant item in the list.

**Implemented method:**

* For each user or query, identify the position of the **first relevant item** in the recommended list.
* Calculate the **reciprocal rank**, taking the reciprocal of the position of the first relevant item for each user or query (i.e., 1/position).\
  Example: if the first relevant item is at the top of the list - the reciprocal rank is 1, if it is on the 2nd position - the reciprocal rank ½, if on the 3rd - ⅓, etc.
* Calculate the **mean reciprocal rank** (MRR). Compute the average reciprocal rank across all users or queries.

$$
\text{MRR} = \frac{1}{U} \sum\_{u=1}^{U}\frac{1}{rank\_i}
$$

Where *U* is the total number of users or queries, and *rank(i)* is the rank of the first relevant item for user *u* in the top-K results.

**Range**: 0 to 1, where 1 indicates that the first recommended item for every user is relevant.

**Interpretation**: A higher MRR indicates that, on average, relevant items are positioned closer to the top of the recommended lists.

**Note**: Only a single top relevant item is considered in this metric, disregarding the position and relevance of other items in the list.

## Diversity

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-911ede695c5b599696a12d64abf9317b3c844154%2Fmetric_diversity-min.png?alt=media\&token=b43f6238-e0a5-48eb-bb9f-3002c5836f64)

**Evidently Metric**: `DiversityMetric`

**Recommendation diversity**: this metric measures the average intra-list diversity at K. It reflects the variety of items within the same user's recommendation list, averaged by all users.

**Implemented method**:

* **Measure the difference between recommended items**. Calculate the Cosine distance for each pair of recommendations inside the top-K in each user's list. The cosine distance serves as a measure of diversity between vectors representing recommended items, and is computed as:

$$
\text{Cosine distance} = 1 - \text{Cosine Similarity}
$$

Link: [Cosine Similarity on Wikipedia](https://en.wikipedia.org/wiki/Cosine_similarity).

* **Intra-list diversity**. Calculate intra-list diversity for each user by averaging the Cosine Distance between each pair of items in the user's top-K list.
* **Overall diversity**. Calculate the overall diversity by averaging the intra-list diversity across all users.

**Range**: The metric is based on Cosine distance, and can take values from 0 to 2.**0:** identical recommendations in top-K.**2:** very diverse recommendations in top-K.

**Interpretation**: the higher the value, the more varied items are shown to each user (e.g. inside a single recommendation block).

**Requirements**: You must pass the `item_features` list to point to numerical columns or embeddings that describe the recommended items. For example, these could be encoded genres that represent each movie. This makes it possible to compare the degree of similarity between different items.

**Notes**:

* This metric does not consider relevance. A recommender system showing varied but irrelevant items will have high diversity.
* This method performs many pairwise calculations between items and can take some time to compute.

## Novelty

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-e856c172f91d9f87980d0e743ea04cd6e458ef07%2Fmetric_novelty-min.png?alt=media\&token=9c294dbc-a61f-4d41-b4c8-cb148778640b)

**Evidently Metric**: `NoveltyMetric`

**Recommendation novelty**: this metric measures the average novelty of recommendations at K. It reflects how unusual top-K items are shown to each user, averaged by all users.

**Implemented method**:

* Measure **novelty of recommended items**. The novelty of an item can be defined based on its popularity in the training set.

$$
\text{novelty}\_i = -\log\_2(p\_i)
$$

where *p* represents the probability that item *i* is observed. It is calculated as the share of users that interacted with an item in the training set.

$$
\text{novelty}\_i = -\log\_2\left(\frac{\text{users who interacted with } i}{\text{number of users}}\right)
$$

High novelty corresponds to long-tail items that few users interacted with, and low novelty values correspond to popular items. If all users had interacted with an item, novelty is 0.

* Measure **novelty by user**. For each user, compute the average item novelty at K, by summing up the novelty of all items and dividing by K.
* **Overall novelty**. Average the novelty by user across all users.

**Range**: 0 to infinity.

**Interpretation**: if the value is higher, the items shown to users are more unusual. If the value is lower, the recommended items are well-known.

**Notes**:

* This metric does not consider relevance. A recommender system showing many irrelevant but unexpected (long tail) items will have high novelty.
* It is not possible to define the novelty of an item absent in the training set. The evaluation only considers items that are present in training.

Further reading: [Castells, P., Vargas, S., & Wang, J. (2011). Novelty and Diversity Metrics for Recommender Systems: Choice, Discovery and Relevance](https://repositorio.uam.es/bitstream/handle/10486/666094/novelty_castells_DDR_2011.pdf)

## Serendipity

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8da7fc058b56cb57d5d94431463c524a33ac809b%2Fmetric_serendipity-min.png?alt=media\&token=66b9f4fa-f582-4147-914e-e43895d0d41e)

**Evidently Metric**: `SerendipityMetric`

Recommendation serendipity: this metric measures how unusual the relevant recommendations are in K, averaged for all users.

Serendipity combines unexpectedness and relevance. It reflects the ability of a recommender system to show relevant items (that get a positive ranking or action) that are unexpected in the context of the user history (= are not similar to previous interactions). For example, a user who usually likes comedies gets recommended and upvotes a thriller.

**Implemented method**.

* Measure the **unexpectedness** of relevant recommendations. The “unexpectedness” is measured using Cosine distance. For every relevant recommendation in top-K, we compute the distance between this item and the previous user interactions in the training set. Higher cosine distance indicates higher unexpectedness.

$$
\text{serendipity}\_i = \text{unexpectedness}\_i\times\text{relevance}\_i
$$

Where *relevance(i)* is equal to 1 if the item is relevant, and is 0 otherwise.

* **Serendipity by user**. Calculate the average of the resulting distances for all relevant recommendations in the user list.
* **Overall serendipity**. Calculate the overall recommendation serendipity by averaging the results across all users.

$$
\text{Serendipity} = 1 - \sum\_{u \in S} \frac{1}{|S| |H\_u|} \sum\_{h \in H\_u} \sum\_{i \in R\_{u,k}} \frac{\text{CosSim}(i, h)}{k}
$$

Where

* *S* is the set of all users.
* *H(u)* is the item history of user *u*.
* *R(u)* Top-K function, where *R(u,k)* gives the top *k* recommended items for user *u*.

**Range**: The metric is based on Cosine distance, and can take values from 0 to 2.

* **0**: only popular, expected relevant recommendations.
* **2**: completely unexpected relevant recommendations.

**Interpretation**: the higher the value, the better the ability of the system to “positively surprise” the user.

**Requirements**: You must pass the `item_features` list to point to the numerical columns or embeddings that describe the recommended items. This allows comparing the degree of similarity between recommended items.

**Notes**:

* This metric is only computed for the users that are present in the training set. If there is no previous recommendation history, these users will be ignored.
* This metric only considers the unexpectedness of relevant items in top-K. Irrelevant recommendations, and their share, are not taken into account.

Further reading: [Zhang, Y., Séaghdha, D., Quercia, D., Jambor, T. (2011). Auralist: introducing serendipity into music recommendation.](http://www.cs.ucl.ac.uk/fileadmin/UCL-CS/research/Research_Notes/RN_11_21.pdf)

## Personalization

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-eb8f132d29885be20af3dc8e4d8923a340f3a279%2Fmetric_personalization-min.png?alt=media\&token=b674a110-1c14-40f1-8e59-01491baef6e7)

**Evidently Metric**: `PersonalizationMetric`

Personalization of recommendations: this metric measures the average uniqueness of each user's recommendations in top-K.

**Implemented method**:

* For every two users, compute the **overlap between top-K recommended items**. (The number of common items in top-K between two lists, divided by K).
* Calculate the **average overlap** across all pairs of users.
* Calculate personalization as:

$$
\text{Personalization} = 1 - \text{average overlap}
$$

The resulting metric reflects the average share of unique recommendations in each user’s list.

**Range**: 0 to 1.

* **0**: Identical recommendations for each user in top-K.
* **1**: Each user’s recommendations in top-K are unique.

**Interpretation**: the higher the value, the more personalized (= different from others) is each user’s list. The metric visualization also shows the top-10 most popular items.

## Popularity Bias

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-2f45f72abc613e41675bcb146bf935a5b7f8aafa%2Fmetric_popularity_bias-min.png?alt=media\&token=6a16323c-76ac-4310-9c2d-08f4d97d0562)

**Evidently Metric**: `PopularityBias`

The recommendation popularity bias is a tendency to favor a few popular items. This metric includes several measurements: ARP, Coverage and Gini index.

### 1. Average Recommendation Popularity (ARP)

ARP reflects the average popularity of the items recommended to the users.

**Implementation**.

* Compute the item popularity as the number of times each item was seen in training.
* Compute the average popularity for each user’s list as a sum of all items’ popularity divided by the number of recommended items.
* Compute the average popularity for all users by averaging the results across all users.

$$
ARP = \frac{1}{|U|} \sum\_{u \in U} \frac{1}{|L\_u|} \sum\_{i \in L\_u} \phi(i) \
$$

Where:

* *U* is the total number of users.
* *L(u)* is the list of items recommended for the user *u*.
* *ϕ(i)* is the number of times item *i* was rated in the training set (popularity of item *i*)

**Range**: 0 to infinity

**Interpretation**: the higher the value, the more popular on average the recommendations are in top-K.

**Note**: This metric is not normalized and depends on the number of recommendations in the training set.

Further reading: [Abdollahpouri, H., Mansoury, M., Burke, R., Mobasher, B., & Malthouse, E. (2021). User-centered Evaluation of Popularity Bias in Recommender Systems](https://dl.acm.org/doi/fullHtml/10.1145/3450613.3456821)

### 2. Coverage

Coverage reflects the item coverage as a proportion of items that has been recommended by the system.

**Implementation**: compute the share of items recommended to the users out of the total number of potential items (as seen in the training dataset).

$$
\text{Coverage} = \frac{\text{Number of unique items recommended} K}{\text{Total number of unique items}}
$$

**Range**: 0 to 1, where 1 means that 100% of items have been recommended to users.

**Interpretation**: the higher the value (usually preferable), the larger the share of items represented in the recommendations. Popularity-based recommenders that only recommend a limited number of popular items will have low coverage.

### 3. Gini index

Gini index: reflects the inequality in the distribution of recommended items shown to different users, as compared to a perfectly equal distribution.

**Implementation**:

$$
Gini(L) = 1 - \frac{1}{|I| - 1} \sum\_{k=1}^{|I|} (2k - |I| - 1) p(i\_k | L) \
$$

Where

* *L* is the combined list of all recommendation lists given to different users (note that an item may appear multiple times in L, if recommended for more than one user).
* *p(i|L)* is the ratio of occurrence of item *i* in *L*.
* *I* is the set of all items in the catalog.

**Range**: 0 to 1, where 0 represents the perfect equality (recommended items are evenly distributed among users), and 1 is complete inequality (the recommendations are concentrated on a single item).

**Interpretation**: the lower the value (usually preferable), the more equal the item distribution in recommendations. If the value is high, a few items are frequently recommended to many users while others are ignored.

Further reading: [Abdollahpouri, H., Mansoury, M., Burke, R., Mobasher, B., & Malthouse, E. (2021). User-centered Evaluation of Popularity Bias in Recommender Systems](https://dl.acm.org/doi/fullHtml/10.1145/3450613.3456821)

## Recommendation table

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-674def86bb99609db946aec9c0617bf6973e50ce%2Fmetric_recsys_table-min.png?alt=media\&token=1502cb77-0285-4ea3-929c-ad720fe2d725)

**Evidently Metric**: `RecCasesTable`

This visual Metric shows the list of recommendations for the specified user IDs (`user_ids: List`). If you do not pass the list of IDs, Evidently will choose 5 random ones.

You can optionally specify:

* The number of training items in the user history to display (`train_item_num`)
* Whether you’d like to additionally include the values of specific features in the table (`display_features: List`)
* The number of items recommended for user to display (`item_num`)

## Score Distribution (Entropy)

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-35b04fe1d78bc63ccea133294415389a0f51192a%2Fmetric_score_distribution-min.png?alt=media\&token=38958b63-810c-45ec-9552-676115236060)

**Evidently Metric**: `ScoreDistribution`

This metric computes the predicted score entropy. It applies only when the `recommendations_type` is a score.

**Implementation**:

* Apply softmax transformation for top-K scores for all users.
* Compute the KL divergence (relative entropy in [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html)).

The visualization shows the distribution of the predicted scores at K (and all scores, if available).

## Item Bias

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-7aaf3120fb54d9c881245e3d95b7ce2e53fdf6ec%2Fmetric_itembias_2-min.png?alt=media\&token=7ceafcc4-0ea5-406a-96b4-4a8edced0d06)

**Evidently Metric**: `ItemBiasMetric`

This visual metric shows the distribution of recommendations by a specified category or numerical value (`column_name`) compared to its distribution in the training set.

This helps compare the model recommendations against what could have been a random classifier that follows the observed distribution of items by a chosen characteristic.

The visualization shows:

* The distribution of items in the training set for the defined `column_name` (with duplicates dropped). This represents the item catalog by this dimension.
* The distribution of the recommended items for the defined `column_name` in the current and reference (if available) datasets.

This visualization helps see the patterns in the model recommendations. In a simplified example, you might observe that the training data contains 3x comedies compared to dramas, but the model recommends dramas 10x more often.

## UserBiasMetric

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-8165b2e4d33e26e57131ca9fe82031b6e5c67ae1%2Fmetric_userbias_1-min.png?alt=media\&token=5f60cb76-b366-437f-9103-f12f07f3b63d)

This visual metric shows the distribution of a specified category or numerical value (`column_name`) compared to its distribution in the training set.

This helps evaluate the distribution of users by a specific category (e.g., age, gender, etc.) in production data compared to training. Before visualization, we drop duplicates in training and current (reference) data.

For example, exploring this metric might help notice that a particular user group is underrepresented in training but is prevalent in production, or vice versa.


# Data drift algorithm

{% hint style="info" %}
**You are looking at the old Evidently documentation**: this API is available with versions 0.6.7 or lower. Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

In some tests and metrics, Evidently uses the default Data Drift Detection algorithm. It helps detect the distribution drift in the individual features, prediction, or target. This page describes how the **default** algorithm works.

## How it works

Evidently compares the distributions of the values in a given column (or columns) of the two datasets. You should pass these datasets as **reference** and **current**. Evidently applies several statistical tests and drift detection methods to detect if the distribution has changed significantly. It returns a "drift detected" or "not detected" result.

There is a default logic to choosing the appropriate drift test for each column. It is based on:

* column type: categorical, numerical, text data or embeddings
* the number of observations in the reference dataset
* the number of unique values in the column (n\_unique)

### Tabular Data

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a75fbd93777e730afb102527718d5b142f6dbfdb%2Fmetric_data_drift_table_2-min.png?alt=media\&token=c60298c6-c2c8-4e28-915a-784607423105)

For **small data with <= 1000 observations** in the reference dataset:

* For numerical columns (n\_unique > 5): [two-sample Kolmogorov-Smirnov test](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test).
* For categorical columns or numerical columns with n\_unique <= 5: [chi-squared test](https://en.wikipedia.org/wiki/Chi-squared_test).
* For binary categorical features (n\_unique <= 2): proportion difference test for independent samples based on Z-score.

All tests use a 0.95 confidence level by default.

For **larger data with > 1000 observations** in the reference dataset:

* For numerical columns (n\_unique > 5):[Wasserstein Distance](https://en.wikipedia.org/wiki/Wasserstein_metric).
* For categorical columns or numerical with n\_unique <= 5):[Jensen--Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence).

All metrics use a threshold = 0.1 by default.

{% hint style="info" %}
**You can always modify this drift detection logic**. You can select any of the statistical tests available in the library (including PSI, K-L divergence, Jensen-Shannon distance, Wasserstein distance, etc.), specify custom thresholds, or pass a custom test. You can read more about using [data drift parameters and available drift detection methods](/user-guide/customization/options-for-statistical-tests).
{% endhint %}

### Text Data

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a57bd7d2e85795086e684893ae26822b75259f1f%2Fmetric_column_drift_text-min.png?alt=media\&token=6f2b72b9-1ba7-433a-b5cc-b1fffdb25b6f)

Text content drift using a **domain classifier**. Evidently trains a binary classification model to discriminate between data from reference and current distributions.

The default for **small data with <= 1000 observations** detects drift if the ROC AUC of the drift detection classifier > possible ROC AUC of the random classifier at a 95th percentile.

The default for **larger data with > 1000 observations** detects drift if the ROC AUC > 0.55.

<details>

<summary>Text content drift detection method</summary>

For small data, the drift score is the ROC-AUC score of the domain classifier computed on a validation dataset. The ROC AUC of the created classifier is compared to the ROC AUC of the random classifier at a set percentile. To ensure the result is statistically meaningful, we repeat the calculation 1000 times with randomly assigned target class probabilities. This produces a distribution with a mean of 0,5. We then take the 95th percentile (default) of this distribution and compare it to the ROC-AUC score of the classifier. If the classifier score is higher, we consider the data drift to be detected. You can also set a different percentile as a parameter.

For large data, the ROC AUC of the obtained classifier is directly compared against the set ROC AUC threshold.

</details>

{% hint style="info" %}
**You can set different thresholds**. You can specify a custom threshold as a [parameter](/user-guide/customization/options-for-statistical-tests).
{% endhint %}

### Text Descriptors Drift

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-a33af2565bb54fe56af397233ce3caead61407a7%2Fmetric_text_descriptors_drift-min.png?alt=media\&token=7cc1f100-1f79-4257-a341-b913f6f43958)

**You can also check for drift in Text Descriptors**. There is an additional method that detects drift in Text Descriptors (such as text length, share of OOV words). This test is available as part of [Text Overview Preset](/presets/text-overview). You can also include it as a **TextDescriptorsDriftMetric()** in a custom Report, or in a Test Suite accordingly.

The descriptors are treated as tabular features. The default drift detection methods for tabular features apply.

### Embeddings

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-c0edce15d405a8aba7653bc4f592e35ccb9d05a4%2Fmetric_embeddings_drift.png?alt=media\&token=f19c1700-7c37-4d54-97d5-3e12bfae14ce)

Embedding drift using a **classifier**. Evidently trains a binary classification model to discriminate between data from reference and current distributions.

The default for **small data with <= 1000 observations** detects drift if the ROC AUC of the drift detection classifier > possible ROC AUC of the random classifier at a 95th percentile.

The default for **larger data with > 1000 observations** detects drift if the ROC AUC > 0.55.

{% hint style="info" %}
**You can choose other embedding drift detection methods**. You can specify custom thresholds and parameters such as dimensionality reduction and choose from other methods, including Euclidean distance, Cosine Similarity, Maximum Mean Discrepancy, and share of drifted embeddings. You must specify this as a [parameter](/user-guide/customization/embeddings-drift-parameters).
{% endhint %}

### Dataset-level drift

With Presets like `DatasetDriftPreset()`, Metrics like `DatasetDriftMetric()` or Tests like `TestShareOfDriftedColumns()` you can also set a rule on top of the individual feature drift results to detect dataset-level drift.

For example, you can declare dataset drift if 50% of all features (columns) drifted or if ⅓ of the most important features drifted.**The default in `DatasetDriftPreset()` is 0.5**.

Note that by default this includes all columns in the dataset. Suppose your dataset contains the prediction column, and you want to separate it from input drift detection. In that case, you should pre-process your dataset to exclude it or specify a list of columns you want to test for drift, and pass the list as a parameter.

![](https://256125905-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FeE67gM4508ESQxkbpOxj%2Fuploads%2Fgit-blob-41d198e2706e14e1a06b53f136be2bd03b5f1c70%2Freport_example_data_drift-min.png?alt=media\&token=95ecebf4-630c-4140-be6d-cff0431a96dd)

{% hint style="info" %}
**You can set different thresholds**. You can specify a custom threshold as a [parameter](/user-guide/customization/options-for-statistical-tests).
{% endhint %}

## Input data requirements

#### Empty columns

To evaluate data or prediction drift in the dataset, you need to ensure that the columns you test for drift are not empty. If these columns are empty in either reference or current data, Evidently will not calculate distribution drift and will raise an error.

#### Empty values

If some columns contain empty or infinite values (+-np.inf), these values will be filtered out when calculating distribution drift in the corresponding column.

By default, drift tests do not react to changes or increases in the number of empty values. Since the high number of nulls can be an important indicator, we recommend grouping the data drift tests (that check for distribution shift) with data integrity tests (that check for a share of nulls). You can choose from several null-related [tests](/reference/all-tests#data-integrity) and metrics and set a threshold.

## Resources

To build up a better intuition for which tests are better in different kinds of use cases, you can read our in-depth blogs with experimental code:

* [Which test is the best? We compared 5 methods to detect data drift on large datasets](https://evidentlyai.com/blog/data-drift-detection-large-datasets).
* [Shift happens: how to detect drift in ML embeddings](https://www.evidentlyai.com/blog/embedding-drift-detection).

Additional links:

* [How to interpret data and prediction drift together?](https://evidentlyai.com/blog/data-and-prediction-drift)
* [Do I need to monitor data drift if I can measure the ML model quality?](https://evidentlyai.com/blog/ml-monitoring-do-i-need-data-drift)
* ["My data drifted. What's next?" How to handle ML model drift in production.](https://evidentlyai.com/blog/ml-monitoring-data-drift-how-to-handle)
* [What is the difference between outlier detection and data drift detection?](https://evidentlyai.com/blog/ml-monitoring-drift-detection-vs-outlier-detection)


# API Reference


# evidently.calculations

## Subpackages

* [evidently.calculations.stattests package](/reference/api-reference/evidently.calculations/evidently.calculations.stattests)
  * [Submodules](/reference/api-reference/evidently.calculations/evidently.calculations.stattests#submodules)
  * [anderson\_darling\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.anderson_darling_stattest)
  * [chisquare\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.chisquare_stattest)
  * [cramer\_von\_mises\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.cramer_von_mises_stattest)
    * [`CramerVonMisesResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.cramer_von_mises_stattest.CramerVonMisesResult)
  * [energy\_distance module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.energy_distance)
  * [epps\_singleton\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.epps_singleton_stattest)
  * [fisher\_exact\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.fisher_exact_stattest)
  * [g\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.g_stattest)
  * [hellinger\_distance module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.hellinger_distance)
  * [jensenshannon module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.jensenshannon)
  * [kl\_div module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.kl_div)
  * [ks\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.ks_stattest)
  * [mann\_whitney\_urank\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.mann_whitney_urank_stattest)
  * [psi module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.psi)
  * [registry module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.registry)
    * [`StatTest`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)
      * [`StatTest.allowed_feature_types`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest.allowed_feature_types)
      * [`StatTest.default_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest.default_threshold)
      * [`StatTest.display_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest.display_name)
      * [`StatTest.func`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest.func)
      * [`StatTest.name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest.name)
    * [`StatTestInvalidFeatureTypeError`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTestInvalidFeatureTypeError)
    * [`StatTestNotFoundError`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTestNotFoundError)
    * [`StatTestResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTestResult)
      * [`StatTestResult.actual_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTestResult.actual_threshold)
      * [`StatTestResult.drift_score`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTestResult.drift_score)
      * [`StatTestResult.drifted`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTestResult.drifted)
    * [`get_stattest()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.get_stattest)
    * [`register_stattest()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.register_stattest)
  * [t\_test module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.t_test)
  * [tvd\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.tvd_stattest)
  * [utils module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.utils)
    * [`generate_fisher2x2_contingency_table()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.utils.generate_fisher2x2_contingency_table)
    * [`get_binned_data()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.utils.get_binned_data)
    * [`get_unique_not_nan_values_list_from_series()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.utils.get_unique_not_nan_values_list_from_series)
    * [`permutation_test()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.utils.permutation_test)
  * [wasserstein\_distance\_norm module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.wasserstein_distance_norm)
  * [z\_stattest module](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#module-evidently.calculations.stattests.z_stattest)
    * [`proportions_diff_z_stat_ind()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.z_stattest.proportions_diff_z_stat_ind)
    * [`proportions_diff_z_test()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.z_stattest.proportions_diff_z_test)

## Submodules

## classification\_performance module <a href="#module-evidently.calculations.classification_performance" id="module-evidently.calculations.classification_performance"></a>

### class ConfusionMatrix(labels: Sequence\[Union\[str, int]], values: list)

Bases: `object`

#### Attributes:

&#x20;    **labels : Sequence\[Union\[str, int]]**

&#x20;    **values : list**

### class DatasetClassificationQuality(accuracy: float, precision: float, recall: float, f1: float, roc\_auc: Optional\[float] = None, log\_loss: Optional\[float] = None, tpr: Optional\[float] = None, tnr: Optional\[float] = None, fpr: Optional\[float] = None, fnr: Optional\[float] = None, rate\_plots\_data: Optional\[Dict] = None, plot\_data: Optional\[Dict] = None)

Bases: `object`

#### Attributes:

&#x20;    **accuracy : float**

&#x20;    **f1 : float**

&#x20;    **fnr : Optional\[float] = None**

&#x20;    **fpr : Optional\[float] = None**

&#x20;    **log\_loss : Optional\[float] = None**

&#x20;    **plot\_data : Optional\[Dict] = None**

&#x20;    **precision : float**

&#x20;    **rate\_plots\_data : Optional\[Dict] = None**

&#x20;    **recall : float**

&#x20;    **roc\_auc : Optional\[float] = None**

&#x20;    **tnr : Optional\[float] = None**

&#x20;    **tpr : Optional\[float] = None**

### class PredictionData(predictions: pandas.core.series.Series, prediction\_probas: Optional\[pandas.core.frame.DataFrame], labels: List\[Union\[str, int]])

Bases: `object`

#### Attributes:

&#x20;    **labels : List\[Union\[str, int]]**

&#x20;    **prediction\_probas : Optional\[DataFrame]**

&#x20;    **predictions : Series**

### calculate\_confusion\_by\_classes(confusion\_matrix: ndarray, class\_names: Sequence\[Union\[str, int]])

Calculate metrics:

* TP (true positive)
* TN (true negative)
* FP (false positive)
* FN (false negative)\
  for each class from confusion matrix.
* **Returns**

  a dict like:

  ```default
  {
      "class_1_name": {
          "tp": 1,
          "tn": 5,
          "fp": 0,
          "fn": 3,
      },
      "class_1_name": {
          "tp": 1,
          "tn": 5,
          "fp": 0,
          "fn": 3,
      },
  }
  ```

### calculate\_matrix(target: Series, prediction: Series, labels: List\[Union\[str, int]])

### calculate\_metrics(column\_mapping: [ColumnMapping](https://docs-old.evidentlyai.com/reference/api-reference/pages/D9Oc6VIiKDafnPidUy9b#evidently.pipeline.column_mapping.ColumnMapping), confusion\_matrix: ConfusionMatrix, target: Series, prediction: PredictionData)

### calculate\_pr\_table(binded)

### collect\_plot\_data(prediction\_probas: DataFrame)

### get\_prediction\_data(data: DataFrame, data\_columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), pos\_label: Optional\[Union\[str, int]], threshold: float = 0.5)

Get predicted values and optional prediction probabilities from source data.\
Also take into account a threshold value - if a probability is less than the value, do not take it into account.

Return and object with predicted values and an optional prediction probabilities.

### k\_probability\_threshold(prediction\_probas: DataFrame, k: Union\[int, float])

### threshold\_probability\_labels(prediction\_probas: DataFrame, pos\_label: Union\[str, int], neg\_label: Union\[str, int], threshold: float)

Get prediction values by probabilities with the threshold apply

## data\_drift module <a href="#module-evidently.calculations.data_drift" id="module-evidently.calculations.data_drift"></a>

Methods and types for data drift calculations.

### class ColumnDataDriftMetrics(column\_name: str, column\_type: str, stattest\_name: str, drift\_score: float, drift\_detected: bool, threshold: float, current\_distribution: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), reference\_distribution: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), current\_small\_distribution: Optional\[list] = None, reference\_small\_distribution: Optional\[list] = None, current\_scatter: Optional\[Dict\[str, list]] = None, x\_name: Optional\[str] = None, plot\_shape: Optional\[Dict\[str, float]] = None, current\_correlations: Optional\[Dict\[str, float]] = None, reference\_correlations: Optional\[Dict\[str, float]] = None)

Bases: `object`

One column drift metrics.

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **column\_type : str**

&#x20;    **current\_correlations : Optional\[Dict\[str, float]] = None**

&#x20;    **current\_distribution :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **current\_scatter : Optional\[Dict\[str, list]] = None**

&#x20;    **current\_small\_distribution : Optional\[list] = None**

&#x20;    **drift\_detected : bool**

&#x20;    **drift\_score : float**

&#x20;    **plot\_shape : Optional\[Dict\[str, float]] = None**

&#x20;    **reference\_correlations : Optional\[Dict\[str, float]] = None**

&#x20;    **reference\_distribution :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **reference\_small\_distribution : Optional\[list] = None**

&#x20;    **stattest\_name : str**

&#x20;    **threshold : float**

&#x20;    **x\_name : Optional\[str] = None**

### class DatasetDrift(number\_of\_drifted\_columns: int, dataset\_drift\_score: float, dataset\_drift: bool)

Bases: `object`

Dataset drift calculation results

#### Attributes:

&#x20;    **dataset\_drift : bool**

&#x20;    **dataset\_drift\_score : float**

&#x20;    **number\_of\_drifted\_columns : int**

### class DatasetDriftMetrics(number\_of\_columns: int, number\_of\_drifted\_columns: int, share\_of\_drifted\_columns: float, dataset\_drift: bool, drift\_by\_columns: Dict\[str, ColumnDataDriftMetrics], options: [DataDriftOptions](https://docs-old.evidentlyai.com/reference/api-reference/pages/ZW3IQh73672cpd3xySyJ#evidently.options.data_drift.DataDriftOptions), dataset\_columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns))

Bases: `object`

#### Attributes:

&#x20;    **dataset\_columns :** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)

&#x20;    **dataset\_drift : bool**

&#x20;    **drift\_by\_columns : Dict\[str, ColumnDataDriftMetrics]**

&#x20;    **number\_of\_columns : int**

&#x20;    **number\_of\_drifted\_columns : int**

&#x20;    **options :** [**DataDriftOptions**](https://docs-old.evidentlyai.com/reference/api-reference/pages/ZW3IQh73672cpd3xySyJ#evidently.options.data_drift.DataDriftOptions)

&#x20;    **share\_of\_drifted\_columns : float**

### ensure\_prediction\_column\_is\_string(\*, prediction\_column: Optional\[Union\[str, Sequence]], current\_data: DataFrame, reference\_data: DataFrame, threshold: float = 0.5)

Update dataset by predictions type:

* if prediction column is None or a string, no dataset changes
* (binary classification) if predictions is a list and its length equals 2

  set predicted\_labels column by threshold
* (multi label classification) if predictions is a list and its length is greater than 2

  set predicted\_labels from probability values in columns by prediction column
* **Returns**

  prediction column name.

### get\_dataset\_drift(drift\_metrics, drift\_share=0.5)

### get\_drift\_for\_columns(\*, current\_data: DataFrame, reference\_data: DataFrame, dataset\_columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), data\_drift\_options: [DataDriftOptions](https://docs-old.evidentlyai.com/reference/api-reference/pages/ZW3IQh73672cpd3xySyJ#evidently.options.data_drift.DataDriftOptions), drift\_share\_threshold: Optional\[float] = None, columns: Optional\[List\[str]] = None)

### get\_one\_column\_drift(\*, current\_data: DataFrame, reference\_data: DataFrame, column\_name: str, options: [DataDriftOptions](https://docs-old.evidentlyai.com/reference/api-reference/pages/ZW3IQh73672cpd3xySyJ#evidently.options.data_drift.DataDriftOptions), dataset\_columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), column\_type: Optional\[str] = None)

## data\_integration module <a href="#module-evidently.calculations.data_integration" id="module-evidently.calculations.data_integration"></a>

### get\_number\_of\_all\_pandas\_missed\_values(dataset: DataFrame)

Calculate the number of missed - nulls by pandas - values in a dataset

### get\_number\_of\_almost\_constant\_columns(dataset: DataFrame, threshold: float)

Calculate the number of almost constant columns in a dataset

### get\_number\_of\_almost\_duplicated\_columns(dataset: DataFrame, threshold: float)

Calculate the number of almost duplicated columns in a dataset

### get\_number\_of\_constant\_columns(dataset: DataFrame)

Calculate the number of constant columns in a dataset

### get\_number\_of\_duplicated\_columns(dataset: DataFrame)

Calculate the number of duplicated columns in a dataset

### get\_number\_of\_empty\_columns(dataset: DataFrame)

Calculate the number of empty columns in a dataset

## data\_quality module <a href="#module-evidently.calculations.data_quality" id="module-evidently.calculations.data_quality"></a>

Methods for overall dataset quality calculations - rows count, a specific values count, etc.

### class ColumnCorrelations(column\_name: str, kind: str, values: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution))

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **kind : str**

&#x20;    **values :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

### class DataQualityGetPlotData()

Bases: `object`

#### Methods:

&#x20;    **calculate\_data\_by\_target(curr: DataFrame, ref: Optional\[DataFrame], feature\_name: str, feature\_type: str, target\_name: str, target\_type: str, merge\_small\_cat: Optional\[int] = 5)**

&#x20;    **calculate\_data\_in\_time(curr: DataFrame, ref: Optional\[DataFrame], feature\_name: str, feature\_type: str, datetime\_name: str, merge\_small\_cat: Optional\[int] = 5)**

&#x20;    **calculate\_main\_plot(curr: DataFrame, ref: Optional\[DataFrame], feature\_name: str, feature\_type: str, merge\_small\_cat: Optional\[int] = 5)**

### class DataQualityPlot(bins\_for\_hist: Dict\[str, pandas.core.frame.DataFrame])

Bases: `object`

#### Attributes:

&#x20;    **bins\_for\_hist : Dict\[str, DataFrame]**

### class DataQualityStats(rows\_count: int, num\_features\_stats: Optional\[Dict\[str, FeatureQualityStats]] = None, cat\_features\_stats: Optional\[Dict\[str, FeatureQualityStats]] = None, datetime\_features\_stats: Optional\[Dict\[str, FeatureQualityStats]] = None, target\_stats: Optional\[Dict\[str, FeatureQualityStats]] = None, prediction\_stats: Optional\[Dict\[str, FeatureQualityStats]] = None)

Bases: `object`

#### Attributes:

&#x20;    **cat\_features\_stats : Optional\[Dict\[str, FeatureQualityStats]] = None**

&#x20;    **datetime\_features\_stats : Optional\[Dict\[str, FeatureQualityStats]] = None**

&#x20;    **num\_features\_stats : Optional\[Dict\[str, FeatureQualityStats]] = None**

&#x20;    **prediction\_stats : Optional\[Dict\[str, FeatureQualityStats]] = None**

&#x20;    **rows\_count : int**

&#x20;    **target\_stats : Optional\[Dict\[str, FeatureQualityStats]] = None**

#### Methods:

&#x20;    **get\_all\_features()**

### class FeatureQualityStats(feature\_type: str, number\_of\_rows: int = 0, count: int = 0, infinite\_count: Optional\[int] = None, infinite\_percentage: Optional\[float] = None, missing\_count: Optional\[int] = None, missing\_percentage: Optional\[float] = None, unique\_count: Optional\[int] = None, unique\_percentage: Optional\[float] = None, percentile\_25: Optional\[float] = None, percentile\_50: Optional\[float] = None, percentile\_75: Optional\[float] = None, max: Optional\[Union\[int, float, bool, str]] = None, min: Optional\[Union\[int, float, bool, str]] = None, mean: Optional\[float] = None, most\_common\_value: Optional\[Union\[int, float, bool, str]] = None, most\_common\_value\_percentage: Optional\[float] = None, std: Optional\[float] = None, most\_common\_not\_null\_value: Optional\[Union\[int, float, bool, str]] = None, most\_common\_not\_null\_value\_percentage: Optional\[float] = None, new\_in\_current\_values\_count: Optional\[int] = None, unused\_in\_current\_values\_count: Optional\[int] = None)

Bases: `object`

Class for all features data quality metrics store.

A type of the feature is stored in feature\_type field.\
Concrete stat kit depends on the feature type. Is a metric is not applicable - leave None value for it.

Metrics for all feature types:

```
- feature type - cat for category, num for numeric, datetime for datetime features

- count - quantity of a meaningful values (do not take into account NaN values)

- missing_count - quantity of meaningless (NaN) values

- missing_percentage - the percentage of the missed values

- unique_count - quantity of unique values

- unique_percentage - the percentage of the unique values

- max - maximum value (not applicable for category features)

- min - minimum value (not applicable for category features)

- most_common_value - the most common value in the feature values

- most_common_value_percentage - the percentage of the most common value

- most_common_not_null_value - if most_common_value equals NaN - the next most common value. Otherwise - None

- most_common_not_null_value_percentage - the percentage of most_common_not_null_value if it is defined.

    If most_common_not_null_value is not defined, equals None too.
```

Metrics for numeric features only:

```
- infinite_count - quantity infinite values (for numeric features only)

- infinite_percentage - the percentage of infinite values (for numeric features only)

- percentile_25 - 25% percentile for meaningful values

- percentile_50 - 50% percentile for meaningful values

- percentile_75 - 75% percentile for meaningful values

- mean - the sum of the meaningful values divided by the number of the meaningful values

- std - standard deviation of the values
```

Metrics for category features only:

* new\_in\_current\_values\_count - quantity of new values in the current dataset after the reference

  ```
    Defined for reference dataset only.
  ```

  * new\_in\_current\_values\_count - quantity of values in the reference dataset that not presented in the current

    Defined for reference dataset only.

#### Attributes:

&#x20;    **count : int = 0**

&#x20;    **feature\_type : str**

&#x20;    **infinite\_count : Optional\[int] = None**

&#x20;    **infinite\_percentage : Optional\[float] = None**

&#x20;    **max : Optional\[Union\[int, float, bool, str]] = None**

&#x20;    **mean : Optional\[float] = None**

&#x20;    **min : Optional\[Union\[int, float, bool, str]] = None**

&#x20;    **missing\_count : Optional\[int] = None**

&#x20;    **missing\_percentage : Optional\[float] = None**

&#x20;    **most\_common\_not\_null\_value : Optional\[Union\[int, float, bool, str]] = None**

&#x20;    **most\_common\_not\_null\_value\_percentage : Optional\[float] = None**

&#x20;    **most\_common\_value : Optional\[Union\[int, float, bool, str]] = None**

&#x20;    **most\_common\_value\_percentage : Optional\[float] = None**

&#x20;    **new\_in\_current\_values\_count : Optional\[int] = None**

&#x20;    **number\_of\_rows : int = 0**

&#x20;    **percentile\_25 : Optional\[float] = None**

&#x20;    **percentile\_50 : Optional\[float] = None**

&#x20;    **percentile\_75 : Optional\[float] = None**

&#x20;    **std : Optional\[float] = None**

&#x20;    **unique\_count : Optional\[int] = None**

&#x20;    **unique\_percentage : Optional\[float] = None**

&#x20;    **unused\_in\_current\_values\_count : Optional\[int] = None**

#### Methods:

&#x20;    **as\_dict()**

&#x20;    **is\_category()**

Checks that the object store stats for a category feature

&#x20;    **is\_datetime()**

Checks that the object store stats for a datetime feature

&#x20;    **is\_numeric()**

Checks that the object store stats for a numeric feature

### calculate\_category\_column\_correlations(column\_name: str, dataset: DataFrame, columns: List\[str])

For category columns calculate cramer\_v correlation

### calculate\_column\_distribution(column: Series, column\_type: str)

### calculate\_correlations(dataset: DataFrame, columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns))

### calculate\_cramer\_v\_correlation(column\_name: str, dataset: DataFrame, columns: List\[str])

### calculate\_data\_quality\_stats(dataset: DataFrame, columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), task: Optional\[str])

### calculate\_numerical\_column\_correlations(column\_name: str, dataset: DataFrame, columns: List\[str])

### get\_features\_stats(feature: Series, feature\_type: str)

### get\_pairwise\_correlation(df, func: Callable\[\[Series, Series], float])

Compute pairwise correlation of columns\
:param df: initial data frame.\
:param func: function for computing pairwise correlation.

* **Returns**

  Correlation matrix.

### get\_rows\_count(data: Union\[DataFrame, Series])

Count quantity of rows in a dataset

## regression\_performance module <a href="#module-evidently.calculations.regression_performance" id="module-evidently.calculations.regression_performance"></a>

### class ErrorWithQuantiles(error, quantile\_top, quantile\_other)

Bases: `object`

### class FeatureBias(feature\_type: str, majority: float, under: float, over: float, range: float)

Bases: `object`

#### Attributes:

&#x20;    **feature\_type : str**

&#x20;    **majority : float**

&#x20;    **over : float**

&#x20;    **range : float**

&#x20;    **under : float**

#### Methods:

&#x20;    **as\_dict(prefix)**

### class RegressionPerformanceMetrics(mean\_error: float, mean\_abs\_error: float, mean\_abs\_perc\_error: float, error\_std: float, abs\_error\_max: float, abs\_error\_std: float, abs\_perc\_error\_std: float, error\_normality: dict, underperformance: dict, error\_bias: dict)

Bases: `object`

#### Attributes:

&#x20;    **abs\_error\_max : float**

&#x20;    **abs\_error\_std : float**

&#x20;    **abs\_perc\_error\_std : float**

&#x20;    **error\_bias : dict**

&#x20;    **error\_normality : dict**

&#x20;    **error\_std : float**

&#x20;    **mean\_abs\_error : float**

&#x20;    **mean\_abs\_perc\_error : float**

&#x20;    **mean\_error : float**

&#x20;    **underperformance : dict**

### calculate\_regression\_performance(dataset: DataFrame, columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), error\_bias\_prefix: str)

### error\_bias\_table(dataset, err\_quantiles, num\_feature\_names, cat\_feature\_names)

### error\_with\_quantiles(dataset, prediction\_column, target\_column, quantile: float)


# evidently.calculations.stattests

Available statistical tests.\
For detailed information about statistical tests see module documentation.

## Submodules

## anderson\_darling\_stattest module <a href="#module-evidently.calculations.stattests.anderson_darling_stattest" id="module-evidently.calculations.stattests.anderson_darling_stattest"></a>

Anderson-Darling test of two samples.

Name: “anderson”

Import:

```python
>>> from evidently.calculations.stattests import anderson_darling_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import anderson_darling_test
>>> options = DataDriftOptions(all_features_stattest=anderson_darling_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="anderson")
```

## chisquare\_stattest module <a href="#module-evidently.calculations.stattests.chisquare_stattest" id="module-evidently.calculations.stattests.chisquare_stattest"></a>

Chisquare test of two samples.

Name: “chisquare”

Import:

```python
>>> from evidently.calculations.stattests import chi_stat_test
```

Properties:

* only for categorical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import chi_stat_test
>>> options = DataDriftOptions(all_features_stattest=chi_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="chisquare")
```

## cramer\_von\_mises\_stattest module <a href="#module-evidently.calculations.stattests.cramer_von_mises_stattest" id="module-evidently.calculations.stattests.cramer_von_mises_stattest"></a>

Cramer-Von-mises test of two samples.

Name: “cramer\_von\_mises”

Import:

```python
>>> from evidently.calculations.stattests import cramer_von_mises
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import cramer_von_mises
>>> options = DataDriftOptions(all_features_stattest=cramer_von_mises)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="cramer_von_mises")
```

### class CramerVonMisesResult(statistic, pvalue)

Bases: `object`

## energy\_distance module <a href="#module-evidently.calculations.stattests.energy_distance" id="module-evidently.calculations.stattests.energy_distance"></a>

Energy-distance test of two samples.

Name: “ed”

Import:

```python
>>> from evidently.calculations.stattests import energy_dist_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import energy_dist_test
>>> options = DataDriftOptions(all_features_stattest=energy_dist_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="ed")
```

## epps\_singleton\_stattest module <a href="#module-evidently.calculations.stattests.epps_singleton_stattest" id="module-evidently.calculations.stattests.epps_singleton_stattest"></a>

Epps-Singleton test of two samples.

Name: “es”

Import:

```python
>>> from evidently.calculations.stattests import epps_singleton_test
```

Properties:

* only for numerical features
* returns p-value
* default threshold 0.05

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import epps_singleton_test
>>> options = DataDriftOptions(all_features_stattest=epps_singleton_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="es")
```

## fisher\_exact\_stattest module <a href="#module-evidently.calculations.stattests.fisher_exact_stattest" id="module-evidently.calculations.stattests.fisher_exact_stattest"></a>

Fisher’s exact test of two samples.

Name: “fisher\_exact”

Import:

```python
>>> from evidently.calculations.stattests import fisher_exact_test
```

Properties:

* only for categorical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import fisher_exact_test
>>> options = DataDriftOptions(all_features_stattest=fisher_exact_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="fisher_exact")
```

## g\_stattest module <a href="#module-evidently.calculations.stattests.g_stattest" id="module-evidently.calculations.stattests.g_stattest"></a>

G-test of two samples.

Name: “g\_test”

Import:

```python
>>> from evidently.calculations.stattests import g_test
```

Properties:

* only for categorical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import g_test
>>> options = DataDriftOptions(all_features_stattest=g_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="g_test")
```

## hellinger\_distance module <a href="#module-evidently.calculations.stattests.hellinger_distance" id="module-evidently.calculations.stattests.hellinger_distance"></a>

Hellinger distance of two samples.

Name: “hellinger”

Import:

```python
>>> from evidently.calculations.stattests import hellinger_stat_test
```

Properties:

* only for categorical and numerical features
* returns distance

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import hellinger_stat_test
>>> options = DataDriftOptions(all_features_stattest=hellinger_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="hellinger")
```

## jensenshannon module <a href="#module-evidently.calculations.stattests.jensenshannon" id="module-evidently.calculations.stattests.jensenshannon"></a>

Jensen-Shannon distance of two samples.

Name: “jensenshannon”

Import:

```python
>>> from evidently.calculations.stattests import jensenshannon_stat_test
```

Properties:

* only for categorical and numerical features
* returns distance

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import jensenshannon_stat_test
>>> options = DataDriftOptions(all_features_stattest=jensenshannon_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="jensenshannon")
```

## kl\_div module <a href="#module-evidently.calculations.stattests.kl_div" id="module-evidently.calculations.stattests.kl_div"></a>

Kullback-Leibler divergence of two samples.

Name: “kl\_div”

Import:

```python
>>> from evidently.calculations.stattests import kl_div_stat_test
```

Properties:

* only for categorical and numerical features
* returns divergence

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import kl_div_stat_test
>>> options = DataDriftOptions(all_features_stattest=kl_div_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="kl_div")
```

## ks\_stattest module <a href="#module-evidently.calculations.stattests.ks_stattest" id="module-evidently.calculations.stattests.ks_stattest"></a>

Kolmogorov-Smirnov test of two samples.

Name: “ks”

Import:

```python
>>> from evidently.calculations.stattests import ks_stat_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import ks_stat_test
>>> options = DataDriftOptions(all_features_stattest=ks_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="ks")
```

## mann\_whitney\_urank\_stattest module <a href="#module-evidently.calculations.stattests.mann_whitney_urank_stattest" id="module-evidently.calculations.stattests.mann_whitney_urank_stattest"></a>

Mann-Whitney U-rank test of two samples.

Name: “mannw”

Import:

```python
>>> from evidently.calculations.stattests import mann_whitney_u_stat_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import mann_whitney_u_stat_test
>>> options = DataDriftOptions(all_features_stattest=mann_whitney_u_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="mannw")
```

## psi module <a href="#module-evidently.calculations.stattests.psi" id="module-evidently.calculations.stattests.psi"></a>

PSI of two samples.

Name: “psi”

Import:

```python
>>> from evidently.calculations.stattests import psi_stat_test
```

Properties:

* only for categorical and numerical features
* returns PSI value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import psi_stat_test
>>> options = DataDriftOptions(all_features_stattest=psi_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="psi")
```

## registry module <a href="#module-evidently.calculations.stattests.registry" id="module-evidently.calculations.stattests.registry"></a>

### class StatTest(name: str, display\_name: str, func: Callable\[\[pandas.core.series.Series, pandas.core.series.Series, str, float], Tuple\[float, bool]], allowed\_feature\_types: List\[str], default\_threshold: float = 0.05)

Bases: `object`

#### Attributes:

&#x20;    **allowed\_feature\_types : List\[str]**

&#x20;    **default\_threshold : float = 0.05**

&#x20;    **display\_name : str**

&#x20;    **func : Callable\[\[Series, Series, str, float], Tuple\[float, bool]]**

&#x20;    **name : str**

### exception StatTestInvalidFeatureTypeError(stattest\_name: str, feature\_type: str)

Bases: `ValueError`

### exception StatTestNotFoundError(stattest\_name: str)

Bases: `ValueError`

### class StatTestResult(drift\_score: float, drifted: bool, actual\_threshold: float)

Bases: `object`

#### Attributes:

&#x20;    **actual\_threshold : float**

&#x20;    **drift\_score : float**

&#x20;    **drifted : bool**

### get\_stattest(reference\_data: Series, current\_data: Series, feature\_type: str, stattest\_func: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], StatTest]])

### register\_stattest(stat\_test: StatTest)

## t\_test module <a href="#module-evidently.calculations.stattests.t_test" id="module-evidently.calculations.stattests.t_test"></a>

T test of two samples.

Name: “t\_test”

Import:

```python
>>> from evidently.calculations.stattests import t_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import t_test
>>> options = DataDriftOptions(all_features_stattest=t_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="t_test")
```

## tvd\_stattest module <a href="#module-evidently.calculations.stattests.tvd_stattest" id="module-evidently.calculations.stattests.tvd_stattest"></a>

Total variation distance of two samples.

Name: “TVD”

Import:

```python
>>> from evidently.calculations.stattests import tvd_test
```

Properties:

* only for numerical features
* returns distance

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import tvd_test
>>> options = DataDriftOptions(all_features_stattest=tvd_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="TVD")
```

## utils module <a href="#module-evidently.calculations.stattests.utils" id="module-evidently.calculations.stattests.utils"></a>

### generate\_fisher2x2\_contingency\_table(reference\_data: Series, current\_data: Series)

Generate 2x2 contingency matrix for fisher exact test\
:param reference\_data: reference data\
:param current\_data: current data

* **Raises**

  `ValueError` – if reference\_data and current\_data are not of equal length
* **Returns**

  contingency\_matrix for binary data
* **Return type**

  contingency\_matrix

### get\_binned\_data(reference\_data: Series, current\_data: Series, feature\_type: str, n: int, feel\_zeroes: bool = True)

Split variable into n buckets based on reference quantiles\
:param reference\_data: reference data\
:param current\_data: current data\
:param feature\_type: feature type\
:param n: number of quantiles

* **Returns**

  % of records in each bucket for reference\
  current\_percents: % of records in each bucket for current
* **Return type**

  reference\_percents

### get\_unique\_not\_nan\_values\_list\_from\_series(current\_data: Series, reference\_data: Series)

Get unique values from current and reference series, drop NaNs

### permutation\_test(reference\_data, current\_data, observed, test\_statistic\_func, iterations=100)

Perform a two-sided permutation test\
:param reference\_data: reference data\
:param current\_data: current data\
:param observed: observed value\
:param test\_statistic\_func: the test statistic function\
:param iterations: number of times to permute

* **Returns**

  two-sided p\_value
* **Return type**

  p\_value

## wasserstein\_distance\_norm module <a href="#module-evidently.calculations.stattests.wasserstein_distance_norm" id="module-evidently.calculations.stattests.wasserstein_distance_norm"></a>

Wasserstein distance of two samples.

Name: “wasserstein”

Import:

```python
>>> from evidently.calculations.stattests import wasserstein_stat_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import wasserstein_stat_test
>>> options = DataDriftOptions(all_features_stattest=wasserstein_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="wasserstein")
```

## z\_stattest module <a href="#module-evidently.calculations.stattests.z_stattest" id="module-evidently.calculations.stattests.z_stattest"></a>

Mann-Whitney U-rank test of two samples.

Name: “mannw”

Import:

```python
>>> from evidently.calculations.stattests import mann_whitney_u_stat_test
```

Properties:

* only for numerical features
* returns p-value

### Example

Using by object:

```python
>>> from evidently.options import DataDriftOptions
>>> from evidently.calculations.stattests import mann_whitney_u_stat_test
>>> options = DataDriftOptions(all_features_stattest=mann_whitney_u_stat_test)
```

Using by name:

```python
>>> from evidently.options import DataDriftOptions
>>> options = DataDriftOptions(all_features_stattest="mannw")
```

### proportions\_diff\_z\_stat\_ind(ref: DataFrame, curr: DataFrame)

### proportions\_diff\_z\_test(z\_stat, alternative='two-sided')


# evidently.metrics

Available metrics for Reports and Tests.\
All metrics is grouped into modules.\
For specific group see module documentation.

## Subpackages

* [evidently.metrics.classification\_performance package](/reference/api-reference/evidently.metrics/evidently.metrics.classification_performance)
  * [Submodules](/reference/api-reference/evidently.metrics/evidently.metrics.classification_performance#submodules)
  * [base\_classification\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.base_classification_metric)
    * [`ThresholdClassificationMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.base_classification_metric.ThresholdClassificationMetric)
      * [`ThresholdClassificationMetric.get_target_prediction_data()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.base_classification_metric.ThresholdClassificationMetric.get_target_prediction_data)
      * [`ThresholdClassificationMetric.k`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.base_classification_metric.ThresholdClassificationMetric.k)
      * [`ThresholdClassificationMetric.probas_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.base_classification_metric.ThresholdClassificationMetric.probas_threshold)
  * [class\_balance\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.class_balance_metric)
    * [`ClassificationClassBalance`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalance)
      * [`ClassificationClassBalance.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalance.calculate)
    * [`ClassificationClassBalanceRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalanceRenderer)
      * [`ClassificationClassBalanceRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalanceRenderer.color_options)
      * [`ClassificationClassBalanceRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalanceRenderer.render_html)
      * [`ClassificationClassBalanceRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalanceRenderer.render_json)
    * [`ClassificationClassBalanceResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalanceResult)
      * [`ClassificationClassBalanceResult.plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_balance_metric.ClassificationClassBalanceResult.plot_data)
  * [class\_separation\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.class_separation_metric)
    * [`ClassificationClassSeparationPlot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlot)
      * [`ClassificationClassSeparationPlot.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlot.calculate)
    * [`ClassificationClassSeparationPlotRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotRenderer)
      * [`ClassificationClassSeparationPlotRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotRenderer.color_options)
      * [`ClassificationClassSeparationPlotRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotRenderer.render_html)
      * [`ClassificationClassSeparationPlotRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotRenderer.render_json)
    * [`ClassificationClassSeparationPlotResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotResults)
      * [`ClassificationClassSeparationPlotResults.current_plot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotResults.current_plot)
      * [`ClassificationClassSeparationPlotResults.reference_plot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotResults.reference_plot)
      * [`ClassificationClassSeparationPlotResults.target_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.class_separation_metric.ClassificationClassSeparationPlotResults.target_name)
  * [classification\_dummy\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.classification_dummy_metric)
    * [`ClassificationDummyMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetric)
      * [`ClassificationDummyMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetric.calculate)
      * [`ClassificationDummyMetric.correction_for_threshold()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetric.correction_for_threshold)
      * [`ClassificationDummyMetric.quality_metric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetric.quality_metric)
    * [`ClassificationDummyMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricRenderer)
      * [`ClassificationDummyMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricRenderer.color_options)
      * [`ClassificationDummyMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricRenderer.render_html)
      * [`ClassificationDummyMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricRenderer.render_json)
    * [`ClassificationDummyMetricResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricResults)
      * [`ClassificationDummyMetricResults.by_reference_dummy`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricResults.by_reference_dummy)
      * [`ClassificationDummyMetricResults.dummy`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricResults.dummy)
      * [`ClassificationDummyMetricResults.metrics_matrix`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricResults.metrics_matrix)
      * [`ClassificationDummyMetricResults.model_quality`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_dummy_metric.ClassificationDummyMetricResults.model_quality)
  * [classification\_quality\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.classification_quality_metric)
    * [`ClassificationQualityMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetric)
      * [`ClassificationQualityMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetric.calculate)
      * [`ClassificationQualityMetric.confusion_matrix_metric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetric.confusion_matrix_metric)
    * [`ClassificationQualityMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricRenderer)
      * [`ClassificationQualityMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricRenderer.color_options)
      * [`ClassificationQualityMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricRenderer.render_html)
      * [`ClassificationQualityMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricRenderer.render_json)
    * [`ClassificationQualityMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricResult)
      * [`ClassificationQualityMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricResult.current)
      * [`ClassificationQualityMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricResult.reference)
      * [`ClassificationQualityMetricResult.target_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.classification_quality_metric.ClassificationQualityMetricResult.target_name)
  * [confusion\_matrix\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.confusion_matrix_metric)
    * [`ClassificationConfusionMatrix`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrix)
      * [`ClassificationConfusionMatrix.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrix.calculate)
      * [`ClassificationConfusionMatrix.k`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrix.k)
      * [`ClassificationConfusionMatrix.probas_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrix.probas_threshold)
    * [`ClassificationConfusionMatrixRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixRenderer)
      * [`ClassificationConfusionMatrixRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixRenderer.color_options)
      * [`ClassificationConfusionMatrixRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixRenderer.render_html)
      * [`ClassificationConfusionMatrixRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixRenderer.render_json)
    * [`ClassificationConfusionMatrixResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixResult)
      * [`ClassificationConfusionMatrixResult.current_matrix`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixResult.current_matrix)
      * [`ClassificationConfusionMatrixResult.reference_matrix`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.confusion_matrix_metric.ClassificationConfusionMatrixResult.reference_matrix)
  * [pr\_curve\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.pr_curve_metric)
    * [`ClassificationPRCurve`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurve)
      * [`ClassificationPRCurve.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurve.calculate)
      * [`ClassificationPRCurve.calculate_metrics()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurve.calculate_metrics)
    * [`ClassificationPRCurveRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveRenderer)
      * [`ClassificationPRCurveRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveRenderer.color_options)
      * [`ClassificationPRCurveRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveRenderer.render_html)
      * [`ClassificationPRCurveRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveRenderer.render_json)
    * [`ClassificationPRCurveResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveResults)
      * [`ClassificationPRCurveResults.current_pr_curve`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveResults.current_pr_curve)
      * [`ClassificationPRCurveResults.reference_pr_curve`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_curve_metric.ClassificationPRCurveResults.reference_pr_curve)
  * [pr\_table\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.pr_table_metric)
    * [`ClassificationPRTable`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTable)
      * [`ClassificationPRTable.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTable.calculate)
      * [`ClassificationPRTable.calculate_metrics()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTable.calculate_metrics)
    * [`ClassificationPRTableRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableRenderer)
      * [`ClassificationPRTableRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableRenderer.color_options)
      * [`ClassificationPRTableRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableRenderer.render_html)
      * [`ClassificationPRTableRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableRenderer.render_json)
    * [`ClassificationPRTableResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableResults)
      * [`ClassificationPRTableResults.current_pr_table`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableResults.current_pr_table)
      * [`ClassificationPRTableResults.reference_pr_table`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.pr_table_metric.ClassificationPRTableResults.reference_pr_table)
  * [probability\_distribution\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.probability_distribution_metric)
    * [`ClassificationProbDistribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistribution)
      * [`ClassificationProbDistribution.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistribution.calculate)
      * [`ClassificationProbDistribution.get_distribution()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistribution.get_distribution)
    * [`ClassificationProbDistributionRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionRenderer)
      * [`ClassificationProbDistributionRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionRenderer.color_options)
      * [`ClassificationProbDistributionRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionRenderer.render_html)
      * [`ClassificationProbDistributionRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionRenderer.render_json)
    * [`ClassificationProbDistributionResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionResults)
      * [`ClassificationProbDistributionResults.current_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionResults.current_distribution)
      * [`ClassificationProbDistributionResults.reference_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.probability_distribution_metric.ClassificationProbDistributionResults.reference_distribution)
  * [quality\_by\_class\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.quality_by_class_metric)
    * [`ClassificationQualityByClass`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClass)
      * [`ClassificationQualityByClass.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClass.calculate)
      * [`ClassificationQualityByClass.k`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClass.k)
      * [`ClassificationQualityByClass.probas_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClass.probas_threshold)
    * [`ClassificationQualityByClassRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassRenderer)
      * [`ClassificationQualityByClassRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassRenderer.color_options)
      * [`ClassificationQualityByClassRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassRenderer.render_html)
      * [`ClassificationQualityByClassRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassRenderer.render_json)
    * [`ClassificationQualityByClassResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassResult)
      * [`ClassificationQualityByClassResult.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassResult.columns)
      * [`ClassificationQualityByClassResult.current_metrics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassResult.current_metrics)
      * [`ClassificationQualityByClassResult.current_roc_aucs`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassResult.current_roc_aucs)
      * [`ClassificationQualityByClassResult.reference_metrics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassResult.reference_metrics)
      * [`ClassificationQualityByClassResult.reference_roc_aucs`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_class_metric.ClassificationQualityByClassResult.reference_roc_aucs)
  * [quality\_by\_feature\_table module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.quality_by_feature_table)
    * [`ClassificationQualityByFeatureTable`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTable)
      * [`ClassificationQualityByFeatureTable.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTable.calculate)
      * [`ClassificationQualityByFeatureTable.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTable.columns)
    * [`ClassificationQualityByFeatureTableRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableRenderer)
      * [`ClassificationQualityByFeatureTableRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableRenderer.color_options)
      * [`ClassificationQualityByFeatureTableRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableRenderer.render_html)
      * [`ClassificationQualityByFeatureTableRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableRenderer.render_json)
    * [`ClassificationQualityByFeatureTableResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults)
      * [`ClassificationQualityByFeatureTableResults.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults.columns)
      * [`ClassificationQualityByFeatureTableResults.curr_predictions`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults.curr_predictions)
      * [`ClassificationQualityByFeatureTableResults.current_plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults.current_plot_data)
      * [`ClassificationQualityByFeatureTableResults.ref_predictions`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults.ref_predictions)
      * [`ClassificationQualityByFeatureTableResults.reference_plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults.reference_plot_data)
      * [`ClassificationQualityByFeatureTableResults.target_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.quality_by_feature_table.ClassificationQualityByFeatureTableResults.target_name)
  * [roc\_curve\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#module-evidently.metrics.classification_performance.roc_curve_metric)
    * [`ClassificationRocCurve`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurve)
      * [`ClassificationRocCurve.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurve.calculate)
      * [`ClassificationRocCurve.calculate_metrics()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurve.calculate_metrics)
    * [`ClassificationRocCurveRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveRenderer)
      * [`ClassificationRocCurveRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveRenderer.color_options)
      * [`ClassificationRocCurveRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveRenderer.render_html)
      * [`ClassificationRocCurveRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveRenderer.render_json)
    * [`ClassificationRocCurveResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveResults)
      * [`ClassificationRocCurveResults.current_roc_curve`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveResults.current_roc_curve)
      * [`ClassificationRocCurveResults.reference_roc_curve`](https://docs-old.evidentlyai.com/reference/api-reference/pages/7pJDNvXyDzKgWqwUj4Fn#evidently.metrics.classification_performance.roc_curve_metric.ClassificationRocCurveResults.reference_roc_curve)
* [evidently.metrics.data\_drift package](/reference/api-reference/evidently.metrics/evidently.metrics.data_drift)
  * [Submodules](/reference/api-reference/evidently.metrics/evidently.metrics.data_drift#submodules)
  * [column\_drift\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#module-evidently.metrics.data_drift.column_drift_metric)
    * [`ColumnDriftMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetric)
      * [`ColumnDriftMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetric.calculate)
      * [`ColumnDriftMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetric.column_name)
      * [`ColumnDriftMetric.get_parameters()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetric.get_parameters)
      * [`ColumnDriftMetric.stattest`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetric.stattest)
      * [`ColumnDriftMetric.stattest_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetric.stattest_threshold)
    * [`ColumnDriftMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricRenderer)
      * [`ColumnDriftMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricRenderer.color_options)
      * [`ColumnDriftMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricRenderer.render_html)
      * [`ColumnDriftMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricRenderer.render_json)
    * [`ColumnDriftMetricResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults)
      * [`ColumnDriftMetricResults.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.column_name)
      * [`ColumnDriftMetricResults.column_type`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.column_type)
      * [`ColumnDriftMetricResults.current_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.current_distribution)
      * [`ColumnDriftMetricResults.current_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.current_scatter)
      * [`ColumnDriftMetricResults.drift_detected`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.drift_detected)
      * [`ColumnDriftMetricResults.drift_score`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.drift_score)
      * [`ColumnDriftMetricResults.plot_shape`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.plot_shape)
      * [`ColumnDriftMetricResults.reference_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.reference_distribution)
      * [`ColumnDriftMetricResults.stattest_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.stattest_name)
      * [`ColumnDriftMetricResults.stattest_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.stattest_threshold)
      * [`ColumnDriftMetricResults.x_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_drift_metric.ColumnDriftMetricResults.x_name)
  * [column\_value\_plot module](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#module-evidently.metrics.data_drift.column_value_plot)
    * [`ColumnValuePlot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlot)
      * [`ColumnValuePlot.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlot.calculate)
      * [`ColumnValuePlot.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlot.column_name)
    * [`ColumnValuePlotRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotRenderer)
      * [`ColumnValuePlotRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotRenderer.color_options)
      * [`ColumnValuePlotRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotRenderer.render_html)
      * [`ColumnValuePlotRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotRenderer.render_json)
    * [`ColumnValuePlotResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotResults)
      * [`ColumnValuePlotResults.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotResults.column_name)
      * [`ColumnValuePlotResults.current_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotResults.current_scatter)
      * [`ColumnValuePlotResults.datetime_column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotResults.datetime_column_name)
      * [`ColumnValuePlotResults.reference_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.column_value_plot.ColumnValuePlotResults.reference_scatter)
  * [data\_drift\_table module](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#module-evidently.metrics.data_drift.data_drift_table)
    * [`DataDriftTable`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTable)
      * [`DataDriftTable.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTable.calculate)
      * [`DataDriftTable.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTable.columns)
      * [`DataDriftTable.get_parameters()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTable.get_parameters)
      * [`DataDriftTable.options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTable.options)
    * [`DataDriftTableRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableRenderer)
      * [`DataDriftTableRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableRenderer.color_options)
      * [`DataDriftTableRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableRenderer.render_html)
      * [`DataDriftTableRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableRenderer.render_json)
    * [`DataDriftTableResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults)
      * [`DataDriftTableResults.dataset_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults.dataset_columns)
      * [`DataDriftTableResults.dataset_drift`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults.dataset_drift)
      * [`DataDriftTableResults.drift_by_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults.drift_by_columns)
      * [`DataDriftTableResults.number_of_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults.number_of_columns)
      * [`DataDriftTableResults.number_of_drifted_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults.number_of_drifted_columns)
      * [`DataDriftTableResults.share_of_drifted_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.data_drift_table.DataDriftTableResults.share_of_drifted_columns)
  * [dataset\_drift\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#module-evidently.metrics.data_drift.dataset_drift_metric)
    * [`DataDriftMetricsRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DataDriftMetricsRenderer)
      * [`DataDriftMetricsRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DataDriftMetricsRenderer.color_options)
      * [`DataDriftMetricsRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DataDriftMetricsRenderer.render_html)
      * [`DataDriftMetricsRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DataDriftMetricsRenderer.render_json)
    * [`DatasetDriftMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetric)
      * [`DatasetDriftMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetric.calculate)
      * [`DatasetDriftMetric.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetric.columns)
      * [`DatasetDriftMetric.drift_share`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetric.drift_share)
      * [`DatasetDriftMetric.get_parameters()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetric.get_parameters)
      * [`DatasetDriftMetric.options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetric.options)
    * [`DatasetDriftMetricResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetricResults)
      * [`DatasetDriftMetricResults.dataset_drift`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetricResults.dataset_drift)
      * [`DatasetDriftMetricResults.drift_share`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetricResults.drift_share)
      * [`DatasetDriftMetricResults.number_of_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetricResults.number_of_columns)
      * [`DatasetDriftMetricResults.number_of_drifted_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetricResults.number_of_drifted_columns)
      * [`DatasetDriftMetricResults.share_of_drifted_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.dataset_drift_metric.DatasetDriftMetricResults.share_of_drifted_columns)
  * [target\_by\_features\_table module](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#module-evidently.metrics.data_drift.target_by_features_table)
    * [`TargetByFeaturesTable`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTable)
      * [`TargetByFeaturesTable.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTable.calculate)
      * [`TargetByFeaturesTable.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTable.columns)
    * [`TargetByFeaturesTableRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableRenderer)
      * [`TargetByFeaturesTableRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableRenderer.color_options)
      * [`TargetByFeaturesTableRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableRenderer.render_html)
      * [`TargetByFeaturesTableRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableRenderer.render_json)
    * [`TargetByFeaturesTableResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults)
      * [`TargetByFeaturesTableResults.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.columns)
      * [`TargetByFeaturesTableResults.curr_predictions`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.curr_predictions)
      * [`TargetByFeaturesTableResults.current_plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.current_plot_data)
      * [`TargetByFeaturesTableResults.ref_predictions`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.ref_predictions)
      * [`TargetByFeaturesTableResults.reference_plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.reference_plot_data)
      * [`TargetByFeaturesTableResults.target_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.target_name)
      * [`TargetByFeaturesTableResults.task`](https://docs-old.evidentlyai.com/reference/api-reference/pages/COTU5JUxoGgFWbdE4JKA#evidently.metrics.data_drift.target_by_features_table.TargetByFeaturesTableResults.task)
* [evidently.metrics.data\_integrity package](/reference/api-reference/evidently.metrics/evidently.metrics.data_integrity)
  * [Submodules](/reference/api-reference/evidently.metrics/evidently.metrics.data_integrity#submodules)
  * [column\_missing\_values\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#module-evidently.metrics.data_integrity.column_missing_values_metric)
    * [`ColumnMissingValues`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValues)
      * [`ColumnMissingValues.different_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValues.different_missing_values)
      * [`ColumnMissingValues.number_of_different_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValues.number_of_different_missing_values)
      * [`ColumnMissingValues.number_of_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValues.number_of_missing_values)
      * [`ColumnMissingValues.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValues.number_of_rows)
      * [`ColumnMissingValues.share_of_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValues.share_of_missing_values)
    * [`ColumnMissingValuesMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetric)
      * [`ColumnMissingValuesMetric.DEFAULT_MISSING_VALUES`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetric.DEFAULT_MISSING_VALUES)
      * [`ColumnMissingValuesMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetric.calculate)
      * [`ColumnMissingValuesMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetric.column_name)
      * [`ColumnMissingValuesMetric.missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetric.missing_values)
    * [`ColumnMissingValuesMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricRenderer)
      * [`ColumnMissingValuesMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricRenderer.color_options)
      * [`ColumnMissingValuesMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricRenderer.render_html)
      * [`ColumnMissingValuesMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricRenderer.render_json)
    * [`ColumnMissingValuesMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricResult)
      * [`ColumnMissingValuesMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricResult.column_name)
      * [`ColumnMissingValuesMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricResult.current)
      * [`ColumnMissingValuesMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_missing_values_metric.ColumnMissingValuesMetricResult.reference)
  * [column\_regexp\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#module-evidently.metrics.data_integrity.column_regexp_metric)
    * [`ColumnRegExpMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetric)
      * [`ColumnRegExpMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetric.calculate)
      * [`ColumnRegExpMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetric.column_name)
      * [`ColumnRegExpMetric.reg_exp`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetric.reg_exp)
      * [`ColumnRegExpMetric.top`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetric.top)
    * [`ColumnRegExpMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetricRenderer)
      * [`ColumnRegExpMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetricRenderer.color_options)
      * [`ColumnRegExpMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetricRenderer.render_html)
      * [`ColumnRegExpMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.ColumnRegExpMetricRenderer.render_json)
    * [`DataIntegrityValueByRegexpMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpMetricResult)
      * [`DataIntegrityValueByRegexpMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpMetricResult.column_name)
      * [`DataIntegrityValueByRegexpMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpMetricResult.current)
      * [`DataIntegrityValueByRegexpMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpMetricResult.reference)
      * [`DataIntegrityValueByRegexpMetricResult.reg_exp`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpMetricResult.reg_exp)
      * [`DataIntegrityValueByRegexpMetricResult.top`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpMetricResult.top)
    * [`DataIntegrityValueByRegexpStat`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpStat)
      * [`DataIntegrityValueByRegexpStat.number_of_matched`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpStat.number_of_matched)
      * [`DataIntegrityValueByRegexpStat.number_of_not_matched`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpStat.number_of_not_matched)
      * [`DataIntegrityValueByRegexpStat.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpStat.number_of_rows)
      * [`DataIntegrityValueByRegexpStat.table_of_matched`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpStat.table_of_matched)
      * [`DataIntegrityValueByRegexpStat.table_of_not_matched`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_regexp_metric.DataIntegrityValueByRegexpStat.table_of_not_matched)
  * [column\_summary\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#module-evidently.metrics.data_integrity.column_summary_metric)
    * [`CategoricalCharacteristics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics)
      * [`CategoricalCharacteristics.count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.count)
      * [`CategoricalCharacteristics.missing`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.missing)
      * [`CategoricalCharacteristics.missing_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.missing_percentage)
      * [`CategoricalCharacteristics.most_common`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.most_common)
      * [`CategoricalCharacteristics.most_common_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.most_common_percentage)
      * [`CategoricalCharacteristics.new_in_current_values_count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.new_in_current_values_count)
      * [`CategoricalCharacteristics.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.number_of_rows)
      * [`CategoricalCharacteristics.unique`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.unique)
      * [`CategoricalCharacteristics.unique_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.unique_percentage)
      * [`CategoricalCharacteristics.unused_in_current_values_count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.CategoricalCharacteristics.unused_in_current_values_count)
    * [`ColumnSummary`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummary)
      * [`ColumnSummary.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummary.column_name)
      * [`ColumnSummary.column_type`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummary.column_type)
      * [`ColumnSummary.current_characteristics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummary.current_characteristics)
      * [`ColumnSummary.plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummary.plot_data)
      * [`ColumnSummary.reference_characteristics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummary.reference_characteristics)
    * [`ColumnSummaryMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetric)
      * [`ColumnSummaryMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetric.calculate)
      * [`ColumnSummaryMetric.map_data()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetric.map_data)
    * [`ColumnSummaryMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetricRenderer)
      * [`ColumnSummaryMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetricRenderer.color_options)
      * [`ColumnSummaryMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetricRenderer.render_html)
      * [`ColumnSummaryMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.ColumnSummaryMetricRenderer.render_json)
    * [`DataByTarget`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataByTarget)
      * [`DataByTarget.data_for_plots`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataByTarget.data_for_plots)
      * [`DataByTarget.target_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataByTarget.target_name)
      * [`DataByTarget.target_type`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataByTarget.target_type)
    * [`DataInTime`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataInTime)
      * [`DataInTime.data_for_plots`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataInTime.data_for_plots)
      * [`DataInTime.datetime_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataInTime.datetime_name)
      * [`DataInTime.freq`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataInTime.freq)
    * [`DataQualityPlot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataQualityPlot)
      * [`DataQualityPlot.bins_for_hist`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataQualityPlot.bins_for_hist)
      * [`DataQualityPlot.counts_of_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataQualityPlot.counts_of_values)
      * [`DataQualityPlot.data_by_target`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataQualityPlot.data_by_target)
      * [`DataQualityPlot.data_in_time`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DataQualityPlot.data_in_time)
    * [`DatetimeCharacteristics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics)
      * [`DatetimeCharacteristics.count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.count)
      * [`DatetimeCharacteristics.first`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.first)
      * [`DatetimeCharacteristics.last`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.last)
      * [`DatetimeCharacteristics.missing`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.missing)
      * [`DatetimeCharacteristics.missing_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.missing_percentage)
      * [`DatetimeCharacteristics.most_common`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.most_common)
      * [`DatetimeCharacteristics.most_common_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.most_common_percentage)
      * [`DatetimeCharacteristics.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.number_of_rows)
      * [`DatetimeCharacteristics.unique`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.unique)
      * [`DatetimeCharacteristics.unique_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.DatetimeCharacteristics.unique_percentage)
    * [`NumericCharacteristics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics)
      * [`NumericCharacteristics.count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.count)
      * [`NumericCharacteristics.infinite_count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.infinite_count)
      * [`NumericCharacteristics.infinite_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.infinite_percentage)
      * [`NumericCharacteristics.max`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.max)
      * [`NumericCharacteristics.mean`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.mean)
      * [`NumericCharacteristics.min`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.min)
      * [`NumericCharacteristics.missing`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.missing)
      * [`NumericCharacteristics.missing_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.missing_percentage)
      * [`NumericCharacteristics.most_common`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.most_common)
      * [`NumericCharacteristics.most_common_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.most_common_percentage)
      * [`NumericCharacteristics.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.number_of_rows)
      * [`NumericCharacteristics.p25`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.p25)
      * [`NumericCharacteristics.p50`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.p50)
      * [`NumericCharacteristics.p75`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.p75)
      * [`NumericCharacteristics.std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.std)
      * [`NumericCharacteristics.unique`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.unique)
      * [`NumericCharacteristics.unique_percentage`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.column_summary_metric.NumericCharacteristics.unique_percentage)
  * [dataset\_missing\_values\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#module-evidently.metrics.data_integrity.dataset_missing_values_metric)
    * [`DatasetMissingValues`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues)
      * [`DatasetMissingValues.columns_with_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.columns_with_missing_values)
      * [`DatasetMissingValues.different_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.different_missing_values)
      * [`DatasetMissingValues.different_missing_values_by_column`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.different_missing_values_by_column)
      * [`DatasetMissingValues.number_of_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_columns)
      * [`DatasetMissingValues.number_of_columns_with_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_columns_with_missing_values)
      * [`DatasetMissingValues.number_of_different_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_different_missing_values)
      * [`DatasetMissingValues.number_of_different_missing_values_by_column`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_different_missing_values_by_column)
      * [`DatasetMissingValues.number_of_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_missing_values)
      * [`DatasetMissingValues.number_of_missing_values_by_column`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_missing_values_by_column)
      * [`DatasetMissingValues.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_rows)
      * [`DatasetMissingValues.number_of_rows_with_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.number_of_rows_with_missing_values)
      * [`DatasetMissingValues.share_of_columns_with_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.share_of_columns_with_missing_values)
      * [`DatasetMissingValues.share_of_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.share_of_missing_values)
      * [`DatasetMissingValues.share_of_missing_values_by_column`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.share_of_missing_values_by_column)
      * [`DatasetMissingValues.share_of_rows_with_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValues.share_of_rows_with_missing_values)
    * [`DatasetMissingValuesMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetric)
      * [`DatasetMissingValuesMetric.DEFAULT_MISSING_VALUES`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetric.DEFAULT_MISSING_VALUES)
      * [`DatasetMissingValuesMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetric.calculate)
      * [`DatasetMissingValuesMetric.missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetric.missing_values)
    * [`DatasetMissingValuesMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricRenderer)
      * [`DatasetMissingValuesMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricRenderer.color_options)
      * [`DatasetMissingValuesMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricRenderer.render_html)
      * [`DatasetMissingValuesMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricRenderer.render_json)
    * [`DatasetMissingValuesMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricResult)
      * [`DatasetMissingValuesMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricResult.current)
      * [`DatasetMissingValuesMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_missing_values_metric.DatasetMissingValuesMetricResult.reference)
  * [dataset\_summary\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#module-evidently.metrics.data_integrity.dataset_summary_metric)
    * [`DatasetSummary`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary)
      * [`DatasetSummary.columns_type`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.columns_type)
      * [`DatasetSummary.date_column`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.date_column)
      * [`DatasetSummary.id_column`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.id_column)
      * [`DatasetSummary.nans_by_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.nans_by_columns)
      * [`DatasetSummary.number_of_almost_constant_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_almost_constant_columns)
      * [`DatasetSummary.number_of_almost_duplicated_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_almost_duplicated_columns)
      * [`DatasetSummary.number_of_categorical_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_categorical_columns)
      * [`DatasetSummary.number_of_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_columns)
      * [`DatasetSummary.number_of_constant_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_constant_columns)
      * [`DatasetSummary.number_of_datetime_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_datetime_columns)
      * [`DatasetSummary.number_of_duplicated_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_duplicated_columns)
      * [`DatasetSummary.number_of_duplicated_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_duplicated_rows)
      * [`DatasetSummary.number_of_empty_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_empty_columns)
      * [`DatasetSummary.number_of_empty_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_empty_rows)
      * [`DatasetSummary.number_of_missing_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_missing_values)
      * [`DatasetSummary.number_of_numeric_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_numeric_columns)
      * [`DatasetSummary.number_of_rows`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_of_rows)
      * [`DatasetSummary.number_uniques_by_columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.number_uniques_by_columns)
      * [`DatasetSummary.prediction`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.prediction)
      * [`DatasetSummary.target`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummary.target)
    * [`DatasetSummaryMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetric)
      * [`DatasetSummaryMetric.almost_constant_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetric.almost_constant_threshold)
      * [`DatasetSummaryMetric.almost_duplicated_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetric.almost_duplicated_threshold)
      * [`DatasetSummaryMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetric.calculate)
    * [`DatasetSummaryMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricRenderer)
      * [`DatasetSummaryMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricRenderer.color_options)
      * [`DatasetSummaryMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricRenderer.render_html)
      * [`DatasetSummaryMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricRenderer.render_json)
    * [`DatasetSummaryMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricResult)
      * [`DatasetSummaryMetricResult.almost_duplicated_threshold`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricResult.almost_duplicated_threshold)
      * [`DatasetSummaryMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricResult.current)
      * [`DatasetSummaryMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/frXIRRjbV3w55KVEnMCt#evidently.metrics.data_integrity.dataset_summary_metric.DatasetSummaryMetricResult.reference)
* [evidently.metrics.data\_quality package](/reference/api-reference/evidently.metrics/evidently.metrics.data_quality)
  * [Submodules](/reference/api-reference/evidently.metrics/evidently.metrics.data_quality#submodules)
  * [column\_correlations\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.column_correlations_metric)
    * [`ColumnCorrelationsMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetric)
      * [`ColumnCorrelationsMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetric.calculate)
      * [`ColumnCorrelationsMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetric.column_name)
    * [`ColumnCorrelationsMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricRenderer)
      * [`ColumnCorrelationsMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricRenderer.color_options)
      * [`ColumnCorrelationsMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricRenderer.render_html)
      * [`ColumnCorrelationsMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricRenderer.render_json)
    * [`ColumnCorrelationsMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricResult)
      * [`ColumnCorrelationsMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricResult.column_name)
      * [`ColumnCorrelationsMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricResult.current)
      * [`ColumnCorrelationsMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_correlations_metric.ColumnCorrelationsMetricResult.reference)
  * [column\_distribution\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.column_distribution_metric)
    * [`ColumnDistributionMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetric)
      * [`ColumnDistributionMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetric.calculate)
      * [`ColumnDistributionMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetric.column_name)
    * [`ColumnDistributionMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricRenderer)
      * [`ColumnDistributionMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricRenderer.color_options)
      * [`ColumnDistributionMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricRenderer.render_html)
      * [`ColumnDistributionMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricRenderer.render_json)
    * [`ColumnDistributionMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricResult)
      * [`ColumnDistributionMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricResult.column_name)
      * [`ColumnDistributionMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricResult.current)
      * [`ColumnDistributionMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_distribution_metric.ColumnDistributionMetricResult.reference)
  * [column\_quantile\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.column_quantile_metric)
    * [`ColumnQuantileMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetric)
      * [`ColumnQuantileMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetric.calculate)
      * [`ColumnQuantileMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetric.column_name)
      * [`ColumnQuantileMetric.quantile`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetric.quantile)
    * [`ColumnQuantileMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricRenderer)
      * [`ColumnQuantileMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricRenderer.color_options)
      * [`ColumnQuantileMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricRenderer.render_html)
      * [`ColumnQuantileMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricRenderer.render_json)
    * [`ColumnQuantileMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult)
      * [`ColumnQuantileMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult.column_name)
      * [`ColumnQuantileMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult.current)
      * [`ColumnQuantileMetricResult.current_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult.current_distribution)
      * [`ColumnQuantileMetricResult.quantile`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult.quantile)
      * [`ColumnQuantileMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult.reference)
      * [`ColumnQuantileMetricResult.reference_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_quantile_metric.ColumnQuantileMetricResult.reference_distribution)
  * [column\_value\_list\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.column_value_list_metric)
    * [`ColumnValueListMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetric)
      * [`ColumnValueListMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetric.calculate)
      * [`ColumnValueListMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetric.column_name)
      * [`ColumnValueListMetric.values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetric.values)
    * [`ColumnValueListMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricRenderer)
      * [`ColumnValueListMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricRenderer.color_options)
      * [`ColumnValueListMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricRenderer.render_html)
      * [`ColumnValueListMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricRenderer.render_json)
    * [`ColumnValueListMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricResult)
      * [`ColumnValueListMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricResult.column_name)
      * [`ColumnValueListMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricResult.current)
      * [`ColumnValueListMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricResult.reference)
      * [`ColumnValueListMetricResult.values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ColumnValueListMetricResult.values)
    * [`ValueListStat`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat)
      * [`ValueListStat.number_in_list`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.number_in_list)
      * [`ValueListStat.number_not_in_list`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.number_not_in_list)
      * [`ValueListStat.rows_count`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.rows_count)
      * [`ValueListStat.share_in_list`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.share_in_list)
      * [`ValueListStat.share_not_in_list`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.share_not_in_list)
      * [`ValueListStat.values_in_list`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.values_in_list)
      * [`ValueListStat.values_not_in_list`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_list_metric.ValueListStat.values_not_in_list)
  * [column\_value\_range\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.column_value_range_metric)
    * [`ColumnValueRangeMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetric)
      * [`ColumnValueRangeMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetric.calculate)
      * [`ColumnValueRangeMetric.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetric.column_name)
      * [`ColumnValueRangeMetric.left`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetric.left)
      * [`ColumnValueRangeMetric.right`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetric.right)
    * [`ColumnValueRangeMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricRenderer)
      * [`ColumnValueRangeMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricRenderer.color_options)
      * [`ColumnValueRangeMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricRenderer.render_html)
      * [`ColumnValueRangeMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricRenderer.render_json)
    * [`ColumnValueRangeMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult)
      * [`ColumnValueRangeMetricResult.column_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.column_name)
      * [`ColumnValueRangeMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.current)
      * [`ColumnValueRangeMetricResult.current_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.current_distribution)
      * [`ColumnValueRangeMetricResult.left`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.left)
      * [`ColumnValueRangeMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.reference)
      * [`ColumnValueRangeMetricResult.reference_distribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.reference_distribution)
      * [`ColumnValueRangeMetricResult.right`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ColumnValueRangeMetricResult.right)
    * [`ValuesInRangeStat`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ValuesInRangeStat)
      * [`ValuesInRangeStat.number_in_range`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ValuesInRangeStat.number_in_range)
      * [`ValuesInRangeStat.number_not_in_range`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ValuesInRangeStat.number_not_in_range)
      * [`ValuesInRangeStat.number_of_values`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ValuesInRangeStat.number_of_values)
      * [`ValuesInRangeStat.share_in_range`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ValuesInRangeStat.share_in_range)
      * [`ValuesInRangeStat.share_not_in_range`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.column_value_range_metric.ValuesInRangeStat.share_not_in_range)
  * [dataset\_correlations\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.dataset_correlations_metric)
    * [`CorrelationStats`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.CorrelationStats)
      * [`CorrelationStats.abs_max_correlation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.CorrelationStats.abs_max_correlation)
      * [`CorrelationStats.abs_max_features_correlation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.CorrelationStats.abs_max_features_correlation)
      * [`CorrelationStats.abs_max_prediction_features_correlation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.CorrelationStats.abs_max_prediction_features_correlation)
      * [`CorrelationStats.abs_max_target_features_correlation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.CorrelationStats.abs_max_target_features_correlation)
      * [`CorrelationStats.target_prediction_correlation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.CorrelationStats.target_prediction_correlation)
    * [`DataQualityCorrelationMetricsRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DataQualityCorrelationMetricsRenderer)
      * [`DataQualityCorrelationMetricsRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DataQualityCorrelationMetricsRenderer.color_options)
      * [`DataQualityCorrelationMetricsRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DataQualityCorrelationMetricsRenderer.render_html)
      * [`DataQualityCorrelationMetricsRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DataQualityCorrelationMetricsRenderer.render_json)
    * [`DatasetCorrelation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelation)
      * [`DatasetCorrelation.correlation`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelation.correlation)
      * [`DatasetCorrelation.stats`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelation.stats)
    * [`DatasetCorrelationsMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelationsMetric)
      * [`DatasetCorrelationsMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelationsMetric.calculate)
    * [`DatasetCorrelationsMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelationsMetricResult)
      * [`DatasetCorrelationsMetricResult.current`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelationsMetricResult.current)
      * [`DatasetCorrelationsMetricResult.reference`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.dataset_correlations_metric.DatasetCorrelationsMetricResult.reference)
  * [stability\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#module-evidently.metrics.data_quality.stability_metric)
    * [`DataQualityStabilityMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetric)
      * [`DataQualityStabilityMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetric.calculate)
    * [`DataQualityStabilityMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricRenderer)
      * [`DataQualityStabilityMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricRenderer.color_options)
      * [`DataQualityStabilityMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricRenderer.render_html)
      * [`DataQualityStabilityMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricRenderer.render_json)
    * [`DataQualityStabilityMetricResult`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricResult)
      * [`DataQualityStabilityMetricResult.number_not_stable_prediction`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricResult.number_not_stable_prediction)
      * [`DataQualityStabilityMetricResult.number_not_stable_target`](https://docs-old.evidentlyai.com/reference/api-reference/pages/DuXwyzcKTfPfwXGvgpwg#evidently.metrics.data_quality.stability_metric.DataQualityStabilityMetricResult.number_not_stable_target)
* [evidently.metrics.regression\_performance package](/reference/api-reference/evidently.metrics/evidently.metrics.regression_performance)
  * [Submodules](/reference/api-reference/evidently.metrics/evidently.metrics.regression_performance#submodules)
  * [abs\_perc\_error\_in\_time module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.abs_perc_error_in_time)
    * [`RegressionAbsPercentageErrorPlot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlot)
      * [`RegressionAbsPercentageErrorPlot.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlot.calculate)
    * [`RegressionAbsPercentageErrorPlotRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotRenderer)
      * [`RegressionAbsPercentageErrorPlotRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotRenderer.color_options)
      * [`RegressionAbsPercentageErrorPlotRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotRenderer.render_html)
      * [`RegressionAbsPercentageErrorPlotRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotRenderer.render_json)
    * [`RegressionAbsPercentageErrorPlotResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotResults)
      * [`RegressionAbsPercentageErrorPlotResults.current_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotResults.current_scatter)
      * [`RegressionAbsPercentageErrorPlotResults.reference_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotResults.reference_scatter)
      * [`RegressionAbsPercentageErrorPlotResults.x_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.abs_perc_error_in_time.RegressionAbsPercentageErrorPlotResults.x_name)
  * [error\_bias\_table module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.error_bias_table)
    * [`RegressionErrorBiasTable`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable)
      * [`RegressionErrorBiasTable.TOP_ERROR_DEFAULT`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable.TOP_ERROR_DEFAULT)
      * [`RegressionErrorBiasTable.TOP_ERROR_MAX`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable.TOP_ERROR_MAX)
      * [`RegressionErrorBiasTable.TOP_ERROR_MIN`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable.TOP_ERROR_MIN)
      * [`RegressionErrorBiasTable.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable.calculate)
      * [`RegressionErrorBiasTable.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable.columns)
      * [`RegressionErrorBiasTable.top_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTable.top_error)
    * [`RegressionErrorBiasTableRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableRenderer)
      * [`RegressionErrorBiasTableRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableRenderer.color_options)
      * [`RegressionErrorBiasTableRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableRenderer.render_html)
      * [`RegressionErrorBiasTableRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableRenderer.render_json)
    * [`RegressionErrorBiasTableResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults)
      * [`RegressionErrorBiasTableResults.cat_feature_names`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.cat_feature_names)
      * [`RegressionErrorBiasTableResults.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.columns)
      * [`RegressionErrorBiasTableResults.current_plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.current_plot_data)
      * [`RegressionErrorBiasTableResults.error_bias`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.error_bias)
      * [`RegressionErrorBiasTableResults.num_feature_names`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.num_feature_names)
      * [`RegressionErrorBiasTableResults.prediction_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.prediction_name)
      * [`RegressionErrorBiasTableResults.reference_plot_data`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.reference_plot_data)
      * [`RegressionErrorBiasTableResults.target_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.target_name)
      * [`RegressionErrorBiasTableResults.top_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_bias_table.RegressionErrorBiasTableResults.top_error)
  * [error\_distribution module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.error_distribution)
    * [`RegressionErrorDistribution`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistribution)
      * [`RegressionErrorDistribution.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistribution.calculate)
    * [`RegressionErrorDistributionRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionRenderer)
      * [`RegressionErrorDistributionRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionRenderer.color_options)
      * [`RegressionErrorDistributionRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionRenderer.render_html)
      * [`RegressionErrorDistributionRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionRenderer.render_json)
    * [`RegressionErrorDistributionResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionResults)
      * [`RegressionErrorDistributionResults.current_bins`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionResults.current_bins)
      * [`RegressionErrorDistributionResults.reference_bins`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_distribution.RegressionErrorDistributionResults.reference_bins)
  * [error\_in\_time module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.error_in_time)
    * [`RegressionErrorPlot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlot)
      * [`RegressionErrorPlot.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlot.calculate)
    * [`RegressionErrorPlotRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotRenderer)
      * [`RegressionErrorPlotRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotRenderer.color_options)
      * [`RegressionErrorPlotRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotRenderer.render_html)
      * [`RegressionErrorPlotRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotRenderer.render_json)
    * [`RegressionErrorPlotResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotResults)
      * [`RegressionErrorPlotResults.current_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotResults.current_scatter)
      * [`RegressionErrorPlotResults.reference_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotResults.reference_scatter)
      * [`RegressionErrorPlotResults.x_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_in_time.RegressionErrorPlotResults.x_name)
  * [error\_normality module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.error_normality)
    * [`RegressionErrorNormality`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormality)
      * [`RegressionErrorNormality.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormality.calculate)
    * [`RegressionErrorNormalityRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityRenderer)
      * [`RegressionErrorNormalityRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityRenderer.color_options)
      * [`RegressionErrorNormalityRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityRenderer.render_html)
      * [`RegressionErrorNormalityRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityRenderer.render_json)
    * [`RegressionErrorNormalityResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityResults)
      * [`RegressionErrorNormalityResults.current_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityResults.current_error)
      * [`RegressionErrorNormalityResults.reference_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.error_normality.RegressionErrorNormalityResults.reference_error)
  * [predicted\_and\_actual\_in\_time module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.predicted_and_actual_in_time)
    * [`RegressionPredictedVsActualPlot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlot)
      * [`RegressionPredictedVsActualPlot.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlot.calculate)
    * [`RegressionPredictedVsActualPlotRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotRenderer)
      * [`RegressionPredictedVsActualPlotRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotRenderer.color_options)
      * [`RegressionPredictedVsActualPlotRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotRenderer.render_html)
      * [`RegressionPredictedVsActualPlotRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotRenderer.render_json)
    * [`RegressionPredictedVsActualPlotResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotResults)
      * [`RegressionPredictedVsActualPlotResults.current_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotResults.current_scatter)
      * [`RegressionPredictedVsActualPlotResults.reference_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotResults.reference_scatter)
      * [`RegressionPredictedVsActualPlotResults.x_name`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_and_actual_in_time.RegressionPredictedVsActualPlotResults.x_name)
  * [predicted\_vs\_actual module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.predicted_vs_actual)
    * [`RegressionPredictedVsActualScatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatter)
      * [`RegressionPredictedVsActualScatter.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatter.calculate)
    * [`RegressionPredictedVsActualScatterRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterRenderer)
      * [`RegressionPredictedVsActualScatterRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterRenderer.color_options)
      * [`RegressionPredictedVsActualScatterRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterRenderer.render_html)
      * [`RegressionPredictedVsActualScatterRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterRenderer.render_json)
    * [`RegressionPredictedVsActualScatterResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterResults)
      * [`RegressionPredictedVsActualScatterResults.current_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterResults.current_scatter)
      * [`RegressionPredictedVsActualScatterResults.reference_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.predicted_vs_actual.RegressionPredictedVsActualScatterResults.reference_scatter)
  * [regression\_dummy\_metric module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.regression_dummy_metric)
    * [`RegressionDummyMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetric)
      * [`RegressionDummyMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetric.calculate)
      * [`RegressionDummyMetric.quality_metric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetric.quality_metric)
    * [`RegressionDummyMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricRenderer)
      * [`RegressionDummyMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricRenderer.color_options)
      * [`RegressionDummyMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricRenderer.render_html)
      * [`RegressionDummyMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricRenderer.render_json)
    * [`RegressionDummyMetricResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults)
      * [`RegressionDummyMetricResults.abs_error_max`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.abs_error_max)
      * [`RegressionDummyMetricResults.abs_error_max_by_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.abs_error_max_by_ref)
      * [`RegressionDummyMetricResults.abs_error_max_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.abs_error_max_default)
      * [`RegressionDummyMetricResults.mean_abs_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.mean_abs_error)
      * [`RegressionDummyMetricResults.mean_abs_error_by_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.mean_abs_error_by_ref)
      * [`RegressionDummyMetricResults.mean_abs_error_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.mean_abs_error_default)
      * [`RegressionDummyMetricResults.mean_abs_perc_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.mean_abs_perc_error)
      * [`RegressionDummyMetricResults.mean_abs_perc_error_by_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.mean_abs_perc_error_by_ref)
      * [`RegressionDummyMetricResults.mean_abs_perc_error_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.mean_abs_perc_error_default)
      * [`RegressionDummyMetricResults.rmse`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.rmse)
      * [`RegressionDummyMetricResults.rmse_by_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.rmse_by_ref)
      * [`RegressionDummyMetricResults.rmse_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_dummy_metric.RegressionDummyMetricResults.rmse_default)
  * [regression\_performance\_metrics module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.regression_performance_metrics)
    * [`RegressionPerformanceMetrics`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetrics)
      * [`RegressionPerformanceMetrics.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetrics.calculate)
      * [`RegressionPerformanceMetrics.get_parameters()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetrics.get_parameters)
    * [`RegressionPerformanceMetricsRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsRenderer)
      * [`RegressionPerformanceMetricsRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsRenderer.color_options)
      * [`RegressionPerformanceMetricsRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsRenderer.render_html)
      * [`RegressionPerformanceMetricsRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsRenderer.render_json)
    * [`RegressionPerformanceMetricsResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults)
      * [`RegressionPerformanceMetricsResults.abs_error_max`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.abs_error_max)
      * [`RegressionPerformanceMetricsResults.abs_error_max_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.abs_error_max_default)
      * [`RegressionPerformanceMetricsResults.abs_error_max_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.abs_error_max_ref)
      * [`RegressionPerformanceMetricsResults.abs_error_std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.abs_error_std)
      * [`RegressionPerformanceMetricsResults.abs_perc_error_std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.abs_perc_error_std)
      * [`RegressionPerformanceMetricsResults.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.columns)
      * [`RegressionPerformanceMetricsResults.error_bias`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.error_bias)
      * [`RegressionPerformanceMetricsResults.error_normality`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.error_normality)
      * [`RegressionPerformanceMetricsResults.error_std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.error_std)
      * [`RegressionPerformanceMetricsResults.hist_for_plot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.hist_for_plot)
      * [`RegressionPerformanceMetricsResults.me_default_sigma`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.me_default_sigma)
      * [`RegressionPerformanceMetricsResults.me_hist_for_plot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.me_hist_for_plot)
      * [`RegressionPerformanceMetricsResults.mean_abs_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_abs_error)
      * [`RegressionPerformanceMetricsResults.mean_abs_error_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_abs_error_default)
      * [`RegressionPerformanceMetricsResults.mean_abs_error_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_abs_error_ref)
      * [`RegressionPerformanceMetricsResults.mean_abs_perc_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_abs_perc_error)
      * [`RegressionPerformanceMetricsResults.mean_abs_perc_error_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_abs_perc_error_default)
      * [`RegressionPerformanceMetricsResults.mean_abs_perc_error_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_abs_perc_error_ref)
      * [`RegressionPerformanceMetricsResults.mean_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_error)
      * [`RegressionPerformanceMetricsResults.mean_error_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.mean_error_ref)
      * [`RegressionPerformanceMetricsResults.r2_score`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.r2_score)
      * [`RegressionPerformanceMetricsResults.r2_score_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.r2_score_ref)
      * [`RegressionPerformanceMetricsResults.rmse`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.rmse)
      * [`RegressionPerformanceMetricsResults.rmse_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.rmse_default)
      * [`RegressionPerformanceMetricsResults.rmse_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.rmse_ref)
      * [`RegressionPerformanceMetricsResults.underperformance`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.underperformance)
      * [`RegressionPerformanceMetricsResults.underperformance_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.underperformance_ref)
      * [`RegressionPerformanceMetricsResults.vals_for_plots`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_performance_metrics.RegressionPerformanceMetricsResults.vals_for_plots)
  * [regression\_quality module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.regression_quality)
    * [`RegressionQualityMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetric)
      * [`RegressionQualityMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetric.calculate)
    * [`RegressionQualityMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricRenderer)
      * [`RegressionQualityMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricRenderer.color_options)
      * [`RegressionQualityMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricRenderer.render_html)
      * [`RegressionQualityMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricRenderer.render_json)
    * [`RegressionQualityMetricResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults)
      * [`RegressionQualityMetricResults.abs_error_max`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_error_max)
      * [`RegressionQualityMetricResults.abs_error_max_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_error_max_default)
      * [`RegressionQualityMetricResults.abs_error_max_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_error_max_ref)
      * [`RegressionQualityMetricResults.abs_error_std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_error_std)
      * [`RegressionQualityMetricResults.abs_error_std_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_error_std_ref)
      * [`RegressionQualityMetricResults.abs_perc_error_std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_perc_error_std)
      * [`RegressionQualityMetricResults.abs_perc_error_std_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.abs_perc_error_std_ref)
      * [`RegressionQualityMetricResults.columns`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.columns)
      * [`RegressionQualityMetricResults.error_bias`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.error_bias)
      * [`RegressionQualityMetricResults.error_normality`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.error_normality)
      * [`RegressionQualityMetricResults.error_std`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.error_std)
      * [`RegressionQualityMetricResults.error_std_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.error_std_ref)
      * [`RegressionQualityMetricResults.hist_for_plot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.hist_for_plot)
      * [`RegressionQualityMetricResults.me_default_sigma`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.me_default_sigma)
      * [`RegressionQualityMetricResults.me_hist_for_plot`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.me_hist_for_plot)
      * [`RegressionQualityMetricResults.mean_abs_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_abs_error)
      * [`RegressionQualityMetricResults.mean_abs_error_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_abs_error_default)
      * [`RegressionQualityMetricResults.mean_abs_error_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_abs_error_ref)
      * [`RegressionQualityMetricResults.mean_abs_perc_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_abs_perc_error)
      * [`RegressionQualityMetricResults.mean_abs_perc_error_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_abs_perc_error_default)
      * [`RegressionQualityMetricResults.mean_abs_perc_error_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_abs_perc_error_ref)
      * [`RegressionQualityMetricResults.mean_error`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_error)
      * [`RegressionQualityMetricResults.mean_error_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.mean_error_ref)
      * [`RegressionQualityMetricResults.r2_score`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.r2_score)
      * [`RegressionQualityMetricResults.r2_score_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.r2_score_ref)
      * [`RegressionQualityMetricResults.rmse`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.rmse)
      * [`RegressionQualityMetricResults.rmse_default`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.rmse_default)
      * [`RegressionQualityMetricResults.rmse_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.rmse_ref)
      * [`RegressionQualityMetricResults.underperformance`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.underperformance)
      * [`RegressionQualityMetricResults.underperformance_ref`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.underperformance_ref)
      * [`RegressionQualityMetricResults.vals_for_plots`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.regression_quality.RegressionQualityMetricResults.vals_for_plots)
  * [top\_error module](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#module-evidently.metrics.regression_performance.top_error)
    * [`RegressionTopErrorMetric`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetric)
      * [`RegressionTopErrorMetric.calculate()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetric.calculate)
    * [`RegressionTopErrorMetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricRenderer)
      * [`RegressionTopErrorMetricRenderer.color_options`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricRenderer.color_options)
      * [`RegressionTopErrorMetricRenderer.render_html()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricRenderer.render_html)
      * [`RegressionTopErrorMetricRenderer.render_json()`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricRenderer.render_json)
    * [`RegressionTopErrorMetricResults`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricResults)
      * [`RegressionTopErrorMetricResults.curr_mean_err_per_group`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricResults.curr_mean_err_per_group)
      * [`RegressionTopErrorMetricResults.curr_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricResults.curr_scatter)
      * [`RegressionTopErrorMetricResults.ref_mean_err_per_group`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricResults.ref_mean_err_per_group)
      * [`RegressionTopErrorMetricResults.ref_scatter`](https://docs-old.evidentlyai.com/reference/api-reference/pages/F8VHapGbLd8I3dwCzP4R#evidently.metrics.regression_performance.top_error.RegressionTopErrorMetricResults.ref_scatter)

## Submodules

## base\_metric module <a href="#module-evidently.metrics.base_metric" id="module-evidently.metrics.base_metric"></a>

### class ErrorResult(exception: BaseException)

Bases: `object`

#### Attributes:

&#x20;    **exception : BaseException**

### class InputData(reference\_data: Optional\[pandas.core.frame.DataFrame], current\_data: pandas.core.frame.DataFrame, column\_mapping: [ColumnMapping](https://docs-old.evidentlyai.com/reference/api-reference/pages/D9Oc6VIiKDafnPidUy9b#evidently.pipeline.column_mapping.ColumnMapping), data\_definition: [DataDefinition](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_preprocessing.DataDefinition))

Bases: `object`

#### Attributes:

&#x20;    **column\_mapping :** [**ColumnMapping**](https://docs-old.evidentlyai.com/reference/api-reference/pages/D9Oc6VIiKDafnPidUy9b#evidently.pipeline.column_mapping.ColumnMapping)

&#x20;    **current\_data : DataFrame**

&#x20;    **data\_definition :** [**DataDefinition**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_preprocessing.DataDefinition)

&#x20;    **reference\_data : Optional\[DataFrame]**

### class Metric()

Bases: `Generic`\[`TResult`]

#### Attributes:

&#x20;    **context = None**

#### Methods:

&#x20;    **abstract calculate(data: InputData)**

&#x20;    **get\_id()**

&#x20;    **get\_parameters()**

&#x20;    **get\_result()**

&#x20;    **set\_context(context)**

### generate\_column\_metrics(metric\_class: Type\[Metric], columns: Optional\[Union\[str, list]] = None, parameters: Optional\[Dict] = None, skip\_id\_column: bool = False)

Function for generating metrics for columns

## utils module <a href="#module-evidently.metrics.utils" id="module-evidently.metrics.utils"></a>

### apply\_func\_to\_binned\_data(df\_for\_bins, func, target\_column, preds\_column, is\_ref\_data=False)

### make\_target\_bins\_for\_reg\_plots(curr: DataFrame, target\_column, preds\_column, ref: Optional\[DataFrame] = None)


# evidently.metrics.classification\_performance

## Submodules

## base\_classification\_metric module <a href="#module-evidently.metrics.classification_performance.base_classification_metric" id="module-evidently.metrics.classification_performance.base_classification_metric"></a>

### class ThresholdClassificationMetric(probas\_threshold: Optional\[float], k: Optional\[Union\[float, int]])

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`TResult`], `ABC`

#### Attributes:

&#x20;    **k : Optional\[Union\[float, int]]**

&#x20;    **probas\_threshold : Optional\[float]**

#### Methods:

&#x20;    **get\_target\_prediction\_data(data: DataFrame, column\_mapping:** [**ColumnMapping**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/D9Oc6VIiKDafnPidUy9b#evidently.pipeline.column_mapping.ColumnMapping)**)**

## class\_balance\_metric module <a href="#module-evidently.metrics.classification_performance.class_balance_metric" id="module-evidently.metrics.classification_performance.class_balance_metric"></a>

### class ClassificationClassBalance()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationClassBalanceResult`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ClassificationClassBalanceRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationClassBalance)**

&#x20;    **render\_json(obj: ClassificationClassBalance)**

### class ClassificationClassBalanceResult(plot\_data: Dict\[str, int])

Bases: `object`

#### Attributes:

&#x20;    **plot\_data : Dict\[str, int]**

## class\_separation\_metric module <a href="#module-evidently.metrics.classification_performance.class_separation_metric" id="module-evidently.metrics.classification_performance.class_separation_metric"></a>

### class ClassificationClassSeparationPlot()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationClassSeparationPlotResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ClassificationClassSeparationPlotRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationClassSeparationPlot)**

&#x20;    **render\_json(obj: ClassificationClassSeparationPlot)**

### class ClassificationClassSeparationPlotResults(target\_name: str, current\_plot: Optional\[pandas.core.frame.DataFrame] = None, reference\_plot: Optional\[pandas.core.frame.DataFrame] = None)

Bases: `object`

#### Attributes:

&#x20;    **current\_plot : Optional\[DataFrame] = None**

&#x20;    **reference\_plot : Optional\[DataFrame] = None**

&#x20;    **target\_name : str**

## classification\_dummy\_metric module <a href="#module-evidently.metrics.classification_performance.classification_dummy_metric" id="module-evidently.metrics.classification_performance.classification_dummy_metric"></a>

### class ClassificationDummyMetric(probas\_threshold: Optional\[float] = None, k: Optional\[Union\[float, int]] = None)

Bases: `ThresholdClassificationMetric`\[`ClassificationDummyMetricResults`]

#### Attributes:

&#x20;    **quality\_metric : ClassificationQualityMetric**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **correction\_for\_threshold(dummy\_results:** [**DatasetClassificationQuality**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)**, threshold: float, target: Series, labels: list, probas\_shape: tuple)**

### class ClassificationDummyMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationDummyMetric)**

&#x20;    **render\_json(obj: ClassificationDummyMetric)**

### class ClassificationDummyMetricResults(dummy: [DatasetClassificationQuality](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality), by\_reference\_dummy: Optional\[[DatasetClassificationQuality](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)], model\_quality: Optional\[[DatasetClassificationQuality](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)], metrics\_matrix: dict)

Bases: `object`

#### Attributes:

&#x20;    **by\_reference\_dummy : Optional\[**[**DatasetClassificationQuality**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)**]**

&#x20;    **dummy :** [**DatasetClassificationQuality**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)

&#x20;    **metrics\_matrix : dict**

&#x20;    **model\_quality : Optional\[**[**DatasetClassificationQuality**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)**]**

## classification\_quality\_metric module <a href="#module-evidently.metrics.classification_performance.classification_quality_metric" id="module-evidently.metrics.classification_performance.classification_quality_metric"></a>

### class ClassificationQualityMetric(probas\_threshold: Optional\[float] = None, k: Optional\[Union\[float, int]] = None)

Bases: `ThresholdClassificationMetric`\[`ClassificationQualityMetricResult`]

#### Attributes:

&#x20;    **confusion\_matrix\_metric : ClassificationConfusionMatrix**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ClassificationQualityMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationQualityMetric)**

&#x20;    **render\_json(obj: ClassificationQualityMetric)**

### class ClassificationQualityMetricResult(current: [DatasetClassificationQuality](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality), reference: Optional\[[DatasetClassificationQuality](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)], target\_name: str)

Bases: `object`

#### Attributes:

&#x20;    **current :** [**DatasetClassificationQuality**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)

&#x20;    **reference : Optional\[**[**DatasetClassificationQuality**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.DatasetClassificationQuality)**]**

&#x20;    **target\_name : str**

## confusion\_matrix\_metric module <a href="#module-evidently.metrics.classification_performance.confusion_matrix_metric" id="module-evidently.metrics.classification_performance.confusion_matrix_metric"></a>

### class ClassificationConfusionMatrix(probas\_threshold: Optional\[float] = None, k: Optional\[Union\[float, int]] = None)

Bases: `ThresholdClassificationMetric`\[`ClassificationConfusionMatrixResult`]

#### Attributes:

&#x20;    **k : Optional\[Union\[float, int]]**

&#x20;    **probas\_threshold : Optional\[float]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ClassificationConfusionMatrixRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationConfusionMatrix)**

&#x20;    **render\_json(obj: ClassificationConfusionMatrix)**

### class ClassificationConfusionMatrixResult(current\_matrix: [ConfusionMatrix](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.ConfusionMatrix), reference\_matrix: Optional\[[ConfusionMatrix](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.ConfusionMatrix)])

Bases: `object`

#### Attributes:

&#x20;    **current\_matrix :** [**ConfusionMatrix**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.ConfusionMatrix)

&#x20;    **reference\_matrix : Optional\[**[**ConfusionMatrix**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.ConfusionMatrix)**]**

## pr\_curve\_metric module <a href="#module-evidently.metrics.classification_performance.pr_curve_metric" id="module-evidently.metrics.classification_performance.pr_curve_metric"></a>

### class ClassificationPRCurve()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationPRCurveResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **calculate\_metrics(target\_data: Series, prediction:** [**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)**)**

### class ClassificationPRCurveRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationPRCurve)**

&#x20;    **render\_json(obj: ClassificationPRCurve)**

### class ClassificationPRCurveResults(current\_pr\_curve: Optional\[dict] = None, reference\_pr\_curve: Optional\[dict] = None)

Bases: `object`

#### Attributes:

&#x20;    **current\_pr\_curve : Optional\[dict] = None**

&#x20;    **reference\_pr\_curve : Optional\[dict] = None**

## pr\_table\_metric module <a href="#module-evidently.metrics.classification_performance.pr_table_metric" id="module-evidently.metrics.classification_performance.pr_table_metric"></a>

### class ClassificationPRTable()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationPRTableResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **calculate\_metrics(target\_data: Series, prediction:** [**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)**)**

### class ClassificationPRTableRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationPRTable)**

&#x20;    **render\_json(obj: ClassificationPRTable)**

### class ClassificationPRTableResults(current\_pr\_table: Optional\[dict] = None, reference\_pr\_table: Optional\[dict] = None)

Bases: `object`

#### Attributes:

&#x20;    **current\_pr\_table : Optional\[dict] = None**

&#x20;    **reference\_pr\_table : Optional\[dict] = None**

## probability\_distribution\_metric module <a href="#module-evidently.metrics.classification_performance.probability_distribution_metric" id="module-evidently.metrics.classification_performance.probability_distribution_metric"></a>

### class ClassificationProbDistribution()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationProbDistributionResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **static get\_distribution(dataset: DataFrame, target\_name: str, prediction\_labels: Iterable)**

### class ClassificationProbDistributionRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationProbDistribution)**

&#x20;    **render\_json(obj: ClassificationProbDistribution)**

### class ClassificationProbDistributionResults(current\_distribution: Optional\[Dict\[str, list]], reference\_distribution: Optional\[Dict\[str, list]])

Bases: `object`

#### Attributes:

&#x20;    **current\_distribution : Optional\[Dict\[str, list]]**

&#x20;    **reference\_distribution : Optional\[Dict\[str, list]]**

## quality\_by\_class\_metric module <a href="#module-evidently.metrics.classification_performance.quality_by_class_metric" id="module-evidently.metrics.classification_performance.quality_by_class_metric"></a>

### class ClassificationQualityByClass(probas\_threshold: Optional\[float] = None, k: Optional\[Union\[float, int]] = None)

Bases: `ThresholdClassificationMetric`\[`ClassificationQualityByClassResult`]

#### Attributes:

&#x20;    **k : Optional\[Union\[float, int]]**

&#x20;    **probas\_threshold : Optional\[float]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ClassificationQualityByClassRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationQualityByClass)**

&#x20;    **render\_json(obj: ClassificationQualityByClass)**

### class ClassificationQualityByClassResult(columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), current\_metrics: dict, current\_roc\_aucs: Optional\[list], reference\_metrics: Optional\[dict], reference\_roc\_aucs: Optional\[dict])

Bases: `object`

#### Attributes:

&#x20;    **columns :** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)

&#x20;    **current\_metrics : dict**

&#x20;    **current\_roc\_aucs : Optional\[list]**

&#x20;    **reference\_metrics : Optional\[dict]**

&#x20;    **reference\_roc\_aucs : Optional\[dict]**

## quality\_by\_feature\_table module <a href="#module-evidently.metrics.classification_performance.quality_by_feature_table" id="module-evidently.metrics.classification_performance.quality_by_feature_table"></a>

### class ClassificationQualityByFeatureTable(columns: Optional\[List\[str]] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationQualityByFeatureTableResults`]

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ClassificationQualityByFeatureTableRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationQualityByFeatureTable)**

&#x20;    **render\_json(obj: ClassificationQualityByFeatureTable)**

### class ClassificationQualityByFeatureTableResults(current\_plot\_data: pandas.core.frame.DataFrame, reference\_plot\_data: Optional\[pandas.core.frame.DataFrame], target\_name: str, curr\_predictions: [PredictionData](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData), ref\_predictions: Optional\[[PredictionData](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)], columns: List\[str])

Bases: `object`

#### Attributes:

&#x20;    **columns : List\[str]**

&#x20;    **curr\_predictions :** [**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)

&#x20;    **current\_plot\_data : DataFrame**

&#x20;    **ref\_predictions : Optional\[**[**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)**]**

&#x20;    **reference\_plot\_data : Optional\[DataFrame]**

&#x20;    **target\_name : str**

## roc\_curve\_metric module <a href="#module-evidently.metrics.classification_performance.roc_curve_metric" id="module-evidently.metrics.classification_performance.roc_curve_metric"></a>

### class ClassificationRocCurve()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ClassificationRocCurveResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **calculate\_metrics(target\_data: Series, prediction:** [**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)**)**

### class ClassificationRocCurveRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ClassificationRocCurve)**

&#x20;    **render\_json(obj: ClassificationRocCurve)**

### class ClassificationRocCurveResults(current\_roc\_curve: Optional\[dict] = None, reference\_roc\_curve: Optional\[dict] = None)

Bases: `object`

#### Attributes:

&#x20;    **current\_roc\_curve : Optional\[dict] = None**

&#x20;    **reference\_roc\_curve : Optional\[dict] = None**


# evidently.metrics.data\_drift

## Submodules

## column\_drift\_metric module <a href="#module-evidently.metrics.data_drift.column_drift_metric" id="module-evidently.metrics.data_drift.column_drift_metric"></a>

### class ColumnDriftMetric(column\_name: str, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, stattest\_threshold: Optional\[float] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnDriftMetricResults`]

Calculate drift metric for a column

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **stattest\_threshold : Optional\[float]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **get\_parameters()**

### class ColumnDriftMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnDriftMetric)**

&#x20;    **render\_json(obj: ColumnDriftMetric)**

### class ColumnDriftMetricResults(column\_name: str, column\_type: str, stattest\_name: str, stattest\_threshold: float, drift\_score: Union\[float, int], drift\_detected: bool, current\_distribution: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), reference\_distribution: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), current\_scatter: Optional\[Dict\[str, list]], x\_name: Optional\[str], plot\_shape: Optional\[Dict\[str, float]])

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **column\_type : str**

&#x20;    **current\_distribution :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **current\_scatter : Optional\[Dict\[str, list]]**

&#x20;    **drift\_detected : bool**

&#x20;    **drift\_score : Union\[float, int]**

&#x20;    **plot\_shape : Optional\[Dict\[str, float]]**

&#x20;    **reference\_distribution :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **stattest\_name : str**

&#x20;    **stattest\_threshold : float**

&#x20;    **x\_name : Optional\[str]**

## column\_value\_plot module <a href="#module-evidently.metrics.data_drift.column_value_plot" id="module-evidently.metrics.data_drift.column_value_plot"></a>

### class ColumnValuePlot(column\_name: str)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnValuePlotResults`]

#### Attributes:

&#x20;    **column\_name : str**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnValuePlotRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnValuePlot)**

&#x20;    **render\_json(obj: ColumnValuePlot)**

### class ColumnValuePlotResults(column\_name: str, datetime\_column\_name: Optional\[str], current\_scatter: pandas.core.frame.DataFrame, reference\_scatter: pandas.core.frame.DataFrame)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current\_scatter : DataFrame**

&#x20;    **datetime\_column\_name : Optional\[str]**

&#x20;    **reference\_scatter : DataFrame**

## data\_drift\_table module <a href="#module-evidently.metrics.data_drift.data_drift_table" id="module-evidently.metrics.data_drift.data_drift_table"></a>

### class DataDriftTable(columns: Optional\[List\[str]] = None, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_column\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, stattest\_threshold: Optional\[float] = None, cat\_stattest\_threshold: Optional\[float] = None, num\_stattest\_threshold: Optional\[float] = None, per\_column\_stattest\_threshold: Optional\[Dict\[str, float]] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DataDriftTableResults`]

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **options :** [**DataDriftOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.data_drift.DataDriftOptions)

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **get\_parameters()**

### class DataDriftTableRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: DataDriftTable)**

&#x20;    **render\_json(obj: DataDriftTable)**

### class DataDriftTableResults(number\_of\_columns: int, number\_of\_drifted\_columns: int, share\_of\_drifted\_columns: float, dataset\_drift: bool, drift\_by\_columns: Dict\[str, [ColumnDataDriftMetrics](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_drift.ColumnDataDriftMetrics)], dataset\_columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns))

Bases: `object`

#### Attributes:

&#x20;    **dataset\_columns :** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)

&#x20;    **dataset\_drift : bool**

&#x20;    **drift\_by\_columns : Dict\[str,** [**ColumnDataDriftMetrics**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_drift.ColumnDataDriftMetrics)**]**

&#x20;    **number\_of\_columns : int**

&#x20;    **number\_of\_drifted\_columns : int**

&#x20;    **share\_of\_drifted\_columns : float**

## dataset\_drift\_metric module <a href="#module-evidently.metrics.data_drift.dataset_drift_metric" id="module-evidently.metrics.data_drift.dataset_drift_metric"></a>

### class DataDriftMetricsRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: DatasetDriftMetric)**

&#x20;    **render\_json(obj: DatasetDriftMetric)**

### class DatasetDriftMetric(columns: Optional\[List\[str]] = None, drift\_share: float = 0.5, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_column\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, stattest\_threshold: Optional\[float] = None, cat\_stattest\_threshold: Optional\[float] = None, num\_stattest\_threshold: Optional\[float] = None, per\_column\_stattest\_threshold: Optional\[Dict\[str, float]] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DatasetDriftMetricResults`]

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **drift\_share : float**

&#x20;    **options :** [**DataDriftOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.data_drift.DataDriftOptions)

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **get\_parameters()**

### class DatasetDriftMetricResults(drift\_share: float, number\_of\_columns: int, number\_of\_drifted\_columns: int, share\_of\_drifted\_columns: float, dataset\_drift: bool)

Bases: `object`

#### Attributes:

&#x20;    **dataset\_drift : bool**

&#x20;    **drift\_share : float**

&#x20;    **number\_of\_columns : int**

&#x20;    **number\_of\_drifted\_columns : int**

&#x20;    **share\_of\_drifted\_columns : float**

## target\_by\_features\_table module <a href="#module-evidently.metrics.data_drift.target_by_features_table" id="module-evidently.metrics.data_drift.target_by_features_table"></a>

### class TargetByFeaturesTable(columns: Optional\[List\[str]] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`TargetByFeaturesTableResults`]

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class TargetByFeaturesTableRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: TargetByFeaturesTable)**

&#x20;    **render\_json(obj: TargetByFeaturesTable)**

### class TargetByFeaturesTableResults(current\_plot\_data: pandas.core.frame.DataFrame, reference\_plot\_data: pandas.core.frame.DataFrame, target\_name: Optional\[str], curr\_predictions: Optional\[[PredictionData](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)], ref\_predictions: Optional\[[PredictionData](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)], columns: List\[str], task: str)

Bases: `object`

#### Attributes:

&#x20;    **columns : List\[str]**

&#x20;    **curr\_predictions : Optional\[**[**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)**]**

&#x20;    **current\_plot\_data : DataFrame**

&#x20;    **ref\_predictions : Optional\[**[**PredictionData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.classification_performance.PredictionData)**]**

&#x20;    **reference\_plot\_data : DataFrame**

&#x20;    **target\_name : Optional\[str]**

&#x20;    **task : str**


# evidently.metrics.data\_integrity

## Submodules

## column\_missing\_values\_metric module <a href="#module-evidently.metrics.data_integrity.column_missing_values_metric" id="module-evidently.metrics.data_integrity.column_missing_values_metric"></a>

### class ColumnMissingValues(number\_of\_rows: int, different\_missing\_values: Dict\[Any, int], number\_of\_different\_missing\_values: int, number\_of\_missing\_values: int, share\_of\_missing\_values: float)

Bases: `object`

Statistics about missing values in a column

#### Attributes:

&#x20;    **different\_missing\_values : Dict\[Any, int]**

&#x20;    **number\_of\_different\_missing\_values : int**

&#x20;    **number\_of\_missing\_values : int**

&#x20;    **number\_of\_rows : int**

&#x20;    **share\_of\_missing\_values : float**

### class ColumnMissingValuesMetric(column\_name: str, missing\_values: Optional\[list] = None, replace: bool = True)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnMissingValuesMetricResult`]

Count missing values in a column.

Missing value is a null or NaN value.

Calculate an amount of missing values kinds and count for such values.\
NA-types like numpy.NaN, pandas.NaT are counted as one type.

You can set you own missing values list with missing\_values parameter.\
Value None in the list means that Pandas null values will be included in the calculation.

If replace parameter is False - add defaults to user’s list.\
If replace parameter is True - use values from missing\_values list only.

#### Attributes:

&#x20;    **DEFAULT\_MISSING\_VALUES = \['', inf, -inf, None]**

&#x20;    **column\_name : str**

&#x20;    **missing\_values : frozenset**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnMissingValuesMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnMissingValuesMetric)**

&#x20;    **render\_json(obj: ColumnMissingValuesMetric)**

### class ColumnMissingValuesMetricResult(column\_name: str, current: ColumnMissingValues, reference: Optional\[ColumnMissingValues] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current : ColumnMissingValues**

&#x20;    **reference : Optional\[ColumnMissingValues] = None**

## column\_regexp\_metric module <a href="#module-evidently.metrics.data_integrity.column_regexp_metric" id="module-evidently.metrics.data_integrity.column_regexp_metric"></a>

### class ColumnRegExpMetric(column\_name: str, reg\_exp: str, top: int = 10)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DataIntegrityValueByRegexpMetricResult`]

Count number of values in a column matched or not by a regular expression (regexp)

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **reg\_exp : str**

&#x20;    **top : int**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnRegExpMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnRegExpMetric)**

&#x20;    **render\_json(obj: ColumnRegExpMetric)**

### class DataIntegrityValueByRegexpMetricResult(column\_name: str, reg\_exp: str, top: int, current: DataIntegrityValueByRegexpStat, reference: Optional\[DataIntegrityValueByRegexpStat] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current : DataIntegrityValueByRegexpStat**

&#x20;    **reference : Optional\[DataIntegrityValueByRegexpStat] = None**

&#x20;    **reg\_exp : str**

&#x20;    **top : int**

### class DataIntegrityValueByRegexpStat(number\_of\_matched: int, number\_of\_not\_matched: int, number\_of\_rows: int, table\_of\_matched: Dict\[str, int], table\_of\_not\_matched: Dict\[str, int])

Bases: `object`

Statistics about matched by a regular expression values in a column for one dataset

#### Attributes:

&#x20;    **number\_of\_matched : int**

&#x20;    **number\_of\_not\_matched : int**

&#x20;    **number\_of\_rows : int**

&#x20;    **table\_of\_matched : Dict\[str, int]**

&#x20;    **table\_of\_not\_matched : Dict\[str, int]**

## column\_summary\_metric module <a href="#module-evidently.metrics.data_integrity.column_summary_metric" id="module-evidently.metrics.data_integrity.column_summary_metric"></a>

### class CategoricalCharacteristics(number\_of\_rows: int, count: int, unique: Optional\[int], unique\_percentage: Optional\[float], most\_common: Optional\[object], most\_common\_percentage: Optional\[float], missing: Optional\[int], missing\_percentage: Optional\[float], new\_in\_current\_values\_count: Optional\[int] = None, unused\_in\_current\_values\_count: Optional\[int] = None)

Bases: `object`

#### Attributes:

&#x20;    **count : int**

&#x20;    **missing : Optional\[int]**

&#x20;    **missing\_percentage : Optional\[float]**

&#x20;    **most\_common : Optional\[object]**

&#x20;    **most\_common\_percentage : Optional\[float]**

&#x20;    **new\_in\_current\_values\_count : Optional\[int] = None**

&#x20;    **number\_of\_rows : int**

&#x20;    **unique : Optional\[int]**

&#x20;    **unique\_percentage : Optional\[float]**

&#x20;    **unused\_in\_current\_values\_count : Optional\[int] = None**

### class ColumnSummary(column\_name: str, column\_type: str, reference\_characteristics: Union\[NumericCharacteristics, CategoricalCharacteristics, DatetimeCharacteristics, NoneType], current\_characteristics: Union\[NumericCharacteristics, CategoricalCharacteristics, DatetimeCharacteristics], plot\_data: DataQualityPlot)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **column\_type : str**

&#x20;    **current\_characteristics : Union\[NumericCharacteristics, CategoricalCharacteristics, DatetimeCharacteristics]**

&#x20;    **plot\_data : DataQualityPlot**

&#x20;    **reference\_characteristics : Optional\[Union\[NumericCharacteristics, CategoricalCharacteristics, DatetimeCharacteristics]]**

### class ColumnSummaryMetric(column\_name: str)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnSummary`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **static map\_data(stats:** [**FeatureQualityStats**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_quality.FeatureQualityStats)**)**

### class ColumnSummaryMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnSummaryMetric)**

&#x20;    **render\_json(obj: ColumnSummaryMetric)**

### class DataByTarget(data\_for\_plots: Dict\[str, Dict\[str, Union\[list, pandas.core.frame.DataFrame]]], target\_name: str, target\_type: str)

Bases: `object`

#### Attributes:

&#x20;    **data\_for\_plots : Dict\[str, Dict\[str, Union\[list, DataFrame]]]**

&#x20;    **target\_name : str**

&#x20;    **target\_type : str**

### class DataInTime(data\_for\_plots: Dict\[str, pandas.core.frame.DataFrame], freq: str, datetime\_name: str)

Bases: `object`

#### Attributes:

&#x20;    **data\_for\_plots : Dict\[str, DataFrame]**

&#x20;    **datetime\_name : str**

&#x20;    **freq : str**

### class DataQualityPlot(bins\_for\_hist: Dict\[str, pandas.core.frame.DataFrame], data\_in\_time: Optional\[DataInTime], data\_by\_target: Optional\[DataByTarget], counts\_of\_values: Optional\[Dict\[str, pandas.core.frame.DataFrame]])

Bases: `object`

#### Attributes:

&#x20;    **bins\_for\_hist : Dict\[str, DataFrame]**

&#x20;    **counts\_of\_values : Optional\[Dict\[str, DataFrame]]**

&#x20;    **data\_by\_target : Optional\[DataByTarget]**

&#x20;    **data\_in\_time : Optional\[DataInTime]**

### class DatetimeCharacteristics(number\_of\_rows: int, count: int, unique: Optional\[int], unique\_percentage: Optional\[float], most\_common: Optional\[object], most\_common\_percentage: Optional\[float], missing: Optional\[int], missing\_percentage: Optional\[float], first: Optional\[str], last: Optional\[str])

Bases: `object`

#### Attributes:

&#x20;    **count : int**

&#x20;    **first : Optional\[str]**

&#x20;    **last : Optional\[str]**

&#x20;    **missing : Optional\[int]**

&#x20;    **missing\_percentage : Optional\[float]**

&#x20;    **most\_common : Optional\[object]**

&#x20;    **most\_common\_percentage : Optional\[float]**

&#x20;    **number\_of\_rows : int**

&#x20;    **unique : Optional\[int]**

&#x20;    **unique\_percentage : Optional\[float]**

### class NumericCharacteristics(number\_of\_rows: int, count: int, mean: Union\[float, int, NoneType], std: Union\[float, int, NoneType], min: Union\[float, int, NoneType], p25: Union\[float, int, NoneType], p50: Union\[float, int, NoneType], p75: Union\[float, int, NoneType], max: Union\[float, int, NoneType], unique: Optional\[int], unique\_percentage: Optional\[float], missing: Optional\[int], missing\_percentage: Optional\[float], infinite\_count: Optional\[int], infinite\_percentage: Optional\[float], most\_common: Union\[float, int, NoneType], most\_common\_percentage: Optional\[float])

Bases: `object`

#### Attributes:

&#x20;    **count : int**

&#x20;    **infinite\_count : Optional\[int]**

&#x20;    **infinite\_percentage : Optional\[float]**

&#x20;    **max : Optional\[Union\[float, int]]**

&#x20;    **mean : Optional\[Union\[float, int]]**

&#x20;    **min : Optional\[Union\[float, int]]**

&#x20;    **missing : Optional\[int]**

&#x20;    **missing\_percentage : Optional\[float]**

&#x20;    **most\_common : Optional\[Union\[float, int]]**

&#x20;    **most\_common\_percentage : Optional\[float]**

&#x20;    **number\_of\_rows : int**

&#x20;    **p25 : Optional\[Union\[float, int]]**

&#x20;    **p50 : Optional\[Union\[float, int]]**

&#x20;    **p75 : Optional\[Union\[float, int]]**

&#x20;    **std : Optional\[Union\[float, int]]**

&#x20;    **unique : Optional\[int]**

&#x20;    **unique\_percentage : Optional\[float]**

## dataset\_missing\_values\_metric module <a href="#module-evidently.metrics.data_integrity.dataset_missing_values_metric" id="module-evidently.metrics.data_integrity.dataset_missing_values_metric"></a>

### class DatasetMissingValues(different\_missing\_values: Dict\[Any, int], number\_of\_different\_missing\_values: int, different\_missing\_values\_by\_column: Dict\[str, Dict\[Any, int]], number\_of\_different\_missing\_values\_by\_column: Dict\[str, int], number\_of\_missing\_values: int, share\_of\_missing\_values: float, number\_of\_missing\_values\_by\_column: Dict\[str, int], share\_of\_missing\_values\_by\_column: Dict\[str, float], number\_of\_rows: int, number\_of\_rows\_with\_missing\_values: int, share\_of\_rows\_with\_missing\_values: float, number\_of\_columns: int, columns\_with\_missing\_values: List\[str], number\_of\_columns\_with\_missing\_values: int, share\_of\_columns\_with\_missing\_values: float)

Bases: `object`

Statistics about missed values in a dataset

#### Attributes:

&#x20;    **columns\_with\_missing\_values : List\[str]**

&#x20;    **different\_missing\_values : Dict\[Any, int]**

&#x20;    **different\_missing\_values\_by\_column : Dict\[str, Dict\[Any, int]]**

&#x20;    **number\_of\_columns : int**

&#x20;    **number\_of\_columns\_with\_missing\_values : int**

&#x20;    **number\_of\_different\_missing\_values : int**

&#x20;    **number\_of\_different\_missing\_values\_by\_column : Dict\[str, int]**

&#x20;    **number\_of\_missing\_values : int**

&#x20;    **number\_of\_missing\_values\_by\_column : Dict\[str, int]**

&#x20;    **number\_of\_rows : int**

&#x20;    **number\_of\_rows\_with\_missing\_values : int**

&#x20;    **share\_of\_columns\_with\_missing\_values : float**

&#x20;    **share\_of\_missing\_values : float**

&#x20;    **share\_of\_missing\_values\_by\_column : Dict\[str, float]**

&#x20;    **share\_of\_rows\_with\_missing\_values : float**

### class DatasetMissingValuesMetric(missing\_values: Optional\[list] = None, replace: bool = True)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DatasetMissingValuesMetricResult`]

Count missing values in a dataset.

Missing value is a null or NaN value.

Calculate an amount of missing values kinds and count for such values.\
NA-types like numpy.NaN, pandas.NaT are counted as one type.

You can set you own missing values list with missing\_values parameter.\
Value None in the list means that Pandas null values will be included in the calculation.

If replace parameter is False - add defaults to user’s list.\
If replace parameter is True - use values from missing\_values list only.

#### Attributes:

&#x20;    **DEFAULT\_MISSING\_VALUES = \['', inf, -inf, None]**

&#x20;    **missing\_values : frozenset**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class DatasetMissingValuesMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: DatasetMissingValuesMetric)**

&#x20;    **render\_json(obj: DatasetMissingValuesMetric)**

### class DatasetMissingValuesMetricResult(current: DatasetMissingValues, reference: Optional\[DatasetMissingValues] = None)

Bases: `object`

#### Attributes:

&#x20;    **current : DatasetMissingValues**

&#x20;    **reference : Optional\[DatasetMissingValues] = None**

## dataset\_summary\_metric module <a href="#module-evidently.metrics.data_integrity.dataset_summary_metric" id="module-evidently.metrics.data_integrity.dataset_summary_metric"></a>

### class DatasetSummary(target: Optional\[str], prediction: Optional\[Union\[str, Sequence\[str]]], date\_column: Optional\[str], id\_column: Optional\[str], number\_of\_columns: int, number\_of\_rows: int, number\_of\_missing\_values: int, number\_of\_categorical\_columns: int, number\_of\_numeric\_columns: int, number\_of\_datetime\_columns: int, number\_of\_constant\_columns: int, number\_of\_almost\_constant\_columns: int, number\_of\_duplicated\_columns: int, number\_of\_almost\_duplicated\_columns: int, number\_of\_empty\_rows: int, number\_of\_empty\_columns: int, number\_of\_duplicated\_rows: int, columns\_type: dict, nans\_by\_columns: dict, number\_uniques\_by\_columns: dict)

Bases: `object`

Columns information in a dataset

#### Attributes:

&#x20;    **columns\_type : dict**

&#x20;    **date\_column : Optional\[str]**

&#x20;    **id\_column : Optional\[str]**

&#x20;    **nans\_by\_columns : dict**

&#x20;    **number\_of\_almost\_constant\_columns : int**

&#x20;    **number\_of\_almost\_duplicated\_columns : int**

&#x20;    **number\_of\_categorical\_columns : int**

&#x20;    **number\_of\_columns : int**

&#x20;    **number\_of\_constant\_columns : int**

&#x20;    **number\_of\_datetime\_columns : int**

&#x20;    **number\_of\_duplicated\_columns : int**

&#x20;    **number\_of\_duplicated\_rows : int**

&#x20;    **number\_of\_empty\_columns : int**

&#x20;    **number\_of\_empty\_rows : int**

&#x20;    **number\_of\_missing\_values : int**

&#x20;    **number\_of\_numeric\_columns : int**

&#x20;    **number\_of\_rows : int**

&#x20;    **number\_uniques\_by\_columns : dict**

&#x20;    **prediction : Optional\[Union\[str, Sequence\[str]]]**

&#x20;    **target : Optional\[str]**

### class DatasetSummaryMetric(almost\_duplicated\_threshold: float = 0.95, almost\_constant\_threshold: float = 0.95)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DatasetSummaryMetricResult`]

Common dataset(s) columns/features characteristics

#### Attributes:

&#x20;    **almost\_constant\_threshold : float**

&#x20;    **almost\_duplicated\_threshold : float**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class DatasetSummaryMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: DatasetSummaryMetric)**

&#x20;    **render\_json(obj: DatasetSummaryMetric)**

### class DatasetSummaryMetricResult(almost\_duplicated\_threshold: float, current: DatasetSummary, reference: Optional\[DatasetSummary] = None)

Bases: `object`

#### Attributes:

&#x20;    **almost\_duplicated\_threshold : float**

&#x20;    **current : DatasetSummary**

&#x20;    **reference : Optional\[DatasetSummary] = None**


# evidently.metrics.data\_quality

## Submodules

## column\_correlations\_metric module <a href="#module-evidently.metrics.data_quality.column_correlations_metric" id="module-evidently.metrics.data_quality.column_correlations_metric"></a>

### class ColumnCorrelationsMetric(column\_name: str)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnCorrelationsMetricResult`]

Calculates correlations between the selected column and all the other columns.\
In the current and reference (if presented) datasets

#### Attributes:

&#x20;    **column\_name : str**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnCorrelationsMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnCorrelationsMetric)**

&#x20;    **render\_json(obj: ColumnCorrelationsMetric)**

### class ColumnCorrelationsMetricResult(column\_name: str, current: Dict\[str, [ColumnCorrelations](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_quality.ColumnCorrelations)], reference: Optional\[Dict\[str, [ColumnCorrelations](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_quality.ColumnCorrelations)]] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current : Dict\[str,** [**ColumnCorrelations**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_quality.ColumnCorrelations)**]**

&#x20;    **reference : Optional\[Dict\[str,** [**ColumnCorrelations**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ICxNS1GschXEUCV60dnZ#evidently.calculations.data_quality.ColumnCorrelations)**]] = None**

## column\_distribution\_metric module <a href="#module-evidently.metrics.data_quality.column_distribution_metric" id="module-evidently.metrics.data_quality.column_distribution_metric"></a>

### class ColumnDistributionMetric(column\_name: str)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnDistributionMetricResult`]

Calculates distribution for the column

#### Attributes:

&#x20;    **column\_name : str**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnDistributionMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnDistributionMetric)**

&#x20;    **render\_json(obj: ColumnDistributionMetric)**

### class ColumnDistributionMetricResult(column\_name: str, current: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), reference: Optional\[[Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **reference : Optional\[**[**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)**] = None**

## column\_quantile\_metric module <a href="#module-evidently.metrics.data_quality.column_quantile_metric" id="module-evidently.metrics.data_quality.column_quantile_metric"></a>

### class ColumnQuantileMetric(column\_name: str, quantile: float)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnQuantileMetricResult`]

Calculates quantile with specified range

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **quantile : float**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnQuantileMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnQuantileMetric)**

&#x20;    **render\_json(obj: ColumnQuantileMetric)**

### class ColumnQuantileMetricResult(column\_name: str, quantile: float, current: float, current\_distribution: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), reference: Optional\[float] = None, reference\_distribution: Optional\[[Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current : float**

&#x20;    **current\_distribution :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **quantile : float**

&#x20;    **reference : Optional\[float] = None**

&#x20;    **reference\_distribution : Optional\[**[**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)**] = None**

## column\_value\_list\_metric module <a href="#module-evidently.metrics.data_quality.column_value_list_metric" id="module-evidently.metrics.data_quality.column_value_list_metric"></a>

### class ColumnValueListMetric(column\_name: str, values: Optional\[list] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnValueListMetricResult`]

Calculates count and shares of values in the predefined values list

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **values : Optional\[list]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnValueListMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnValueListMetric)**

&#x20;    **render\_json(obj: ColumnValueListMetric)**

### class ColumnValueListMetricResult(column\_name: str, values: List\[Any], current: ValueListStat, reference: Optional\[ValueListStat] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current : ValueListStat**

&#x20;    **reference : Optional\[ValueListStat] = None**

&#x20;    **values : List\[Any]**

### class ValueListStat(number\_in\_list: int, number\_not\_in\_list: int, share\_in\_list: float, share\_not\_in\_list: float, values\_in\_list: Dict\[Any, int], values\_not\_in\_list: Dict\[Any, int], rows\_count: int)

Bases: `object`

#### Attributes:

&#x20;    **number\_in\_list : int**

&#x20;    **number\_not\_in\_list : int**

&#x20;    **rows\_count : int**

&#x20;    **share\_in\_list : float**

&#x20;    **share\_not\_in\_list : float**

&#x20;    **values\_in\_list : Dict\[Any, int]**

&#x20;    **values\_not\_in\_list : Dict\[Any, int]**

## column\_value\_range\_metric module <a href="#module-evidently.metrics.data_quality.column_value_range_metric" id="module-evidently.metrics.data_quality.column_value_range_metric"></a>

### class ColumnValueRangeMetric(column\_name: str, left: Optional\[Union\[float, int]] = None, right: Optional\[Union\[float, int]] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`ColumnValueRangeMetricResult`]

Calculates count and shares of values in the predefined values range

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **left : Optional\[Union\[float, int]]**

&#x20;    **right : Optional\[Union\[float, int]]**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class ColumnValueRangeMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: ColumnValueRangeMetric)**

&#x20;    **render\_json(obj: ColumnValueRangeMetric)**

### class ColumnValueRangeMetricResult(column\_name: str, left: Union\[float, int], right: Union\[float, int], current: ValuesInRangeStat, current\_distribution: [Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution), reference: Optional\[ValuesInRangeStat] = None, reference\_distribution: Optional\[[Distribution](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)] = None)

Bases: `object`

#### Attributes:

&#x20;    **column\_name : str**

&#x20;    **current : ValuesInRangeStat**

&#x20;    **current\_distribution :** [**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)

&#x20;    **left : Union\[float, int]**

&#x20;    **reference : Optional\[ValuesInRangeStat] = None**

&#x20;    **reference\_distribution : Optional\[**[**Distribution**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.visualizations.Distribution)**] = None**

&#x20;    **right : Union\[float, int]**

### class ValuesInRangeStat(number\_in\_range: int, number\_not\_in\_range: int, share\_in\_range: float, share\_not\_in\_range: float, number\_of\_values: int)

Bases: `object`

#### Attributes:

&#x20;    **number\_in\_range : int**

&#x20;    **number\_not\_in\_range : int**

&#x20;    **number\_of\_values : int**

&#x20;    **share\_in\_range : float**

&#x20;    **share\_not\_in\_range : float**

## dataset\_correlations\_metric module <a href="#module-evidently.metrics.data_quality.dataset_correlations_metric" id="module-evidently.metrics.data_quality.dataset_correlations_metric"></a>

### class CorrelationStats(target\_prediction\_correlation: Optional\[float] = None, abs\_max\_target\_features\_correlation: Optional\[float] = None, abs\_max\_prediction\_features\_correlation: Optional\[float] = None, abs\_max\_correlation: Optional\[float] = None, abs\_max\_features\_correlation: Optional\[float] = None)

Bases: `object`

#### Attributes:

&#x20;    **abs\_max\_correlation : Optional\[float] = None**

&#x20;    **abs\_max\_features\_correlation : Optional\[float] = None**

&#x20;    **abs\_max\_prediction\_features\_correlation : Optional\[float] = None**

&#x20;    **abs\_max\_target\_features\_correlation : Optional\[float] = None**

&#x20;    **target\_prediction\_correlation : Optional\[float] = None**

### class DataQualityCorrelationMetricsRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: DatasetCorrelationsMetric)**

&#x20;    **render\_json(obj: DatasetCorrelationsMetric)**

### class DatasetCorrelation(correlation: Dict\[str, pandas.core.frame.DataFrame], stats: Dict\[str, CorrelationStats])

Bases: `object`

#### Attributes:

&#x20;    **correlation : Dict\[str, DataFrame]**

&#x20;    **stats : Dict\[str, CorrelationStats]**

### class DatasetCorrelationsMetric()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DatasetCorrelationsMetricResult`]

Calculate different correlations with target, predictions and features

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class DatasetCorrelationsMetricResult(current: DatasetCorrelation, reference: Optional\[DatasetCorrelation])

Bases: `object`

#### Attributes:

&#x20;    **current : DatasetCorrelation**

&#x20;    **reference : Optional\[DatasetCorrelation]**

## stability\_metric module <a href="#module-evidently.metrics.data_quality.stability_metric" id="module-evidently.metrics.data_quality.stability_metric"></a>

### class DataQualityStabilityMetric()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`DataQualityStabilityMetricResult`]

Calculates stability by target and prediction

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class DataQualityStabilityMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: DataQualityStabilityMetric)**

&#x20;    **render\_json(obj: DataQualityStabilityMetric)**

### class DataQualityStabilityMetricResult(number\_not\_stable\_target: Optional\[int] = None, number\_not\_stable\_prediction: Optional\[int] = None)

Bases: `object`

#### Attributes:

&#x20;    **number\_not\_stable\_prediction : Optional\[int] = None**

&#x20;    **number\_not\_stable\_target : Optional\[int] = None**


# evidently.metrics.regression\_performance

## Submodules

## abs\_perc\_error\_in\_time module <a href="#module-evidently.metrics.regression_performance.abs_perc_error_in_time" id="module-evidently.metrics.regression_performance.abs_perc_error_in_time"></a>

### class RegressionAbsPercentageErrorPlot()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionAbsPercentageErrorPlotResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionAbsPercentageErrorPlotRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionAbsPercentageErrorPlot)**

&#x20;    **render\_json(obj: RegressionAbsPercentageErrorPlot)**

### class RegressionAbsPercentageErrorPlotResults(current\_scatter: Dict\[str, pandas.core.series.Series], reference\_scatter: Optional\[Dict\[str, pandas.core.series.Series]], x\_name: str)

Bases: `object`

#### Attributes:

&#x20;    **current\_scatter : Dict\[str, Series]**

&#x20;    **reference\_scatter : Optional\[Dict\[str, Series]]**

&#x20;    **x\_name : str**

## error\_bias\_table module <a href="#module-evidently.metrics.regression_performance.error_bias_table" id="module-evidently.metrics.regression_performance.error_bias_table"></a>

### class RegressionErrorBiasTable(columns: Optional\[List\[str]] = None, top\_error: Optional\[float] = None)

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionErrorBiasTableResults`]

#### Attributes:

&#x20;    **TOP\_ERROR\_DEFAULT = 0.05**

&#x20;    **TOP\_ERROR\_MAX = 0.5**

&#x20;    **TOP\_ERROR\_MIN = 0**

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **top\_error : float**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionErrorBiasTableRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionErrorBiasTable)**

&#x20;    **render\_json(obj: RegressionErrorBiasTable)**

### class RegressionErrorBiasTableResults(top\_error: float, current\_plot\_data: pandas.core.frame.DataFrame, reference\_plot\_data: Optional\[pandas.core.frame.DataFrame], target\_name: str, prediction\_name: str, num\_feature\_names: List\[str], cat\_feature\_names: List\[str], error\_bias: Optional\[dict] = None, columns: Optional\[List\[str]] = None)

Bases: `object`

#### Attributes:

&#x20;    **cat\_feature\_names : List\[str]**

&#x20;    **columns : Optional\[List\[str]] = None**

&#x20;    **current\_plot\_data : DataFrame**

&#x20;    **error\_bias : Optional\[dict] = None**

&#x20;    **num\_feature\_names : List\[str]**

&#x20;    **prediction\_name : str**

&#x20;    **reference\_plot\_data : Optional\[DataFrame]**

&#x20;    **target\_name : str**

&#x20;    **top\_error : float**

## error\_distribution module <a href="#module-evidently.metrics.regression_performance.error_distribution" id="module-evidently.metrics.regression_performance.error_distribution"></a>

### class RegressionErrorDistribution()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionErrorDistributionResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionErrorDistributionRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionErrorDistribution)**

&#x20;    **render\_json(obj: RegressionErrorDistribution)**

### class RegressionErrorDistributionResults(current\_bins: pandas.core.frame.DataFrame, reference\_bins: Optional\[pandas.core.frame.DataFrame])

Bases: `object`

#### Attributes:

&#x20;    **current\_bins : DataFrame**

&#x20;    **reference\_bins : Optional\[DataFrame]**

## error\_in\_time module <a href="#module-evidently.metrics.regression_performance.error_in_time" id="module-evidently.metrics.regression_performance.error_in_time"></a>

### class RegressionErrorPlot()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionErrorPlotResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionErrorPlotRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionErrorPlot)**

&#x20;    **render\_json(obj: RegressionErrorPlot)**

### class RegressionErrorPlotResults(current\_scatter: Dict\[str, pandas.core.series.Series], reference\_scatter: Optional\[Dict\[str, pandas.core.series.Series]], x\_name: str)

Bases: `object`

#### Attributes:

&#x20;    **current\_scatter : Dict\[str, Series]**

&#x20;    **reference\_scatter : Optional\[Dict\[str, Series]]**

&#x20;    **x\_name : str**

## error\_normality module <a href="#module-evidently.metrics.regression_performance.error_normality" id="module-evidently.metrics.regression_performance.error_normality"></a>

### class RegressionErrorNormality()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionErrorNormalityResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionErrorNormalityRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionErrorNormality)**

&#x20;    **render\_json(obj: RegressionErrorNormality)**

### class RegressionErrorNormalityResults(current\_error: pandas.core.series.Series, reference\_error: Optional\[pandas.core.series.Series])

Bases: `object`

#### Attributes:

&#x20;    **current\_error : Series**

&#x20;    **reference\_error : Optional\[Series]**

## predicted\_and\_actual\_in\_time module <a href="#module-evidently.metrics.regression_performance.predicted_and_actual_in_time" id="module-evidently.metrics.regression_performance.predicted_and_actual_in_time"></a>

### class RegressionPredictedVsActualPlot()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionPredictedVsActualPlotResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionPredictedVsActualPlotRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionPredictedVsActualPlot)**

&#x20;    **render\_json(obj: RegressionPredictedVsActualPlot)**

### class RegressionPredictedVsActualPlotResults(current\_scatter: Dict\[str, pandas.core.series.Series], reference\_scatter: Optional\[Dict\[str, pandas.core.series.Series]], x\_name: str)

Bases: `object`

#### Attributes:

&#x20;    **current\_scatter : Dict\[str, Series]**

&#x20;    **reference\_scatter : Optional\[Dict\[str, Series]]**

&#x20;    **x\_name : str**

## predicted\_vs\_actual module <a href="#module-evidently.metrics.regression_performance.predicted_vs_actual" id="module-evidently.metrics.regression_performance.predicted_vs_actual"></a>

### class RegressionPredictedVsActualScatter()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionPredictedVsActualScatterResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionPredictedVsActualScatterRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionPredictedVsActualScatter)**

&#x20;    **render\_json(obj: RegressionPredictedVsActualScatter)**

### class RegressionPredictedVsActualScatterResults(current\_scatter: Dict\[str, pandas.core.series.Series], reference\_scatter: Optional\[Dict\[str, pandas.core.series.Series]])

Bases: `object`

#### Attributes:

&#x20;    **current\_scatter : Dict\[str, Series]**

&#x20;    **reference\_scatter : Optional\[Dict\[str, Series]]**

## regression\_dummy\_metric module <a href="#module-evidently.metrics.regression_performance.regression_dummy_metric" id="module-evidently.metrics.regression_performance.regression_dummy_metric"></a>

### class RegressionDummyMetric()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionDummyMetricResults`]

#### Attributes:

&#x20;    **quality\_metric : RegressionQualityMetric**

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionDummyMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionDummyMetric)**

&#x20;    **render\_json(obj: RegressionDummyMetric)**

### class RegressionDummyMetricResults(rmse\_default: float, mean\_abs\_error\_default: float, mean\_abs\_perc\_error\_default: float, abs\_error\_max\_default: float, mean\_abs\_error\_by\_ref: Optional\[float] = None, mean\_abs\_error: Optional\[float] = None, mean\_abs\_perc\_error\_by\_ref: Optional\[float] = None, mean\_abs\_perc\_error: Optional\[float] = None, rmse\_by\_ref: Optional\[float] = None, rmse: Optional\[float] = None, abs\_error\_max\_by\_ref: Optional\[float] = None, abs\_error\_max: Optional\[float] = None)

Bases: `object`

#### Attributes:

&#x20;    **abs\_error\_max : Optional\[float] = None**

&#x20;    **abs\_error\_max\_by\_ref : Optional\[float] = None**

&#x20;    **abs\_error\_max\_default : float**

&#x20;    **mean\_abs\_error : Optional\[float] = None**

&#x20;    **mean\_abs\_error\_by\_ref : Optional\[float] = None**

&#x20;    **mean\_abs\_error\_default : float**

&#x20;    **mean\_abs\_perc\_error : Optional\[float] = None**

&#x20;    **mean\_abs\_perc\_error\_by\_ref : Optional\[float] = None**

&#x20;    **mean\_abs\_perc\_error\_default : float**

&#x20;    **rmse : Optional\[float] = None**

&#x20;    **rmse\_by\_ref : Optional\[float] = None**

&#x20;    **rmse\_default : float**

## regression\_performance\_metrics module <a href="#module-evidently.metrics.regression_performance.regression_performance_metrics" id="module-evidently.metrics.regression_performance.regression_performance_metrics"></a>

### class RegressionPerformanceMetrics()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionPerformanceMetricsResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

&#x20;    **get\_parameters()**

### class RegressionPerformanceMetricsRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionPerformanceMetrics)**

&#x20;    **render\_json(obj: RegressionPerformanceMetrics)**

### class RegressionPerformanceMetricsResults(columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), r2\_score: float, rmse: float, rmse\_default: float, mean\_error: float, me\_default\_sigma: float, me\_hist\_for\_plot: Dict\[str, Union\[pandas.core.series.Series, pandas.core.frame.DataFrame]], mean\_abs\_error: float, mean\_abs\_error\_default: float, mean\_abs\_perc\_error: float, mean\_abs\_perc\_error\_default: float, abs\_error\_max: float, abs\_error\_max\_default: float, error\_std: float, abs\_error\_std: float, abs\_perc\_error\_std: float, error\_normality: dict, underperformance: dict, hist\_for\_plot: Dict\[str, pandas.core.series.Series], vals\_for\_plots: Dict\[str, Dict\[str, pandas.core.series.Series]], error\_bias: Optional\[dict] = None, mean\_error\_ref: Optional\[float] = None, mean\_abs\_error\_ref: Optional\[float] = None, mean\_abs\_perc\_error\_ref: Optional\[float] = None, rmse\_ref: Optional\[float] = None, r2\_score\_ref: Optional\[float] = None, abs\_error\_max\_ref: Optional\[float] = None, underperformance\_ref: Optional\[dict] = None)

Bases: `object`

#### Attributes:

&#x20;    **abs\_error\_max : float**

&#x20;    **abs\_error\_max\_default : float**

&#x20;    **abs\_error\_max\_ref : Optional\[float] = None**

&#x20;    **abs\_error\_std : float**

&#x20;    **abs\_perc\_error\_std : float**

&#x20;    **columns :** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)

&#x20;    **error\_bias : Optional\[dict] = None**

&#x20;    **error\_normality : dict**

&#x20;    **error\_std : float**

&#x20;    **hist\_for\_plot : Dict\[str, Series]**

&#x20;    **me\_default\_sigma : float**

&#x20;    **me\_hist\_for\_plot : Dict\[str, Union\[Series, DataFrame]]**

&#x20;    **mean\_abs\_error : float**

&#x20;    **mean\_abs\_error\_default : float**

&#x20;    **mean\_abs\_error\_ref : Optional\[float] = None**

&#x20;    **mean\_abs\_perc\_error : float**

&#x20;    **mean\_abs\_perc\_error\_default : float**

&#x20;    **mean\_abs\_perc\_error\_ref : Optional\[float] = None**

&#x20;    **mean\_error : float**

&#x20;    **mean\_error\_ref : Optional\[float] = None**

&#x20;    **r2\_score : float**

&#x20;    **r2\_score\_ref : Optional\[float] = None**

&#x20;    **rmse : float**

&#x20;    **rmse\_default : float**

&#x20;    **rmse\_ref : Optional\[float] = None**

&#x20;    **underperformance : dict**

&#x20;    **underperformance\_ref : Optional\[dict] = None**

&#x20;    **vals\_for\_plots : Dict\[str, Dict\[str, Series]]**

## regression\_quality module <a href="#module-evidently.metrics.regression_performance.regression_quality" id="module-evidently.metrics.regression_performance.regression_quality"></a>

### class RegressionQualityMetric()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionQualityMetricResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionQualityMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionQualityMetric)**

&#x20;    **render\_json(obj: RegressionQualityMetric)**

### class RegressionQualityMetricResults(columns: [DatasetColumns](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns), r2\_score: float, rmse: float, rmse\_default: float, mean\_error: float, me\_default\_sigma: float, me\_hist\_for\_plot: Dict\[str, pandas.core.series.Series], mean\_abs\_error: float, mean\_abs\_error\_default: float, mean\_abs\_perc\_error: float, mean\_abs\_perc\_error\_default: float, abs\_error\_max: float, abs\_error\_max\_default: float, error\_std: float, abs\_error\_std: float, abs\_perc\_error\_std: float, error\_normality: dict, underperformance: dict, hist\_for\_plot: Dict\[str, pandas.core.series.Series], vals\_for\_plots: Dict\[str, Dict\[str, pandas.core.series.Series]], error\_bias: Optional\[dict] = None, mean\_error\_ref: Optional\[float] = None, mean\_abs\_error\_ref: Optional\[float] = None, mean\_abs\_perc\_error\_ref: Optional\[float] = None, rmse\_ref: Optional\[float] = None, r2\_score\_ref: Optional\[float] = None, abs\_error\_max\_ref: Optional\[float] = None, underperformance\_ref: Optional\[dict] = None, error\_std\_ref: Optional\[float] = None, abs\_error\_std\_ref: Optional\[float] = None, abs\_perc\_error\_std\_ref: Optional\[float] = None)

Bases: `object`

#### Attributes:

&#x20;    **abs\_error\_max : float**

&#x20;    **abs\_error\_max\_default : float**

&#x20;    **abs\_error\_max\_ref : Optional\[float] = None**

&#x20;    **abs\_error\_std : float**

&#x20;    **abs\_error\_std\_ref : Optional\[float] = None**

&#x20;    **abs\_perc\_error\_std : float**

&#x20;    **abs\_perc\_error\_std\_ref : Optional\[float] = None**

&#x20;    **columns :** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)

&#x20;    **error\_bias : Optional\[dict] = None**

&#x20;    **error\_normality : dict**

&#x20;    **error\_std : float**

&#x20;    **error\_std\_ref : Optional\[float] = None**

&#x20;    **hist\_for\_plot : Dict\[str, Series]**

&#x20;    **me\_default\_sigma : float**

&#x20;    **me\_hist\_for\_plot : Dict\[str, Series]**

&#x20;    **mean\_abs\_error : float**

&#x20;    **mean\_abs\_error\_default : float**

&#x20;    **mean\_abs\_error\_ref : Optional\[float] = None**

&#x20;    **mean\_abs\_perc\_error : float**

&#x20;    **mean\_abs\_perc\_error\_default : float**

&#x20;    **mean\_abs\_perc\_error\_ref : Optional\[float] = None**

&#x20;    **mean\_error : float**

&#x20;    **mean\_error\_ref : Optional\[float] = None**

&#x20;    **r2\_score : float**

&#x20;    **r2\_score\_ref : Optional\[float] = None**

&#x20;    **rmse : float**

&#x20;    **rmse\_default : float**

&#x20;    **rmse\_ref : Optional\[float] = None**

&#x20;    **underperformance : dict**

&#x20;    **underperformance\_ref : Optional\[dict] = None**

&#x20;    **vals\_for\_plots : Dict\[str, Dict\[str, Series]]**

## top\_error module <a href="#module-evidently.metrics.regression_performance.top_error" id="module-evidently.metrics.regression_performance.top_error"></a>

### class RegressionTopErrorMetric()

Bases: [`Metric`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.Metric)\[`RegressionTopErrorMetricResults`]

#### Methods:

&#x20;    **calculate(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**)**

### class RegressionTopErrorMetricRenderer(color\_options: Optional\[[ColorOptions](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)] = None)

Bases: [`MetricRenderer`](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/M9iMLhGad2yNU1Kd04Dv#evidently.renderers.base_renderer.MetricRenderer)

#### Attributes:

&#x20;    **color\_options :** [**ColorOptions**](https://docs-old.evidentlyai.com/reference/api-reference/evidently.metrics/pages/ZW3IQh73672cpd3xySyJ#evidently.options.color_scheme.ColorOptions)

#### Methods:

&#x20;    **render\_html(obj: RegressionTopErrorMetric)**

&#x20;    **render\_json(obj: RegressionTopErrorMetric)**

### class RegressionTopErrorMetricResults(curr\_mean\_err\_per\_group: Dict\[str, Dict\[str, float]], curr\_scatter: Dict\[str, Dict\[str, pandas.core.series.Series]], ref\_mean\_err\_per\_group: Optional\[Dict\[str, Dict\[str, float]]], ref\_scatter: Optional\[Dict\[str, Dict\[str, pandas.core.series.Series]]])

Bases: `object`

#### Attributes:

&#x20;    **curr\_mean\_err\_per\_group : Dict\[str, Dict\[str, float]]**

&#x20;    **curr\_scatter : Dict\[str, Dict\[str, Series]]**

&#x20;    **ref\_mean\_err\_per\_group : Optional\[Dict\[str, Dict\[str, float]]]**

&#x20;    **ref\_scatter : Optional\[Dict\[str, Dict\[str, Series]]]**


# evidently.metric\_preset

### class ClassificationPreset(columns: Optional\[List\[str]] = None, probas\_threshold: Optional\[float] = None, k: Optional\[int] = None)

Bases: `MetricPreset`

Metrics preset for classification performance.

Contains metrics:

* ClassificationQualityMetric
* ClassificationClassBalance
* ClassificationConfusionMatrix
* ClassificationQualityByClass

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **k : Optional\[int]**

&#x20;    **probas\_threshold : Optional\[float]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

### class DataDriftPreset(columns: Optional\[List\[str]] = None, drift\_share: float = 0.5, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_column\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, stattest\_threshold: Optional\[float] = None, cat\_stattest\_threshold: Optional\[float] = None, num\_stattest\_threshold: Optional\[float] = None, per\_column\_stattest\_threshold: Optional\[Dict\[str, float]] = None)

Bases: `MetricPreset`

Metric Preset for Data Drift analysis.

Contains metrics:

* DatasetDriftMetric
* DataDriftTable

#### Attributes:

&#x20;    **cat\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **cat\_stattest\_threshold : Optional\[float]**

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **drift\_share : float**

&#x20;    **num\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **num\_stattest\_threshold : Optional\[float]**

&#x20;    **per\_column\_stattest : Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]]**

&#x20;    **per\_column\_stattest\_threshold : Optional\[Dict\[str, float]]**

&#x20;    **stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **stattest\_threshold : Optional\[float]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

### class DataQualityPreset(columns: Optional\[List\[str]] = None)

Bases: `MetricPreset`

Metric preset for Data Quality analysis.

Contains metrics:

* DatasetSummaryMetric
* ColumnSummaryMetric for each column
* DatasetMissingValuesMetric
* DatasetCorrelationsMetric
* **Parameters**

  `columns` – list of columns for analysis.

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

### class RegressionPreset(columns: Optional\[List\[str]] = None)

Bases: `MetricPreset`

Metric preset for Regression performance analysis.

Contains metrics:

* RegressionQualityMetric
* RegressionPredictedVsActualScatter
* RegressionPredictedVsActualPlot
* RegressionErrorPlot
* RegressionAbsPercentageErrorPlot
* RegressionErrorDistribution
* RegressionErrorNormality
* RegressionTopErrorMetric
* RegressionErrorBiasTable

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

### class TargetDriftPreset(columns: Optional\[List\[str]] = None, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_column\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, stattest\_threshold: Optional\[float] = None, cat\_stattest\_threshold: Optional\[float] = None, num\_stattest\_threshold: Optional\[float] = None, per\_column\_stattest\_threshold: Optional\[Dict\[str, float]] = None)

Bases: `MetricPreset`

Metric preset for Target Drift analysis.

Contains metrics:

* ColumnDriftMetric - for target and prediction if present in datasets.
* ColumnValuePlot - if task is regression.
* ColumnCorrelationsMetric - for target and prediction if present in datasets.
* TargetByFeaturesTable

#### Attributes:

&#x20;    **cat\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **cat\_stattest\_threshold : Optional\[float]**

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **num\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **num\_stattest\_threshold : Optional\[float]**

&#x20;    **per\_column\_stattest : Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]]**

&#x20;    **per\_column\_stattest\_threshold : Optional\[Dict\[str, float]]**

&#x20;    **stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **stattest\_threshold : Optional\[float]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

## Submodules

## classification\_performance module <a href="#module-evidently.metric_preset.classification_performance" id="module-evidently.metric_preset.classification_performance"></a>

### class ClassificationPreset(columns: Optional\[List\[str]] = None, probas\_threshold: Optional\[float] = None, k: Optional\[int] = None)

Bases: `MetricPreset`

Metrics preset for classification performance.

Contains metrics:

* ClassificationQualityMetric
* ClassificationClassBalance
* ClassificationConfusionMatrix
* ClassificationQualityByClass

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **k : Optional\[int]**

&#x20;    **probas\_threshold : Optional\[float]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

## data\_drift module <a href="#module-evidently.metric_preset.data_drift" id="module-evidently.metric_preset.data_drift"></a>

### class DataDriftPreset(columns: Optional\[List\[str]] = None, drift\_share: float = 0.5, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_column\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, stattest\_threshold: Optional\[float] = None, cat\_stattest\_threshold: Optional\[float] = None, num\_stattest\_threshold: Optional\[float] = None, per\_column\_stattest\_threshold: Optional\[Dict\[str, float]] = None)

Bases: `MetricPreset`

Metric Preset for Data Drift analysis.

Contains metrics:

* DatasetDriftMetric
* DataDriftTable

#### Attributes:

&#x20;    **cat\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **cat\_stattest\_threshold : Optional\[float]**

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **drift\_share : float**

&#x20;    **num\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **num\_stattest\_threshold : Optional\[float]**

&#x20;    **per\_column\_stattest : Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]]**

&#x20;    **per\_column\_stattest\_threshold : Optional\[Dict\[str, float]]**

&#x20;    **stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **stattest\_threshold : Optional\[float]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

## data\_quality module <a href="#module-evidently.metric_preset.data_quality" id="module-evidently.metric_preset.data_quality"></a>

### class DataQualityPreset(columns: Optional\[List\[str]] = None)

Bases: `MetricPreset`

Metric preset for Data Quality analysis.

Contains metrics:

* DatasetSummaryMetric
* ColumnSummaryMetric for each column
* DatasetMissingValuesMetric
* DatasetCorrelationsMetric
* **Parameters**

  `columns` – list of columns for analysis.

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

## metric\_preset module <a href="#module-evidently.metric_preset.metric_preset" id="module-evidently.metric_preset.metric_preset"></a>

### class MetricPreset()

Bases: `object`

Base class for metric presets

#### Methods:

&#x20;    **abstract generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

## regression\_performance module <a href="#module-evidently.metric_preset.regression_performance" id="module-evidently.metric_preset.regression_performance"></a>

### class RegressionPreset(columns: Optional\[List\[str]] = None)

Bases: `MetricPreset`

Metric preset for Regression performance analysis.

Contains metrics:

* RegressionQualityMetric
* RegressionPredictedVsActualScatter
* RegressionPredictedVsActualPlot
* RegressionErrorPlot
* RegressionAbsPercentageErrorPlot
* RegressionErrorDistribution
* RegressionErrorNormality
* RegressionTopErrorMetric
* RegressionErrorBiasTable

#### Attributes:

&#x20;    **columns : Optional\[List\[str]]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**

## target\_drift module <a href="#module-evidently.metric_preset.target_drift" id="module-evidently.metric_preset.target_drift"></a>

### class TargetDriftPreset(columns: Optional\[List\[str]] = None, stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_column\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, stattest\_threshold: Optional\[float] = None, cat\_stattest\_threshold: Optional\[float] = None, num\_stattest\_threshold: Optional\[float] = None, per\_column\_stattest\_threshold: Optional\[Dict\[str, float]] = None)

Bases: `MetricPreset`

Metric preset for Target Drift analysis.

Contains metrics:

* ColumnDriftMetric - for target and prediction if present in datasets.
* ColumnValuePlot - if task is regression.
* ColumnCorrelationsMetric - for target and prediction if present in datasets.
* TargetByFeaturesTable

#### Attributes:

&#x20;    **cat\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **cat\_stattest\_threshold : Optional\[float]**

&#x20;    **columns : Optional\[List\[str]]**

&#x20;    **num\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **num\_stattest\_threshold : Optional\[float]**

&#x20;    **per\_column\_stattest : Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]]**

&#x20;    **per\_column\_stattest\_threshold : Optional\[Dict\[str, float]]**

&#x20;    **stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]**

&#x20;    **stattest\_threshold : Optional\[float]**

#### Methods:

&#x20;    **generate\_metrics(data:** [**InputData**](https://docs-old.evidentlyai.com/reference/api-reference/pages/PNL2TuiArYt5TU085k7F#evidently.metrics.base_metric.InputData)**, columns:** [**DatasetColumns**](https://docs-old.evidentlyai.com/reference/api-reference/pages/9O7LQ1v4Gn98SSrnqpix#evidently.utils.data_operations.DatasetColumns)**)**


# evidently.options

### class OptionsProvider()

Bases: `object`

#### Methods:

&#x20;    **add(options)**

&#x20;    **get(options\_type: Type\[TypeParam])**

## Submodules

## color\_scheme module <a href="#module-evidently.options.color_scheme" id="module-evidently.options.color_scheme"></a>

### class ColorOptions(primary\_color: str = '#ed0400', secondary\_color: str = '#4d4d4d', current\_data\_color: Optional\[str] = None, reference\_data\_color: Optional\[str] = None, color\_sequence: Sequence\[str] = ('#ed0400', '#0a5f38', '#6c3461', '#71aa34', '#d8dcd6', '#6b8ba4'), fill\_color: str = 'LightGreen', zero\_line\_color: str = 'green', non\_visible\_color: str = 'white', underestimation\_color: str = '#6574f7', overestimation\_color: str = '#ee5540', majority\_color: str = '#1acc98', vertical\_lines: str = 'green', heatmap: str = 'RdBu\_r')

Bases: `object`

Collection of colors for data visualization

* primary\_color - basic color for data visualization.

  Uses by default for all bars and lines for widgets with one dataset and as a default for current data.
* secondary\_color - basic color for second data visualization if we have two data sets, for example, reference data.
* current\_data\_color - color for all current data, by default primary color is used
* reference\_data\_color - color for reference data, by default secondary color is used
* color\_sequence - set of colors for drawing a number of lines in one graph, in for data quality, for example
* fill\_color - fill color for areas in line graphs
* zero\_line\_color - color for base, zero line in line graphs
* non\_visible\_color - color for technical, not visible dots or points for better scalability
* underestimation\_color - color for underestimation line in regression
* overestimation\_color - color for overestimation line in regression
* majority\_color - color for majority line in regression
* lines - color for vertical lines
* heatmap\_colors - colors for heatmap

#### Attributes:

&#x20;    **color\_sequence : Sequence\[str] = ('#ed0400', '#0a5f38', '#6c3461', '#71aa34', '#d8dcd6', '#6b8ba4')**

&#x20;    **current\_data\_color : Optional\[str] = None**

&#x20;    **fill\_color : str = 'LightGreen'**

&#x20;    **heatmap : str = 'RdBu\_r'**

&#x20;    **majority\_color : str = '#1acc98'**

&#x20;    **non\_visible\_color : str = 'white'**

&#x20;    **overestimation\_color : str = '#ee5540'**

&#x20;    **primary\_color : str = '#ed0400'**

&#x20;    **reference\_data\_color : Optional\[str] = None**

&#x20;    **secondary\_color : str = '#4d4d4d'**

&#x20;    **underestimation\_color : str = '#6574f7'**

&#x20;    **vertical\_lines : str = 'green'**

&#x20;    **zero\_line\_color : str = 'green'**

#### Methods:

&#x20;    **get\_current\_data\_color()**

&#x20;    **get\_reference\_data\_color()**

## data\_drift module <a href="#module-evidently.options.data_drift" id="module-evidently.options.data_drift"></a>

### class DataDriftOptions(confidence: Optional\[Union\[float, Dict\[str, float]]] = None, threshold: Optional\[Union\[float, Dict\[str, float]]] = None, drift\_share: float = 0.5, nbinsx: Union\[int, Dict\[str, int]] = 10, xbins: Optional\[Dict\[str, int]] = None, feature\_stattest\_func: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest), Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]]] = None, all\_features\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, cat\_features\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_features\_stattest: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, per\_feature\_stattest: Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]]] = None, all\_features\_threshold: Optional\[float] = None, cat\_features\_threshold: Optional\[float] = None, num\_features\_threshold: Optional\[float] = None, per\_feature\_threshold: Optional\[Dict\[str, float]] = None, cat\_target\_threshold: Optional\[float] = None, num\_target\_threshold: Optional\[float] = None, cat\_target\_stattest\_func: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None, num\_target\_stattest\_func: Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]], [StatTest](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)]] = None)

Bases: `object`

Configuration for Data Drift calculations.

* **Parameters**
  * `confidence` – Defines the confidence level for statistical tests.\
    Applies to all features (if passed as float) or certain features (if passed as dictionary).\
    (Deprecated) Use threshold to define confidence level for statistical\
    tests as more universal solution.
  * `threshold` – Defines thresholds for statistical tests.\
    Applies to all features (if passed as float) or certain features (if passed as dictionary).
  * `drift_share` – Sets the share of drifting features as a condition for Dataset Drift in the Data Drift report.
  * `nbinsx` – Defines the number of bins in a histogram.\
    Applies to all features (if passed as int) or certain features (if passed as dictionary).
  * `xbins` – Defines the boundaries for the size of a specific bin in a histogram.
  * `feature_stattest_func` – Defines a custom statistical test for drift detection in the Data Drift report.\
    Applies to all features (if passed as a function) or individual features (if a dict).\
    (Deprecated) Use all\_features\_stattest or per\_feature\_stattest.
  * `all_features_stattest` – Defines a custom statistical test for drift detection in the Data Drift report\
    for all features.
  * `cat_features_stattest` – Defines a custom statistical test for drift detection in the Data Drift report\
    for categorical features only.
  * `num_features_stattest` – Defines a custom statistical test for drift detection in the Data Drift report\
    for numerical features only.
  * `per_feature_stattest` – Defines a custom statistical test for drift detection in the Data Drift report\
    per feature.
  * `cat_target_stattest_func` – Defines a custom statistical test to detect target drift in category target.
  * `num_target_stattest_func` – Defines a custom statistical test to detect target drift in numeric target.

#### Attributes:

&#x20;    **all\_features\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]] = None**

&#x20;    **all\_features\_threshold : Optional\[float] = None**

&#x20;    **cat\_features\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]] = None**

&#x20;    **cat\_features\_threshold : Optional\[float] = None**

&#x20;    **cat\_target\_stattest\_func : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]] = None**

&#x20;    **cat\_target\_threshold : Optional\[float] = None**

&#x20;    **confidence : Optional\[Union\[float, Dict\[str, float]]] = None**

&#x20;    **drift\_share : float = 0.5**

&#x20;    **feature\_stattest\_func : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**, Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]]] = None**

&#x20;    **nbinsx : Union\[int, Dict\[str, int]] = 10**

&#x20;    **num\_features\_stattest : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]] = None**

&#x20;    **num\_features\_threshold : Optional\[float] = None**

&#x20;    **num\_target\_stattest\_func : Optional\[Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]] = None**

&#x20;    **num\_target\_threshold : Optional\[float] = None**

&#x20;    **per\_feature\_stattest : Optional\[Dict\[str, Union\[str, Callable\[\[Series, Series, str, float], Tuple\[float, bool]],** [**StatTest**](https://docs-old.evidentlyai.com/reference/api-reference/pages/Rb3BF2Y7DDvjAuK838aV#evidently.calculations.stattests.registry.StatTest)**]]] = None**

&#x20;    **per\_feature\_threshold : Optional\[Dict\[str, float]] = None**

&#x20;    **threshold : Optional\[Union\[float, Dict\[str, float]]] = None**

&#x20;    **xbins : Optional\[Dict\[str, int]] = None**

#### Methods:

&#x20;    **as\_dict()**

&#x20;    **get\_feature\_stattest\_func(feature\_name: str, feature\_type: str)**

&#x20;    **get\_nbinsx(feature\_name: str)**

&#x20;    **get\_threshold(feature\_name: str, feature\_type: str)**

## quality\_metrics module <a href="#module-evidently.options.quality_metrics" id="module-evidently.options.quality_metrics"></a>

### class QualityMetricsOptions(conf\_interval\_n\_sigmas: int = 1, classification\_threshold: float = 0.5, cut\_quantile: Union\[NoneType, Tuple\[str, float], Dict\[str, Tuple\[str, float]]] = None)

Bases: `object`

#### Attributes:

&#x20;    **classification\_threshold : float = 0.5**

&#x20;    **conf\_interval\_n\_sigmas : int = 1**

&#x20;    **cut\_quantile : Union\[None, Tuple\[str, float], Dict\[str, Tuple\[str, float]]] = None**

#### Methods:

&#x20;    **as\_dict()**

&#x20;    **get\_cut\_quantile(feature\_name: str)**


# evidently.pipeline

## Submodules

## column\_mapping module <a href="#module-evidently.pipeline.column_mapping" id="module-evidently.pipeline.column_mapping"></a>

### class ColumnMapping(target: Optional\[str] = 'target', prediction: Union\[str, int, Sequence\[str], Sequence\[int], NoneType] = 'prediction', datetime: Optional\[str] = 'datetime', id: Optional\[str] = None, numerical\_features: Optional\[List\[str]] = None, categorical\_features: Optional\[List\[str]] = None, datetime\_features: Optional\[List\[str]] = None, target\_names: Optional\[List\[str]] = None, task: Optional\[str] = None, pos\_label: Union\[str, int, NoneType] = 1)

Bases: `object`

#### Attributes:

&#x20;    **categorical\_features : Optional\[List\[str]] = None**

&#x20;    **datetime : Optional\[str] = 'datetime'**

&#x20;    **datetime\_features : Optional\[List\[str]] = None**

&#x20;    **id : Optional\[str] = None**

&#x20;    **numerical\_features : Optional\[List\[str]] = None**

&#x20;    **pos\_label : Optional\[Union\[str, int]] = 1**

&#x20;    **prediction : Optional\[Union\[str, int, Sequence\[str], Sequence\[int]]] = 'prediction'**

&#x20;    **target : Optional\[str] = 'target'**

&#x20;    **target\_names : Optional\[List\[str]] = None**

&#x20;    **task : Optional\[str] = None**

#### Methods:

&#x20;    **is\_classification\_task()**

&#x20;    **is\_regression\_task()**

### class TaskType()

Bases: `object`

#### Attributes:

&#x20;    **CLASSIFICATION\_TASK : str = 'classification'**

&#x20;    **REGRESSION\_TASK : str = 'regression'**

## pipeline module <a href="#module-evidently.pipeline.pipeline" id="module-evidently.pipeline.pipeline"></a>

### class Pipeline(stages: Sequence\[PipelineStage], options: list)

Bases: `object`

#### Attributes:

&#x20;    **analyzers\_results : Dict\[Type\[Analyzer], object]**

&#x20;    **options\_provider :** [**OptionsProvider**](https://docs-old.evidentlyai.com/reference/api-reference/pages/ZW3IQh73672cpd3xySyJ#evidently.options.OptionsProvider)

&#x20;    **stages : Sequence\[PipelineStage]**

#### Methods:

&#x20;    **execute(reference\_data: DataFrame, current\_data: Optional\[DataFrame] = None, column\_mapping: Optional\[ColumnMapping] = None)**

&#x20;    **get\_analyzers()**

## stage module <a href="#module-evidently.pipeline.stage" id="module-evidently.pipeline.stage"></a>

### class PipelineStage()

Bases: `object`

#### Attributes:

&#x20;    **options\_provider :** [**OptionsProvider**](https://docs-old.evidentlyai.com/reference/api-reference/pages/ZW3IQh73672cpd3xySyJ#evidently.options.OptionsProvider)

#### Methods:

&#x20;    **add\_analyzer(analyzer\_type: Type\[Analyzer])**

&#x20;    **analyzers()**

&#x20;    **abstract calculate(reference\_data: DataFrame, current\_data: DataFrame, column\_mapping: ColumnMapping, analyzers\_results: Dict\[Type\[Analyzer], Any])**




---

[Next Page](/llms-full.txt/1)