> Source: https://builder-docs.ema.ai/testing-operations/evaluating
> Title: Evaluating AI Employees

# Evaluating AI Employees

Evaluations allow you to measure how an AI Employee performs against a large dataset using defined scoring criteria. This helps validate response quality, consistency, and alignment with expected behavior before or after launch.

## Supported AI Employees

Evaluations are currently supported only on **chat-triggered** AI Employees.

## Key Concepts

Concept

Description

**Evaluation**

A single execution that runs against a dataset and scores each response using defined criteria.

**Dataset**

A CSV file containing test queries. Each row represents a single workflow run.

**User Query Column**

The dataset column whose values are used to trigger AI Employee workflow runs.

**Evaluation Criteria**

Defines how responses are assessed. Criteria include evaluation context and one or more scoring guidelines.

**Scoring Guideline**

A rule that defines how a response is scored on a numeric scale. Each guideline produces a separate score.

**LLM Evaluator**

The large language model used to evaluate and score AI Employee responses.

**Evaluation Results**

Row-level outputs containing scores. Results can be viewed in the UI or downloaded as CSV.

## Permissions

Only users with the **AI Employee Manager** or **AI Employee Admin** role can view and run evaluations in the Evaluations tab.

## Creating an Evaluation

Evaluations are created from the **Evaluations** tab of a chat-triggered AI Employee by clicking **New evaluation**.

## Evaluation Setup

### Dataset Selection

You can either:

-   Select an existing dataset (previously uploaded), or
-   Upload a new dataset.

### Input Mapping

Each evaluation requires mapping one dataset column as the **user query**. Values from this column are used to trigger the AI Employee for evaluation.

## Evaluation Criteria

You can either:

-   Use an existing default evaluation criteria or previously saved criteria, or
-   Create a new evaluation criteria.

### Evaluation Context

The evaluation context defines the instructions and variables used to guide scoring.

**Pills:** Pills can be selected from the left panel and inserted into the evaluation context. Each pill represents a variable and is replaced with its corresponding value before being sent to the LLM evaluator.

### Scoring Guidelines

At least one scoring guideline is required per evaluation criteria. Each scoring guideline includes:

-   **Name** (required)
-   **Description** (optional)
-   **Scale** -- An integer range for the scoring scale (e.g., 1 to 3, 1 to 5, or custom ranges).

Each numeric value within the selected scale must have a clearly defined criterion to ensure consistent and reliable scoring.

### LLM Evaluator Selection

Each evaluation requires selecting an LLM evaluator. The selected model is used as a judge to score all dataset rows. The dropdown displays all available models, including any custom models added on the AI Employee's configuration page.

## Running Evaluations

1.  Configure the dataset and evaluation criteria.
2.  Click **Run Evaluation** to start.

### Monitoring Progress

-   Each dataset row displays an execution status.
-   Rows marked **Success** indicate completed scoring.

## Results and Metrics

### Evaluation Results

After completion:

-   Full results are available in the evaluation view.
-   Results can be downloaded as a CSV file.

### Metrics

For each scoring guideline, the following metrics are generated:

-   Highest score
-   Lowest score
-   Average score
-   Score distribution

### Notes

Notes can be added to evaluations for documentation and future reference.

## Past Evaluations

The **Evaluations** tab lists all historical evaluations for the AI Employee, making it easy to compare performance over time.