> Source: https://builder-docs.ema.ai/agent-qa/setup
> Title: Setting Up Agent QA

# Setting Up Agent QA

This section walks you through configuring Agent QA from scratch. By the end, you will have a fully working QA system evaluating your customer service calls automatically.

## 2.1 Prerequisites

Before starting setup, prepare three sets of files. The quality of these inputs directly determines the accuracy of your evaluations.

### Golden Call Recordings

Representative audio files that cover your typical call scenarios.

Requirement

Detail

**Content**

Diverse scenarios -- routine calls, escalations, edge cases, different agents, different outcomes

**Supported formats**

MP3, WAV, Opus

**Maximum file size**

25 MB per file

**Audio quality**

Clear speech, reasonable duration, minimal background noise

> [TIP]
> **Tips for selecting golden recordings:** Include calls with diverse outcomes (successful resolutions, escalations, transfers, unresolved issues). Cover all major contact reason categories. Balance between high-performing and low-performing examples. Include calls from multiple agents to avoid bias. Aim for 20 to 50 examples per QA parameter for best calibration results.

### Label Data CSV

A spreadsheet containing your human reviewers' Pass/Fail/N/A grades for each golden call recording.

Requirement

Detail

**Format**

CSV file, UTF-8 encoded

**Required column**

`Filename` -- must match audio file names without the file extension

**Parameter columns**

One column per QA parameter, with values of exactly `Pass`, `Fail`, or `N/A`

**No empty cells**

Every parameter cell must contain `Pass`, `Fail`, or `N/A`

**Example:**

```
UCID,Transcript,Greeting,Customer Verification,Issue Resolution,Empathy,Hold Time
call-001,"Good morning, thank you for calling...",Pass,Pass,Fail,Pass,N/A
call-002,"Yeah what do you need?",Fail,Pass,Pass,Fail,Pass
call-003,"Hello, this is Sarah from Support...",Pass,N/A,Pass,Pass,Pass
```

> [INFO]
> The `UCID` value `call-001` must correspond to an audio file named `call-001.mp3`, `call-001.wav`, or `call-001.opus`. The `Transcript` column is optional -- if omitted, the system transcribes from audio. Column headers become your parameter names, so use clear, descriptive names.

### QA Form

Requirement

Detail

**Format**

CSV

**Content**

Parameter names, detailed descriptions, scoring criteria (what constitutes Pass vs. Fail), weight assignments, and criticality indicators

## 2.2 Create Your AI Employee

1.  Log in at **app.ema.co**
2.  Navigate to **AI Employees** in the left sidebar
3.  Click **Create** on "Pre-built AI Employee"
4.  Select the **Agent QA** template from the catalog
5.  Enter a **Name** (e.g., "QA Assistant - Customer Service")
6.  Enter a **Description** (e.g., "Evaluates all inbound customer service calls against our 25-parameter QA scorecard")
7.  Select the **Language** for evaluations (this determines the language used in scorecard reasoning and insights)
8.  Click **Continue**

Your AI Employee is created in **Disabled** state. You must complete all configuration steps before enabling it.

After creation, you will see six tabs:

Tab

Purpose

**Configuration**

Set up QA parameters, data sources, integrations, and contact reasons

**Imports**

Bring call data into the AI Employee -- manual upload, CCaaS connectors, and agent-mapping management

**Audit**

Review evaluated interactions with scorecards and transcripts (empty until processing begins)

**Metrics**

View aggregate analytics -- agent performance trends, contact reason breakdowns, resolution rates, sentiment distributions

**Insights**

Browse AI-generated patterns -- strengths, improvement areas, product feedback, competitive intelligence

**Permissions**

Manage user access -- invite users, assign roles, control visibility

## 2.3 Import Your Data

Before Agent QA can evaluate anything, conversations have to reach it. The **Imports** tab is where that happens -- pull recordings automatically from Genesys Cloud, upload a batch of audio files or transcripts yourself, or re-process interactions you already imported.

Imports also depends on **agent mapping**: the link between the agent identifier your source system uses and a known Ema employee. Set the identity schema up before your first import, or agents cannot be resolved and imports will pause waiting for you.

> [INFO]
> Imports, the Genesys Cloud connection, and agent mapping are covered in full in [3\. Imports, Genesys, and Agent Mapping](/legacy-docs/agent-qa/imports) -- including window rules, file limits, the metadata CSV format, job and file statuses, and how to unstick a paused import. Come back here for 2.4 Configure QA Parameters with Auto-Learning once your data is flowing.

## 2.4 Configure QA Parameters with Auto-Learning

Auto-Learning is the fastest and most accurate way to configure your evaluation rules. It uses your golden dataset to generate, calibrate, and iteratively refine parameter instructions until they align with your human reviewers' grades.

### Upload Your Files

1.  Open your AI Employee and click the **Configuration** tab
2.  Scroll to **QA Parameters** and click **Setup QA Parameters**
3.  Upload all files:

File

Instructions

**Audio files**

Select your golden call recordings. Supported formats: MP3, WAV, Opus. Max 25 MB per file.

**Transcript**

Select your golden transcripts. Supported formats: JSON, XML, HTML.

**Label data**

Click **Download sample** first to confirm the expected format. CSV with Pass/Fail/N/A values.

**QA form**

Click **Download sample** to check the expected format. Upload the QA form CSV.

Previously uploaded files are visible in this screen after a page reload or in a new session -- the setup shows your **most recent** upload, not a full history. Reopening the setup pre-populates those files by default; to run with different data, delete the previous files and re-upload. A persona whose Auto-Learning has never run opens with no pre-populated state.

### Validation Before Auto-Learning Can Start

Validation

Requirement

**Column matching**

Parameter columns in your label data CSV must match the parameter definitions in your QA form CSV exactly

**Transcription readiness**

All uploaded audio files must have completed transcription before Auto-Learning can start

**Label data values**

Every cell must contain exactly `Pass`, `Fail`, `N/A`, or `NOT_APPLICABLE`

**No concurrent pipelines**

Only one Auto-Learning pipeline can actively run per AI Employee at a time. A pending review (a completed run awaiting approval) does not block starting another append run -- see Appending New Parameters.

**Files must finish processing first**

You cannot trigger a run while any uploaded golden-data or QA-form file is still being processed. Wait for processing to finish, then retry.

### Run Auto-Learning

1.  After all files are uploaded and validated, click **Generate Parameters**
2.  Auto-Learning begins processing through multiple stages

**What you see during processing:**

-   A split-panel view: the left pane displays the parameters list with per-iteration accuracy data, while the right pane shows an overall accuracy chart tracking improvement across iterations
-   You can click **Run in background** to continue configuring other settings while it runs
-   If processing fails, a **Retry** button appears to restart the pipeline
-   Processing typically takes 15 to 45 minutes depending on the number of golden recordings
-   While a run is active, manual rule editing (add/edit/delete) is disabled with an explanatory notice -- it re-enables once the run finishes

**What happens behind the scenes:** The system runs up to 3 improvement iterations by default, stopping early if accuracy converges (i.e., improvement between consecutive iterations falls below a 1% threshold). Parameters with accuracy below 90% are targeted for instruction rewriting in each iteration.

After the initial run completes, you can click **Deep Optimize** to run additional iterations (up to 20) if accuracy on certain parameters is below your target. Deep Optimize also uses convergence-based early stopping.

> [INFO]
> If you have unsaved manual edits to existing parameters when you start a run, you'll be prompted to save or discard them first -- a run never silently overwrites edits you haven't saved.

### Review and Approve Parameters

After Auto-Learning completes:

1.  Click **Review Parameters**
2.  For each parameter, review:

Field

Description

**Parameter Name**

The name of the QA criterion (must be unique)

**Category**

Grouping label for related parameters

**Weight**

How much this parameter contributes to the overall score (must be greater than 0)

**Criticality**

**Critical** = auto-fail if this parameter fails; **Non-Critical** = weighted contribution only

**Mechanism**

Transcript Only, Knowledge Verified, or Action Based

**Generated Instructions**

A pass/fail instruction and a not-applicable instruction generated by Auto-Learning

**Accuracy**

Percentage match between the system's evaluation and your human grades

3.  Edit any parameter by clicking on it -- you can modify instruction text, weight, criticality, or mechanism
4.  All parameters are selected by default. Deselect any you want to discard, then click **Approve Parameters**

### Understanding Accuracy Results

Accuracy

Interpretation

Recommended Action

**95-100%**

Excellent alignment with human grades

No action needed

**90-94%**

Strong alignment

No action needed

**85-89%**

Acceptable

Consider adding more golden data examples or refining instruction text

**Below 85%**

Needs attention

Review golden data quality; consider splitting complex parameters into simpler ones

### Appending New Parameters

Auto-Learning is not a one-time, one-shot setup -- you can add more rules to a persona that already has approved parameters:

1.  Upload additional golden data (audio/transcripts, label data, and, if needed, an updated QA form covering only the **new** parameters)
2.  Click **Generate Parameters** as before

Auto-Learning runs only for the newly added parameters -- your existing approved parameters, their results, and their pipeline state are untouched. Once the run completes, approving it **merges** the new parameters into your existing set; it never replaces or clears what's already there. You get the same options as a first-ever run: approve as-is, or Deep Optimize the new parameters further before approving.

> [WARNING]
> A new parameter name that exactly matches an existing one is rejected before anything runs. A name that's merely similar produces a non-blocking warning you can proceed past.

### Adding Parameters Manually

You can also add parameters without Auto-Learning:

1.  In Configuration > QA Parameters, click **Add**
2.  Define the parameter name, category, weight, criticality, mechanism, pass/fail instruction, and N/A instruction
3.  Click **Save**

> [INFO]
> Manual parameters do not have accuracy scores since they have not been tested against golden data. Consider running Auto-Learning again after adding manual parameters to validate their performance.

### Bulk Delete and Reset

Select multiple parameters via checkboxes (or select all) to delete them at once -- including all of them, down to an empty parameter set. Resetting to zero shows a warning first, since any continuous-learning corrections tied to the deleted parameters are lost. From an empty set, you can rebuild via Auto-Learning (as a fresh first run) or by adding parameters manually.

## 2.5 Add a Knowledge Base

> [INFO]
> **Skip this step** if all of your parameters use the Transcript Only mechanism. A knowledge base is only required for parameters using the **Knowledge Verified** mechanism.

### Setting Up Data Sources

1.  In the **Configuration** tab, scroll to **Data Sources**
2.  Click **Add** and select your source type:

Source Type

Best For

Sync Behavior

**File Upload**

Quick setup, static documents

Manual -- re-upload when documents change

**Google Drive**

Organizations using Google Workspace

Automatic -- refreshes within ~10 minutes of changes

**SharePoint**

Organizations using Microsoft 365

Automatic sync

**Confluence**

Organizations using Atlassian tools

Automatic sync

**URL Scraping**

Public-facing knowledge bases, help centers

Crawls up to two levels deep from each seed URL

3.  Create a folder name to organize your documents (e.g., "Return Policies," "Product Guides")
4.  Upload or connect your files
5.  Optionally, assign **tags** to documents for precise filtering when connecting folders to specific parameters

> [INFO]
> Adding a large number of links to a URL Scraping folder -- several hundred or more, such as an entire support-site crawl -- is best done as an assisted bulk import. Contact your Ema representative rather than adding them one at a time.

### Supported File Formats

Category

Formats

**Production-ready**

PDF, DOCX, PPTX, TXT, HTML, MD

**Limited support**

XLSB, CSV, XLSX, XML, JSON

**Not supported**

Legacy Office binaries (.doc, .xls), source code files, audio/video, native Google Docs/Sheets/Slides

Scanned documents pass through OCR automatically. Diagrams are not interpreted -- add text captions for important visual content.

## 2.6 Add Integrations

> [INFO]
> **Skip this step** if none of your parameters use the Action Based mechanism.

### Available Integrations

Integration

Typical Use Cases

**Salesforce**

Case creation, CRM field updates, opportunity logging

**Zendesk**

Ticket creation and update verification

**Genesys**

Call metadata retrieval, disposition codes

### Setup Steps

1.  In the **Configuration** tab, scroll to **Integrations**
2.  Click **Add Integration** and select the type
3.  Enter credentials -- API URL, API Key or OAuth credentials, integration-specific settings
4.  Click **Test Connection** to verify connectivity
5.  Configure field mappings -- specify which data fields to retrieve and how to match them to call interactions
6.  Click **Save**
7.  Connect the integration to your Action Based parameters in the QA Parameters section

## 2.7 Configure Contact Reasons

Contact reasons categorize **why** customers are calling. This classification powers the analytics in your Metrics dashboard.

> [INFO]
> Each level of your contact reason hierarchy -- and the Call Driver and Competitor Names categorizations -- runs as its own agent in the underlying workflow, wired together automatically when you use the setup below. You don't need to build these agents yourself.

### The Three-Level Hierarchy

Level

Description

Example

**L1**

High-level category

Reservations

**L2**

Detailed breakdown within L1

Modify Reservation

**L3 (Call Driver)**

Specific root cause

Voluntary Date Change

### Configure Non-Resolution Reasons

Non-resolution reasons categorize **why** an issue was not resolved. The default L1 categories are:

L1 Category

Description

Example L2 Subcategories

**People**

Agent-related issues

Knowledge gap, communication issues, insufficient training

**Process**

Workflow or policy limitations

Scope limitation, policy restriction, approval required

**Technical**

System issues

System downtime, integration failure, tool malfunction

**End User**

Customer-related factors

Customer unresponsive, call disconnected, customer declined solution

### Where to Make Changes

What You Want to Do

Where to Do It

**Edit** existing category instructions

Configuration page within your AI Employee

**Add or delete** categories (new L1/L2/L3 entries)

Workflow Builder -- click "Go to AI Employee builder" from the Configuration page

> [TIP]
> Keep categories mutually exclusive -- each call should clearly belong to one category at each level. Aim for 5 to 10 L1 categories, 3 to 8 L2 per L1, and specific L3 call drivers under each L2. Review and update categories quarterly.

> [INFO]
> For a very large or multi-brand/multi-level taxonomy, rebuilding the whole categorization structure in bulk is available -- contact your Ema representative rather than adding entries one by one in the Workflow Builder.

## 2.8 Additional Configuration Tab Settings

### Glossary

Upload company-specific terms and their definitions so Ema understands your organization's vocabulary -- product names, internal abbreviations, or jargon that a general-purpose model wouldn't otherwise recognize.

### Call Processing

Controls how call audio is processed before it undergoes QA evaluation.

By default, audio calls are redacted with **two independent passes** for personally identifiable information (PII) -- rather than a single pass -- so that PII missed by one pass can still be caught by the other. Redacted values are shown as their specific category (e.g. `[FIRST_NAME]`, `[SSN]`) rather than a generic placeholder.

> [INFO]
> A **Hide sensitive information from audit** toggle controls whether redaction is applied, and an expandable **Redacted entity types** list (31 entity types by default) lets you see and adjust which categories of information are removed. Changes apply only to new calls processed after saving -- past interactions are unaffected.

### Output Language

Sets the output language for this persona. Insights, Audit rationale, CSAT, and Coaching Reports are generated in the selected language. The transcript and rule definitions are never translated, and verbatim quotes from the transcript remain in their original language. Changes apply only to new calls processed after saving.

### Configure EmaFusion

By default, Ema intelligently combines multiple large language models to give your persona the best performance. You can instead choose a specific set of models, or bring your own via API details. If you limit to specific models, choose an **Optimization Priority**: Auto (balances accuracy, latency, and cost by task type), Fastest, Cheapest, or Most Accurate. This setting applies to every agent in the persona by default; you can override it per agent if needed.

### Data Retention

Raw call audio and raw uploaded transcripts are automatically deleted after a retention period -- 30 days by default, configurable per tenant or per persona (a persona-level setting overrides the tenant default, which overrides the platform default). Processed transcripts and evaluation records (scores, scorecards, insights) have **no** retention limit and are kept indefinitely.

> [WARNING]
> Once a raw recording or transcript passes its retention period, it can no longer be re-processed -- this is also why **Rerun** (see [4.3 Rerunning an Interaction](/legacy-docs/agent-qa/audit)) is only available for interactions still inside their retention window.

## 2.9 Set Up Permissions

Agent QA uses role-based access control with five roles. See [Roles and Permissions](/legacy-docs/agent-qa/roles) for the full reference.

1.  Click the **Permissions** tab in your AI Employee
2.  Click **Add User**
3.  Enter the user's email address
4.  Select the appropriate role
5.  Click **Send Invitation**

## 2.10 Enable and Send Your First Call

### Pre-Enable Checklist

-   QA parameters configured and approved (via Auto-Learning or manually)
-   Knowledge base connected and documents indexed (if using Knowledge Verified parameters)
-   Integrations added and connections tested (if using Action Based parameters)
-   Contact reasons configured at all three levels (L1, L2, L3)
-   Non-resolution reasons configured (L1 and L2)
-   Permissions assigned and invitations sent

### Enable Your AI Employee

1.  Navigate to the **Configuration** tab
2.  Click **Save changes** if any pending changes are indicated
3.  Click the **Enable** toggle at the top of the page and confirm

### Verify with a Test Call

Upload a test call via the API to verify end-to-end processing:

**API endpoints:**

-   Audio Files: `POST https://api.ema.co/api/v1/external/upload/file`
-   Transcript File: `POST https://api.ema.co/api/v1/external/upload/transcript`

**Required headers:**

Header

Value

`x-persona-id`

Your AI Employee's persona ID (found in Configuration tab URL or settings)

`Authorization`

`Bearer <your-jwt-token>`

**Required form fields:**

Field

Type

Description

`resource_id`

string

A unique identifier for the call (e.g., `test-call-001`)

`agent_id`

string

The agent's UUID in your system

`agent_email`

string

The agent's email address (used for Agent-role permission filtering)

`channels`

string

Number of audio channels (`"1"` for mono, `"2"` for stereo)

`timestamp`

string

Call timestamp in `YYYY-MM-DD HH:MM:SS` format

`is_golden_data`

string

`"false"` for regular evaluation calls

`file`

file

The audio file (MP3, WAV, or Opus, max 25 MB)

**Example (Python):**

```
import requests

API_URL = "https://api.ema.co/api/v1/external/upload/file"
PERSONA_ID = "your-persona-id"
BEARER_TOKEN = "your-jwt-token"

headers = {
 "x-persona-id": PERSONA_ID,
 "Authorization": f"Bearer {BEARER_TOKEN}",
}

data = {
 "resource_id": "test-call-001",
 "agent_id": "agent-uuid",
 "agent_email": "agent@example.com",
 "channels": "1",
 "timestamp": "2024-01-15 14:30:00",
 "is_golden_data": "false",
}

with open("test_call.wav", "rb") as f:
 files = {"file": ("test_call.wav", f, "audio/wav")}
 response = requests.post(API_URL, headers=headers, data=data, files=files)

print(response.status_code, response.text)
```

**Expected response:** HTTP 200 with a confirmation message.

### Verify the Results

1.  Wait 2 to 5 minutes for processing
2.  Navigate to the **Audit** tab
3.  Your test interaction should appear with a status of "QA completed"
4.  Click **View Details** to review the scorecard, transcript, and reasoning

### Troubleshooting a Failed Test Call

Issue

What to Check

**Unsupported format**

Audio must be MP3, WAV, or Opus. Transcript can be JSON, XML, or HTML.

**File too large**

Must be under 25 MB. Compress or split longer recordings.

**Poor audio quality**

Ensure clear speech. Very short recordings (under 10 seconds) or recordings with no speech may fail.

**Authentication error**

Verify your bearer token is valid and the `x-persona-id` header matches your AI Employee.

**AI Employee disabled**

Confirm the Enable toggle is set to Enabled.