> Source: https://builder-docs.ema.ai/builder-guides/advanced/scraping-web-links
> Title: Scraping Web Links

# Scraping Web Links

> This guide explains how to configure an AI Employee to crawl and ingest content from specific web pages. Web scraping lets you use publicly available web content as a data source for your workflows, keeping your AI Employee's knowledge base current with information from websites, help centers, and documentation portals.

## Prerequisites

-   The specific URLs you wish to scrape and ingest
-   Access to the Ema platform dashboard

## Part 1: Creating the AI Employee

### Step 1: Create a New AI Employee

Navigate to Ema's **AI Employees** page via the main dashboard. Click the **Create New Employee** button.

### Step 2: Configure the AI Employee Profile

Select an AI Employee template (for example, Conversation AI Employee). Fill in the required fields: name, description, and type. Proceed to the configuration area once the basic profile is set up.

## Part 2: Workflow Configuration and Web Scraping

### Step 3: Access Shared Configurations

On the AI Employee configuration page, navigate to the **Workflow Builder**. Locate and click the **Shared Configurations** button in the top-left corner of the interface.

### Step 4: Add a Web Scraping Data Source

In the Shared Configuration modal, click the **Add** button and select the option to **Scrape Web Links**.

### Step 5: Configure Ingestion and Import Settings

Paste the specific web links you wish to crawl into the **Base URL** field. A URL is required to proceed.

Before starting the scrape, configure your **Import Settings**:

Setting

Description

Default

**Maximum pages**

The total number of web pages to import from this URL.

200 (minimum: 1)

**Website navigation depth**

How many folder levels deep the crawler should explore. A depth of 2 includes paths like `/policies/hr/`.

2 (minimum: 1)

> [INFO]
> If no custom values are provided, the system uses the defaults listed above. Ema clicks on all links on the page and scrapes downstream pages up to the configured depth. Ema also downloads any documents such as PDFs encountered during the scraping.

### Step 6: Ingest Web Links

After configuring import settings, the system begins processing the URLs to extract and ingest the content.

## Part 3: Verification and Status

### Step 7: Verify Completion

Wait for the processing status to update. You will see a confirmation status indicating **Completed** once the scraping process finishes. Web link scraping can take a few minutes depending on the number of pages and depth configured.

## Part 4: Synchronization Settings

### Step 8: Configure Sync Options

Once the web links have been successfully scraped, you can manage their synchronization frequency to keep the AI Employee's knowledge up to date.

-   **Manual sync** -- Click the three dots (**...**) next to the source in the root folder and select **Sync Now** to refresh data manually.
-   **Automatic sync** -- Click the **Auto Sync** dropdown and select a sync interval: **1, 7, 15, or 30 days**.

> [TIP]
> For frequently updated websites such as help centers or documentation portals, set automatic sync to 1 or 7 days. For relatively static content, 15 or 30 days is sufficient.