Web crawler

NOTE: Pages to crawl must be publicly accessible.

Automatically gather and synchronize content from specified, publicly accessible websites into your assistant. Theta Assist's web crawler connects your assistant to your chosen web sources, for accurate, relevant and referenced responses based on your website content.

In this article

  • Key features
  • Getting started
    • Adding URLs to Crawl
    • Starting a web crawl
  • Monitoring crawl progress
    • History page
    • Details page
    • Status definitions
  • Resource limitations
  • File synchronization
  • Troubleshooting
    • Common Issues
  • Cancelling a crawl
  • Deleting a crawl 
  • Best practices
  • Advanced features
    • URL run limits
    • Queue position tracking

Key features

  • Multi-URL support: Add multiple websites to be crawled

  • Selective link following: Choose whether to follow links, per URL

  • Exclusions: Prevent specific URLs from being crawled

  • Smart updates: Detect and sync only changed content

  • Detailed progress monitoring: Track crawl status and progress in real-time

  • Queue management: Schedule and prioritize crawls when system resources are limited

  • Cancellation: Stop crawls in progress if needed

Getting started

Adding URLs to Crawl

  1. If you haven't already, select Show Advanced Options - this appears below the Instructions field when editing your assistant. Click Save.

  2. Navigate to your assistant's Web Crawl tab, which should now be visible.

  3. Add URLs you want to include in the crawl. Note that the pages must be publicly accessible to be able to be crawled.

    • Enter the complete URL (e.g. https://your-website.com/documentation)

    • Select Follow Links if you want the crawler to follow links on this page

    • Each URL will be assigned a unique identifier for tracking

  4. Optionally, add URLs to exclude from the crawl and select Exclude.
    Note, you can also use * at the end of a URL path for wildcard matching. For example, https://your-website.com/news/* will exclude any pages below this, eg https://your-website.com/news/article-one etc.

Starting a web crawl

  1. After configuring your URLs, click Start Web Crawl (bottom right)

  2. The system will initiate a new crawl job

  3. You'll be redirected to the web crawl history page, where you can monitor progress

Monitoring crawl progress

History page

The Web crawl history page displays a list of all your crawl runs with:

  • Creation date & time

  • Current status (Pending, Processing, Completed, Error, or Cancelled)

  • Completion date & time (if applicable)

  • A Details button to view full information

 

 

Details page

The details page provides comprehensive information about a specific crawl:

Crawl overview

  • Status: Current state of the crawl

  • Created: When the crawl was initiated

  • Last Activity: Most recent update time

  • Completed: When the crawl finished (if applicable)

  • Total Duration: How long the crawl took to complete

  • Overall Progress: Percentage and number of pages visited vs. total pages

Queue position

If your crawl is pending, you'll see:

  • Current position in the queue

  • Total crawls waiting

  • Number of crawls currently running

  • Maximum parallel crawls allowed by the system

URL progress details

A table showing for each URL:

  • URL being crawled

  • Status

  • Whether links are being followed

  • Progress percentage

  • Detailed counts (visited pages, queued pages)

  • Limit indicators (if the maximum page limit was reached)

Log information

A detailed log showing:

  • Timestamped entries of crawl activities

  • Information about files uploaded

  • Any errors or warnings encountered

  • Link discovery information

Status definitions

  • Pending: Crawl is waiting in queue to start

  • Processing: Crawl is actively running

  • Completed: Crawl finished successfully

  • Error: Crawl encountered an error

  • Cancelled: Crawl was manually cancelled

Resource limitations

The system has several built-in limitations to ensure optimal performance:

  • Maximum URL Limit: Each URL can crawl up to a predetermined number of pages (specified in your instance's configuration)

  • Parallel Crawl Limit: The system limits how many crawls can run simultaneously

  • Queue System: When the parallel limit is reached, new crawls are placed in a queue

File synchronization

The web crawler performs smart synchronization:

  1. Change Detection: Files are only updated when content has changed (using hash comparison)

  2. Pruning: Files that no longer exist on the website are automatically removed

  3. Progress Updates: All synchronization activities are logged

To synchronize your assistant with the latest version of your web pages, start a new crawl.

Troubleshooting

Common Issues

Crawl Stuck in Pending Status

  • This is normal when the system has reached its maximum parallel crawl limit

  • Check the Queue Position section to see your place in line

  • The crawl will start automatically when resources become available

Crawl completed but missing content

  • Check if the "Limit Reached" indicator appears for any URLs

  • Consider breaking your crawl into smaller, more targeted URLs

  • Check the logs for any errors or skipped pages

Crawl ended with error status

  • Review the Log Information section for specific error details

  • Common causes include website authentication requirements, robots.txt restrictions, or network issues

Cancelling a crawl

If you need to stop a crawl in progress:

  1. Navigate to the Web Crawl Details page for the running crawl

  2. Click the "Cancel Crawl" button in the top right

  3. Confirm the cancellation in the dialog

  4. The crawl will stop and show "Cancelled" status

Deleting a crawl 

If you no longer wish your assistant to have access to the knowledge generated in a web crawl, you can delete the associated file.

Navigate to your assistant's Files tab to delete the relevant file.

 

Best practices

  1. Start Small: Begin with a few key pages rather than entire websites

  2. Use exclusion patterns: Use /* at the end of a URL path to exclude large sections of your site that are not relevant

  3. Monitor progress: Check the details page during large crawls to ensure it's progressing as expected

  4. Schedule during off-hours: Large crawls may be best run during low-traffic periods

  5. Regular Updates: Schedule periodic re-crawls to keep your assistant's knowledge fresh

Advanced features

URL run limits

Each URL has a maximum page limit defined in your system configuration. When this limit is reached:

  • A "Limit Reached" indicator will appear in the URL progress section

  • The crawler will stop processing new links for that URL

  • Other URLs in the same crawl will continue processing

Queue position tracking

For pending crawls, the system provides detailed queue information:

  • Your position in the queue

  • Total number of crawls waiting

  • Number of crawls currently running

  • Maximum parallel crawls allowed

This helps you estimate when your crawl will start processing.