Skip to main content
POST
/
1
/
crawlers
/
{id}
/
test
Test crawl a URL
curl --request POST \
  --url https://crawler.algolia.com/api/1/crawlers/{id}/test \
  --header 'Authorization: Basic <encoded-value>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://www.algolia.com/blog"
}
'
{
  "startDate": "2024-04-02T15:34:29Z",
  "endDate": "2024-04-02T15:34:29Z",
  "logs": [
    [
      "Processing url 'https://www.algolia.com/blog'"
    ]
  ],
  "records": [
    {
      "indexName": "testIndex",
      "records": [
        {
          "objectID": "https://www.algolia.com/blog",
          "numberOfLinks": 2
        }
      ],
      "recordsPerExtractor": [
        {
          "index": 0,
          "type": "custom",
          "records": [
            {
              "objectID": "https://www.algolia.com/blog"
            }
          ]
        }
      ]
    }
  ],
  "links": [
    "https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine",
    "https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity"
  ],
  "externalData": {
    "externalData1": {
      "data1": "val1",
      "data2": "val2"
    },
    "externalData2": {
      "data1": "val1",
      "data2": "val2"
    }
  },
  "error": {}
}

Authorizations

Authorization
string
header
required

Basic authentication header of the form Basic <encoded-value>, where <encoded-value> is the base64-encoded string username:password.

Path Parameters

id
string
required

Crawler ID. Universally unique identifier (UUID) of the crawler.

Example:

"e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"

Body

application/json
url
string
required

URL to test.

Example:

"https://www.algolia.com/blog"

config
object

Crawler configuration to update. You can only update top-level configuration properties. To update a nested configuration, such as actions.recordExtractor, you must provide the complete top-level object such as actions.

Response

OK

startDate
string
required

Date and time when the test crawl started, in RFC 3339 format.

Example:

"2024-04-02T15:34:29Z"

endDate
string
required

Date and time when the test crawl finished, in RFC 3339 format.

Example:

"2024-04-02T15:34:29Z"

logs
string[][]
required

Logs from the record extraction.

Parameters associated with each log.

records
extractedRecords Ā· object[]
required

Extracted records from the URL.

Links found on the page, which match the configuration and would be processed.

Example:
[
"https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine",
"https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity"
]
externalData
object

External data associated with the tested URL.

External data is refreshed automatically at the beginning of the crawl.

Example:
{
"externalData1": { "data1": "val1", "data2": "val2" },
"externalData2": { "data1": "val1", "data2": "val2" }
}
error
urlTestError Ā· object

An error.

Example:
{}
Last modified on February 20, 2026