> **Building with AI coding agents?** Install the authstack plugin with one command. This equips your agent with accurate Scalekit implementation patterns. > > **Recommended** (the `-y` flags skip prompts, so the command runs without an interactive terminal): > ```bash > npx -y @scalekit-inc/cli setup -y > ``` > > Global: > ```bash > npm install -g @scalekit-inc/cli > scalekit setup -y > ``` > > Supports Claude Code, Cursor, GitHub Copilot, Codex + skills for other Agent Skills-compatible agents in `~/.agents/skills`. > Skills: integrate-agentkit, implement-saaskit, add-mcp-oauth, implement-sso, implement-scim. > [Full setup guide](https://docs.scalekit.com/dev-kit/build-with-ai/) --- # Connect AI agents to Algolia Crawler The Algolia Crawler connector lets your AI agent act in each user's Algolia Crawler account. Each user connects their own Algolia Crawler username and password once, and Scalekit sends it with every call, so your agent never handles credentials. It comes with 20 tools. **Authentication:** Basic Auth **Categories:** Search, Developer Tools **Tools:** 20: 9 read, 9 write, 2 destructive **Users sign in with:** Username and password **Built by:** Scalekit connector **Try it:** [Playground in the Scalekit dashboard](https://app.scalekit.com/ws/signup?sk_intent=playground&provider=ALGOLIACRAWLER) ## Setup 1. ### Install the SDK **Node.js** ```bash npm install @scalekit-sdk/node dotenv ``` **Python** ```bash pip install scalekit-sdk-python python-dotenv ``` 2. ### Set your credentials Add your Scalekit credentials to your `.env` file. Find values in **[app.scalekit.com](https://app.scalekit.com)** > **Developers** > **API Credentials**. ```sh title=".env" SCALEKIT_ENVIRONMENT_URL= SCALEKIT_CLIENT_ID= SCALEKIT_CLIENT_SECRET= ``` 3. ### Create the Algolia Crawler connection In **AgentKit > Connections**, create an Algolia Crawler connection. The name you give it is the `connection_name` your code passes. See [Configure connections](/agentkit/connections/). ## Tools Pass the exact name to `execute_tool`, with an input like each tool's example. To ask for a tool that's missing, use the [request form](https://scalekitsupport.portal.usepylon.com/forms/request-a-connector-tool). **Node.js** ```typescript const result = await actions.executeTool({ toolName: 'algoliacrawler_get_config_version', toolInput: { id: '', version: 3, }, connector: 'algoliacrawler', identifier: 'user_123', }) ``` **Python** ```python result = actions.execute_tool( tool_name="algoliacrawler_get_config_version", tool_input={ "id": "", "version": 3, }, connection_name="algoliacrawler", identifier="user_123", ) ``` `result.data` is the app's response as JSON, and `result.execution_id` (`executionId` in Node.js) is the ID of the call. ### `algoliacrawler_get_config_version` Get Config Version · Read-only Retrieve one saved version of a crawler's configuration. Returns the version number, creation time, author id, and the full configuration object for that version (start URLs, actions, index prefix, schedule, rate limit, and other crawler settings). Use this to inspect or restore an earlier configuration. Use list_config_versions to find version numbers and get_crawler for the current configuration. Requires a crawler id from list_crawlers and a version number from list_config_versions. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `version` (`integer`, required): Version number of the crawler configuration to retrieve. Version 1 is the initial configuration used when the crawler was created. Use list_config_versions to find valid numbers. Example: 3. Example input: `{"id":"","version":3}` ### `algoliacrawler_get_crawl_run_file` Get Crawl Run File · Read-only Retrieve the downloadable log file for a single crawl run of a crawler. Returns a JSON object with a file string holding the log file content for that run. Use this to debug or audit one crawl run. Use list_crawl_runs first to find the log id, and get_url_stats for aggregate counts instead of per-run logs. Requires a crawler id from list_crawlers and a log id from list_crawl_runs. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `logId` (`string`, required): Unique ID (UUID) of the crawler log (crawl run) to download. Use list_crawl_runs to find it. Example: a2ebb507-ef64-4b6b-9d84-ef66baaa7a80. Example input: `{"id":"","logId":""}` ### `algoliacrawler_get_crawler` Get Crawler · Read-only Get the details of one Algolia crawler by id, optionally including its full configuration. Returns the crawler name, created and updated timestamps, whether it is running, reindexing, or blocked (with the blocking error and blocking task id when blocked), and the last reindex start and end times; with the configuration option set, also returns the configuration. Use this for the state or configuration of a known crawler. Use list_crawlers to find ids and get_task_status to follow an asynchronous task. Requires a crawler id from list_crawlers or create_crawler. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to retrieve. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `withConfig` (`boolean`, optional): Set to true to include the crawler's full configuration in the response. Leave empty or false to return only status information such as name, timestamps, and running state. Example: true. Example input: `{"id":""}` ### `algoliacrawler_get_task_status` Get Task Status · Read-only Retrieve the status of a crawler task, showing whether it is still pending or has completed. Returns an object with a pending boolean. Use this to poll a task started by another crawler tool, such as a run or reindex, until pending is false. Use cancel_task to unblock a crawler whose task failed. Requires a crawler id from list_crawlers and a task id returned by the tool that started the task. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `taskID` (`string`, required): Unique ID (UUID) of the task to check. It is returned when a crawler action such as run, reindex, or crawl URLs is started. Example: 98458796-b7bb-4703-8b1b-785c1080b110. Example input: `{"id":"","taskID":""}` ### `algoliacrawler_get_url_stats` Get URL Stats · Read-only Retrieve crawl statistics for a crawler, broken down by URL status. Returns the total count of crawled URLs and a data array where each entry has a status (DONE, SKIPPED, or FAILED), a category (fetch, extraction, indexing, or success), a reason, a readable explanation, and the number of URLs with that status. Use this to see why URLs were skipped or failed. Use list_crawl_runs for per-run history and get_crawler for configuration. Requires a crawler id from list_crawlers. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. Example input: `{"id":""}` ### `algoliacrawler_list_config_versions` List Config Versions · Read-only List the saved configuration versions of a crawler, including who authored each change. Returns a paginated response with the current page, items per page, total count, and an items array of version number, creation time, and author id. Every configuration update adds a new version. Use this to find a version number before calling get_config_version. The list does not include the configuration itself. Requires a crawler id from list_crawlers. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `itemsPerPage` (`integer`, optional): Number of configuration versions to return per page, from 1 to 100. Defaults to 20. Use together with page to walk through large result sets. Example: 20. Default: `20`. - `page` (`integer`, optional): Page of results to retrieve, from 1 to 100. Defaults to 1. Compare the total in the response with itemsPerPage to decide whether more pages exist. Example: 1. Default: `1`. Example input: `{"id":""}` ### `algoliacrawler_list_crawl_runs` List Crawl Runs · Read-only List the recorded crawl runs (crawler logs) for a crawler, optionally filtered by date range or URL status. Returns a logs array and a meta object with the total number of matching records. Each log has its id, config id, reindex id, start and completion times, file sizes, expiry, status, access count, and counts of done, skipped, and failed URLs. Use this to find a log id before calling get_crawl_run_file or delete_crawl_runs. Use get_url_stats for aggregate URL status counts across the crawler. Requires a crawler id from list_crawlers. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `from` (`string`, optional): Only return runs on or after this date, as a Unix timestamp in seconds, passed as a string. Leave empty for no lower bound. Example: 1762264044. - `limit` (`integer`, optional): Maximum number of runs to return, from 1 to 1000. Defaults to 10. Use together with offset to page through results. Example: 10. Default: `10`. - `offset` (`integer`, optional): Number of runs to skip before returning results, used for pagination together with limit. Leave empty to start from the first run. Example: 10. - `order` (`string`, optional): Sort direction of the results, either ASC or DESC. Leave empty to use the server default order. Example: DESC. One of: `ASC`, `DESC`. - `status` (`string`, optional): Filter runs by crawled URL status. Must be one of DONE, SKIPPED, or FAILED. Leave empty to return runs regardless of status. Example: FAILED. One of: `DONE`, `SKIPPED`, `FAILED`. - `until` (`string`, optional): Only return runs on or before this date, as a Unix timestamp in seconds, passed as a string. Leave empty for no upper bound. Example: 1762264044. Example input: `{"id":""}` ### `algoliacrawler_list_crawlers` List Crawlers · Read-only List the Algolia crawlers on the account, optionally filtered by crawler name or Algolia application ID. Returns a paginated response with the current page, items per page, total count, and an items array of crawler ids and names. Use this to find a crawler id before calling a tool that needs one. The list only carries id and name, so use the get-crawler tool for configuration details. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `itemsPerPage` (`integer`, optional): Number of crawlers to return per page, from 1 to 100. Defaults to 20. Use together with page to walk through large result sets. Default: `20`. - `name` (`string`, optional): Crawler name used to filter the response, up to 64 characters. Leave empty to list crawlers regardless of name. Example: test-crawler. - `page` (`integer`, optional): Page of results to retrieve, from 1 to 100. Defaults to 1. Compare the total in the response with itemsPerPage to decide whether more pages exist. Default: `1`. Example input: `{}` ### `algoliacrawler_list_domains` List Domains · Read-only List the domains registered for crawling on the account, optionally filtered by Algolia application ID. Returns a paginated response with the current page, items per page, total count, and an items array of domain name, Algolia application ID, and whether the domain is validated. Use this to check which domains crawlers are allowed to run against, since crawlers only run if their URLs match a registered domain. Use list_crawlers to list crawlers instead. Read only: yes. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `itemsPerPage` (`integer`, optional): Number of domains to return per page, from 1 to 100. Defaults to 20. Use together with page to walk through large result sets. Example: 20. Default: `20`. - `page` (`integer`, optional): Page of results to retrieve, from 1 to 100. Defaults to 1. Compare the total in the response with itemsPerPage to decide whether more pages exist. Example: 1. Default: `1`. Example input: `{}` ### `algoliacrawler_cancel_task` Cancel Task · Write Cancel a blocking task on a crawler so its schedule can resume. Returns an empty successful response when the task is cancelled. Use this when a task ran into an error and is blocking the crawler's schedule. Use get_task_status to check whether a task is still pending before cancelling. Requires a crawler id from list_crawlers and the id of the blocking task. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `taskID` (`string`, required): Unique ID (UUID) of the blocking task to cancel. Tasks that ran into an error block the crawler schedule until cancelled. Example: 98458796-b7bb-4703-8b1b-785c1080b110. Example input: `{"id":"","taskID":""}` ### `algoliacrawler_crawl_urls` Crawl URLs · Write Crawl the specified URLs, extract records from them, and add the records to the crawler's index. Returns a taskId acknowledging the action. Use this to refresh specific pages without a full crawl; use start_reindex for a full crawl and test_url to preview extraction without indexing. If a crawl is already running, the records go to a temporary index. This operation is rate limited to 500 requests every 24 hours. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to crawl URLs with. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `urls` (`array`, required): List of full URLs to crawl, each including the scheme. Provide at least one. Example: ["https://www.algolia.com/products/crawler/"]. - `save` (`boolean`, optional): Whether to add the URLs to the crawler configuration's extra URLs list. Set true to always save them, false to never save them. If left empty, the URLs are saved only if they were not indexed during the last reindex. Example: true. Example input: `{"id":"","urls":["https://www.algolia.com/products/crawler/"]}` ### `algoliacrawler_create_crawler` Create Crawler · Write Create a new Algolia crawler from a name and a full configuration. Returns the id (UUID) of the new crawler. Use this to set up a crawler that does not exist yet. Use list_crawlers first to check for an existing crawler with the same name, and update_crawler or update_crawler_config to change one that exists. To start crawling afterwards, use start_reindex. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `config` (`object`, required): Full crawler configuration object. A configuration must name the Algolia application the crawler writes to, a rate limit from 1 to 100 that sets how many concurrent tasks run per second (start low, for example 2 to 4), and a non-empty list of actions (up to 30). Each action names the target index and carries a record extractor, which is an object marked as a function whose source is a JavaScript function string that turns a crawled page into Algolia records; an action can also limit which URLs it applies to with path patterns. Add start URLs or sitemaps so the crawler knows where to begin. The Algolia API key is optional on creation (the Crawler generates one if omitted) and must never be an Admin API key. Minimal working example: {"appId": "YourApplicationID", "apiKey": "YourCrawlerIndexingApiKey", "rateLimit": 4, "startUrls": ["https://www.example.com"], "actions": [{"indexName": "example_index", "pathsToMatch": ["https://www.example.com/**"], "recordExtractor": {"__type": "function", "source": "({ url, $ }) => [{ objectID: url.href, title: $('head title').text() }]"}}]} - `name` (`string`, required): Name for the new crawler, up to 64 characters. Use a short descriptive label so it is easy to find with the list crawlers tool. Example: test-crawler. Example input: `{"config":{"rateLimit":8,"maxUrls":5000},"name":""}` ### `algoliacrawler_pause_crawler` Pause Crawler · Write Pause the specified crawler. Returns a taskId acknowledging the action. Use this to stop a crawler temporarily; use run_crawler to resume it. Use delete_crawler to remove a crawler that is no longer needed. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to pause. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. Example input: `{"id":""}` ### `algoliacrawler_run_crawler` Unpause Crawler · Write Unpause the specified crawler so it resumes work. Returns a taskId acknowledging the action. Use this to undo pause_crawler: previously ongoing crawls resume, otherwise the crawler waits for its next scheduled run. Use start_reindex to begin a new crawl right away. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to unpause. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. Example input: `{"id":""}` ### `algoliacrawler_start_reindex` Start Reindex · Write Start or resume a crawl of the crawler's configured URLs. Returns a taskId acknowledging the action. Use this to run a full crawl now instead of waiting for the schedule. Use crawl_urls to crawl only specific URLs, and run_crawler only to unpause a paused crawler. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to start a crawl for. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. Example input: `{"id":""}` ### `algoliacrawler_test_url` Test URL · Write Test crawl one URL with the crawler's configuration, optionally with configuration overrides, and show the records it would extract. Returns the test start and end dates, extraction logs, the extracted records grouped by index name, links found on the page, any external data, and an error if one occurred. Use this to preview a configuration change before saving it with update_crawler_config. It does not add records to the index; use crawl_urls to actually index URLs. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to test with. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `url` (`string`, required): Full URL of the web page to test crawl, including the scheme. It should match the crawler's configured start URLs or path patterns. Example: https://www.algolia.com/blog. - `config` (`object`, optional): Optional configuration overrides to try during the test, as a JSON object. Only top-level configuration properties can be overridden: to change something nested, such as a record extractor inside actions, send the complete top-level property (the whole actions list). The published API schema lists the application ID, rate limit, and actions as required for a configuration, so if a partial override is rejected, send a complete configuration. Leave empty to test with the crawler's saved configuration. Example overriding the actions: {"actions": [{"indexName": "example_index", "pathsToMatch": ["https://www.example.com/**"], "recordExtractor": {"__type": "function", "source": "({ url, $ }) => [{ objectID: url.href, title: $('head title').text() }]"}}]} Example input: `{"id":"","url":""}` ### `algoliacrawler_update_crawler` Update Crawler · Write Rename a crawler or replace its entire configuration with a new one. Returns a taskId for the asynchronous update. Use this to rename a crawler, or to overwrite the whole configuration when versioning is not needed. Use update_crawler_config instead to change individual top-level settings with a new configuration version recorded. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to update. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `config` (`object`, optional): Complete replacement configuration for the crawler. Because this operation replaces the whole configuration, send every setting you want to keep, not only the ones you are changing; anything omitted is dropped. Changes made here are not versioned. A configuration must name the Algolia application the crawler writes to, a rate limit from 1 to 100 that sets how many concurrent tasks run per second (start low, for example 2 to 4), and a non-empty list of actions (up to 30). Each action names the target index and carries a record extractor, which is an object marked as a function whose source is a JavaScript function string that turns a crawled page into Algolia records; an action can also limit which URLs it applies to with path patterns. Add start URLs or sitemaps so the crawler knows where to begin. The Algolia API key is optional on creation (the Crawler generates one if omitted) and must never be an Admin API key. Minimal working example: {"appId": "YourApplicationID", "apiKey": "YourCrawlerIndexingApiKey", "rateLimit": 4, "startUrls": ["https://www.example.com"], "actions": [{"indexName": "example_index", "pathsToMatch": ["https://www.example.com/**"], "recordExtractor": {"__type": "function", "source": "({ url, $ }) => [{ objectID: url.href, title: $('head title').text() }]"}}]} - `name` (`string`, optional): New name for the crawler, up to 64 characters. Provide this to rename the crawler; leave empty to keep the current name. Example: test-crawler. Example input: `{"id":""}` ### `algoliacrawler_update_crawler_config` Update Crawler Configuration · Write Update top-level settings of a crawler's configuration, creating a new configuration version. Returns a taskId for the asynchronous update. Use this for versioned configuration changes. Use update_crawler to rename a crawler or replace its whole configuration without versioning, and list_config_versions to review earlier versions. Requires a crawler id from list_crawlers. Read only: no. Destructive: no. Idempotent (safe to retry): no. Inputs: - `config` (`object`, required): Configuration properties to update, as a JSON object. Only top-level configuration properties can be changed: to change something nested, such as a record extractor inside actions, send the complete top-level property (the whole actions list). Properties you do not include are left as they are. The published API schema lists the application ID, rate limit, and actions as required for a configuration, so if a partial update is rejected, send the full configuration including them. Each update creates a new configuration version. Example: {"rateLimit": 8, "maxUrls": 5000} - `id` (`string`, required): Crawler ID (UUID) of the crawler to update the configuration of. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. Example input: `{"config":{"rateLimit":8,"maxUrls":5000},"id":""}` ### `algoliacrawler_delete_crawl_runs` Delete Crawl Runs · Destructive Delete the recorded crawl run logs with the given log ids from a crawler. This removes the log records only, not the crawler itself. Returns an array of the log ids that were submitted. Use this to clean up old run logs. Use list_crawl_runs to find log ids, and delete_crawler to remove the crawler. Deletion cannot be undone. Requires a crawler id from list_crawlers and one or more log ids from list_crawl_runs. Read only: no. Destructive: yes. Idempotent (safe to retry): no. Inputs: - `id` (`string`, required): Unique ID (UUID) of the crawler. Use list_crawlers to find it. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. - `logIds` (`array`, required): List of crawler log (crawl run) UUIDs to delete. Use list_crawl_runs to find them. Example: ["a2ebb507-ef64-4b6b-9d84-ef66baaa7a80"]. Example input: `{"id":"","logIds":["a2ebb507-ef64-4b6b-9d84-ef66baaa7a80"]}` ### `algoliacrawler_delete_crawler` Delete Crawler · Destructive Delete the specified Algolia crawler. Returns a taskId acknowledging the deletion. Use this only when the crawler is no longer needed. Use pause_crawler to stop a crawler temporarily instead. Requires a crawler id from list_crawlers. Read only: no. Destructive: yes. Idempotent (safe to retry): yes. Inputs: - `id` (`string`, required): Crawler ID (UUID) of the crawler to delete. Get it from the list crawlers tool or from the response of the create crawler tool. Example: e0f6db8a-24f5-4092-83a4-1b2c6cb6d809. Example input: `{"id":""}` ## Related Other Search connectors ([all 35](/agentkit/connectors/?category=search)): - [Supadata](/agentkit/connectors/supadata/): Scalekit connector, API key, 21 tools - [Exa](/agentkit/connectors/exa/): Scalekit connector, API key, 21 tools - [Brave Search](/agentkit/connectors/brave/): Scalekit connector, API key, 18 tools --- ## More Scalekit documentation | Resource | What it contains | When to use it | |----------|-----------------|----------------| | [/llms.txt](/llms.txt) | Structured index with routing hints per product area | Start here — find which documentation set covers your topic before loading full content | | [/llms-full.txt](/llms-full.txt) | Complete documentation for all Scalekit products in one file | Use when you need exhaustive context across multiple products or when the topic spans several areas | | [sitemap-0.xml](https://docs.scalekit.com/sitemap-0.xml) | Full URL list of every documentation page | Use to discover specific page URLs you can fetch for targeted, page-level answers |