The Crawlbase connector lets your AI agent act in each user's Crawlbase account. Each user connects their own Crawlbase API key once, and Scalekit sends it with every call, so your agent never handles credentials. It comes with 14 tools.
- Tools
- 14
- What they doRead · write · destructive
- 8 · 4 · 28 read4 write2 destructive
- Users sign in with
- API key
Setup
Install the SDK
Terminal window npm install @scalekit-sdk/node dotenvTerminal window pip install scalekit-sdk-python python-dotenvSet your credentials
Add your Scalekit credentials to your
.envfile. Find values in app.scalekit.com > Developers > API Credentials..env SCALEKIT_ENVIRONMENT_URL=<your-environment-url>SCALEKIT_CLIENT_ID=<your-client-id>SCALEKIT_CLIENT_SECRET=<your-client-secret>Create the Crawlbase connection
In AgentKit > Connections, create a Crawlbase connection. The name you give it is the
connection_nameyour code passes. See Configure connections.Authorize a user and make your first call
quickstart.mts import { ScalekitClient } from '@scalekit-sdk/node'import 'dotenv/config'import { createInterface } from 'node:readline/promises'const scalekit = new ScalekitClient(process.env.SCALEKIT_ENVIRONMENT_URL,process.env.SCALEKIT_CLIENT_ID,process.env.SCALEKIT_CLIENT_SECRET,)const actions = scalekit.actionsconst connector = 'crawlbase'const identifier = 'user_123'// Generate an authorization link for the userconst { link } = await actions.getAuthorizationLink({ connectionName: connector, identifier })console.log('Authorize Crawlbase:', link)const rl = createInterface({ input: process.stdin, output: process.stdout })await rl.question('Press Enter after authorizing...')rl.close()// Make your first callconst result = await actions.executeTool({connector,identifier,toolName: 'crawlbase_storage_get',toolInput: {},})console.log(result)Terminal window npx tsx quickstart.mtsquickstart.py import osfrom scalekit import ScalekitClientfrom dotenv import load_dotenvload_dotenv()scalekit_client = ScalekitClient(env_url=os.getenv("SCALEKIT_ENVIRONMENT_URL"),client_id=os.getenv("SCALEKIT_CLIENT_ID"),client_secret=os.getenv("SCALEKIT_CLIENT_SECRET"),)actions = scalekit_client.actionsconnection_name = "crawlbase"identifier = "user_123"# Generate an authorization link for the userlink_response = actions.get_authorization_link(connection_name=connection_name,identifier=identifier,)print("Authorize Crawlbase:", link_response.link)input("Press Enter after authorizing...")# Make your first callresult = actions.execute_tool(tool_input={},tool_name="crawlbase_storage_get",connection_name=connection_name,identifier=identifier,)print(result)Terminal window python quickstart.pyEach user opens the link once and enters their Crawlbase credentials there. See Authorize a user for the full flow and statuses.
Tools
Pass the exact name toexecute_toolcrawlbase_account_usage_getGet this month's usage statistics for one Crawlbase product.Read-onlyGet Account Usage
Get this month's usage statistics for one Crawlbase product. Returns totalSuccess, totalFailed, totalDue, remainingCredits (subscription products) and domainStats per domain. Use this to monitor credit consumption. Rate limited to 1 request every 5 minutes, so cache the result.
Inputs
productstringrequired- Product whose usage to return. Example: crawling-api.one of
crawling-apicrawlersmartproxyscraper-apileads-apiscreenshot-api previous_monthboolean- Set to true to include the previous month's statistics too. Example: true.
crawlbase_page_crawlFetch any web page through Crawlbase's proxy network, with optional headless-browser rendering, anti-bot bypass and geo-routing.Read-onlyCrawl Page
Fetch any web page through Crawlbase's proxy network, with optional headless-browser rendering, anti-bot bypass and geo-routing. Returns, with the default format json, an object with cb_status, original_status, url, body (the page HTML) and rid when stored; format html returns raw HTML and format md returns Markdown. Use this for raw page content. Use crawlbase_page_markdown_get for clean article text, crawlbase_page_scrape for structured data from a supported site and crawlbase_page_crawl_async for slow pages.
Inputs
urlstringrequired- Full URL of the page to crawl, including http:// or https://. The platform URL-encodes it. Example: https://www.example.com/products/123.
ajax_waitboolean- Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
cookiesstring- Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
cookies_sessionstring- Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
countrystring- Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
css_click_selectorstring- CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
custom_success_codesstring- Comma-separated HTTP status codes from the site to treat as success instead of retrying. These requests are charged. Example: 403,429.
devicestring- Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of
desktoptabletmobile formatstring- Response envelope: json (page plus metadata, default for this tool), html (raw page) or md (GitHub-flavored Markdown). Example: json.one of
jsonhtmlmddefaultjson get_cookiesboolean- Set to true to include the site's Set-Cookie values (as original_set_cookie). Example: true.
get_headersboolean- Set to true to include the site's response headers (under original_headers with format json). Example: true.
heightinteger- Screenshot height in pixels with mode viewport. Example: 800.
javascriptboolean- Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
modestring- With screenshot true, viewport captures only the visible area (see width and height); the default is the full page. Example: viewport.one of
viewport page_waitinteger- Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
prettyboolean- Set to true to pretty-print the JSON envelope. Example: false.
request_headersstring- Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
screenshotboolean- Set to true to also capture a JPEG of the rendered page; its temporary screenshot_url (valid one hour) comes back in the JSON. JavaScript rendering only. Example: true.
scrollboolean- Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
scroll_intervalinteger- Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
storeboolean- Set to true to also save the result in Crawlbase Cloud Storage and get back an rid for crawlbase_storage_get. Example: true.
tor_networkboolean- Set to true to route over the Tor network, for .onion sites only. Example: false.
user_agentstring- Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
widthinteger- Screenshot width in pixels with mode viewport. Example: 1200.
crawlbase_page_markdown_getFetch a web page through Crawlbase and convert it to GitHub-flavored Markdown, optionally keeping only the main article content.Read-onlyGet Page as Markdown
Fetch a web page through Crawlbase and convert it to GitHub-flavored Markdown, optionally keeping only the main article content. Returns the page as Markdown text. Use this to read articles, docs or blog posts as clean LLM context. Use crawlbase_page_crawl for HTML or metadata.
Inputs
urlstringrequired- Full URL of the page to crawl, including http:// or https://. The platform URL-encodes it. Example: https://www.example.com/products/123.
ajax_waitboolean- Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
cookiesstring- Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
cookies_sessionstring- Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
countrystring- Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
css_click_selectorstring- CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
devicestring- Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of
desktoptabletmobile javascriptboolean- Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
md_readabilityboolean- Set to true (default) to drop navigation, sidebars, footers and ads and keep the main content before converting. Example: true.default
true page_waitinteger- Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
request_headersstring- Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
scrollboolean- Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
scroll_intervalinteger- Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
user_agentstring- Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
crawlbase_page_scrapeFetch a page from a supported site and extract structured JSON with one of Crawlbase's built-in scrapers, or auto-detect the page type.Read-onlyScrape Page Data
Fetch a page from a supported site and extract structured JSON with one of Crawlbase's built-in scrapers, or auto-detect the page type. Returns the scraper's parsed fields as JSON (for example title, price, rating and reviews for a product page). Use this for Amazon, Google, eBay, LinkedIn, Instagram, Facebook, TikTok and other supported pages. Use crawlbase_page_crawl for unsupported sites.
Inputs
urlstringrequired- Full URL of the page to scrape, including https://, for a site the chosen scraper supports. Example: https://www.example.com/products/123.
ajax_waitboolean- Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
autoparseboolean- Set to true to detect the page type and apply the matching scraper automatically. Example: true.
cookiesstring- Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
cookies_sessionstring- Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
countrystring- Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
css_click_selectorstring- CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
devicestring- Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of
desktoptabletmobile javascriptboolean- Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
page_waitinteger- Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
request_headersstring- Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
scraperstring- Built-in scraper to apply, for example amazon-product-details, amazon-serp, google-serp, ebay-product, linkedin-profile, linkedin-company, instagram-profile, facebook-page, tiktok-profile or email-extractor. Leave empty and set autoparse to detect it. Full list: https://crawlbase.com/docs/scrapers. Example: amazon-product-details.
scrollboolean- Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
scroll_intervalinteger- Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
user_agentstring- Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
crawlbase_storage_getGet a page saved in Crawlbase Cloud Storage, by request id or by original URL.Read-onlyGet Stored Page
Get a page saved in Crawlbase Cloud Storage, by request id or by original URL. Returns the stored page; with format json an object with the body and its metadata (url, original_status, rid, stored date). Use this to read async crawls, crawler results or pages crawled with store true. Send rid or url. Retrieval is free.
Inputs
formatstring- Response envelope: json (body plus metadata, default for this tool) or html. Example: json.one of
jsonhtmldefaultjson ridstring- Request id returned when the page was stored. Send this or url. Example: a1B2c3D4e5F6.
urlstring- Original URL of the stored page; returns the latest stored version. Send this or rid. Example: https://www.example.com/products/123.
crawlbase_storage_rids_listList the request ids of pages in your Crawlbase Cloud Storage, newest first by default, with scroll pagination.Read-onlyList Stored Page IDs
List the request ids of pages in your Crawlbase Cloud Storage, newest first by default, with scroll pagination. Returns an array of rids and, when scroll is true, a scroll_id for the next page. Use this to inventory stored results before crawlbase_storage_bulk_get. A scroll_id expires after about 15 seconds of inactivity.
Inputs
limitinteger- Maximum number of rids to return, up to 10000. Set this explicitly; there is no default. Example: 100.
scrollboolean- Set to true on the first call to start a scroll session and get a scroll_id. Example: true.
scroll_idstring- scroll_id from the previous response, to get the next page. Do not send scroll with it. Example: dXVlcnlUaGVuRmV0Y2g7.
scroll_orderstring- Order by stored time: desc (newest first, default) or asc. Example: desc.one of
ascdesc
crawlbase_storage_total_count_getGet how many pages are currently in your Crawlbase Cloud Storage.Read-onlyGet Stored Page Count
Get how many pages are currently in your Crawlbase Cloud Storage. Returns a single integer count. Use this to check storage size. Use crawlbase_storage_rids_list to list the ids.
Inputs
This tool takes no inputs.
crawlbase_user_agents_getGet random, realistic browser User-Agent strings for crawling outside Crawlbase.Read-onlyGet Random User Agents
Get random, realistic browser User-Agent strings for crawling outside Crawlbase. Returns one or more User-Agent strings. Use this only for your own crawlers; crawlbase_page_crawl already rotates User-Agents. Free, limited to 1 request a second.
Inputs
devicestring- Device class to pick User-Agents for. Default desktop. Example: mobile.one of
desktoptabletmobile sizeinteger- Number of User-Agents to return, 1 to 10. Default 1. Example: 5.
crawlbase_crawler_url_pushAdd a URL to one of your Enterprise Crawler queues so Crawlbase crawls it in the background.WritePush URL to Crawler
Add a URL to one of your Enterprise Crawler queues so Crawlbase crawls it in the background. Returns rid for the queued URL. Use this to feed a crawler created in the Crawlbase console; results go to that crawler's webhook or to Cloud Storage (read them with crawlbase_storage_get). Push rate is capped at 30 URLs a second.
Inputs
crawlerstringrequired- Name of an Enterprise Crawler created in the Crawlbase console. Example: product-monitor.
urlstringrequired- Full URL to enqueue, including https://. Example: https://www.example.com/products/123.
callback_headersstring- Custom header to add to the webhook delivery, as Name|value, for example to authenticate it. Example: X-Webhook-Token|s3kret.
countrystring- Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
devicestring- Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of
desktoptabletmobile formatstring- Shape of the delivered result: html or json. Example: json.one of
htmljson
crawlbase_page_crawl_asyncQueue a page crawl and return immediately with a request id instead of waiting for the page.WriteCrawl Page Asynchronously
Queue a page crawl and return immediately with a request id instead of waiting for the page. Returns rid. The result goes to the callback URL if set, otherwise it is stored in Cloud Storage. Use this for slow or heavy pages, then read the result with crawlbase_storage_get by rid. Use crawlbase_page_crawl to wait for the page inline.
Inputs
urlstringrequired- Full URL of the page to crawl, including http:// or https://. The platform URL-encodes it. Example: https://www.example.com/products/123.
ajax_waitboolean- Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
callbackstring- Webhook URL that receives the crawl result. Leave empty to keep the result in Cloud Storage. Example: https://example.com/hooks/crawlbase.
cookiesstring- Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
cookies_sessionstring- Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
countrystring- Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
css_click_selectorstring- CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
devicestring- Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of
desktoptabletmobile formatstring- Envelope of the delivered result: html or json. Example: json.one of
htmljson javascriptboolean- Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
page_waitinteger- Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
request_headersstring- Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
scrollboolean- Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
scroll_intervalinteger- Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
user_agentstring- Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
crawlbase_page_postSend a JSON POST request to a URL through Crawlbase's proxy network and anti-bot handling.WriteSend POST Through Crawlbase
Send a JSON POST request to a URL through Crawlbase's proxy network and anti-bot handling. Returns the target's response body; metadata such as original_status comes in response headers. Use this for JSON APIs or GraphQL endpoints you are allowed to call. Works with the normal token only; Crawlbase suspends accounts that use POST for spam or abuse.
Inputs
bodyobjectrequired- JSON object to send to the target unchanged. Example: {"query": "{ viewer { login } }"}
urlstringrequired- Full URL of the endpoint to send the POST to, including https://. Example: https://www.example.com/products/123.
countrystring- Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
custom_success_codesstring- Comma-separated HTTP status codes from the site to treat as success instead of retrying. These requests are charged. Example: 403,429.
devicestring- Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of
desktoptabletmobile get_headersboolean- Set to true to include the site's response headers (under original_headers with format json). Example: true.
crawlbase_storage_bulk_getGet up to 100 stored pages from Cloud Storage by request id in one call, optionally deleting them as they are returned.WriteGet Stored Pages in Bulk
Get up to 100 stored pages from Cloud Storage by request id in one call, optionally deleting them as they are returned. Returns an array with one object per found rid; each body is base64-encoded gzip data. Use this to drain many crawler results at once. Use crawlbase_storage_get for one readable page.
Inputs
ridsarrayrequired- Request ids to fetch, at most 100; extra ids are ignored. Example: ["a1B2c3D4e5F6", "g7H8i9J0k1L2"]
auto_deleteboolean- Set to true to delete each returned page from storage in the same call. Example: false.
crawlbase_storage_bulk_deletePermanently delete several pages from Crawlbase Cloud Storage by request id.DestructiveDelete Stored Pages in Bulk
Permanently delete several pages from Crawlbase Cloud Storage by request id. Returns an array with status true for each deleted rid, or status false with result Not Found. Use this to clean up stored results. Deletion cannot be undone; use crawlbase_storage_delete for a single page.
Inputs
ridsarrayrequired- Request ids of the stored pages to delete. Example: ["a1B2c3D4e5F6", "g7H8i9J0k1L2"]
crawlbase_storage_deletePermanently delete one page from Crawlbase Cloud Storage by request id.DestructiveDelete Stored Page
Permanently delete one page from Crawlbase Cloud Storage by request id. Returns success, or an error of Not Found when the rid is not in storage. Use crawlbase_storage_bulk_delete for many pages. Deletion cannot be undone.
- Idempotent
Inputs
ridstringrequired- Request id of the stored page to delete. Example: a1B2c3D4e5F6.
No tools match.