Skip to content
Scalekit Docs
Talk to an Engineer Dashboard

Connect AI agents to Crawlbase

Scalekit connector
Open markdown

The Crawlbase connector lets your AI agent act in each user's Crawlbase account. Each user connects their own Crawlbase API key once, and Scalekit sends it with every call, so your agent never handles credentials. It comes with 14 tools.

Tools
14
What they doRead · write · destructive
8 · 4 · 28 read4 write2 destructive
Users sign in with

Setup

  1. Install the SDK

    Terminal window
    npm install @scalekit-sdk/node dotenv
  2. Set your credentials

    Add your Scalekit credentials to your .env file. Find values in app.scalekit.com > Developers > API Credentials.

    .env
    SCALEKIT_ENVIRONMENT_URL=<your-environment-url>
    SCALEKIT_CLIENT_ID=<your-client-id>
    SCALEKIT_CLIENT_SECRET=<your-client-secret>
  3. Create the Crawlbase connection

    In AgentKit > Connections, create a Crawlbase connection. The name you give it is the connection_name your code passes. See Configure connections.

  4. Authorize a user and make your first call

    quickstart.mts
    import { ScalekitClient } from '@scalekit-sdk/node'
    import 'dotenv/config'
    import { createInterface } from 'node:readline/promises'
    const scalekit = new ScalekitClient(
    process.env.SCALEKIT_ENVIRONMENT_URL,
    process.env.SCALEKIT_CLIENT_ID,
    process.env.SCALEKIT_CLIENT_SECRET,
    )
    const actions = scalekit.actions
    const connector = 'crawlbase'
    const identifier = 'user_123'
    // Generate an authorization link for the user
    const { link } = await actions.getAuthorizationLink({ connectionName: connector, identifier })
    console.log('Authorize Crawlbase:', link)
    const rl = createInterface({ input: process.stdin, output: process.stdout })
    await rl.question('Press Enter after authorizing...')
    rl.close()
    // Make your first call
    const result = await actions.executeTool({
    connector,
    identifier,
    toolName: 'crawlbase_storage_get',
    toolInput: {},
    })
    console.log(result)
    Terminal window
    npx tsx quickstart.mts

    Each user opens the link once and enters their Crawlbase credentials there. See Authorize a user for the full flow and statuses.

Tools

Pass the exact name to execute_tool
Try in PlaygroundRequest a tool
  • crawlbase_account_usage_getGet this month's usage statistics for one Crawlbase product.Read-only

    Get Account Usage

    Get this month's usage statistics for one Crawlbase product. Returns totalSuccess, totalFailed, totalDue, remainingCredits (subscription products) and domainStats per domain. Use this to monitor credit consumption. Rate limited to 1 request every 5 minutes, so cache the result.

    Inputs

    productstringrequired
    Product whose usage to return. Example: crawling-api.one of crawling-apicrawlersmartproxyscraper-apileads-apiscreenshot-api
    previous_monthboolean
    Set to true to include the previous month's statistics too. Example: true.
  • crawlbase_page_crawlFetch any web page through Crawlbase's proxy network, with optional headless-browser rendering, anti-bot bypass and geo-routing.Read-only

    Crawl Page

    Fetch any web page through Crawlbase's proxy network, with optional headless-browser rendering, anti-bot bypass and geo-routing. Returns, with the default format json, an object with cb_status, original_status, url, body (the page HTML) and rid when stored; format html returns raw HTML and format md returns Markdown. Use this for raw page content. Use crawlbase_page_markdown_get for clean article text, crawlbase_page_scrape for structured data from a supported site and crawlbase_page_crawl_async for slow pages.

    Inputs

    urlstringrequired
    Full URL of the page to crawl, including http:// or https://. The platform URL-encodes it. Example: https://www.example.com/products/123.
    ajax_waitboolean
    Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
    cookiesstring
    Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
    cookies_sessionstring
    Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
    countrystring
    Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
    css_click_selectorstring
    CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
    custom_success_codesstring
    Comma-separated HTTP status codes from the site to treat as success instead of retrying. These requests are charged. Example: 403,429.
    devicestring
    Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of desktoptabletmobile
    formatstring
    Response envelope: json (page plus metadata, default for this tool), html (raw page) or md (GitHub-flavored Markdown). Example: json.one of jsonhtmlmddefault json
    get_cookiesboolean
    Set to true to include the site's Set-Cookie values (as original_set_cookie). Example: true.
    get_headersboolean
    Set to true to include the site's response headers (under original_headers with format json). Example: true.
    heightinteger
    Screenshot height in pixels with mode viewport. Example: 800.
    javascriptboolean
    Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
    modestring
    With screenshot true, viewport captures only the visible area (see width and height); the default is the full page. Example: viewport.one of viewport
    page_waitinteger
    Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
    prettyboolean
    Set to true to pretty-print the JSON envelope. Example: false.
    request_headersstring
    Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
    screenshotboolean
    Set to true to also capture a JPEG of the rendered page; its temporary screenshot_url (valid one hour) comes back in the JSON. JavaScript rendering only. Example: true.
    scrollboolean
    Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
    scroll_intervalinteger
    Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
    storeboolean
    Set to true to also save the result in Crawlbase Cloud Storage and get back an rid for crawlbase_storage_get. Example: true.
    tor_networkboolean
    Set to true to route over the Tor network, for .onion sites only. Example: false.
    user_agentstring
    Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
    widthinteger
    Screenshot width in pixels with mode viewport. Example: 1200.
  • crawlbase_page_markdown_getFetch a web page through Crawlbase and convert it to GitHub-flavored Markdown, optionally keeping only the main article content.Read-only

    Get Page as Markdown

    Fetch a web page through Crawlbase and convert it to GitHub-flavored Markdown, optionally keeping only the main article content. Returns the page as Markdown text. Use this to read articles, docs or blog posts as clean LLM context. Use crawlbase_page_crawl for HTML or metadata.

    Inputs

    urlstringrequired
    Full URL of the page to crawl, including http:// or https://. The platform URL-encodes it. Example: https://www.example.com/products/123.
    ajax_waitboolean
    Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
    cookiesstring
    Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
    cookies_sessionstring
    Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
    countrystring
    Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
    css_click_selectorstring
    CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
    devicestring
    Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of desktoptabletmobile
    javascriptboolean
    Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
    md_readabilityboolean
    Set to true (default) to drop navigation, sidebars, footers and ads and keep the main content before converting. Example: true.default true
    page_waitinteger
    Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
    request_headersstring
    Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
    scrollboolean
    Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
    scroll_intervalinteger
    Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
    user_agentstring
    Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
  • crawlbase_page_scrapeFetch a page from a supported site and extract structured JSON with one of Crawlbase's built-in scrapers, or auto-detect the page type.Read-only

    Scrape Page Data

    Fetch a page from a supported site and extract structured JSON with one of Crawlbase's built-in scrapers, or auto-detect the page type. Returns the scraper's parsed fields as JSON (for example title, price, rating and reviews for a product page). Use this for Amazon, Google, eBay, LinkedIn, Instagram, Facebook, TikTok and other supported pages. Use crawlbase_page_crawl for unsupported sites.

    Inputs

    urlstringrequired
    Full URL of the page to scrape, including https://, for a site the chosen scraper supports. Example: https://www.example.com/products/123.
    ajax_waitboolean
    Set to true to wait until the page's network is idle before capturing. JavaScript rendering only. Example: true.
    autoparseboolean
    Set to true to detect the page type and apply the matching scraper automatically. Example: true.
    cookiesstring
    Cookies to forward to the site, in Cookie header form. Example: session=abc123; theme=dark.
    cookies_sessionstring
    Sticky cookie session name, up to 32 characters. Calls sharing the value replay cookies set by earlier calls. Sessions expire 300 seconds after the last call. Example: checkout-flow-1.
    countrystring
    Two-letter country code to route the crawl through that country's exit nodes. Supported: AE, BR, CA, CN, DE, DK, ES, FI, FR, GB, IL, JP, KZ, MD, NL, PK, PL, RU, SE, TR, UA, US. Leave empty for automatic selection; Crawlbase may override it for some sites. Example: US.
    css_click_selectorstring
    CSS selector to click before capturing; separate several with a pipe to click them in order. If a selector is missing the request fails with cb_status 595, so add ,body as a fallback. JavaScript rendering only. Example: #load-more,body.
    devicestring
    Device class whose User-Agent and viewport to emulate. Default desktop. Example: mobile.one of desktoptabletmobile
    javascriptboolean
    Set to true to render the page in a headless browser with the account's JavaScript token, even when connected with the normal token. Billed as a JavaScript request. Needed for page_wait, ajax_wait, scroll and css_click_selector unless the connection uses the JavaScript token. Example: true.
    page_waitinteger
    Milliseconds to wait after page load before capturing. JavaScript rendering only. Example: 2000.
    request_headersstring
    Headers to forward to the site, pipe-separated name:value pairs. Example: accept-language:en-GB|accept-encoding:gzip.
    scraperstring
    Built-in scraper to apply, for example amazon-product-details, amazon-serp, google-serp, ebay-product, linkedin-profile, linkedin-company, instagram-profile, facebook-page, tiktok-profile or email-extractor. Leave empty and set autoparse to detect it. Full list: https://crawlbase.com/docs/scrapers. Example: amazon-product-details.
    scrollboolean
    Set to true to scroll to the bottom before capturing, to trigger lazy loading. JavaScript rendering only; billed by processing time. Example: true.
    scroll_intervalinteger
    Maximum seconds to spend scrolling when scroll is true, up to 60. Default 10. Example: 10.
    user_agentstring
    Override the User-Agent header sent to the site. Use sparingly; the defaults are tuned per site. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36.
  • crawlbase_storage_getGet a page saved in Crawlbase Cloud Storage, by request id or by original URL.Read-only

    Get Stored Page

    Get a page saved in Crawlbase Cloud Storage, by request id or by original URL. Returns the stored page; with format json an object with the body and its metadata (url, original_status, rid, stored date). Use this to read async crawls, crawler results or pages crawled with store true. Send rid or url. Retrieval is free.

    Inputs

    formatstring
    Response envelope: json (body plus metadata, default for this tool) or html. Example: json.one of jsonhtmldefault json
    ridstring
    Request id returned when the page was stored. Send this or url. Example: a1B2c3D4e5F6.
    urlstring
    Original URL of the stored page; returns the latest stored version. Send this or rid. Example: https://www.example.com/products/123.
  • crawlbase_storage_rids_listList the request ids of pages in your Crawlbase Cloud Storage, newest first by default, with scroll pagination.Read-only

    List Stored Page IDs

    List the request ids of pages in your Crawlbase Cloud Storage, newest first by default, with scroll pagination. Returns an array of rids and, when scroll is true, a scroll_id for the next page. Use this to inventory stored results before crawlbase_storage_bulk_get. A scroll_id expires after about 15 seconds of inactivity.

    Inputs

    limitinteger
    Maximum number of rids to return, up to 10000. Set this explicitly; there is no default. Example: 100.
    scrollboolean
    Set to true on the first call to start a scroll session and get a scroll_id. Example: true.
    scroll_idstring
    scroll_id from the previous response, to get the next page. Do not send scroll with it. Example: dXVlcnlUaGVuRmV0Y2g7.
    scroll_orderstring
    Order by stored time: desc (newest first, default) or asc. Example: desc.one of ascdesc
  • crawlbase_storage_total_count_getGet how many pages are currently in your Crawlbase Cloud Storage.Read-only

    Get Stored Page Count

    Get how many pages are currently in your Crawlbase Cloud Storage. Returns a single integer count. Use this to check storage size. Use crawlbase_storage_rids_list to list the ids.

    Inputs

    This tool takes no inputs.

  • crawlbase_user_agents_getGet random, realistic browser User-Agent strings for crawling outside Crawlbase.Read-only

    Get Random User Agents

    Get random, realistic browser User-Agent strings for crawling outside Crawlbase. Returns one or more User-Agent strings. Use this only for your own crawlers; crawlbase_page_crawl already rotates User-Agents. Free, limited to 1 request a second.

    Inputs

    devicestring
    Device class to pick User-Agents for. Default desktop. Example: mobile.one of desktoptabletmobile
    sizeinteger
    Number of User-Agents to return, 1 to 10. Default 1. Example: 5.