The Diffbot connector lets your AI agent act in each user's Diffbot account. Each user connects their own Diffbot API key once, and Scalekit sends it with every call, so your agent never handles credentials. It comes with 39 tools.
- Tools
- 39
- What they doRead · write · destructive
- 25 · 9 · 525 read9 write5 destructive
- Users sign in with
- API key
Setup
Install the SDK
Terminal window npm install @scalekit-sdk/node dotenvTerminal window pip install scalekit-sdk-python python-dotenvSet your credentials
Add your Scalekit credentials to your
.envfile. Find values in app.scalekit.com > Developers > API Credentials..env SCALEKIT_ENVIRONMENT_URL=<your-environment-url>SCALEKIT_CLIENT_ID=<your-client-id>SCALEKIT_CLIENT_SECRET=<your-client-secret>Create the Diffbot connection
In AgentKit > Connections, create a Diffbot connection. The name you give it is the
connection_nameyour code passes. See Configure connections.Authorize a user and make your first call
quickstart.mts import { ScalekitClient } from '@scalekit-sdk/node'import 'dotenv/config'import { createInterface } from 'node:readline/promises'const scalekit = new ScalekitClient(process.env.SCALEKIT_ENVIRONMENT_URL,process.env.SCALEKIT_CLIENT_ID,process.env.SCALEKIT_CLIENT_SECRET,)const actions = scalekit.actionsconst connector = 'diffbot'const identifier = 'user_123'// Generate an authorization link for the userconst { link } = await actions.getAuthorizationLink({ connectionName: connector, identifier })console.log('Authorize Diffbot:', link)const rl = createInterface({ input: process.stdin, output: process.stdout })await rl.question('Press Enter after authorizing...')rl.close()// Make your first callconst result = await actions.executeTool({connector,identifier,toolName: 'diffbot_account_get',toolInput: {},})console.log(result)Terminal window npx tsx quickstart.mtsquickstart.py import osfrom scalekit import ScalekitClientfrom dotenv import load_dotenvload_dotenv()scalekit_client = ScalekitClient(env_url=os.getenv("SCALEKIT_ENVIRONMENT_URL"),client_id=os.getenv("SCALEKIT_CLIENT_ID"),client_secret=os.getenv("SCALEKIT_CLIENT_SECRET"),)actions = scalekit_client.actionsconnection_name = "diffbot"identifier = "user_123"# Generate an authorization link for the userlink_response = actions.get_authorization_link(connection_name=connection_name,identifier=identifier,)print("Authorize Diffbot:", link_response.link)input("Press Enter after authorizing...")# Make your first callresult = actions.execute_tool(tool_input={},tool_name="diffbot_account_get",connection_name=connection_name,identifier=identifier,)print(result)Terminal window python quickstart.pyEach user opens the link once and enters their Diffbot credentials there. See Authorize a user for the full flow and statuses.
Tools
Pass the exact name toexecute_tooldiffbot_account_getGet the Diffbot token's account details and API usage.Read-onlyGet Account
Get the Diffbot token's account details and API usage. Returns the plan, token status, credit usage and daily API call volumes. Use it to check the token works and how many credits are left.
Inputs
daysinteger- Number of days of API call volume to return. Defaults to 31. Example: 31.
diffbot_bulk_data_getDownload a bulk job's extracted data, or its URL report.Read-onlyGet Bulk Data
Download a bulk job's extracted data, or its URL report. Returns the extracted objects as JSON (or CSV). Use List Bulk Jobs to check the job is done.
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
formatstring- csv for CSV with top-level fields only. Defaults to JSON. Example: csv.one of
csv numinteger- Only the most recently processed N URLs. Example: 100.
typestring- urls returns the URL report (CSV) instead of extracted data. Example: urls.one of
urls
diffbot_bulk_jobs_listList the token's bulk jobs, or one job by name, with their status.Read-onlyList Bulk Jobs
List the token's bulk jobs, or one job by name, with their status. Returns jobs with name, status, URLs processed and download URLs. Use Get Bulk Data to download a job's results.
Inputs
namestring- Only return this job. Example: acme-products-batch.
diffbot_crawl_data_getDownload a crawl job's extracted data, or its URL report.Read-onlyGet Crawl Data
Download a crawl job's extracted data, or its URL report. Returns the extracted objects as JSON (or CSV). Use Search with DQL (type crawl) to query a subset across crawl jobs.
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
formatstring- csv for CSV with top-level fields only. Defaults to JSON. Example: csv.one of
csv numinteger- Only the most recently processed N URLs. Example: 100.
typestring- urls returns the URL report (CSV) instead of extracted data. Example: urls.one of
urls
diffbot_crawl_jobs_listList the token's crawl jobs, or one job by name, with their status.Read-onlyList Crawl Jobs
List the token's crawl jobs, or one job by name, with their status. Returns jobs with name, status, pages crawled and processed, limits and download URLs. Use Get Crawl Data to download a job's results.
Inputs
namestring- Only return this job. Example: acme-blog-crawl.
diffbot_custom_api_extractExtract a page with one of your Custom APIs, which apply your own CSS-selector rules.Read-onlyExtract with Custom API
Extract a page with one of your Custom APIs, which apply your own CSS-selector rules. Returns an objects array shaped by your rules. Use List Custom API Rules to see your Custom APIs.
Inputs
apistringrequired- Name of your Custom API (the name you gave its ruleset). Example: my-store-products.
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_custom_api_rules_listList the custom extraction rulesets defined on the token.Read-onlyList Custom API Rules
List the custom extraction rulesets defined on the token. Returns each ruleset with its API, URL pattern, test URL, rules and prefilters. Use Save Custom API Rules to add or change one.
Inputs
This tool takes no inputs.
diffbot_dql_report_getGet a DQL coverage report by its id.Read-onlyGet DQL Coverage Report
Get a DQL coverage report by its id. Returns per-field coverage for the query. Use Create DQL Coverage Report to make one.
Inputs
report_idstringrequired- Report id, from Create DQL Coverage Report. Example: rpt_9f8e7d6c.
diffbot_dql_searchSearch the Diffbot Knowledge Graph (organizations, people, articles, products and more) or your crawl collections with a DQL query.Read-onlySearch with DQL
Search the Diffbot Knowledge Graph (organizations, people, articles, products and more) or your crawl collections with a DQL query. Returns hits and a data array of matching entities with their facts. Use Enhance Entity to match one known person or organization.
Inputs
querystringrequired- DQL query, for example type:Organization locations.city.name:"San Francisco" nbEmployees>5000. Example: type:Organization name:"Diffbot".
clusterstring- Articles only: all tags each article with a clusterId, best keeps one per cluster, dedupe removes near-duplicates. Example: best.one of
allbestdedupe colstring- With type crawl: comma-separated crawl job names to search, or all. Example: acme-blog-crawl.
filterstring- Semicolon-separated fields to keep in the response, as dot paths or JsonPath, for example $.name;$.homepageUri. Example: $.name;$.homepageUri;$.nbEmployees.
filter_excludestring- Semicolon-separated fields to drop from the response, as dot paths or JsonPath. Example: $.descriptors.
frominteger- Index of the first result (not for Article queries). Example: 0.
jsonmodestring- extended adds fact origins; id returns only Diffbot IDs and origins. Example: extended.one of
extendedid no_dedup_articlesboolean- Return duplicate articles instead of deduplicating them. Example: false.
non_canonical_factsboolean- Also return non-canonical facts. Example: false.
sizeinteger- Maximum results; -1 returns all. Defaults to 50. Example: 10.
typestring- query (default) searches the Knowledge Graph; crawl searches your crawl job collections. Example: query.one of
querycrawl
diffbot_enhanceMatch a person or organization from partial details (name, URL, email, employer and so on) to the Knowledge Graph.Read-onlyEnhance Entity
Match a person or organization from partial details (name, URL, email, employer and so on) to the Knowledge Graph. Returns the best matching entities with score and full facts. Use Search with DQL for open-ended queries. Use Create Enhance Bulk Job for many records.
Inputs
typestringrequired- Entity type: Person or Organization. Example: Organization.one of
PersonOrganization custom_idstring- Your own ID, echoed back for correlation. Example: crm-00123.
descriptionstring- Free-text description of the entity. Example: Knowledge graph and web extraction company.
emailarray- Email address(es). Person only. Example: ["jane@example.com"].
employerstring- Employer name. Person only. Example: Diffbot.
filterstring- Semicolon-separated fields to keep, as dot paths or JsonPath. Example: $.name;$.homepageUri.
filter_excludestring- Semicolon-separated fields to drop. Example: $.descriptors.
idstring- Diffbot ID of a known entity. Example: EYX1i02YVPsuT7fPLUYgRhQ.
ipstring- IP address of the entity. Example: 203.0.113.10.
jsonmodestring- extended adds fact origins. Example: extended.one of
extended locationstring- Location of the entity. Example: Menlo Park, California.
namearray- Name(s) of the person or organization. Example: ["Diffbot"].
non_canonical_factsboolean- Also return non-canonical facts. Example: false.
phonestring- Phone number of the entity. Example: +1 650 555 0100.
refreshboolean- Recrawl the entity's sources and rebuild it before returning. Example: false.
schoolstring- School attended. Person only. Example: Stanford University.
searchboolean- Also search the web for sources and merge them in. Example: false.
sizeinteger- Maximum matches to return. Defaults to 1. Example: 1.
thresholdnumber- Minimum similarity score for a match. Example: 0.5.
titlestring- Job title. Person only. Example: VP Engineering.
urlarray- Homepage or profile URL(s) of the entity. Example: ["diffbot.com"].
diffbot_enhance_bulk_coverage_report_getGet a coverage report for an Enhance bulk job.Read-onlyGet Enhance Coverage Report
Get a coverage report for an Enhance bulk job. Returns per-field coverage of the matched entities. Use Get Enhance Bulk Results for the records themselves.
Inputs
bulkjob_idstringrequired- Enhance bulk job ID, returned by Create Enhance Bulk Job. Example: a1b2c3d4e5.
report_idstringrequired- Enhance coverage report ID. Example: rpt_1a2b3c.
diffbot_enhance_bulk_jobs_listList the token's Enhance bulk jobs with their status.Read-onlyList Enhance Bulk Jobs
List the token's Enhance bulk jobs with their status. Returns each job's ID, name, status and record counts. Use Get Enhance Bulk Job Status for one job.
Inputs
This tool takes no inputs.
diffbot_enhance_bulk_result_getGet the result for one record of an Enhance bulk job.Read-onlyGet Enhance Bulk Result
Get the result for one record of an Enhance bulk job. Returns that record's matches with score and entity. Use Get Enhance Bulk Results for all records.
Inputs
bulkjob_idstringrequired- Enhance bulk job ID, returned by Create Enhance Bulk Job. Example: a1b2c3d4e5.
job_indexintegerrequired- Index of the record within the bulk job. Example: 0.
diffbot_enhance_bulk_results_getDownload the results of a completed Enhance bulk job.Read-onlyGet Enhance Bulk Results
Download the results of a completed Enhance bulk job. Returns one result per record, each with score and matched entity (JSON lines by default). Use Get Enhance Bulk Result for one record.
Inputs
bulkjob_idstringrequired- Enhance bulk job ID, returned by Create Enhance Bulk Job. Example: a1b2c3d4e5.
filterstring- Semicolon-separated fields to keep in the response, as dot paths or JsonPath, for example $.name;$.homepageUri. Example: $.name;$.homepageUri;$.nbEmployees.
filter_excludestring- Semicolon-separated fields to drop from the response, as dot paths or JsonPath. Example: $.descriptors.
formatstring- Output format: json, jsonl (default), csv, xls or xlsx. Example: json.one of
jsonjsonlcsvxlsxlsx frominteger- 1-based index of the first result. Example: 1.
only_matchesstring- true returns only records with a match. Example: true.one of
truefalse sizeinteger- Number of results from that position; -1 for all. Example: 100.
diffbot_enhance_bulk_status_getGet one Enhance bulk job's status and progress.Read-onlyGet Enhance Bulk Job Status
Get one Enhance bulk job's status and progress. Returns its status and counts of processed and matched records. Use Get Enhance Bulk Results once it completes.
Inputs
bulkjob_idstringrequired- Enhance bulk job ID, returned by Create Enhance Bulk Job. Example: a1b2c3d4e5.
diffbot_extract_analyzeClassify a web page's type (article, product, discussion and so on) and extract it with the matching Extract API.Read-onlyAnalyze Page
Classify a web page's type (article, product, discussion and so on) and extract it with the matching Extract API. Returns the detected type and an objects array of structured data. Use a specific Extract tool (for example Extract Article) when you know the page type.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
discussionboolean- false skips extracting comments or reviews. Example: false.
fallbackstring- Extract API to use when the page type is other. Example: article.one of
articleproductdiscussionimagevideolist fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
modestring- Force a specific Extract API instead of auto-detection: article, product, discussion, image, video, list, event or job. Example: article.one of
articleproductdiscussionimagevideolisteventjob proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_articleExtract the article (title, author, date, text, images, tags) from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Article
Extract the article (title, author, date, text, images, tags) from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with title, author, date, text, html, images, tags, categories and sentiment. Use Analyze Page when unsure of the page type.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
category_confidencenumber- Minimum category score, 0.0 to 1.0. Defaults to 0.5. Example: 0.5.
discussionboolean- false skips extracting comments. Example: false.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring, breadcrumb or quotes. Example: meta,querystring.
max_tagsinteger- Maximum tags to return. Defaults to 10. Example: 10.
natural_languagestring- Run the extracted text through the Natural Language API: comma-separated entities, sentiment, summary, facts, openFacts, records, categories, sentences or language. Example: entities,sentiment.
pagingboolean- true joins multi-page articles into one result. Example: true.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
summary_num_sentencesinteger- Sentences in the summary when natural_language includes summary. Defaults to 3. Example: 3.
tag_confidencenumber- Minimum tag score, 0.0 to 1.0. Defaults to 0.5. Example: 0.5.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_discussionExtract a forum thread, comments or reviews from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Discussion
Extract a forum thread, comments or reviews from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with the thread title and posts, each with author, date, text and votes. Use Extract Article for the article a comment thread belongs to.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
max_pagesstring- Pages of a thread to join, or all. Defaults to 1. Example: all.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_eventExtract event details from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Event
Extract event details from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with event title, start and end dates, venue, location, organizer and description. Use Analyze Page when unsure of the page type.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_imageExtract the main images from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Images
Extract the main images from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with image URLs, sizes, titles and captions. Use Extract Video for videos.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_jobExtract a job posting from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Job Posting
Extract a job posting from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with job title, employer, location, salary, employment type, skills and description. Use Analyze Page when unsure of the page type.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_listExtract the items of a list page (search results, product listings, article indexes) from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract List
Extract the items of a list page (search results, product listings, article indexes) from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with the list items, each with title, link, summary and image. Use Crawl tools to follow every link on a site.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_productExtract product details (name, price, availability, specs) from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Product
Extract product details (name, price, availability, specs) from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with title, brand, offer and regular price, availability, SKU, specifications, images and reviews. Use Analyze Page when unsure of the page type.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
discussionboolean- false skips extracting reviews. Example: false.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_extract_videoExtract video details from a web page URL with Diffbot's AI, with no rules needed.Read-onlyExtract Video
Extract video details from a web page URL with Diffbot's AI, with no rules needed. Returns an objects array with title, video URL, duration, author, date, view count and thumbnail. Use Extract Images for images.
Inputs
urlstringrequired- URL of the web page to extract. Example: https://www.example.com/blog/launch-announcement.
fieldsstring- Comma-separated optional fields to add, such as links, meta, querystring or breadcrumb. Example: meta,querystring.
proxystring- IP address (and port) of your own proxy to fetch the page through. Example: 203.0.113.10:8080.
proxy_authstring- Credentials for your own proxy, as username:password. Example: user:pass.
timeoutinteger- Milliseconds to wait for the page to be fetched. Defaults to 30000. Example: 30000.
use_proxystring- default uses Diffbot's datacenter proxy for this request; none disables proxies even if enabled globally. Example: default.one of
defaultnone
diffbot_natural_language_processAnalyze raw text with Diffbot Natural Language: entities, sentiment, facts, relationships and more.Read-onlyProcess Text
Analyze raw text with Diffbot Natural Language: entities, sentiment, facts, relationships and more. Returns entities (with Knowledge Graph links), sentiment, facts and sentences for the text. Use Extract Article with natural_language to analyze a web page instead.
Inputs
contentstringrequired- Text to analyze, 1 to 100,000 characters. Example: Diffbot is a startup based in Menlo Park..
document_typestring- Optional hint about the kind of document. Example: news.
fieldsstring- Comma-separated results to return: entities, sentiment, facts, records, sentences, categories, summary, openFacts or language. Defaults to entities,sentiment,facts,records,sentences. Example: entities,sentiment,facts.
formatstring- plain text (default) or plain text with title (title, two line breaks, then body). Example: plain text.one of
plain textplain text with title langstring- ISO 639-1 language code. Defaults to auto-detect. Example: en.
summary_sentencesinteger- Maximum sentences in the summary, 1 to 10. Defaults to 3. Example: 3.
diffbot_bulk_job_createStart a bulk job that extracts a list of URLs with an Extract API, without crawling.WriteCreate Bulk Job
Start a bulk job that extracts a list of URLs with an Extract API, without crawling. Returns the job's settings and status. Use List Bulk Jobs to follow progress and Get Bulk Data to download results.
Inputs
api_urlstringrequired- Extract or Custom API URL to process each URL with, optionally with parameters. Example: https://api.diffbot.com/v3/product.
namestringrequired- Unique job name, used to manage the job and get its data. Example: acme-products-batch.
urlsstringrequired- Space-separated URLs to process. Startup plans need at least 50 URLs. Example: https://www.example.com/p/1 https://www.example.com/p/2.
notify_emailstring- Email notified when the job completes. Example: ops@example.com.
notify_webhookstring- URL that receives a POST when the job completes. Example: https://example.com/hooks/diffbot.
obey_robotsstring- 0 ignores robots.txt. Example: 1.one of
01 page_process_patternstring- ||-separated strings the page HTML must contain to be processed. Example: <article.
repeatnumber- Repeat the job every this many days. Example: 7.0.
diffbot_bulk_job_pausePause a bulk job, or resume a paused one.WritePause or Resume Bulk Job
Pause a bulk job, or resume a paused one. Returns the job's updated status. Use List Bulk Jobs to check the status.
- Idempotent
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
pauseintegerrequired- 1 pauses the job; 0 resumes it. Example: 1.one of
01
diffbot_crawl_createStart a crawl that spiders a site from seed URLs and extracts matching pages with an Extract API.WriteCreate Crawl
Start a crawl that spiders a site from seed URLs and extracts matching pages with an Extract API. Returns the job's settings and status. Use List Crawl Jobs to follow progress and Get Crawl Data to download results.
Inputs
api_urlstringrequired- Extract API URL to process pages with, optionally with parameters. Example: https://api.diffbot.com/v3/analyze.
namestringrequired- Unique job name, used to manage the crawl and get its data. Example: acme-blog-crawl.
seedsstringrequired- Seed URL(s), separated by spaces. A non-www subdomain seed limits the crawl to that subdomain. Example: https://blog.example.com.
crawl_delaynumber- Seconds to wait between URLs from one IP. Example: 0.25.
max_hopsinteger- Link depth from the seeds; 0 processes only seeds, -1 (default) is unlimited. Example: 2.
max_roundsinteger- Maximum repeat rounds; 0 (default) repeats indefinitely. Example: 4.
max_to_crawlinteger- Maximum pages to spider. Defaults to 100000. Example: 1000.
max_to_processinteger- Maximum pages to process. Defaults to 100000. Example: 500.
notify_emailstring- Email notified when the crawl completes or hits a limit. Example: ops@example.com.
notify_webhookstring- URL that receives a POST when the crawl completes or hits a limit. Example: https://example.com/hooks/diffbot.
obey_robotsinteger- 1 (default) obeys robots.txt; 0 ignores it. Example: 1.one of
01 only_process_if_newinteger- 1 (default) processes only new pages on repeat rounds; 0 processes all. Example: 1.one of
01 page_process_patternstring- ||-separated strings the page HTML must contain to be processed. Example: <article.
repeatnumber- Repeat the crawl every this many days (for example 7.0). Example: 7.0.
restrict_domaininteger- 1 (default) keeps the crawl on the seed domains; 0 allows limited cross-domain crawling. Example: 1.one of
01 url_crawl_patternstring- ||-separated strings a URL must contain to be crawled; ! negates, ^ and $ anchor. Example: /blog/||!/tag/.
url_crawl_regexstring- Regex a URL must match to be crawled. Overrides url_crawl_pattern. Example: ^https://blog\.example\.com/20\d\d/.
url_process_patternstring- ||-separated strings a URL must contain to be processed. Example: /blog/20.
url_process_regexstring- Regex a URL must match to be processed. Overrides url_process_pattern. Example: /blog/\d{4}/.
use_proxiesinteger- 1 forces proxy IPs for crawling and processing. Example: 0.one of
01
diffbot_crawl_job_pausePause a crawl job, or resume a paused one.WritePause or Resume Crawl
Pause a crawl job, or resume a paused one. Returns the job's updated status. Use List Crawl Jobs to check the status.
- Idempotent
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
pauseintegerrequired- 1 pauses the job; 0 resumes it. Example: 1.one of
01
diffbot_crawl_job_round_startStart a new round of a crawl job now (repeat the crawl manually).WriteStart Crawl Round
Start a new round of a crawl job now (repeat the crawl manually). Returns the job's updated status. By default only new pages are processed in the new round.
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
round_startintegerrequired- Must be 1. Example: 1.one of
1
diffbot_custom_api_rules_saveCreate or update a Custom API ruleset that extracts fields with CSS selectors for URLs matching a pattern.WriteSave Custom API Rules
Create or update a Custom API ruleset that extracts fields with CSS selectors for URLs matching a pattern. Returns the saved ruleset. Use List Custom API Rules to see existing rulesets.
Inputs
apistringrequired- API the rules apply to, prefixed with /api/ (for example /api/article), or all. Example: /api/product.
rulesarrayrequired- Rules, each with a field name and a CSS selector, plus optional type, args and filters. Example: [{"name": "price", "selector": ".price"}].
url_patternstringrequired- Regex of the URLs the ruleset applies to. Example: ^https?://(www\.)?example\.com/products/.*.
notesarray- Free-text notes on the ruleset. Example: ["Added for the product catalog"].
prefiltersarray- CSS selectors removed from the page before extraction. Example: [".cookie-banner"].
render_optionsstring- Rendering options. Example: wait=2000.
test_urlstring- URL used to check the rules still work. Example: https://www.example.com/products/widget.
use_proxystring- none disables proxies for these URLs when proxies are set globally. Example: none.
diffbot_dql_report_createGenerate a coverage report for a DQL query, showing how fully each field is filled.WriteCreate DQL Coverage Report
Generate a coverage report for a DQL query, showing how fully each field is filled. Returns the report with its id and per-field coverage. Use Get DQL Coverage Report to fetch it again later.
Inputs
querystringrequired- DQL query to report on. Example: type:Organization industries:"Software Companies".
exportspecstring- Columns for CSV-style export, for example name;summary. Example: name;homepageUri.
filterstring- Semicolon-separated fields to keep in the response, as dot paths or JsonPath, for example $.name;$.homepageUri. Example: $.name;$.homepageUri;$.nbEmployees.
filter_excludestring- Semicolon-separated fields to drop from the response, as dot paths or JsonPath. Example: $.descriptors.
frominteger- Index of the first result. Example: 0.
sizeinteger- Maximum results to cover; -1 for all. Defaults to 50. Example: 100.
diffbot_enhance_bulk_createStart an asynchronous job that enhances many person or organization records at once.WriteCreate Enhance Bulk Job
Start an asynchronous job that enhances many person or organization records at once. Returns the bulk job ID (202). Use Get Enhance Bulk Job Status, then Get Enhance Bulk Results.
Inputs
recordsarrayrequired- Records to enhance. Each has a type (Person or Organization) and any of the Enhance inputs: name, url, location, email, employer, title, school, phone, description, ip, id. Example: [{"type": "Organization", "name": "Diffbot"}, {"type": "Organization", "name": "Readwise"}].
namestring- Name for the bulk job. Example: crm-enrichment-nov.
refreshboolean- Recrawl sources before returning. Example: false.
searchboolean- Also search the web for sources. Example: false.
sizeinteger- Matches per record. Defaults to 1. Example: 1.
thresholdnumber- Minimum similarity score for a match. Example: 0.5.
webhook_urlstring- URL notified when the job finishes. Example: https://example.com/hooks/diffbot.
diffbot_enhance_bulk_stopStop a running Enhance bulk job.WriteStop Enhance Bulk Job
Stop a running Enhance bulk job. Returns the job's updated status. Use Get Enhance Bulk Job Status to check it.
- Idempotent
Inputs
bulkjob_idstringrequired- Enhance bulk job ID, returned by Create Enhance Bulk Job. Example: a1b2c3d4e5.
diffbot_bulk_job_deleteDelete a bulk job and all its data.DestructiveDelete Bulk Job
Delete a bulk job and all its data. Returns a confirmation. Use Get Bulk Data first to keep the results.
- Idempotent
Inputs
deleteintegerrequired- Must be 1. Example: 1.one of
1 namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
diffbot_bulk_job_restartRestart a bulk job, erasing its processed data and reprocessing all URLs.DestructiveRestart Bulk Job
Restart a bulk job, erasing its processed data and reprocessing all URLs. Returns the job's updated status. Use Get Bulk Data first to keep the current results.
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
restartintegerrequired- Must be 1. Example: 1.one of
1
diffbot_crawl_job_deleteDelete a crawl job and all its data.DestructiveDelete Crawl
Delete a crawl job and all its data. Returns a confirmation. Use Get Crawl Data first to keep the results.
- Idempotent
Inputs
deleteintegerrequired- Must be 1. Example: 1.one of
1 namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
diffbot_crawl_job_restartRestart a crawl job from scratch, erasing its processed data and reprocessing all URLs.DestructiveRestart Crawl
Restart a crawl job from scratch, erasing its processed data and reprocessing all URLs. Returns the job's updated status. Use Start Crawl Round to repeat a crawl without erasing data.
Inputs
namestringrequired- Name of the job, as given when it was created. Example: acme-blog-crawl.
restartintegerrequired- Must be 1. Example: 1.one of
1
diffbot_enhance_bulk_deleteDelete an Enhance bulk job and its results.DestructiveDelete Enhance Bulk Job
Delete an Enhance bulk job and its results. Returns a confirmation. Use Get Enhance Bulk Results first to keep the results.
- Idempotent
Inputs
bulkjob_idstringrequired- Enhance bulk job ID, returned by Create Enhance Bulk Job. Example: a1b2c3d4e5.
No tools match.