subimpact WhatsApp us

Crawl4AI LLM Scraper: Markdown for AI

The Crawl4AI LLM Scraper renders pages in a real browser and returns LLM-ready Markdown with a link graph, plus optional raw HTML, text, JSON-LD and OpenGraph entities lifted into price, author and rating columns. Watch mode rechecks pages over plain HTTP and bills $0.002 for an unchanged page, $0.05 for a detected change. Rendered pages cost $0.005, the first three per run are free. Run it from Apify Console or the API.

from $0.05 per change detected, $50.00 per 1,000 · Updated Nov 2026, version 0.3.6

Key facts

Price
from $0.05 per change detected, $50.00 per 1,000
Pricing model
Pay per event
Output
JSON, CSV, Excel, XML, RSS via Apify datasets
Access
Apify Console, REST API, Python and JS clients
Version
Updated Nov 2026, version 0.3.6
Usage
96.5% successful runs (30 days)
Actor
subimpact/crawl4ai-llm-scraper

Who it is for

  • AI engineers building RAG corpora from documentation sites
  • Platform teams watching vendor pricing and terms pages
  • SEO and content analysts auditing broken links and duplicate pages
  • Analysts extracting product and price entities at scale

Use cases

Build a RAG corpus in one run

Set startUrls to a docs root, maxDepth to 2 and maxPages to 200, then turn on aiPackage. The run writes ai-package.jsonl with one record per page, an llms.txt link index and a manifest.json describing the bundle. Point your pipeline at that key-value store file instead of paging a dataset and reshaping records.

Watch pricing pages cheaply

Set mode watch, keep changeDetection on fast and give the run a watchId. Each check fetches the page over plain HTTP, strips scripts and CSRF nonces, and hashes it. Unchanged pages cost $0.002 and skip the browser entirely, while a material change costs $0.05 and reports addedLines, removedLines and diffRatio.

Extract product entities from a shop crawl

Set extract to auto and every page returns structured data with types, jsonld and og blocks, plus entityType, price, currency and author lifted to the top level. Billing is per page where a type was actually found, so a catalogue crawl does not pay for empty pages. A price that cannot be parsed confidently stays a string rather than a wrong number.

Alert a channel when a page moves

Set alertWebhookUrl to an HTTPS endpoint and each detected change is POSTed with url, changeType, diffRatio, title and changedAt. A Slack incoming webhook is recognised and sent as Slack blocks. Delivery is billed at $0.02 only on a 2xx response, and a failed delivery is logged without failing the run.

Audit a site you already crawled

Turn on siteIntel and the run summary adds brokenLinks with status codes, orphanCount, duplicateGroups, tokenEstimate and the ten largest pages. It is computed from pages the run already fetched, so there is no extra crawling and no extra charge. Duplicate content and a 404 behind an internal link both surface without a separate audit tool.

Output fields

FieldTypeDescription
url any
title any
format any
chars any
linkCount any
depth any
error any

Sample output

From one of our own test runs on 20 Nov 2026, trimmed. Personal data is redacted.

2 rows, first 3 shown

[
  {
    "recordType": "page",
    "url": "https://crawlee.dev",
    "title": "Crawlee · Build reliable crawlers. Fast.",
    "format": "markdown",
    "markdown": "[Skip to main content](https://crawlee.dev/#__docusaurus_skipToContent_fallback)\n[![Crawlee](https://crawlee.dev/img/crawlee-light.svg)![Crawlee](https://crawlee.dev/img/crawlee-dark.svg)](https://crawlee.dev/)\n[![Crawlee JavaScript](htt...",
    "links": {
      "internal": [
        "https://crawlee.dev/",
        "https://crawlee.dev/js",
        "https://crawlee.dev/python",
        "..."
      ],
      "external": [
        "https://github.com/apify/crawlee",
        "https://github.com/apify/crawlee-python",
        "https://console.apify.com/actors/6i5QsHBMtm3hKph70?asrc=run_on_apify&runConfig=eyJ1IjoiRWdQdHczb2VqNlRhRHQ1cW4iLCJ2IjoxfQ.eyJpbnB1dCI6IntcbiAgICBcImNvZGVcIjogXCJpbXBvcnQgeyBQbGF5d3JpZ2h0Q3Jhd2xlciB9IGZyb20gJ2NyYXdsZWUnO1xcblxcbi8vIENyYXd...",
        "..."
      ]
    },
    "linkCount": 17,
    "depth": 0,
    "chars": 6083,
    "crawledAt": "2026-11-20T02:43:14Z",
    "detectedVia": "render",
    "previousCrawledAt": "2026-11-19T02:38:18Z",
    "charsDelta": -43,
    "diffRatio": 0.0364,
    "addedLines": [
      "[![Crawlee](https://crawlee.dev/img/crawlee-light.svg)![Crawlee](https://crawlee.dev/img/crawlee-dark.svg)](https://crawlee.dev/)",
      "[![Crawlee JavaScript](https://crawlee.dev/img/crawlee-javascript-light.svg)![Crawlee JavaScript](https://crawlee.dev/img/crawlee-javascript-dark.svg)](https://crawlee.dev/js)[![Crawlee Python](https://crawlee.dev/img/crawlee-python-ligh...",
      "![Crawlee JavaScript](https://crawlee.dev/img/crawlee-javascript-light.svg)![Crawlee JavaScript](https://crawlee.dev/img/crawlee-javascript-dark.svg)",
      "..."
    ],
    "removedLines": [
      "[![Crawlee](https://crawlee.dev/img/crawlee-light.svg)](https://crawlee.dev/)",
      "[![Crawlee JavaScript](https://crawlee.dev/img/crawlee-javascript-light.svg)](https://crawlee.dev/js)[![Crawlee Python](https://crawlee.dev/img/crawlee-python-light.svg)](https://crawlee.dev/python)[![Crawlee](https://crawlee.dev/img/cra...",
      "[![Crawlee](https://crawlee.dev/img/crawlee-light.svg)](https://crawlee.dev/)",
      "..."
    ],
    "diffTruncated": false,
    "changeType": "changed",
    "changeSignal": {
      "changedSincePrevious": true,
      "locked": true
    }
  },
  {
    "recordType": "summary",
    "mode": "crawl",
    "watchId": "4da88c409a2c8f41",
    "checked": 1,
    "unchanged": 0,
    "changed": 1,
    "new": 0,
    "removed": 0,
    "failed": 0,
    "crawlComplete": true,
    "extract": "none",
    "elapsedSeconds": 2.8,
    "finishedAt": "2026-11-20T02:43:14Z",
    "estimatedChargeUsd": 0,
    "chargedUsd": 0,
    "eventCounts": {},
    "chargeCapReached": false,
    "estimateOnly": false,
    "freeMode": true
  }
]

Input

FieldTypeRequiredDescriptionDefault
startUrls array Yes List of pages to scrape or watch. Each entry: {"url": "https://..."} [{"url":"https://crawlee.dev"}]
mode string No Crawl = scrape every page. Watch = check each page for changes and only render the ones that moved. crawl
maxDepth integer No How many levels of internal links to follow (0 = only the start URLs) 1
maxPages integer No Hard cap on the number of pages processed per run 10
outputFormat string No Format of the page content stored in the dataset markdown
includeRawHtml boolean No Also store the raw (unrendered) HTML for each page false
pageLoadTimeoutMs integer No Max time to wait for a page to render 30000
renderConcurrency integer No How many pages to render in parallel. Automatically reduced to fit the run's memory limit. 5
extract string No Pull structured data out of every page. JSON-LD reads blocks; OpenGraph reads og:/twitter: meta tags plus the canonical URL. Auto does both. Billed at $0.010 per page where a type is actually found — pages with no structured data cost n... none
aiPackage boolean No Write ai-package.jsonl (one JSON record per page), llms.txt (the link index) and manifest.json to this run's key-value store, ready for a RAG pipeline. $0.100 once per run. Not available on free runs. false
siteIntel boolean No Add an audit block to the run summary: broken links, orphan pages, duplicate content, token estimate and the largest pages. Computed from what the run already fetched — no extra crawling, no extra charge. false
watchId string No Name of the state store that remembers previous page fingerprints. Runs sharing a watch ID compare against each other. Defaults to a hash of the start URLs. -
changeDetection string No Fast = plain HTTP fetch first, render only on a mismatch (much cheaper). Rendered = always render before comparing (needed for JS-built pages). fast
changeThreshold number No How different a page must be to count as changed, 0-1. 0.02 = 2% of lines differ. 0.02
skipUnchanged boolean No In crawl mode, skip rendering pages whose content has not changed since the last run with the same watch ID. false
alertWebhookUrl string No HTTPS URL to POST to when a change is detected. Slack incoming webhooks are auto-formatted as Slack messages. -
maxChargeUsd number No Hard ceiling on what this run may charge. Once reached, no further events are billed. 5
estimateOnly boolean No Run the full crawl and report what it would cost, without charging anything. false
proxyConfiguration object No Apify proxy for IP rotation. Leave default for automatic proxy. -
verifyProxyRotation boolean No Fetch an IP echo endpoint twice on two different proxy sessions and report both exit IPs to the dataset. Use this to confirm rotation is actually happening. false

Example input

{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ]
}

Pricing

EventWhat it meansPriceLowest tier
change-detected Per page with a material content change in watch mode. $0.05 $0.05
actor-start One charge per run (skipped in free mode). $0.01 $0.01
ai-package Once per run when the RAG bundle is written. $0.1 $0.1
alert-delivered Per successful webhook/Slack alert delivery. $0.02 $0.02
page-monitored Per page checked in watch mode with no change found. $0.002 $0.002
page-scraped Per rendered page (first 3 per run free). $0.005 $0.005
structured-extract Per page where structured data (JSON-LD/OpenGraph) was found. $0.01 $0.01

$50.00 for 1,000 change detected units at the listed price.

Prices from the Apify Store on 20 November 2026. Apify shows the final price before each run.

How it compares with other Apify actors

Price per 1,000 pages at Apify Store list prices. Every row is a live listing on the Store, read on 20 Nov 2026, so a buyer can check each number at the source.

Actor Price per 1,000 pages (Free plan) Higher plans (Gold) Start fee per run Platform usage Users Rating Last build
Crawl4AI LLM Scraper — Markdown for AI This API Pay per event $5.00 $5.00 $0.01 Included 2 No reviews yet 18 Nov 2026
Website to Markdown Scraper for AI & LLMs Direct match, pay per event $30.00 $24.00 $0.00005 Included 947 5.0 (1 review) 25 Oct 2026
Website Content Crawler Pro Direct match, pay per event $2.99 $2.97 None Included 575 3.7 (3 reviews) 13 Dec 2025
Website Content to Markdown for LLM Training Direct match, pay per event $2.99 $2.99 $0.09 Paid separately 341 5.0 (2 reviews) 25 Nov 2025
Website Content Crawler - Markdown, 45 Fields Direct match, pay per event $25.00 $20.00 $0.005 Included 222 No reviews yet 3 Nov 2026
Where Website to Markdown Scraper for AI & LLMs is stronger

Where Website to Markdown Scraper for AI & LLMs is stronger

  • Larger user base than the other markdown crawlers listed here
  • Depth and URL caps keep a same domain crawl to a few inputs
  • Headless Chrome rendering, per the listing

Where this API is stronger

  • Change detection with webhook alerts for pages you monitor
  • Optional structured extraction and an AI ready package per run
  • Each row records link count and crawl depth alongside the content
Where Website Content Crawler Pro is stronger

Where Website Content Crawler Pro is stronger

  • Bulk processing from one start URL input
  • Markdown, JSON or plain text output for each page
  • Lists integrations with LangChain and LlamaIndex

Where this API is stronger

  • Monitoring mode with change detection and alert webhooks
  • Depth crawling with a page cap and a per run charge limit
  • Optional structured extraction and AI ready package output
Where Website Content to Markdown for LLM Training is stronger

Where Website Content to Markdown for LLM Training is stronger

  • Simple flow aimed at building training data from pages
  • Main content extraction keeps page boilerplate out of the markdown
  • Several pages can be converted in a single run

Where this API is stronger

  • Rows carry link count, character count and crawl depth
  • Change detection and webhook alerts for monitored pages
  • Output format can be Markdown, HTML or plain text per run
Where Website Content Crawler - Markdown, 45 Fields is stronger

Where Website Content Crawler - Markdown, 45 Fields is stronger

  • Wide per page field set including meta tags, headings and Open Graph data
  • Word count, language, author and published time per page
  • Depth, domain and URL pattern limits for the crawl

Where this API is stronger

  • Change detection and alerting for the pages you watch
  • Proxy rotation with an option to verify that rotation works
  • Optional structured extraction and AI ready package output

Website to Markdown is a crowded category on the Store, and every actor listed here bills per page. quaking_pail and datascoutapi focus on clean Markdown for retrieval, parseforge adds a wide metadata field set, and easyapi keeps a simple conversion flow. Ours adds monitoring with change detection, alerts and optional structured extraction.

Prices are Apify Store list prices, checked 20 Nov 2026. Platform usage 'Paid separately' means Apify bills compute on top of the event price. Check each listing before you buy.

How to run it

  1. List pages in startUrls as objects with a url field, for example https://example.com/pricing, and set maxDepth and maxPages for the crawl size.
  2. Set outputFormat to markdown, html or text, and turn on aiPackage if a JSONL corpus and llms.txt index are what you need.
  3. For monitoring, switch mode to watch, name a watchId and choose changeDetection fast or rendered.
  4. Use estimateOnly true on a full-size run first: it crawls everything, charges nothing and reports the bill it would have run up.
  5. Run the actor, then read the page rows and summary row in the dataset, or the ai-package files in the key-value store.

cURL

curl -X POST "https://api.apify.com/v2/acts/subimpact~crawl4ai-llm-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawlee.dev"}]}'

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("subimpact/crawl4ai-llm-scraper").call(run_input={"startUrls":[{"url":"https://crawlee.dev"}]})

for item in client.dataset(run["default_dataset_id"]).iterate_items():
    print(item)

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('subimpact/crawl4ai-llm-scraper').call({"startUrls":[{"url":"https://crawlee.dev"}]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

console.log(items);

n8n

Use the Apify node (Run an Actor).
Actor ID: subimpact/crawl4ai-llm-scraper
Input JSON: paste the example input above.

Limits

  • maxPages is capped at 1,000 per run, so very large sites need several runs.
  • Page fingerprints above 200 KB are compared by hash only, without a line diff.
  • Alert webhooks must be https and resolve to a public address, with no redirects followed.
  • An AI corpus above 8 MB is truncated rather than failing, and the manifest says so.

Questions people ask

What does a page cost?

A rendered page costs $0.005, and the first three pages of a run are free. In watch mode an unchanged check costs $0.002 and a detected change costs $0.05. The run start is $0.01, the AI-ready package $0.10 once per run, and alerts $0.02 per delivered webhook.

Can I try it without paying?

Yes. Runs with maxPages at 3 or fewer, no watch, alert, incremental or aiPackage features, no residential proxy and up to 4 GB memory are not billed. They still show which structured types a page exposes and whether it changed, with the values locked. estimateOnly gives a full-size price without charging.

How is watch mode cheaper than re-scraping?

A watch check fetches HTML without a browser, strips scripts, styles and CSRF nonces, and compares a hash to the stored fingerprint. Pages that match cost $0.002 and skip rendering entirely. A browser only starts when the fingerprint moves, so a 1,000 page daily watch at 3% change costs far less than rendering all of them.

What is in the AI-ready package?

Three files in the run's key-value store: ai-package.jsonl with url, title, markdown, structured data and crawledAt per page, llms.txt as a link index in llms.txt format, and manifest.json with the actor version, run id, page count and schema. It costs $0.10 once per run, charged only after the files are written.

Does it handle JavaScript-built pages?

Yes. Pages are rendered in a real browser through the Crawl4AI engine, and changeDetection can be set to rendered when a plain HTTP fetch would not show the real content. Structured extraction reads the post-JavaScript DOM with script tags intact, so JSON-LD injected by client-side JavaScript is picked up too.

Can I cap the spend of a run?

maxChargeUsd is enforced inside the run. Once the ceiling is reached nothing further is billed and the summary says so. Charges are also de-duplicated per URL and event, so an Apify migration mid-run cannot bill you twice for the same page.

How do I know the crawl used rotating IPs?

Set verifyProxyRotation true and the run fetches an IP echo endpoint on two different sessions, writing both exit IPs and a rotating flag to the dataset. Requests already go through Apify Proxy sessions rotated per request, with a fresh session on each retry, and up to three attempts per page.

Need a data source that is not here?

We build and run scrapers for sites that have no public API. Tell us the site, the fields and how often you need them.