Crawl4AI LLM Scraper: Markdown for AI
The Crawl4AI LLM Scraper renders pages in a real browser and returns LLM-ready Markdown with a link graph, plus optional raw HTML, text, JSON-LD and OpenGraph entities lifted into price, author and rating columns. Watch mode rechecks pages over plain HTTP and bills $0.002 for an unchanged page, $0.05 for a detected change. Rendered pages cost $0.005, the first three per run are free. Run it from Apify Console or the API.
from $0.05 per change detected, $50.00 per 1,000 · Updated Nov 2026, version 0.3.6
Key facts
- Price
- from $0.05 per change detected, $50.00 per 1,000
- Pricing model
- Pay per event
- Output
- JSON, CSV, Excel, XML, RSS via Apify datasets
- Access
- Apify Console, REST API, Python and JS clients
- Version
- Updated Nov 2026, version 0.3.6
- Usage
- 96.5% successful runs (30 days)
- Actor
subimpact/crawl4ai-llm-scraper
Who it is for
- AI engineers building RAG corpora from documentation sites
- Platform teams watching vendor pricing and terms pages
- SEO and content analysts auditing broken links and duplicate pages
- Analysts extracting product and price entities at scale
Use cases
Build a RAG corpus in one run
Set startUrls to a docs root, maxDepth to 2 and maxPages to 200, then turn on aiPackage. The run writes ai-package.jsonl with one record per page, an llms.txt link index and a manifest.json describing the bundle. Point your pipeline at that key-value store file instead of paging a dataset and reshaping records.
Watch pricing pages cheaply
Set mode watch, keep changeDetection on fast and give the run a watchId. Each check fetches the page over plain HTTP, strips scripts and CSRF nonces, and hashes it. Unchanged pages cost $0.002 and skip the browser entirely, while a material change costs $0.05 and reports addedLines, removedLines and diffRatio.
Extract product entities from a shop crawl
Set extract to auto and every page returns structured data with types, jsonld and og blocks, plus entityType, price, currency and author lifted to the top level. Billing is per page where a type was actually found, so a catalogue crawl does not pay for empty pages. A price that cannot be parsed confidently stays a string rather than a wrong number.
Alert a channel when a page moves
Set alertWebhookUrl to an HTTPS endpoint and each detected change is POSTed with url, changeType, diffRatio, title and changedAt. A Slack incoming webhook is recognised and sent as Slack blocks. Delivery is billed at $0.02 only on a 2xx response, and a failed delivery is logged without failing the run.
Audit a site you already crawled
Turn on siteIntel and the run summary adds brokenLinks with status codes, orphanCount, duplicateGroups, tokenEstimate and the ten largest pages. It is computed from pages the run already fetched, so there is no extra crawling and no extra charge. Duplicate content and a 404 behind an internal link both surface without a separate audit tool.
Output fields
| Field | Type | Description |
|---|---|---|
url | any | |
title | any | |
format | any | |
chars | any | |
linkCount | any | |
depth | any | |
error | any |
Sample output
From one of our own test runs on 20 Nov 2026, trimmed. Personal data is redacted.
2 rows, first 3 shown
[
{
"recordType": "page",
"url": "https://crawlee.dev",
"title": "Crawlee · Build reliable crawlers. Fast.",
"format": "markdown",
"markdown": "[Skip to main content](https://crawlee.dev/#__docusaurus_skipToContent_fallback)\n[](https://crawlee.dev/)\n[](https://crawlee.dev/)",
"[](https://crawlee.dev/js)[",
"..."
],
"removedLines": [
"[](https://crawlee.dev/)",
"[](https://crawlee.dev/js)[](https://crawlee.dev/python)[](https://crawlee.dev/)",
"..."
],
"diffTruncated": false,
"changeType": "changed",
"changeSignal": {
"changedSincePrevious": true,
"locked": true
}
},
{
"recordType": "summary",
"mode": "crawl",
"watchId": "4da88c409a2c8f41",
"checked": 1,
"unchanged": 0,
"changed": 1,
"new": 0,
"removed": 0,
"failed": 0,
"crawlComplete": true,
"extract": "none",
"elapsedSeconds": 2.8,
"finishedAt": "2026-11-20T02:43:14Z",
"estimatedChargeUsd": 0,
"chargedUsd": 0,
"eventCounts": {},
"chargeCapReached": false,
"estimateOnly": false,
"freeMode": true
}
]Input
| Field | Type | Required | Description | Default |
|---|---|---|---|---|
startUrls | array | Yes | List of pages to scrape or watch. Each entry: {"url": "https://..."} | [{"url":"https://crawlee.dev"}] |
mode | string | No | Crawl = scrape every page. Watch = check each page for changes and only render the ones that moved. | crawl |
maxDepth | integer | No | How many levels of internal links to follow (0 = only the start URLs) | 1 |
maxPages | integer | No | Hard cap on the number of pages processed per run | 10 |
outputFormat | string | No | Format of the page content stored in the dataset | markdown |
includeRawHtml | boolean | No | Also store the raw (unrendered) HTML for each page | false |
pageLoadTimeoutMs | integer | No | Max time to wait for a page to render | 30000 |
renderConcurrency | integer | No | How many pages to render in parallel. Automatically reduced to fit the run's memory limit. | 5 |
extract | string | No | Pull structured data out of every page. JSON-LD reads blocks; OpenGraph reads og:/twitter: meta tags plus the canonical URL. Auto does both. Billed at $0.010 per page where a type is actually found — pages with no structured data cost n... | none |
aiPackage | boolean | No | Write ai-package.jsonl (one JSON record per page), llms.txt (the link index) and manifest.json to this run's key-value store, ready for a RAG pipeline. $0.100 once per run. Not available on free runs. | false |
siteIntel | boolean | No | Add an audit block to the run summary: broken links, orphan pages, duplicate content, token estimate and the largest pages. Computed from what the run already fetched — no extra crawling, no extra charge. | false |
watchId | string | No | Name of the state store that remembers previous page fingerprints. Runs sharing a watch ID compare against each other. Defaults to a hash of the start URLs. | - |
changeDetection | string | No | Fast = plain HTTP fetch first, render only on a mismatch (much cheaper). Rendered = always render before comparing (needed for JS-built pages). | fast |
changeThreshold | number | No | How different a page must be to count as changed, 0-1. 0.02 = 2% of lines differ. | 0.02 |
skipUnchanged | boolean | No | In crawl mode, skip rendering pages whose content has not changed since the last run with the same watch ID. | false |
alertWebhookUrl | string | No | HTTPS URL to POST to when a change is detected. Slack incoming webhooks are auto-formatted as Slack messages. | - |
maxChargeUsd | number | No | Hard ceiling on what this run may charge. Once reached, no further events are billed. | 5 |
estimateOnly | boolean | No | Run the full crawl and report what it would cost, without charging anything. | false |
proxyConfiguration | object | No | Apify proxy for IP rotation. Leave default for automatic proxy. | - |
verifyProxyRotation | boolean | No | Fetch an IP echo endpoint twice on two different proxy sessions and report both exit IPs to the dataset. Use this to confirm rotation is actually happening. | false |
Example input
{
"startUrls": [
{
"url": "https://crawlee.dev"
}
]
}Pricing
| Event | What it means | Price | Lowest tier |
|---|---|---|---|
change-detected | Per page with a material content change in watch mode. | $0.05 | $0.05 |
actor-start | One charge per run (skipped in free mode). | $0.01 | $0.01 |
ai-package | Once per run when the RAG bundle is written. | $0.1 | $0.1 |
alert-delivered | Per successful webhook/Slack alert delivery. | $0.02 | $0.02 |
page-monitored | Per page checked in watch mode with no change found. | $0.002 | $0.002 |
page-scraped | Per rendered page (first 3 per run free). | $0.005 | $0.005 |
structured-extract | Per page where structured data (JSON-LD/OpenGraph) was found. | $0.01 | $0.01 |
$50.00 for 1,000 change detected units at the listed price.
Prices from the Apify Store on 20 November 2026. Apify shows the final price before each run.
How it compares with other Apify actors
Price per 1,000 pages at Apify Store list prices. Every row is a live listing on the Store, read on 20 Nov 2026, so a buyer can check each number at the source.
| Actor | Price per 1,000 pages (Free plan) | Higher plans (Gold) | Start fee per run | Platform usage | Users | Rating | Last build |
|---|---|---|---|---|---|---|---|
| Crawl4AI LLM Scraper — Markdown for AI This API | $5.00 | $5.00 | $0.01 | Included | 2 | No reviews yet | 18 Nov 2026 |
| Website to Markdown Scraper for AI & LLMs | $30.00 | $24.00 | $0.00005 | Included | 947 | 5.0 (1 review) | 25 Oct 2026 |
| Website Content Crawler Pro | $2.99 | $2.97 | None | Included | 575 | 3.7 (3 reviews) | 13 Dec 2025 |
| Website Content to Markdown for LLM Training | $2.99 | $2.99 | $0.09 | Paid separately | 341 | 5.0 (2 reviews) | 25 Nov 2025 |
| Website Content Crawler - Markdown, 45 Fields | $25.00 | $20.00 | $0.005 | Included | 222 | No reviews yet | 3 Nov 2026 |
Where Website to Markdown Scraper for AI & LLMs is stronger
Where Website to Markdown Scraper for AI & LLMs is stronger
- Larger user base than the other markdown crawlers listed here
- Depth and URL caps keep a same domain crawl to a few inputs
- Headless Chrome rendering, per the listing
Where this API is stronger
- Change detection with webhook alerts for pages you monitor
- Optional structured extraction and an AI ready package per run
- Each row records link count and crawl depth alongside the content
Where Website Content Crawler Pro is stronger
Where Website Content Crawler Pro is stronger
- Bulk processing from one start URL input
- Markdown, JSON or plain text output for each page
- Lists integrations with LangChain and LlamaIndex
Where this API is stronger
- Monitoring mode with change detection and alert webhooks
- Depth crawling with a page cap and a per run charge limit
- Optional structured extraction and AI ready package output
Where Website Content to Markdown for LLM Training is stronger
Where Website Content to Markdown for LLM Training is stronger
- Simple flow aimed at building training data from pages
- Main content extraction keeps page boilerplate out of the markdown
- Several pages can be converted in a single run
Where this API is stronger
- Rows carry link count, character count and crawl depth
- Change detection and webhook alerts for monitored pages
- Output format can be Markdown, HTML or plain text per run
Where Website Content Crawler - Markdown, 45 Fields is stronger
Where Website Content Crawler - Markdown, 45 Fields is stronger
- Wide per page field set including meta tags, headings and Open Graph data
- Word count, language, author and published time per page
- Depth, domain and URL pattern limits for the crawl
Where this API is stronger
- Change detection and alerting for the pages you watch
- Proxy rotation with an option to verify that rotation works
- Optional structured extraction and AI ready package output
Website to Markdown is a crowded category on the Store, and every actor listed here bills per page. quaking_pail and datascoutapi focus on clean Markdown for retrieval, parseforge adds a wide metadata field set, and easyapi keeps a simple conversion flow. Ours adds monitoring with change detection, alerts and optional structured extraction.
Prices are Apify Store list prices, checked 20 Nov 2026. Platform usage 'Paid separately' means Apify bills compute on top of the event price. Check each listing before you buy.
How to run it
- List pages in startUrls as objects with a url field, for example https://example.com/pricing, and set maxDepth and maxPages for the crawl size.
- Set outputFormat to markdown, html or text, and turn on aiPackage if a JSONL corpus and llms.txt index are what you need.
- For monitoring, switch mode to watch, name a watchId and choose changeDetection fast or rendered.
- Use estimateOnly true on a full-size run first: it crawls everything, charges nothing and reports the bill it would have run up.
- Run the actor, then read the page rows and summary row in the dataset, or the ai-package files in the key-value store.
cURL
curl -X POST "https://api.apify.com/v2/acts/subimpact~crawl4ai-llm-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrls":[{"url":"https://crawlee.dev"}]}'Python
from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("subimpact/crawl4ai-llm-scraper").call(run_input={"startUrls":[{"url":"https://crawlee.dev"}]})
for item in client.dataset(run["default_dataset_id"]).iterate_items():
print(item)JavaScript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('subimpact/crawl4ai-llm-scraper').call({"startUrls":[{"url":"https://crawlee.dev"}]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);n8n
Use the Apify node (Run an Actor).
Actor ID: subimpact/crawl4ai-llm-scraper
Input JSON: paste the example input above.Limits
- maxPages is capped at 1,000 per run, so very large sites need several runs.
- Page fingerprints above 200 KB are compared by hash only, without a line diff.
- Alert webhooks must be https and resolve to a public address, with no redirects followed.
- An AI corpus above 8 MB is truncated rather than failing, and the manifest says so.
Questions people ask
What does a page cost?
A rendered page costs $0.005, and the first three pages of a run are free. In watch mode an unchanged check costs $0.002 and a detected change costs $0.05. The run start is $0.01, the AI-ready package $0.10 once per run, and alerts $0.02 per delivered webhook.
Can I try it without paying?
Yes. Runs with maxPages at 3 or fewer, no watch, alert, incremental or aiPackage features, no residential proxy and up to 4 GB memory are not billed. They still show which structured types a page exposes and whether it changed, with the values locked. estimateOnly gives a full-size price without charging.
How is watch mode cheaper than re-scraping?
A watch check fetches HTML without a browser, strips scripts, styles and CSRF nonces, and compares a hash to the stored fingerprint. Pages that match cost $0.002 and skip rendering entirely. A browser only starts when the fingerprint moves, so a 1,000 page daily watch at 3% change costs far less than rendering all of them.
What is in the AI-ready package?
Three files in the run's key-value store: ai-package.jsonl with url, title, markdown, structured data and crawledAt per page, llms.txt as a link index in llms.txt format, and manifest.json with the actor version, run id, page count and schema. It costs $0.10 once per run, charged only after the files are written.
Does it handle JavaScript-built pages?
Yes. Pages are rendered in a real browser through the Crawl4AI engine, and changeDetection can be set to rendered when a plain HTTP fetch would not show the real content. Structured extraction reads the post-JavaScript DOM with script tags intact, so JSON-LD injected by client-side JavaScript is picked up too.
Can I cap the spend of a run?
maxChargeUsd is enforced inside the run. Once the ceiling is reached nothing further is billed and the summary says so. Charges are also de-duplicated per URL and event, so an Apify migration mid-run cannot bill you twice for the same page.
How do I know the crawl used rotating IPs?
Set verifyProxyRotation true and the run fetches an IP echo endpoint on two different sessions, writing both exit IPs and a rotating flag to the dataset. Requests already go through Apify Proxy sessions rotated per request, with a fresh session on each retry, and up to three attempts per page.
Need a data source that is not here?
We build and run scrapers for sites that have no public API. Tell us the site, the fields and how often you need them.