subimpact WhatsApp us

Sitemap URL Extractor

Sitemap URL Extractor finds a site's sitemaps through robots.txt and the usual sitemap paths, follows nested sitemap indexes up to five levels, unpacks gzip and reads RSS, Atom and plain-text sitemaps, then returns one JSON row per URL with the source sitemap, lastmod, changefreq, priority, image URLs and hreflang alternates. It costs $0.0003 per URL ($0.30 per 1,000), with an optional status check at $0.0005 per URL.

from $0.0003 per url extracted, $0.3 per 1,000, down to $0.0002 on higher Apify plans · Updated Nov 2026, version 0.1.5

Key facts

Price
from $0.0003 per url extracted, $0.3 per 1,000, down to $0.0002 on higher Apify plans
Pricing model
Pay per event
Output
JSON, CSV, Excel, XML, RSS via Apify datasets
Access
Apify Console, REST API, Python and JS clients, MCP tool sitemap_urls
Version
Updated Nov 2026, version 0.1.5
Actor
subimpact/sitemap-url-extractor

Who it is for

  • SEO teams auditing which URLs a site actually publishes
  • Crawler builders who need a seed list before scraping
  • Content migration teams comparing old and new URL sets
  • Site owners finding 404s and redirects at scale

Use cases

Get a complete crawl seed list

Point startUrls at a domain and the run reads robots.txt Sitemap lines first, then probes the common sitemap paths, following indexes up to five levels deep. The dataset becomes a deduplicated URL list you can feed to any crawler, with maxUrls as a hard cap so a huge site does not turn into an unbounded job.

Keep only the section you audit

Set includePatterns to a regex such as /blog/ and excludePatterns to something like \.(jpg|jpeg|png|webp|pdf)$ to drop media. Filters are applied before delivery, so you pay only for the URLs that reach the dataset. That is how a content team extracts just the blog posts from a sitemap index holding the whole site.

Find stale or broken pages

Turn on checkStatus and each delivered URL gets an HTTP status code via a HEAD request with a GET fallback. Sorting the rows by statusCode surfaces 404s and redirects in the same run that extracted them. The status check is billed separately per URL, so you can run extraction alone first and add checks on a filtered subset.

See what changed since a date

lastmodAfter takes an ISO date and keeps only sitemap entries whose lastmod is on or after it. Entries with no lastmod are dropped while the filter is set, so the result is a genuine changed-since list. That is the input for a reindex request or for checking whether a CMS migration actually updated timestamps.

Audit a multilingual site

Each row carries the xhtml:link hreflang alternates that multilingual sites publish, along with image counts and the image URLs themselves. That shows whether every language version is listed and correctly paired, which is otherwise assembled by hand from sitemap XML. Rows also record the sitemap they came from, so a missing locale points at the file that should have carried it.

Output fields

FieldTypeDescription
url string/null Absolute URL found in a sitemap, fragment removed. null on error rows.
sitemapUrl string/null The sitemap file this URL came from (the final URL after redirects).
lastmod string/null Sitemap <lastmod> value exactly as published (W3C datetime, date-only, or RSS pubDate / Atom updated).
changefreq string/null Sitemap <changefreq> value as published (always, hourly, daily, weekly, monthly, yearly, never).
priority number/null Sitemap <priority> value as a number between 0 and 1.
images integer/null How many <image:image> entries the sitemap lists for this URL (0 when the sitemap has no image data).
imageUrls array/null The <image:loc> values listed for this URL, in sitemap order. null when the sitemap lists none.
alternates array/null xhtml:link rel="alternate" entries for this URL: the hreflang code and the URL it points at. null when the sitemap has no language variants.
statusCode integer/null HTTP status code returned for this URL when checkStatus is on, without following redirects (301/302 means the URL redirects). null when checkStatus is off or the request failed.
scrapedAt string ISO 8601 timestamp of when this row was delivered.
status string ok for a URL found in a sitemap, error for a sitemap (or discovery step) that could not be read. Error rows are free.
error string/null Why a sitemap could not be read (HTTP status, timeout, not a sitemap, empty). null on URL rows.

Sample output

From one of our own test runs on 20 Nov 2026, trimmed. Personal data is redacted.

2 rows, first 3 shown

[
  {
    "url": "https://apify.com/",
    "sitemapUrl": "https://apify.com/sitemap/pages.xml",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "images": 0,
    "imageUrls": null,
    "alternates": null,
    "statusCode": null,
    "scrapedAt": "2026-11-20T02:53:30.759Z",
    "status": "ok",
    "error": null
  },
  {
    "url": "https://apify.com/about",
    "sitemapUrl": "https://apify.com/sitemap/pages.xml",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "images": 0,
    "imageUrls": null,
    "alternates": null,
    "statusCode": null,
    "scrapedAt": "2026-11-20T02:53:30.759Z",
    "status": "ok",
    "error": null
  }
]

Input

FieldTypeRequiredDescriptionDefault
startUrls array No Website roots (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). For a site root the actor reads robots.txt for Sitemap: lines and then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitem... -
includePatterns array No Keep only URLs matching at least one of these regular expressions. Example: /blog/ to keep blog URLs only. Empty = keep everything. []
excludePatterns array No Drop URLs matching any of these regular expressions. Example: \.(jpg|png|pdf)$ or /tag/. Excludes are applied after includes. []
lastmodAfter string No ISO date (2026-01-31 or 2026-01-31T12:00:00Z). Keeps only sitemap entries whose is on or after this date. Entries without a lastmod are dropped when this is set.
maxUrls integer No Hard cap on delivered URL rows (1-1,000,000). The run stops as soon as it is reached, so a default run stays cheap. Charged at $0.0003 per delivered URL. 1000
checkStatus boolean No Send a HEAD request (GET fallback) to each delivered URL and fill in statusCode, so you can spot 404s and redirects. Billed separately at $0.50 per 1,000 checked URLs. false
maxSitemaps integer No How many sitemap files (index children included) one run may read (1-10,000). Default 200 is plenty for almost every site. 200
requestTimeoutSecs integer No How long to wait for a sitemap or robots.txt response (5-300 seconds). 60
proxyConfiguration object No Optional. Sitemaps are public, so no proxy is used by default. Add an Apify proxy for sites that block datacenter IPs. -

Example input

{
  "startUrls": [
    "https://www.smashingmagazine.com"
  ]
}

Pricing

EventWhat it meansPriceLowest tier
url-extracted One URL found in a sitemap and delivered as a dataset row. Platform usage included. $0.0003 $0.0002
url-status-checked One HTTP status check (HEAD, GET fallback) for a delivered URL when checkStatus is on. Platform usage included. $0.0005 $0.00035

$0.3 at the listed price, or $0.2 on higher Apify plans, for 1,000 url extracted units.

Prices from the Apify Store on 20 November 2026. Apify shows the final price before each run.

How it compares with other Apify actors

Price per 1,000 urls at Apify Store list prices. Every row is a live listing on the Store, read on 20 Nov 2026, so a buyer can check each number at the source.

Actor Price per 1,000 urls (Free plan) Higher plans (Gold) Start fee per run Platform usage Users Rating Last build
Sitemap URL Extractor — every URL in a site's sitemaps This API Pay per event $0.3 $0.2 None Included 1 No reviews yet 18 Nov 2026
Sitemap Extractor Direct match, pay per event $0.5 $0.1 None Included 473 3.2 (8 reviews) 27 Oct 2026
Sitemap URL Extractor Direct match, pay per event $30.00 $1.00 $0.00005 Included 668 5.0 (3 reviews) 8 Nov 2026
Website URL Extractor - Get All Site URLs Partial match, pay per event $0.35 $0.2 $0.05 Included 244 No reviews yet 15 Jun 2026
Sitemap to URL Crawler — Extract Sitemap.xml URLs Direct match, pay per event $1.50 $0.5 None Included 64 No reviews yet 14 Nov 2026
Where Sitemap Extractor is stronger

Where Sitemap Extractor is stronger

  • Larger user base and maintained by Apify itself
  • Status checking plus crawl depth and retry controls
  • Compact output of url, status and lastmod

Where this API is stronger

  • Nested sitemap indexes, gzip files and RSS or Atom feeds found from robots.txt
  • Priority, changefreq, image counts and hreflang alternates per URL
  • Include and exclude patterns plus a lastmod cutoff applied before extraction
Where Sitemap URL Extractor is stronger

Where Sitemap URL Extractor is stronger

  • Larger user base on the Store
  • Very short input: one or more sitemap URLs and nothing else

Where this API is stronger

  • Sitemaps discovered from robots.txt rather than needing exact sitemap URLs
  • Optional HTTP status check per URL as a separate event
  • Pattern filters, lastmod cutoff and a maximum URL count
Where Website URL Extractor - Get All Site URLs is stronger

Where Website URL Extractor - Get All Site URLs is stronger

  • Crawls pages as well as parsing sitemaps
  • Keyword filter and result cap for quick audits
  • No code needed and a simple input form

Where this API is stronger

  • Nested sitemap indexes, gzip and RSS or Atom feed handling
  • Image counts and hreflang alternates on each URL
  • No start fee before the first URL
Where Sitemap to URL Crawler — Extract Sitemap.xml URLs is stronger

Where Sitemap to URL Crawler — Extract Sitemap.xml URLs is stronger

  • Accepts a domain or a sitemap URL as the starting point
  • Very small input set for a fast export
  • Built for RAG and SEO export workflows

Where this API is stronger

  • Filters, lastmod cutoff and status checks in the same actor
  • Richer rows with priority, changefreq, images and alternates
  • Handles gzip sitemaps and RSS or Atom feeds

Sitemap extraction is a crowded, simple lane and the actors differ in how much they discover and how much they return. The Apify listing is the best known with status checks, and the others keep setup minimal. Ours goes further on discovery and detail: robots.txt lookup, nested indexes, gzip and feeds, with images, alternates and filtering per URL.

Prices are Apify Store list prices, checked 20 Nov 2026. Platform usage 'Paid separately' means Apify bills compute on top of the event price. Check each listing before you buy.

How to run it

  1. Put a site root or a direct sitemap URL in startUrls, for example ["https://www.smashingmagazine.com"] or ["https://example.com/sitemap.xml"].
  2. Optional: set includePatterns and excludePatterns to keep only the sections you want, for example /blog/ and \.(jpg|png|pdf)$.
  3. Set maxUrls, 1,000 by default, as the delivered row cap, and maxSitemaps if the site has a large index.
  4. Turn on checkStatus only if you also want an HTTP status code per URL, since that is billed per check.
  5. Start the run in the Apify Console; rows stream in batches of up to 500 while the sitemap is still downloading.
  6. Export as JSON, CSV or Excel, and read RUN_SUMMARY for sitemaps read, URLs seen, duplicates and filter counts.

cURL

curl -X POST "https://api.apify.com/v2/acts/subimpact~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://www.smashingmagazine.com"]}'

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("subimpact/sitemap-url-extractor").call(run_input={"startUrls":["https://www.smashingmagazine.com"]})

for item in client.dataset(run["default_dataset_id"]).iterate_items():
    print(item)

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('subimpact/sitemap-url-extractor').call({"startUrls":["https://www.smashingmagazine.com"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

console.log(items);

MCP

claude mcp add subimpact-data -e APIFY_TOKEN=<YOUR_APIFY_TOKEN> -- npx -y @subimpact/data-mcp

# Then ask your agent to use the sitemap_urls tool.

n8n

Use the Apify node (Run an Actor).
Actor ID: subimpact/sitemap-url-extractor
Input JSON: paste the example input above.

MCP users: @subimpact/data-mcp exposes these actors as tools, so an agent can call sitemap_urls directly.

Limits

  • A single sitemap is capped at 100 MB decompressed, and nested indexes at five levels.
  • Entries without a lastmod are dropped whenever lastmodAfter is set.
  • statusCode checks do not follow redirects, and are billed per URL whether the code is 200 or 404.
  • The Actor reads sitemaps only; it does not crawl pages or discover URLs by following links.

Questions people ask

What does a URL cost?

$0.0003 per delivered URL, which is $0.30 per 1,000 URLs, and $0.0005 per URL when checkStatus adds a status code. There is no start fee. Rows are pushed before they are charged, so URLs removed by your filters, duplicates and error rows cost nothing.

Do I need to know the sitemap URL?

No. Give it the site root and it reads every Sitemap line in robots.txt, then probes /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml in order. A direct sitemap URL is also accepted and read as-is.

Does it unpack gzip and read feeds?

Yes. .xml.gz files are gunzipped, and RSS and Atom documents listed as sitemaps are parsed as URL sources. A body that is not XML is read as a plain-text sitemap with one URL per line, and an HTML error page is reported as not a sitemap rather than as zero URLs.

What is in each row?

The URL, the sitemap it came from, lastmod, changefreq, priority, the number of image entries with their URLs, the hreflang alternates, and statusCode when checkStatus is on. URLs are normalised, relative locations resolved and fragments removed before delivery.

How deep does it follow sitemap indexes?

Up to five levels, which is Google's own limit, and up to maxSitemaps files, 200 by default. Each sitemap URL is fetched once per run, so a file linked from two places is not read twice or billed twice.

What happens with a broken sitemap?

It becomes one free error row and the run continues with the next sitemap. A 404 is not retried, 408, 429 and 5xx responses are retried twice with backoff, and a connection stalled for 30 seconds without data is dropped. One bad file does not lose the rest of the run.

Can it handle very large sitemaps?

Yes. Sitemaps are streamed through a tokenizer, so a 50,000 URL file never has to fit in memory, and rows are delivered in batches while the file is still downloading. The recorded runs peaked around 100 MB of memory for 1,000 URLs and 125 MB for 20,000.

Need a data source that is not here?

We build and run scrapers for sites that have no public API. Tell us the site, the fields and how often you need them.