# Sitemap URL Extractor

Sitemap URL Extractor finds a site's sitemaps through robots.txt and the usual sitemap paths, follows nested sitemap indexes up to five levels, unpacks gzip and reads RSS, Atom and plain-text sitemaps, then returns one JSON row per URL with the source sitemap, lastmod, changefreq, priority, image URLs and hreflang alternates. It costs $0.0003 per URL ($0.30 per 1,000), with an optional status check at $0.0005 per URL.

## Key facts

- Price: from $0.0003 per url extracted, $0.3 per 1,000, down to $0.0002 on higher Apify plans
- Pricing model: Pay per event
- Output: JSON, CSV, Excel, XML, RSS via Apify datasets
- Access: Apify Console, REST API, Python and JS clients, MCP tool sitemap_urls
- Version: Updated Nov 2026, build 0.1.5
- Actor: subimpact/sitemap-url-extractor

## Who it is for

- SEO teams auditing which URLs a site actually publishes
- Crawler builders who need a seed list before scraping
- Content migration teams comparing old and new URL sets
- Site owners finding 404s and redirects at scale

## Use cases

### Get a complete crawl seed list

Point startUrls at a domain and the run reads robots.txt Sitemap lines first, then probes the common sitemap paths, following indexes up to five levels deep. The dataset becomes a deduplicated URL list you can feed to any crawler, with maxUrls as a hard cap so a huge site does not turn into an unbounded job.

### Keep only the section you audit

Set includePatterns to a regex such as /blog/ and excludePatterns to something like \.(jpg|jpeg|png|webp|pdf)$ to drop media. Filters are applied before delivery, so you pay only for the URLs that reach the dataset. That is how a content team extracts just the blog posts from a sitemap index holding the whole site.

### Find stale or broken pages

Turn on checkStatus and each delivered URL gets an HTTP status code via a HEAD request with a GET fallback. Sorting the rows by statusCode surfaces 404s and redirects in the same run that extracted them. The status check is billed separately per URL, so you can run extraction alone first and add checks on a filtered subset.

### See what changed since a date

lastmodAfter takes an ISO date and keeps only sitemap entries whose lastmod is on or after it. Entries with no lastmod are dropped while the filter is set, so the result is a genuine changed-since list. That is the input for a reindex request or for checking whether a CMS migration actually updated timestamps.

### Audit a multilingual site

Each row carries the xhtml:link hreflang alternates that multilingual sites publish, along with image counts and the image URLs themselves. That shows whether every language version is listed and correctly paired, which is otherwise assembled by hand from sitemap XML. Rows also record the sitemap they came from, so a missing locale points at the file that should have carried it.

## Price events

- URL extracted (`url-extracted`): $0.0003, lowest tier $0.0002. One URL found in a sitemap and delivered as a dataset row. Platform usage included.
- URL status checked (`url-status-checked`): $0.0005, lowest tier $0.00035. One HTTP status check (HEAD, GET fallback) for a delivered URL when checkStatus is on. Platform usage included.

Prices from the Apify Store. Apify shows the final price before each run.

## How it compares with other Apify actors

Price per 1,000 urls at Apify Store list prices, checked 20 Nov 2026.

| Actor | Price per 1,000 urls (Free plan) | Higher plans (Gold) | Start fee per run | Platform usage | Users | Rating | Last build |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Sitemap URL Extractor — every URL in a site's sitemaps (this API) | $0.3 | $0.2 | None | Included | 1 | No reviews yet | 18 Nov 2026 |
| [Sitemap Extractor](https://apify.com/apify/sitemap-extractor) | $0.5 | $0.1 | None | Included | 473 | 3.2 (8 reviews) | 27 Oct 2026 |
| [Sitemap URL Extractor](https://apify.com/onescales/sitemap-url-extractor) | $30.00 | $1.00 | $0.00005 | Included | 668 | 5.0 (3 reviews) | 8 Nov 2026 |
| [Website URL Extractor - Get All Site URLs](https://apify.com/lofomachines/urls-extractor) | $0.35 | $0.2 | $0.05 | Included | 244 | No reviews yet | 15 Jun 2026 |
| [Sitemap to URL Crawler — Extract Sitemap.xml URLs](https://apify.com/logiover/sitemap-to-url-crawler) | $1.50 | $0.5 | None | Included | 64 | No reviews yet | 14 Nov 2026 |

### Where Sitemap Extractor is stronger

- Larger user base and maintained by Apify itself
- Status checking plus crawl depth and retry controls
- Compact output of url, status and lastmod

### Where this API is stronger

- Nested sitemap indexes, gzip files and RSS or Atom feeds found from robots.txt
- Priority, changefreq, image counts and hreflang alternates per URL
- Include and exclude patterns plus a lastmod cutoff applied before extraction

### Where Sitemap URL Extractor is stronger

- Larger user base on the Store
- Very short input: one or more sitemap URLs and nothing else

### Where this API is stronger

- Sitemaps discovered from robots.txt rather than needing exact sitemap URLs
- Optional HTTP status check per URL as a separate event
- Pattern filters, lastmod cutoff and a maximum URL count

### Where Website URL Extractor - Get All Site URLs is stronger

- Crawls pages as well as parsing sitemaps
- Keyword filter and result cap for quick audits
- No code needed and a simple input form

### Where this API is stronger

- Nested sitemap indexes, gzip and RSS or Atom feed handling
- Image counts and hreflang alternates on each URL
- No start fee before the first URL

### Where Sitemap to URL Crawler — Extract Sitemap.xml URLs is stronger

- Accepts a domain or a sitemap URL as the starting point
- Very small input set for a fast export
- Built for RAG and SEO export workflows

### Where this API is stronger

- Filters, lastmod cutoff and status checks in the same actor
- Richer rows with priority, changefreq, images and alternates
- Handles gzip sitemaps and RSS or Atom feeds

Sitemap extraction is a crowded, simple lane and the actors differ in how much they discover and how much they return. The Apify listing is the best known with status checks, and the others keep setup minimal. Ours goes further on discovery and detail: robots.txt lookup, nested indexes, gzip and feeds, with images, alternates and filtering per URL.

Prices are Apify Store list prices, checked 20 Nov 2026. Platform usage 'Paid separately' means Apify bills compute on top of the event price. Check each listing before you buy.

## Input

- `startUrls` (array, optional): Website roots (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). For a site root the actor reads robots.txt for Sitemap: lines and then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitem...
- `includePatterns` (array, optional): Keep only URLs matching at least one of these regular expressions. Example: /blog/ to keep blog URLs only. Empty = keep everything.
- `excludePatterns` (array, optional): Drop URLs matching any of these regular expressions. Example: \.(jpg|png|pdf)$ or /tag/. Excludes are applied after includes.
- `lastmodAfter` (string, optional): ISO date (2026-01-31 or 2026-01-31T12:00:00Z). Keeps only sitemap entries whose  is on or after this date. Entries without a lastmod are dropped when this is set.
- `maxUrls` (integer, optional): Hard cap on delivered URL rows (1-1,000,000). The run stops as soon as it is reached, so a default run stays cheap. Charged at $0.0003 per delivered URL.
- `checkStatus` (boolean, optional): Send a HEAD request (GET fallback) to each delivered URL and fill in statusCode, so you can spot 404s and redirects. Billed separately at $0.50 per 1,000 checked URLs.
- `maxSitemaps` (integer, optional): How many sitemap files (index children included) one run may read (1-10,000). Default 200 is plenty for almost every site.
- `requestTimeoutSecs` (integer, optional): How long to wait for a sitemap or robots.txt response (5-300 seconds).
- `proxyConfiguration` (object, optional): Optional. Sitemaps are public, so no proxy is used by default. Add an Apify proxy for sites that block datacenter IPs.

### Example input

```json
{
  "startUrls": [
    "https://www.smashingmagazine.com"
  ]
}
```

## Output fields

- `url` (string/null): Absolute URL found in a sitemap, fragment removed. null on error rows.
- `sitemapUrl` (string/null): The sitemap file this URL came from (the final URL after redirects).
- `lastmod` (string/null): Sitemap <lastmod> value exactly as published (W3C datetime, date-only, or RSS pubDate / Atom updated).
- `changefreq` (string/null): Sitemap <changefreq> value as published (always, hourly, daily, weekly, monthly, yearly, never).
- `priority` (number/null): Sitemap <priority> value as a number between 0 and 1.
- `images` (integer/null): How many <image:image> entries the sitemap lists for this URL (0 when the sitemap has no image data).
- `imageUrls` (array/null): The <image:loc> values listed for this URL, in sitemap order. null when the sitemap lists none.
- `alternates` (array/null): xhtml:link rel="alternate" entries for this URL: the hreflang code and the URL it points at. null when the sitemap has no language variants.
- `statusCode` (integer/null): HTTP status code returned for this URL when checkStatus is on, without following redirects (301/302 means the URL redirects). null when checkStatus is off or the request failed.
- `scrapedAt` (string): ISO 8601 timestamp of when this row was delivered.
- `status` (string): ok for a URL found in a sitemap, error for a sitemap (or discovery step) that could not be read. Error rows are free.
- `error` (string/null): Why a sitemap could not be read (HTTP status, timeout, not a sitemap, empty). null on URL rows.

## Sample output

Trimmed from one of our own test runs. Personal data is redacted.

```json
[
  {
    "url": "https://apify.com/",
    "sitemapUrl": "https://apify.com/sitemap/pages.xml",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "images": 0,
    "imageUrls": null,
    "alternates": null,
    "statusCode": null,
    "scrapedAt": "2026-11-20T02:53:30.759Z",
    "status": "ok",
    "error": null
  },
  {
    "url": "https://apify.com/about",
    "sitemapUrl": "https://apify.com/sitemap/pages.xml",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "images": 0,
    "imageUrls": null,
    "alternates": null,
    "statusCode": null,
    "scrapedAt": "2026-11-20T02:53:30.759Z",
    "status": "ok",
    "error": null
  }
]
```

## How to run it

1. Put a site root or a direct sitemap URL in startUrls, for example ["https://www.smashingmagazine.com"] or ["https://example.com/sitemap.xml"].
2. Optional: set includePatterns and excludePatterns to keep only the sections you want, for example /blog/ and \.(jpg|png|pdf)$.
3. Set maxUrls, 1,000 by default, as the delivered row cap, and maxSitemaps if the site has a large index.
4. Turn on checkStatus only if you also want an HTTP status code per URL, since that is billed per check.
5. Start the run in the Apify Console; rows stream in batches of up to 500 while the sitemap is still downloading.
6. Export as JSON, CSV or Excel, and read RUN_SUMMARY for sitemaps read, URLs seen, duplicates and filter counts.

### cURL

```bash
curl -X POST "https://api.apify.com/v2/acts/subimpact~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://www.smashingmagazine.com"]}'
```

### Python

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("subimpact/sitemap-url-extractor").call(run_input={
      "startUrls": [
        "https://www.smashingmagazine.com"
      ]
    })

for item in client.dataset(run["default_dataset_id"]).iterate_items():
    print(item)
```

### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('subimpact/sitemap-url-extractor').call({
  "startUrls": [
    "https://www.smashingmagazine.com"
  ]
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

console.log(items);
```

### MCP

```bash
claude mcp add subimpact-data -e APIFY_TOKEN=<YOUR_APIFY_TOKEN> -- npx -y @subimpact/data-mcp

# Then ask your agent to use the sitemap_urls tool.
```

### n8n

Use the Apify node (Run an Actor).
Actor ID: subimpact/sitemap-url-extractor
Input JSON: paste the example input below.

## Limits

- A single sitemap is capped at 100 MB decompressed, and nested indexes at five levels.
- Entries without a lastmod are dropped whenever lastmodAfter is set.
- statusCode checks do not follow redirects, and are billed per URL whether the code is 200 or 404.
- The Actor reads sitemaps only; it does not crawl pages or discover URLs by following links.

## FAQ

### What does a URL cost?

$0.0003 per delivered URL, which is $0.30 per 1,000 URLs, and $0.0005 per URL when checkStatus adds a status code. There is no start fee. Rows are pushed before they are charged, so URLs removed by your filters, duplicates and error rows cost nothing.

### Do I need to know the sitemap URL?

No. Give it the site root and it reads every Sitemap line in robots.txt, then probes /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml in order. A direct sitemap URL is also accepted and read as-is.

### Does it unpack gzip and read feeds?

Yes. .xml.gz files are gunzipped, and RSS and Atom documents listed as sitemaps are parsed as URL sources. A body that is not XML is read as a plain-text sitemap with one URL per line, and an HTML error page is reported as not a sitemap rather than as zero URLs.

### What is in each row?

The URL, the sitemap it came from, lastmod, changefreq, priority, the number of image entries with their URLs, the hreflang alternates, and statusCode when checkStatus is on. URLs are normalised, relative locations resolved and fragments removed before delivery.

### How deep does it follow sitemap indexes?

Up to five levels, which is Google's own limit, and up to maxSitemaps files, 200 by default. Each sitemap URL is fetched once per run, so a file linked from two places is not read twice or billed twice.

### What happens with a broken sitemap?

It becomes one free error row and the run continues with the next sitemap. A 404 is not retried, 408, 429 and 5xx responses are retried twice with backoff, and a connection stalled for 30 seconds without data is dropped. One bad file does not lose the rest of the run.

### Can it handle very large sitemaps?

Yes. Sitemaps are streamed through a tokenizer, so a 50,000 URL file never has to fit in memory, and rows are delivered in batches while the file is still downloading. The recorded runs peaked around 100 MB of memory for 1,000 URLs and 125 MB for 20,000.

## Links

- Page: https://subimpact.net/apis/sitemap-url-extractor/
- Category: https://subimpact.net/apis/category/search-seo/
- Run on Apify: https://apify.com/subimpact/sitemap-url-extractor
- All data APIs: https://subimpact.net/apis/
- Catalogue JSON: https://subimpact.net/apis/catalogue.json
