Read a sitemap and output a list of URLs.
sitemap2urllist is a CLI tool for parsing a sitemap and outputting a simple list of URLs, which can easily be piped into other tools (e.g., lychee).
cargo install --locked sitemap2urllist
Read a sitemap and output a list of URLs.
Usage: sitemap2urllist [OPTIONS] <URL>
Arguments:
<URL> The URL to a sitemap
Options:
-c, --cache Use request cache stored on disk at `.sitemapcache` (recommended)
--max-cache-age <MAX_CACHE_AGE> Discard all cached requests older than this duration [default: 14d]
-v, --verbose... Increase logging verbosity
-q, --quiet... Decrease logging verbosity
-h, --help Print help (see more with '--help')
-V, --version Print version
At some point, it is likely link checkers like lychee obviate the need for this tool by implementing recursive link checking.
In the meantime, it is easy to run a link check from your local machine on an entire website as defined by its sitemap by doing something like the following.
sitemap2urllist https://www.numbersstation.ai/sitemap.xml --cache | xargs lychee --cache
Note you can combine this with lychee's configuration to do things like cache or ignore certain errors, etc.
- Sitemap-to-Urllist (rust/shell/typescript): Simple sitemap.xml to urllist.txt converter (abandoned)