Supported Formats Reference

Overview of Parsers

Format

Index

Pages

Parser

Object

Parser

Object

Website [*]

IndexWebsiteSitemap

Robots.txt

IndexRobotsTxtSitemapParser

IndexRobotsTxtSitemap

Plain text

PlainTextSitemapParser

PagesTextSitemap

XML

XMLSitemapParser

XMLSitemapParser

Sitemap

IndexXMLSitemapParser

IndexXMLSitemap

PagesXMLSitemapParser

PagesXMLSitemap

RSS 2.0

PagesRSSSitemapParser

PagesRSSSitemap

Atom 0.3/1.0

PagesAtomSitemapParser

PagesAtomSitemap

Unknown

InvalidSitemap

InvalidSitemap

Robots.txt

Example
# An example robots.txt file
# Some sitemap declarations, Google allows all of these
Sitemap: https://example.org/sitemap.xml
sitemap: https://example.org/weirdly-named-sitemap.xml
site-map: https://example.org/another-sitemap.xml
sItE-mAp: https://example.org/and-another-sitemap.xml

# And other stuff too
User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

The robots.txt parser implements the same logic to detect Sitemap keys as Google’s parser, namely it is case insensitive and supports Sitemap or Site-map.

Plain Text

Example
Yes, there are plain text sitemaps and they could just contain random text.

https://example.org/page1
https://example.org/page2

The plain text parser reads the file line by line and considers anything that appears to be a useful URL a page. Specifically, it looks for lines that appear to be URLs, can be parsed successfully by urllib.parse.urlparse(), and have the HTTP or HTTPS protocol and has a non-empty hostname. This means that non-URLs in the file will simply be ignored, which is more permissive than the either standard.

XML Sitemap

Examples
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <sitemap>
        <loc>https://example.org/sitemap1.xml</loc>
        <lastmod>2024-01-01</lastmod>
    </sitemap>
    <sitemap>
        <loc>https://example.org/sitemap2.xml</loc>
        <lastmod>2024-01-02</lastmod>
    </sitemap>
</sitemapindex>

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9/">
    <url>
        <loc>https://example.org/page1</loc>
        <lastmod>2024-01-01</lastmod>
    </url>
    <url>
        <loc>https://example.org/page2</loc>
        <lastmod>2024-01-02</lastmod>
    </url>
</urlset>

Sitemaps XML (not to be confused with other sitemap formats that happen to be in XML) is the most common kind of sitemap.

The Sitemaps XML parser supports both the Sitemap and Sitemap index formats.

Supports the following non-standard features:

  • Truncated files (perhaps because the web server timed out while serving the file) will be parsed as much as possible

  • Any unexpected tags are ignored

  • Timestamps are parsed flexibly

Note

Namespaces must be declared to parse the sitemap and any extensions correctly. Any unrecognised namespaces will be ignored.

XML Sitemap Extensions

Note

The Google Video extension is not currently supported, and only the standard part of the sitemap will be parsed.

Google News

Example
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
    xmlns:news="http://www.google.com/schemas/sitemap-news/0.9">
    <url>
        <loc>https://example.org/page1</loc>
        <lastmod>2024-01-01</lastmod>

        <news:news>
            <news:publication>
                <news:name>Example.org News</news:name>
            </news:publication>
            <news:publication_date>2024-01-01</news:publication_date>
            <news:title>News Article One</news:title>
        </news:news>
    </url>
    <url>
        <loc>https://example.org/page2</loc>
        <lastmod>2024-01-02</lastmod>

        <news:news>
            <news:publication>
                <news:name>Example.org News</news:name>
            </news:publication>
            <news:publication_date>2024-01-02</news:publication_date>
            <news:title>News Article Two</news:title>
        </news:news>
    </url>
</urlset>

The Google News extension provides additional information to describe the news story which a webpage represents, in addition to the page itself.

If the page contains Google News data, it is stored as a SitemapNewsStory object in SitemapPage.news_story.

Google Image

Example
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
    xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">

    <url>
        <loc>https://example.org/news/one</loc>

        <image:image>
            <image:loc>https://example.com/image.jpg</image:loc>
        </image:image>
        <image:image>
          <image:loc>https://example.com/photo.jpg</image:loc>
        </image:image>
    </url>

    <url>
        <loc>https://example.org/news/two</loc>

        <image:image>
            <image:loc>https://example.com/image2.jpg</image:loc>
        </image:image>
    </url>
</urlset>

The Google Image extension provides additional information to describe images on the page.

If the page contains Google Image data, it is stored as a list of SitemapImage objects in SitemapPage.images.

Additional Features

Beyond the Sitemap specification, USP also supports some non-standard features used by large sitemap consumers (e.g. Google).

Alternate Localised Pages

Example
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9/"
    xmlns:xhtml="http://www.w3.org/1999/xhtml">
    <url>
        <loc>https://example.org/en/page</loc>
        <lastmod>2024-01-01</lastmod>
        <xhtml:link
            rel="alternate"
            hreflang="fr-FR"
            href="https://example.org/fr/page"/>
    </url>
    <url>
        <loc>https://example.org/fr/page</loc>
        <lastmod>2024-01-02</lastmod>
        <xhtml:link
            rel="alternate"
            hreflang="en-GB"
            href="https://example.org/en/page"/>
    </url>
</urlset>

Alternate localised pages specified with the <link> tag will be stored as a list in SitemapPage.alternates. Language codes are not validated.

Date Time Parsing

It is relatively common for sitemaps to not correctly follow the W3C Datetime format (a subset of ISO 8601). To handle this, date times are parsed flexibly with fallbacks. This is done in two steps to allow the faster, more reliable parser to be used where possible.

First, an attempt is made with a full ISO 8601 parser:

If this is unsuccessful, dateutil.parser.parse() is tried, which is able to parse most standard forms of date, but is slower and is more likely to mis-parse.

Without trying the optimised parser first, in large sitemaps, datetime parsing would take a significant proportion of the total runtime.

RSS 2.0

Example
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
    <channel>
        <title>Example</title>
        <link>https://example.org/</link>
        <description>Example</description>
        <item>
            <title>Page 1</title>
            <link>https://example.org/page1</link>
            <pubDate>Mon, 01 Jan 2024 12:00:00 UTC</pubDate>
        </item>
        <item>
            <title>Page 2</title>
            <link>https://example.org/page2</link>
            <pubDate>Tue, 02 Jan 2024 14:00:00 UTC</pubDate>
        </item>
    </channel>
</rss>

Implementation details:

  • Per the specification, <item> elements without a <title> or <description> are invalid and ignored.

  • Although the specification states <link> is optional, we ignore an <item> if it does not contain one

  • Dates are parsed flexibly

Note

mRSS is not currently supported and will be ignored.

Date Time Parsing

It is relatively common for feeds to not correctly follow the RFC 2822 format. To handle this, date times are parsed with dateutil.parser.parse(), which is able to parse most standard forms of date. Given that feeds should be short, the performance impact of this is minimal.

Atom 0.3/1.0

Examples
<?xml version="1.0" encoding="UTF-8"?>
<feed version="0.3" xmlns="http://purl.org/atom/ns#">
    <title>Example</title>
    <link rel="alternate" type="text/html" href="https://example.org/" />
    <modified>2024-01-01</modified>
    <entry>
        <title>Page 1</title>
        <link rel="alternate" type="text/html" href="https://example.org/page1" />
        <id>https://example.org/page1</id>
        <issued>2024-01-01</issued>
    </entry>
    <entry>
        <title>Page 2</title>
        <link rel="alternate" type="text/html" href="https://example.org/page2" />
        <id>https://example.org/page2</id>
        <issued>2024-01-02</issued>
    </entry>
</feed>

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>Example</title>
    <link href="https://example.org/" />
    <updated>2024-01-01</updated>
    <entry>
        <title>Page 1</title>
        <link href="https://example.org/page1" />
        <id>https://example.org/page1</id>
        <updated>2024-01-01</updated>
    </entry>
    <entry>
        <title>Page 2</title>
        <link href="https://example.org/page2" />
        <id>https://example.org/page2</id>
        <updated>2024-01-02</updated>
    </entry>
</feed>

Implementation details:

  • The same parser is used for 0.3 and 1.0, and it does not attempt to detect the version, therefore it can accept invalid feeds which are a mixture of both

  • Dates are parsed flexibly

Date Time Parsing

Atom 0.3 follows the W3C Datetime (a subset of ISO 8601) format, and Atom 1.0 follows RFC 3339 (which is similar but not entirely equivalent to ISO 8601 [2]). In either case, dateutil.parser.parse() is used to parse the date, which is able to parse most standard forms of date. Given that feeds should be short, the performance impact of this is minimal.

Footnotes