Fetch and Parse Process

When calling sitemap_tree_for_homepage(), USP will try several methods to find sitemaps and recurse through sub-sitemaps.

Broadly the process is as follows:

  1. Attempt to fetch https://example.org/robots.txt and parse for Sitemap: statements. We consider robots.txt to be an index-type sitemap (as it lists other sitemaps)

  2. Fetch and parse each discovered sitemap URL. If a sitemap is an index-type sitemap, recurse into it.

  3. Try to fetch known sitemap locations like /sitemap.xml and /sitemap_index.xml, excluding those already declared in robots.txt.

  4. Create a top-level dummy sitemap to act as the parent of robots.txt and discovered sitemaps.

See also

Reference of formats, parsing and representation classes

All fetching is done through the SitemapFetcher class, which is responsible for fetching and choosing the appropriate parser for the content.

The fetcher then attempts to parse using the process shown in this flowchart:

Show Parse Flowchart
%3 fetch Fetch with SitemapFetcher usp.fetch_parse.SitemapFetcher httpsucc Did request succeed? fetch->httpsucc returninvalid Return InvalidSitemap usp.objects.sitemap.InvalidSitemap httpsucc->returninvalid No seemsxml Starts with '<' char? httpsucc->seemsxml Yes userobots IndexRobotsTxtSitemapParser usp.fetch_parse.IndexRobotsTxtSitemapParser isrobots URL ends with robots.txt? seemsxml->isrobots No xmlparse XMLSitemapParser usp.fetch_parse.XMLSitemapParser seemsxml->xmlparse Yes isrobots->userobots Yes useplaintext PlainTextSitemapParser usp.fetch_parse.PlainTextSitemapParser isrobots->useplaintext No isurlset Has <urlset>? xmlparse->isurlset concretepages PagesXMLSitemapParser usp.fetch_parse.PagesXMLSitemapParser isurlset->concretepages Yes issitemapindex Has <sitemapindex>? isurlset->issitemapindex No concreteindex IndexXMLSitemapParser usp.fetch_parse.IndexXMLSitemapParser issitemapindex->concreteindex Yes isrss Has <rss>? issitemapindex->isrss No concreterss PagesRSSSitemapParser usp.fetch_parse.PagesRSSSitemapParser isrss->concreterss Yes isatom Has <feed>? isrss->isatom No concreteatom PagesAtomSitemapParser usp.fetch_parse.PagesAtomSitemapParser isatom->concreteatom Yes error Raise SitemapXMLParsingException usp.exceptions.SitemapXMLParsingException isatom->error No

Non-XML documents are parsed directly with their respective parser. For XML documents, the XMLSitemapParser parses the document to determine the type of the XML document and select the appropriate parser (the concrete parser) to actually extract information.

XML documents are detected with a heuristic (the document, when leading whitespace is trimmed, starts with <) to avoid issues with incorrect content types.

Index-type parsers instantiate the appropriate class from usp.objects.sitemap and another SitemapFetcher to fetch each of their children. This allows a sitemap of one type (e.g. robots.txt) to contain sitemaps of another type (e.g. XML). Duplicate declarations of sub-sitemaps within the same index-type sitemap are ignored, but otherwise order is preserved.

Page-type parsers instantiate the appropriate class from usp.objects.sitemap and instantiate instances of their internal page class (e.g. PagesXMLSitemapParser.Page). These are not converted to the public class SitemapPage until the end of the fetch process. The order sub-sitemaps or pages are declare in is preserved.

Tree Construction

See also

Sitemap Tree

Each parser instance returns an object inheriting from AbstractSitemap after the parse process (including any child fetch-and-parses), constructing the tree from the bottom up. The top IndexWebsiteSitemap is then created to act as the parent of robots.txt and all well-known-path discovered sitemaps.

Deduplication

During the parse process, some de-duplication is performed within each individual sitemap. In an index sitemap, only the first declaration of a sub-sitemap is fetched. In a page sitemap, only the first declaration of a page is included.

However, this means that if a sub-sitemap is declared in multiple index sitemaps, or a page is declared in multiple page sitemaps, it will be included multiple times.