Despite evidence to the contrary from major search engines, SEO experts will universally demand that your website has a robots.txt and a sitemap.xml file. With a multi-site instance of Sitecore, this becomes a little tricky, since all those sites share the same file system. We can’t have one file to rule them all.
Constellation.Foundation.SitemapXml is a robust solution that was designed from the ground-up to support multi-site Sitecore without messing with the file system. robots.txt and sitemap.xml are handled as discrete dynamic calls which allows us to bring some Sitecore context into play and serve hostname-relevant documents back to the requesting agent.
This is a Foundation-level component, and although it does work out of the box, it was designed to allow developers to customize logic around which pages get included in the Sitemap, what information is spit out, and it lets you do it on a site-by-site basis.
There’s also robust support for site-by-site robots.txt excludes.
In a hurry and don’t want to have to code against the API? You can also get a completely self-contained sitemap.xml kit that includes sitemap inclusion flags and a host of other SEO meta-information for your pages. All you have to do is add our Templates as bases to your Page Template.