Infrastructure

Support for sitemaps landed last fall in Metacat: https://github.com/NCEAS/metacat/pull/1283. Sitemaps are good for users but especially for search engines and DataONE's Search Catalog could benefit from having sitemaps enabled. A sitemap could help crawlers discover all of the datasets in DataONE. The CNs already run Metacat and should use Metacat's sitemaps ability to generate sitemaps for all content.

To enable sitemaps on the CNs, a few things seem to be needed. I'm not very familiar with how the CNs get built so I may be wrong or be missing things:

• Sitemaps rely on two properties in Metacat's metacat.properties file, which should have values: sitemap.location.base=https://search.dataone.org/ and sitemap.entry.base=https://search.dataone.org/view
• The Apache config the CNs are built with need to serve the sitemap_index.xml and individual sitemaps from the Tomcat webapps dir. Metacat generates sitemaps in the sitemaps subfolder (e.g., /usr/lib/tomcat8/webapps/metacat/sitemaps). A Directory directive should work so long as filesystem permissions are set up for Apache to see the files.
• We need a robots.txt that points at the sitemap index file at search.dataone.org which provides the entrypoint to sitemap_index.xml
• Metacat generates sitemaps with a recurring job mechanism that's internal to Metacat. AFAIK this job isn't turned on when Tomcat loads Metacat and a request has to get sent to the admin API which turns this job on as a side-effect. We might want to change this to reduce maintenance burden or chance of having stale sitemaps

Dave nominated Jing for this work and has targeted this for the next CCI release. I'm not sure which that is so please select whichever one is appropriate.

Note this relates to https://redmine.dataone.org/issues/8693 which we've delayed because Google's crawler infrastructure has changed and DataONE is now visible by Google. Google staff have indicated we only need to send them a robots.txt that points to our sitemaps for them to begin crawling.