Create a compliant robots.txt file with search bot presets, AI scraper blocking, and XML sitemap directives.
