Both files get lumped together as “stuff for search engines,” but sitemap and robots.txt point in opposite directions. robots.txt is control — “please don’t come here” — while the sitemap is discovery — “there are pages over here.”
What not to put in robots.txt
Blocking a page you don’t want indexed with a Disallow rule is a mistake: it only forbids crawling, and the page can remain indexed regardless. The proper signal for exclusion is a noindex meta tag on the page itself; robots.txt should be reserved for areas that never need crawling at all, such as admin routes or internal search results.
Running the sitemap
Hand the job to your static site generator’s official plugin and you get everything generated automatically, including the xhtml:link entries for multilingual pages. The fiddly hand-tuning of the old days is mostly gone.
lastmodusually reflects build time, not git history — if you need precision, that takes extra workpriorityandchangefreqare ignored by the major engines, so their absence costs nothing
Takeaway
“Control is robots, discovery is sitemap, exclusion is noindex” — keep that three-way split and there is nothing left to agonize over. Beyond it, the plugin defaults were enough.