> won't try to fetch the default path until after the <head> section of the page...

		andrewxdiamond on Nov 1, 2024 \| parent \| context \| favorite \| on: Tell HN: Robots.txt pitfalls – what I learned the ... > won't try to fetch the default path until after the <head> section of the page loads for HTML content That's a really interesting optimization. How did you discover this? Reading source?