Robots.txt: Mastering Crawler Control
•
6 min read
What is a Robots.txt File?
A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests.
Caution: Misconfiguring robots.txt can accidentally de-index your entire website from Google!