Robots.txt is a tiny text file with an outsized ability to ruin your week. Used right, it guides crawlers politely. Used wrong, it can quietly tell Google to ignore your entire website.
The short version
A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which areas they are allowed or not allowed to crawl.
What it is good for
Keeping crawlers out of admin or staging areas
Reducing crawl waste on low value sections
Pointing crawlers to your sitemap
The danger
One overly broad disallow rule can block your whole site from being crawled. It also does not reliably keep pages out of the index, that is what other controls are for. Handle it carefully and double check before deploying.
FAQ
Does robots.txt remove a page from Google?
Not reliably. It controls crawling, not indexing. To keep a page out of results you generally need a different method, such as a noindex directive.