Robots.txt

A text file at the root of a website that tells search engine and AI crawlers which pages they may access.