What does Robots.txt mean?

Robots.txt is a plain text file placed in the root directory of your website that communicates with search engine bots. It contains rules that specify which parts of your site should be crawled or ignored. For example, you might block bots from accessing duplicate pages, admin panels, or development folders.

It’s a key part of technical SEO, ensuring that search engines focus their attention (and your crawl budget) on the pages that matter most.

Example

“We updated our robots.txt file to block crawlers from accessing our staging site and to prioritize indexing key landing pages.”

What are ways to use Robots.txt in your business?

Use robots.txt to prevent search engines from crawling low-value or private pages that shouldn’t appear in search results. Be cautious - blocking a page in robots.txt means search engines won’t crawl it, but it may still appear in results if linked elsewhere unless a noindex tag is also used.

Test your robots.txt file using Google Search Console’s robots.txt tester to ensure your rules are working correctly.

Pro Tip

Never block pages in robots.txt if you also want to use a noindex meta tag - bots can’t read the tag if the page is blocked from crawling in the first place.