A robots.txt file is so unassuming that you could easily miss it. Yet…
Since I'm nice, I'll give you the recipe to get a robot for free. As you may have guessed, this is actually a file intended for the robots that crawl your website at regular intervals, like Google's. This tiny, unassuming file can make all the difference because it tells search engines the best route to take when touring the site. Let's go…
Recipe No. 13 Invest in an SEO kitchen robot, it's not a luxury!
This file is very simple. It must be placed at your site's root and be named robots.txt; but it's easy to add because we'll do it using a very popular plugin: Yoast
We'll configure this plugin in a later recipe, but as it stands it will be installed and ready to use.
- 1 Yoast SEO plugin
- Install the Yoast SEO plugin
- Activate it, then go to the new SEO menu that appeared on the left and at the top of your admin dashboard.
- Click on General Settings, then on the Features tab
- In the active list, set the XML Sitemap feature to Enabled
There, it's done. You're surprised, right? And what's it for? As I said in the introduction, it's like a road sign for Google. It shows the available routes to visit your site and the no-go areas. It may seem trivial, but it's essential. You're welcome!
