Secure and SEO friendly robots.txt

0
120
Secure and SEO friendly robots.txt
robots.txt

Creating a robots.txt file

A robots.txt files tells search engine crawlers like Google where they can and cannot index content for inclusion in their search results. You may think granting them full access will help your SEO efforts, but there are certain places you simply should not allow them access to, and not limiting their access is, as you may have guessed, a security threat.

The robots.txt file should be added to the root of your site, where WordPress is installed. Here is an example of a good robots.txt file:

User-agent:*
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Allow: /wp-content/uploads

LEAVE A REPLY

Please enter your comment!
Please enter your name here