Creating a robots.txt file
A robots.txt files tells search engine crawlers like Google where they can and cannot index content for inclusion in their search results. You may think granting them full access will help your SEO efforts, but there are certain places you simply should not allow them access to, and not limiting their access is, as you may have guessed, a security threat.
The robots.txt file should be added to the root of your site, where WordPress is installed. Here is an example of a good robots.txt file: