Grab Your Free 17-Point WordPress Pre-Launch PDF Checklist: http://vid.io/xqRL
Check The Robots.txt File For Potential WordPress SEO Issues | WP Learning Lab
The Robots.txt is an important file for controlling access to your website. Inside this file you can block access from different user agents, IP addresses or block access to specific files and folders.
Here’s where it can be a problem.
When a website is in development the developer often restricts access to your site to just a few IPs so that they can work on it and you can view it, but regular traffic isn’t able to access it.
It makes sense, you don’t want your visitors and future customers to see a half-finished website.
On the rare occasion a website will launch without these restrictions being removed from the robots.txt. That destroys your SEO and organic traffic because your site can’t be accessed! Even though everything looks fine to you because your IP is allowed access.
Luckily it’s easy to fix. There are two…