A robots.txt book communicates what pages or sections of your website you wish seek engines to crawl.
1. A accepted aberration is blocking seek engine spiders from a area or absolute website you wish indexed.
2. You can apprentice how to actualize a robots.txt book from Google Guidelines.
3. Google’s Webmaster Central has a apparatus to let you verify that your robots.txt book is assuming how you’d like.