Create and use robots.txt file blogger

Here are instructions for creating and using robots.txt file for websites, to the search engines index the content management of your website

robots.txt is a text file is structured, when the spider (bot, crawler) of the SE (Search engine) on the website to collect the data will robots.txt file to see the instructions in this file.

robots.txt can define each of the different bot SE can vary in each area of ​​the website or website?

Some of the SE bots: Googlebot (Google), Googlebot-Image (Google), Yandex (Russia SE), Bingbot (Bing) / Yahoo Slurp (Yahoo) ...

The common syntax of robots.txt file

User-agent: audience acceptance bot Disallow / Allow: URL you want to block / allow

*: Represent all

Example: User-agent: * (It means to accept all types of bot.)

Locks entire site
Disallow: /

Blocking a folder and everything in it
Disallow: / wp-admin /

Block 1
Disallow: / private_file.html

Removing a picture from Google Images
User-agent: Googlebot-Image Disallow: / images / sexy.jpg

Remove all images from Google Images:
User-agent: Googlebot-Image Disallow: /

Blocking any one image, for example. Gif
User-agent: Googlebot Disallow: / *. gif $

Things to avoid in the robots.txt file

- Distinguish case sensitive.

- Do not write balance, lack of white terms.

- Do not insert any character other than the command syntax.

- Each statement should be written on one line.

How to create and placement robots.txt file

- Use notepad or any other program that created the file, then rename the file robots.txt.

- Put in the root directory of the website. (Http://www.tin24h.us/robots.txt)