Creating a Proper robots.txt and sitemap.xml · English

The goal is to prevent search engines from indexing certain directories in order to get rid of duplicate content. For example, the same article can be reached through both /tag/ and /category/.

Why is duplicate content bad?

Here is what Google has to say about it (translation). You can also find information about the Yandex search engine saying that its algorithms ban sites with duplicate content. I don’t know whether that’s true or not; I’m not an SEO specialist (if any experts are reading this, leave a comment), but I’ll block the duplicates just in case!

Search with Google or Yandex, and you’ll find a wealth of information on this topic.

UPDATE: About duplicate content in WordPress.

Robots.txt

robots.txt is a file that restricts robots’ access to content on an HTTP server. The file must be located in the site’s root directory (i.e., its path relative to the site name must be /robots.txt). If there are multiple subdomains, the file must be placed in the root directory of each one. This file complements the Sitemaps standard, which serves exactly the opposite purpose: making it easier for robots to access content. http://ru.wikipedia.org/wiki/Robots.txt

There is also some information on the MasterTalk forum: What robots.txt is and what it is for, as well as a website dedicated to this file: Everything about robots.txt in Russian.

Here are some more experiments by Denis Boltikov on the ideal robots.txt for WordPress: The ideal robots.txt for WordPress and The ideal robots.txt for WordPress — 2.

My Robots.txt

Here’s what my robots.txt looks like:



User-agent: *
Disallow: /wp-
Disallow: /xmlrpc.php
Disallow: /category/
Disallow: /page/
Disallow: /trackback/
Disallow: /feed
Disallow: /comments/
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=
Disallow: /?s=
Allow: /wp-content/uploads/
Sitemap: /sitemap.xml

I’ve allowed search bots to index my site only through “Tags” and “Archives”. Note that I use the /wp- pattern so that URLs such as wp-login.php, wp-register.php, wp-admin, etc., are covered by the indexing restriction (Disallow).

I’ve also allowed “Yandex Images” and “Google Images” to index my "/wp-content/uploads/", since it mostly contains images. And I’ve made “sitemap.xml” available for better indexing.

There are a number of specialized online services for checking the syntax and structure of a robots.txt file:

Sitemaps

You should also remember the sitemap.xml file.

A Sitemap is an XML file containing information for search engines (such as Google, Yahoo, Ask.com, MSN, and Yandex) about the pages of a website that should be indexed. Sitemaps can help search engines determine the location of a site’s pages, when they were last updated, how frequently they are updated, and their importance relative to other pages on the site, so that the search engine can index the site more intelligently. http://ru.wikipedia.org/wiki/Sitemaps

A sitemap is not required, and search engines do not guarantee that they will use it. It merely helps search bots with indexing, but I think it’s very useful!

There is a plugin for WordPress called Google XML Sitemaps (other useful WordPress plugins I use). This plugin generates a sitemap.xml file for you, containing information for search bots on indexing the site. You can configure it yourself and specify priority percentages for posts, pages, and so on.

Google and Yandex Webmaster Centers

To speed up indexing, it is recommended that you notify Google and Yandex about your robots.txt and sitemap.xml files. There are dedicated services for this:

Google Webmaster Tools
Yandex.Webmaster (after adding your site, you’ll be able to attach sitemap.xml)

Using HTTP pings to notify search engines of updates to sitemaps

You can tell search engines where your Sitemap file is located using an HTTP request:

Google http://google.com/webmasters/sitemaps/ping?sitemap=<sitemap_location> Yandex http://webmaster.yandex.ru/wmconsole/sitemap_list.xml?host=<sitemap_location> Yahoo! http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=SitemapWriter&url=<sitemap_location> http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=<sitemap_location> Ask.com http://submissions.ask.com/ping?sitemap=<sitemap_location> Live Search http://webmaster.live.com/ping.aspx?siteMap=<sitemap_location>

Here, <sitemap_location> must be replaced with the full URL of the Sitemap file. For example, to add the file http://www.example.com/sitemap.xml to the Google search engine, you need to send the HTTP request http://www.google.com/webmasters/sitemaps/ping?sitemap=http://www.example.com/sitemap.xml.

The information on pings was taken from http://ru.wikipedia.org/wiki/Sitemaps

Google recommends pinging it no more than once an hour; here’s the source link.