A site map informs search engines which URLs on your site you want them to index. At a minimum, a site map consists of a list of the URLs on your site. It can also include when each URL was last updated, how often the URL changes, and how important it is in relation to other URLs in the site. This enables search engines to crawl the site more intelligently.
On some sites, not all pages are available through the human-browseable interface, that is, not all pages can be accessed by clicking on a link. If a human can't click on the link, then neither can the search engine crawler. On these sites, a site map is especially useful. The site map ensures that all pages will be crawled.
Using Magento, you can generate a site map containing all accessible URLs on your store's site. Then, you can instruct the search engines to index those URLs. Google, MSN, Yahoo, and Ask use the same protocol now.
Note
A site map increases the chance that all your pages will be included in search indexes. It does not influence the way that pages are ranked in search results.
In this section, we will cover generating a site map and pointing search engines towards the site map.
Before you can complete this task, you must have FTP access to your Magento site, that is, you must be able to upload, delete, and edit files in your Magento site.
Note
Although it's called a "Google Sitemap", the major search engines use the same protocol. So this site map will work for all of them.
3. Click on the Add Sitemap button. The New Sitemap page displays.
4. In the Filename field, enter a name for the site map.
5. In the Path field, enter the directory on your Magento server where you want the site map. To place it in the root (home) directory of your site, enter a forward slash like this: /
6. Click on the Save & Generate button, and the new site map is generated.
7. You can see a URL in the Link for Google column. Either copy and paste this URL into a document so that it can be easily retrieved, or minimize this window without closing it. Later, you will want to copy and paste this link.
8. Launch your FTP client, and navigate to the root directory of your Magento site:
In our example, you can see the two fi les that we will deal with are in the root directory: robots.txt and sitemap.xml.
9. Open the file
robots.txtfor editing. Exactly how you do this depends upon which TP client you are using. If you are using WinSCP, as shown here, then you can just double-click on the file, and it will open for editing:
10. The file might already have some text in it. That's is okay. Add this line to the file:
Sitemap: http://www.yourdomain.com/sitemap.xmlwhere:
www.yourdomain.com is your domain. In our example, it's www.brew-me-a-cup.com.
/sitemap.xmlis the path and filename for the site map. In our example, the site map is in the root directory. If we put Magento in a subdirectory, it might look like this: www.brew-me-a-cup.com/magento/sitemap.xml.11. Save the file.
When a search engine robot or crawler visits your site, the first file it looks for is robots.txt. This file tells the crawler which pages to visit and which it should not visit. Robots are free to abide by or ignore robots.txt. All of the major search engines will abide by robots.txt. This ensures that they visit the pages you want them to index. It does not ensure that they will actually index them or rank them higher.