The site map process has matured considerably in the past few years, beginning when Google introduced Sitemaps 0.84 in June 2005 to accommodate the publishing of lists of links for an entire business website. That was when this idea progressed to pushing your website URLs out to the general digital community using server based files. The server based files come in a variety of formats including HTML, XML, ROR, Text and zipped files as .gz format.

Soon, support for sitemaps began to gain acceptance by the larger search engines. In November 2006, Google, MSN and Yahoo announced joint support for Sitemap protocol. At that time, the version was called Sitemap 0.90 and the website Sitemaps-dot-org was created. In April 2007, Ask and IBM announced support for Sitemaps. At the same time Google, Yahoo, MSN announced their search robots would auto-discover sitemaps through a command in the robots-dot-txt.

With these sitemaps search engines can more intelligently index the website. Sitemaps are an easy way for a business website to inform search engines about pages on their websites. The sitemap is now generally accepted as an XML file that lists URLs for a site along with additional metadata about each URL. The metadata includes when the sitemap was last updated, how often it changes, and how important it is relative to other URLs in the website.

Although search engine spiders discover pages from links within the website, sitemaps supplement this data to allow the search engine spiders that support sitemaps to pick up all URLs. A website can learn about those URLs using the associated metadata. Using the sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for search engine spiders to do a better job of indexing your website. The process is more efficient and as a business website, you can push your content out to the search engines.

How To Create Your Sitemap

The simplest way of creating and maintaining your website is to use a sitemap generator. This is a program that you upload to your website's hosting server, which will create the sitemap in the appropriate formats that include XML, ROR, Text, HTML and .GZ zipped file. The value in using a sitemap generator of this nature is that you can setup a scheduler to automatically run the sitemap at any frequency you desire (e.g. once a week on Saturday's at 3 a.m.). Upon completion of the process, it will then 'ping' Google and Yahoo to inform them the sitemap has changed. They will also provide you a web based interface for any maintenance such as checking on broken links.

Sitemap generators can be found, as a manual process, for free that you have to remember to run - or, you can spend a nominal fee of $15 and buy the version that will automate the process for you. If you consider that you are helping to facilitate proper and efficient indexation of your website's pages by the search engines to generate revenue, the $15 investment is money well spent.

What is ROR XML Sitemap?

ROR (Resources of a Resource) is a rapidly growing independent XML format for describing any object of your website's content in a generic fashion. This generic description allows search engines to better understand the content. ROR promotes the concept of structured feeds, which is similar to the concept of structured blogging. It enables search engines to complement text search with structured information to better understand meaning of a website. In order to inform search engine spiders of this format, you will need to include the following tag in the headers of your home page:

...link rel="alternate" type="application/rss+xml" title="ROR" href="ror.xml"...

Changes to Your Robots-dot-txt File

The robots-dot-txt file is placed in the main folder of your website. It is not something that is linked to. Instead, it is a file that search engine spiders look for upon arriving to your website. This file will guide them to what they can or cannot index about your website. The robots.txt file is something that should be on every website and is certainly required when using sitemaps. Assuming that you want the search engine spiders to index all pages of your website, then the contents of your robots.txt file with the sitemap instruction would contain the following lines:

Sitemap: http://www.domain.com/sitemap.xml

User-agent: *

Disallow:

Submitting Your Website to Google

The simplest thing to do is log into your Google account. If you do not have one, create one and go to "My Account". You can then go to Webmaster tools and follow the instructions to submit your sitemap to Google. If you do not see the webmaster tools link in your "My Account" area, you can go to "About Google" from their home page, then go to "Webmaster Central". At some point Google includes this in your "My Account" area.

Google takes the "Webmaster Tools" quite a number of steps further beyond what Yahoo and certainly, what MSN provides. There are several statistics about the interaction with your sitemap, which types of searches are being done about your site, how Google perceives the visible content, and much, much more. A few days (or weeks) after you submit your sitemap, this information will start to show up.

Submitting Your Sitemap to Yahoo

The process is similar with Yahoo. Log into your Yahoo account, click on Suggest a Site (all the way at the bottom of the home page), then click on "Submit your site for free", then enter the web address for your sitemap into the "Submit Site Feed".

If you use a paid sitemap generator, you will also need a Yahoo API code so the sitemap generator can 'ping' Yahoo after it runs on its scheduled time. You can obtain a Yahoo API code by going to Yahoo's Developer website and clicking on "Get Application ID". You can then place the Yahoo API code in the appropriate place in the sitemap generator program.

Submitting Your Sitemap to Other Locations

Through the robots.txt file, the ROR tag in the headers, and submitting your site to Google and Yahoo, you will have tackled the majority of the marketing. There are other locations such as submitting the sitemap to MoreOver and Ask. You can search on sitemap submission to identify further locations if you wish to expand your reach further.

As of this writing, submitting your sitemap to MSN is not working even though MSN agreed to the protocol with Google and Yahoo. We were not able to readily locate any updates on this matter at MSN. Perhaps this is similar to MSN's problem using the "link:" command in their research results.