Many webmasters believed that great content is the key to Search Engine Optimization but they have forgotten an even more important factor, how easy can the bots crawl their sites? Making a 100% crawlable site should be the top priority for all the webmasters. There is no point having a unique and fresh content but cannot be searched from the search engine index.



In order to ensure that bots can crawl a site successfully, one must make sure that all pages can be found by going from pages to pages via the inter linking mechanism. It is recommended to use text links as a form of linking between all the internal pages. Webmasters can also provide a sitemap that lists down all the pages’ links in a page to facilitate this indexing. One point to take note is that if the site has more than 100 links, it is advisable to split the sitemap into several pages with each page containing no more than 100 links. A page with more than 100 links may be classified as ‘Links Farm’ by the search engine. This is stated officially in the Google Webmaster Guidelines.

Google Sitemap is another great tool for them to index a new site. Creating a Google sitemap is encouraged because it tells the search engine what are the pages in a site and how often are the content updated. This is particularly useful when some pages are not linked to within the site.

Many sites have faced crawling problems because of the way they linked up the internal pages. Search Engine bots have difficulties indexing JavaScript and Flash navigation menu. If it is not possible to remove this kind of navigation in a site, it is advised to implement a text-link navigation system in the footer of a site. This will help the bots to index the site easily.

According to Google Webmaster Guidelines, it is advised to have a static and short HTML link destination rather than a dynamic URL. Dynamic URL especially those with tagged session identifiers will not be indexed because bots will ignore these pages. It is also encouraged to avoid using the ‘&id=’ parameter when passing variables between pages as Google does not include them into the index. If parameters are needed to pass between pages, meaningful parameter like ‘&count=’ can be used instead.

When the bots conduct a visit, they will crawl the page like what a general user will see in their browser. Therefore, the bots will never index a password-protected page. If the objective is to get the site indexed by the search engine, it is advisable to remove any password-protected pages to allow access for the bots.

A site should always be search engine friendly so that users can locate the content easily. To conclude, search engine likes to index simple and content rich pages. A simple and crawlable site with great content is the essential criteria to Search Engine Optimization.