Among the specialized bots Google uses include: The web spider Googlebot, the Adsense spider MediaBot, the image spider ImageBot, the AdWords spider AdsBot, the RSS feed spider FeedFetcher-Google, and Googlebot-Mobile spider for mobile devices. MSN & Yahoo, the other two of the 'big three' have their own proprietary versions of spiders.

Why is it important for an Internet Marketer to know how spiders crawl your website?

A search engine crawler is your best visitor. Giving a crawler easy and uninhibited movement in your website is necessary for good search engine rankings.

Your website must be spider (search engine) friendly if you want any traffic from the search engines. A search engine spider does not read your website the way we humans do. The spider reads web-page source code (HTML) that renders your page, therefore ‘bad code' can be an impediment to the spider, sometimes causing it to give up crawling your website.

Spiders love content (text) and do not read JavaScript at all, therefore a website that is packed with images with no ALT tags to assist the spiders, and heaps of JavaScript may not be indexed successfully. So, when designing your website you must incorporate structural website design principals that elicit search engine friendliness.

An astute marketer should also desire to see how search engines see his or her site. This may be accomplished by a Lynx Viewer which is a text-mode web browser. Additionally, a Lynx Viewer can help you determine if your web pages are accessible to the vision impaired, an assemblage of visitors that should not be ignored ---yes, there are millions of visually impaired people surfing the Internet regularly.

A quick search in Google for "Lynx Viewer" will yield numerous sources from which you can download this important tool for your use.

Even though you must design your website with your visitors in mind first, it is crucial that you accord the search engines top level priority too, since the vast majority of these visitors will arrive via search engines. Practice good SEO (Search Engine Optimization) but not at the expense of your visitors' experience -- it is a balancing act that must be accomplished with prudence.

Also of significant importance is the fact that web browser standards are not yet fully harmonized. A web page that looks great in Internet Explorer might look atrocious in a Mozilla based browser like Firefox or Netscape. Additionally, with the proliferation of hand held devices for browsing the Internet, compliance with W3C standards is becoming more and more critical. A marketer must therefore be conversant with the intricacies of cross-browser design -- designing for one browser (IE) is no longer ideal, as the Google backed FireFox is eating up Microsoft's browser turf at an alarming rate.

Anybody can "whip up" a web page in FrontPage without sufficient knowledge of HTML, but may not be able detect and correct the messy code that FrontPage generates underneath the page, some of which is proprietary to Microsoft. Consequently a website that looks superb in Microsoft Internet Explorer may look and load dreadfully in Opera and/or some other browser, denying you visitor traffic.

Never use a Word Processor to design your website. Word Processing software generate tremendous amounts of code that is not search engine friendly. If you cannot hand-code using a text editor then it is necessary that you use authentic and industry standard web design software that incorporate the most up to date design principles. Macromedia's Dreamweaver and the latest version of Microsoft FrontPage are good candidates with Dreamweaver getting my partisan nod.

A first-rate design strategy should include the use of CSS (Cascading Style Sheets) and valid XHTML, the most current in the HTML generation of standards. Websites designed in strict W3C standards tend to be lighter, faster and cross-browser compatible. This is not to insinuate that table based design is going anywhere anytime soon, for it is my humble disputation that if strict W3C standards were to be enforced in browsers, 95% percent of websites would go out of business, furthermore the lack of inter-browser synchronization just worsens things.

According to some surveys, more than 86% of all people arrive at websites through search engines. In 2006, PC World, arguably the most authoritative and widely-read computer and business magazine, reports that Google remains the site of choice for most surfers.

"The double-digit increase in online search activity marks a significant milestone in the evolution of Internet consumer behavior," says Ken Cassar, senior director of analytics at Nielsen/NetRatings. "Online search is the primary tool most people rely on to do everyday research," he says.

So, from a marketing perspective unless your site appears in the top 30 listings of the major search engines & directories, you will hardly get any worthwhile traffic. Therefore, Search Engine Positioning is vital to your marketing success on the web!

A top ranking in the search engines can bring you highly targeted traffic. If someone visits your site after searching for a product or service that you are selling, it means that he/she is interested in what you are selling and hence is a potential customer for you. Thus, search engines send pre-qualified customers to you.

You can sell virtually anything on the Internet, but in order to succeed you must bring "targeted traffic" to your website....people who are ready to buy your products and services, the vast majority of who will arrive at your website through search engines. If your website is not designed suitably, Google and other search engines will disrespect your website. Respect brings in traffic which translates into the all important Dollars, "Kwacha," Euro or whatever you wish to call money.

Remember, search engine bots are your most important visitors, you must seduce, "open your doors" and accommodate them in order to gain any measurable success in your Internet Marketing endeavor.