To facilitate the efficient functioning of directories search engines have been armed with content filters. This removes or filters duplicate content from pages it's indexing. And the most hurtful penalty is lower rankings.
Unfortunately, these filters not only catch rogues but web pages that are genuine too. What webmasters need to do is understand how filters function and know what action is to be taken to avoid being filtered out.
When a search engine sends out spiders the filters leave out or sieve:
Websites that feature identical content. And when within a site the webmaster includes many copies or versions of pages to cheat the search engines. Filters are also extremely sensitive to "doorway" pages.
Content masked by different packaging. Known as "scraped content" this duplication of pages with little or no relevant changes falls prey to filters.
Product descriptions featured by e-commerce sites. Most e-commerce sites publish alongside a product the manufacturer's description of the product and this content then appears on zillions of e-commerce sites falling victim to filters.
Articles distributed widely over the net. While some engines are programmed to find the origin of the article there are others who may not be able to source the origins.
Pages that are not duplicates but contain the same core material written by different people.
To get the better of filters you need to:
Use a tool like the Similar Page Checker http://www.webconfs.com/similar-page-checker.php to ensure that the pages in your site are not mirroring content from elsewhere. In case there are other URLS with similar or identical content the tool will reveal them to you and you will be able to make changes in your pages.
Be vigilant and know who has "helped" themselves to your content. By using www.copyscape.com you can determine which websites have stolen or copied your work.
Even if you do use distributed content you can add a commentary or make changes to the page focusing on its relevance to your site. By making any content your own you are making it unique and different and this will ensure that the pages are not filtered by search engines.
Even if you are running an e-commerce site you must include product descriptions that are distinctively yours and not run of the mill.
Lean as much as you can about duplicate content and its dangers. Read the issues that were discussed at the SES 2006 New York Session and other forums. Remember most search engines, Google, Yahoo, or Open Directory Project do not want to be flooded by duplicate content and web pages.
Jake Baillie, President of TrueLocal listed the duplicate content mistakes to be: circular navigation; printer friendly pages; inconsistent linking; product only pages; transparent serving domains; and bad cloaking.
It is important for sites to get high ranking through fair and not foul means
