How to make your site crawler friendly?

Crawlers, also known as spiders, are used by the search engines to discover new web pages. When the spiders discover the new web pages it is indexed and that page will be available for the search results.

Minimise Flash and JavaScript Use

OK, I want to make one think clear here. Using JavaScript or Flash does not prevent your site to get indexed. But, if you use JavaScript or Flash for a link, then that link can not be able to followed by the search engine spiders. Or, if you create the entire site from Flash, then that site will not be indexed as the spiders cannot read Flash. So if you are planning to build a site with Flash, also build an HTML version of that site.

Don’t hide pages behind Forms and Search

Filling forms or conducting searches are another process search engine spiders cannot perform. Although, they play great role for you to make money, search engine robots are not as clever as humans. If you have page that is accessed by either filling a form or selecting from a drop down menu, that page cannot be crawled, therefore cannot be indexed by the search engines. Also, if your pages are generated depending on the search queries onsite, then spider will not be able to access your page.

Always use HTML for links

Links that are created by using JavaScript can easily be replaced by HTML links. This way, spiders will have no problem to follow the link to a page.

Make sure your Robot.txt allows spiders

You can tell search engines robots not to crawl your site by including the following META tag: <META NAME=”ROBOTS” CONTENT=”NOINDEX”>

You need to make sure it is like CONTENT=INDEX, if you want the search engines to crawl.

Avoid Links in Frames

Sometimes people choose to use frames for navigation links. Although these links are accessible by the spiders, it is complicated to allow spiders to crawl the frames. So, if you are a novice developer, then you should avoid them.

If you follow these guidelines, you website should not have any problem to get crawled by the search engine spiders. Please feel free to comment, if you want add something that would help the sites get crawled by the search engine spiders.

Technorati Tags: indexing problems, search engine spider problesm, search enging crawler problem

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.