8 tips that'll make your site easier to spider

Article by Scott Morgan, neurocyber search engine optimisation

An area that's often overlooked in regards to obtaining search engine listings, is the matter of just how easy you've made it for the search engine spiders to crawl your web site. For the most part you need a site that allows search engine spiders to enter by any page and then find any other page on your site, if not vast parts of your site will remain unindexed and you'll loose the potential of getting your products and services in front of thousands of searchers. Below you'll find tips to help ensure that your web site and its pages do not remain invisible to the all important search engine spiders

1. Use a robots.txt file - This is a text file that resides in the root of your server i.e. www.neurocyber.co.uk/robots.txt. When spiders or robots come to your site they look for this file and if it is present they read it and obey the instructions within it (at least the reputable ones do). These instructions state which directories and files spiders are permitted to crawl as well as which spiders are permitted to enter your site. You don't need to state all the directories, files and spiders that are allowed, but rather just state what is not allowed, for instance you may not want spiders to crawl your cgi bin or any other sensitive directories. If you want more information on implementing a robots.txt file then read - How to make a robots.txt file

2. Use the minimum amount of meta-tags in the head - Apart from the title tag it's best to just use the meta-keyword and meta-description tag in the head of your HTML, most of the other meta-tags are just perceived as noise by search engine spiders

3. Make sure your navigation isn't stopping spiders in their tracks - Some search engine spiders may have a problem with your site if you're using Java, JavaScript or Flash navigation. That's because these spiders have trouble extracting the urls from these technologies. You should consider supplementing any script based navigation with text links, that way you can ensure that your whole site is reachable by the search engines and that they aren't being stopped in their tracks. Including text links at the bottom of pages to all the other pages on your site is a good idea

4. Place all lengthy JavaScript and CSS code in external files - This tip has many benefits. First off placing JavaScript and CSS code into external files not only makes your pages less likely to foul up search engine spiders, but also makes your pages load faster, as a browser only has to load one instance of the JavaScript/CSS file and not load the entire code every time it loads a page. As if that wasn't reason enough to use external script files you can also save time by only having to update one file when you make any changes to your code, instead of potentially hundreds of page files

5. Use a site map - Doing so ensures that spiders can explore you whole site easily. A site map should have text links to every page on your site and should itself be linked from your sites index page. Doing this ensures a clean path through your site, from index page right through to every page on your site. People also find site maps useful as it gives them an alternative way to navigate your site

6. Validate your html - It may appear that your site works perfectly well. However, what you see in a browser such as Internet Explorer can be misleading. The reason being that browsers are inherently forgiving, trying to display any page a user may be requesting. Search engine spiders on the other hand adhere to W3C standards and are therefore stricter in how they interpret a HTML page. It is well worth running your site through a Validator to see if there are any show stopping problems you need to fix. A good place to start is at the W3C Markup Validation Service

7. Eliminate broken links - Your web site is judged amongst other things on how broken the link structure is. A site with lots of broken links can seem like a badly maintained or updated site and may well be given a lower priority crawl status. Run link checking software regularly to see if any of the links on your site need fixing

8. Be wary of dynamic URL's - Although search engines are getting better at spidering dynamic url's, they can still encounter problems. A dynamic url with a session id or a complex and long string can especially cause problems for search engine spiders. A way round this is to use Apache mod_rewrite if you're on a UNIX server or ISAPI Rewrite if you're on a windows server to show search engine friendly url's while redirecting to the dynamic url

8. Get plenty of links - This tip won't help robots spider your site any easier but it will make it easier for them to find you. The more inbound links you have, the more likely search engine spiders are to find you. Find out more information about link building strategies?

Taking the time to ensure your whole site is spiderable will pay off in the long run. As long as attention has been made to other on page optimisation factors people will now be able to find your site under previously unindexed keyword phrases, this subsequently should increase your visitor traffic and boost your sales!