July 2004
No entry to my website
Is your website giving 'no entry' signs to the search engines?
You've done a fantastic optimisation job on your website - researched all your main key phrases, incorporated them into your page content, and are building incoming links to your website. But you're still getting no results. So what could be happening? Despite optimising your website for the search engines, there could be 'no entry' signals to the search engine spiders, preventing them from indexing it. Here are a few of some of the most common causes and how to deal with them:
1. Website is in frames
If your website is in frames, this can cause problems to search engine spiders unless you help them along their way. It is not impossible to get your framed website to rank well, as long as you optimise it in the right way. Most framed websites fail to optimise the 'frameset' page, which is the first page of code the spiders will come across. If it has not been properly optimised, the spiders will find themselves in a dead end.
To get around this, use the 'no frames' tag after the 'frameset' tag. Here is an example (replace the "{}" with "<>"):
{NOFRAMES}
{body}
{p} Type in a description of your website here, incorporating key phrases used in the home page, and use links into some of the pages on your website, such as 'Marketing Karma' offers a selection of articles, and {a href="http://www.marketingkarma.co.uk/tips.html"}tips{/a} to help you improve your online marketing campaigns.
{/font}{/p}
{/body}
{/NOFRAMES}
Do not be tempted to put in words such as "this website uses frames but your browser doesn't support them" or check your page design software hasn't put it in without you knowing - words like this are definitely going to put visitors off browsing your website.
By incorporating links into the body content, you are giving something for the search engine spiders to follow into the website. Also include a title tag and meta description tag in the source code of this page as usual.
2. Flash pages / splash pages
Flash is still a problem for most search engines to index as they cannot read flash files, and often they are embedded into the source code with the file held externally. The best thing with Flash is to avoid using it unless it really enhances the website or is something a target audience may particularly want to see. If the only reason you are using Flash is to show off, then remove it, or just use it in part of the web page. Sites built entirely in Flash are notoriously bad at ranking well, and in fact cause extra work for the web designers as an html version also has to be designed for those who do not want to download a Flash site as they still use slower modems.
Splash entry pages are also pointless. They usually have very little content, apart from a button or image to click to enter the main website, and are an extra click away from the main content for your visitors. They are user-unfriendly, not only to your visitors, but to search engine spiders who have to delve further into the website to find your content.
3. Web pages are dynamically generated
Web pages which are generated dynamically with content that changes on a regular basis also cause headaches to search engine spiders. They are getting better at indexing dynamic pages, but the more you can do to help with the indexing of the website, the better.
When designing a database driven website, it really does help if a small area on each page contains static content that will not change, with the dynamic content appearing below it. This gives you an opportunity to write a paragraph incorporating your main key phrases, otherwise it is difficult for the search engines to keep up with your page changes and decide how relevant the page is to the key phrases being typed into a search query. Another way of helping the search engines to crawl your dynamic pages is to create a static page sitemap (link to it on every page), which includes your main dynamic urls, particularly the ones you want to be indexed.
4. Meta refresh tags
Meta refresh tags are sometimes used when the page you want the visitor to see is not the same as the url they are clicking on - working in a similar way to a redirect. Depending on how the website is built, one page may refresh to another, for example if the url the visitor clicks on is short, but in reality the page is buried into the website with a much longer url. It is a very long winded way of redirecting a page, but is bad news if you are using it. Meta refresh tags have a very bad reputation with search engines, as spammers and porn sites used them to trick the search engines in earlier days of the internet. Search engine spiders cannot get past the meta refresh and you will be preventing your website being indexed if you are using them. Remove any meta refreshes on your website and if you do need to redirect a url, best use a permanent 301 redirect.
5. Robots.txt
The robots.txt file gives instructions to the search engine spiders about indexing your website. It should be included in your website and located in the root directory. However, if the wrong instructions are given to the search engines, you could find you are inadvertently blocking them without knowing. The most common instruction is to give access to all search engines (user-agent) to index all your pages:
User-agent: *
Disallow:
If you want to prevent certain files from being indexed, instruct the spiders as follows:
User-agent: *
Disallow: /testfile/
Disallow: /testfile2/
You need to list each directory you want to be disallowed separately.
Do not use:
Disallow: /
As you are telling the search engine spiders not to index any files in your website.
The best resource for more information on robots.txt and how to use them is Robotstxt.org.
Marketing Karma's robots.txt file is located at www.marketingkarma.co.uk/robots.txt
Making a few simple changes to your website, such as those mentioned above, really can make all the difference to the success of your website in the search engines.
