Have you ever done a search on something in Google or Bing and then clicked on a link only to find the website is “unavailable” or the page is missing “404 error?”  Unfortunately, it’s an all too common experience.  Some of these crawl errors are absolutely preventable, while others are backend server problems, usually dealt with by contacting your hosting company.

But, before I go further with website errors, it might help to first give a basic explanation of how search engines find your website in the first place.  The most basic functions are to use “crawling bots” to find your website and “crawl” through every page and link on your website to build an index of the pages and content.  The index is used to make searching for information easier, faster, and relevant for humans.  When you do a search, the indices are used to present the user with search results that are not only relevant to your search terms, but ranked appropriately.

So, what happens when you have crawl errors on your website?

There are all kinds of reasons for errors.  Some are really bad (server 500 error), like having your website completely down and unavailable, which means visitors can’t see your site and search engines can’t crawl any of your pages.  Other errors are more common, like (error 404) page not found, where the page or file has been deleted or moved.  Here’s a more comprehensive list of common errors that can occur on web servers:

• 400 Bad Request – invalid syntax occurs from invalid or incorrect address.
• 401 Unauthorized – password protected page or file.
• 403 Forbidden – Another permission based error.
• 404 Not Found – Occurs when the page or file has been deleted or moved and the old link can’t be resolved.
• 500 Internal Server Error – Usually a php type problem, but the website is unreachable.
• 502 Bad Gateway – Usually a network connection between backend servers (contact your hosting company).
• 503 Service Unavailable – Server resource issue resulting from an overload of traffic.
• 504 Gateway Timeout – Server performance or network issue.

Obviously, having any kind of 500 error (500, 502, 503, or 504) is not good, because it means your entire website is NOT accessible to visitors or search engines.  400 type errors are much more common and are usually only page or file specific, but the rest of your website is available.

User Experience & Search Rankings

While 400 type errors are page or file specific, they are a problem for someone coming to your website, because they see an error and not the content they were expecting.  It’s good to fix these errors, because it’s likely that visitors will click away from your site, rather than attempt to find the right content.  Google considers this a poor user experience and can result in lower rankings if your website has too many of these errors.

Regular Maintenance is Important!

Keeping a regular website maintenance schedule is important, because it can help reduce or eliminate broken internal links and page / file errors.  You can use Google & Bing’s FREE webmaster tools to keep track of search errors and then fix them.  400 type errors can be fixed with a 301 page redirect, which takes the visitor to another page or to the new location of the content automatically.

While it can be time consuming to do this type of maintenance, it’s definitely worth the effort.  If you have questions about website maintenance, feel free to contact us.