So what is the real harm?

The most obvious, and urgent harm, comes from exposing your website to potential comprimise. If a vulnerability is found in a WordPress plugin it can take just a few seconds to find a host of web sites  to attack. Using Google’s inurl command a simple search of inurl:wp-content/plugins returns more than 8 million results for a cracker to start his or her search for likely targets.

A dedicated cracker will comprimise your site, but there is no reason to make it easy for them.

Another less obvious problem is created by Google itself. In just this one simple search we’ve seen more than 8 million web pages that have no reason to be in the index. They serve no useful purpose other than to show how invasive Google can be with it’s crawler. It also demonstrates a duplicate content issue that needs to be addressed.

The real problem, however, is the harm this can cause each website this happens to.

It is known that Google may not index all of the pages in a website for various reasons. Assume you have a website with 100 pages. Yet Google decided to index 30 pages of your /wp-content or wp-admin/ folders. You have lost the postential for 30% of your pages to be indexed in favor of pages that should never have been in the index at all. I have seen sites with more than 50% of their indexed pages coming from the back end of WordPress.

What can you do about it?

There are two things that you should do to help secure your site from search engines exploring where they don’t belong.

1. Robots.txt: With every WordPress install I do these days I add this to my robots.txt file.

User-agent: *
Disallow: /blog/wp-admin/
Disallow: /blog/wp-content/
Disallow: /blog/wp-includes/

Be sure to adjust the URL for your site’s install folders.

2. Google’s Webmaster Tools: If you find these pages indexed for your site first install the robots.txt file. Once that is done you should enter your GWT account and remove those pages from the index. Once removed the robots.txt should keep them from being re-indexed.

Unfortunately, from then on you will see and error message in your GWT account. You can ignore this error.

What does all of this tell us? The biggest thing it tells us is that the Google spiders are not as smart as everyone, including Google, would like us to believe. Indexing these pages serves no purpose, and it shows that the bots can and will go to places that they really should not be in and you must be proactive in protecting your website from them. A person would know that there is no reason to index more than 8 million of the exact same pages. An algorithm cannot make that decision.

First of all there are two types of sitemaps. One sitemap for users and one for Search engines. A sitemap for users may be beneficial in that it allows you to put every page on your web site just one click away for end users. Fewer clicks tend to help conversions, and on large web sites they may provide for an easy way for users to find exactly what they are looking for with the fewest clicks.

Sitemaps for search engines are another matter entirely. Originally the xml sitemap, as proposed by Google, was designed to help their spider crawl complex URLs that might have been missed by their spiders. These URLs tended to be in shopping cart systems and database driven sites developed by programmers that had little regard for the real world needs of a modern web site. Urls like domain.com/id7734&ask?whyam-i:like&this made deep crawls by spiders nearly impossible.

Since then having an xml sitemap has been trotted out as the solution to a long list of problems with websites when it comes to them not ranking well, or not having as many pages in the Google index . And for the vast majority of sites this isn’t true.

One of the deep indexing factors that Google does rely on is linking. On Matt Cutt’s blog in 2006 a discussion was raging about the supplemental index and Matt mentioned that the number of links pointing toward a site was a factor in how often it was crawled and how deeply the site was indexed. No mention of the, by then, 1 year old sitemap.xml.

Google itself even says “We don’t guarantee that we’ll crawl or index all of your URLs¹” and that the sitemap is just used to learn about your site’s structure¹.

One of the major reasons that I recommend against xml sitemaps is that they require constant updating every time you update your website. And while there are tools available to help automate the process², that in itself is reason enough not to do it, more work for questionable benefits. The only time an xml sitemap might be justified is when your site navigation is less than useful and is a challenge for search engine spiders to crawl. Unfortunately this may mean that your site navigation is a challenge to follow for end users as well, so it should be dealt with by fixing the navigation, not slapping a sitemap on it and calling it “fixed”.

Another reason that I don’t recommend xml sitemaps, in most cases, is that Google has had a long standing policy that we should build web sites for our users and not for the search engines. Unless their spider isn’t advanced enough to do the job, then webmasters are expected to step up and make things easier for them. Xml sitemaps and nofollow are just a couple of examples of building something for search engines only that has no benefit as all for the user viewing your website.

So to answer the question, Do I really need a sitemap?, no, you don’t. You need clean navigation on a well structured web site. However, if you have complex URLs on an incredibly large web site with lousy navigation, it can’t hurt, but it also probably won’t help much either.

1. From Will Google crawl and index all of the URLs in my Sitemap?
2. Free Sitemap Generators

Danny Sullivan in the address bar hits Google search results and has Search Engine Land as #1. Fair Enough. My name brings up my LinkedIn profile in the search results, still no direct push to my website. Maybe someone more famous than me or Danny will help. George Bush? Nope. Hillary Clinton? Nope. At least there is no political favoritism going on. Maybe another Googler? Sergy Brin pops me over to the Biography Channel – UK and Larry Page lands us over at wikipedia. Hmm, beginning to see a trend here.

Time to uninstall the toolbar. . . . . . .

Hmm, no toolbar and Yahoo! set as the default search and still redirected to Matt’s blog. It’s beginning to look like a Firefox admiration (funding?) issue. Damn, not even Marc Andreessen gets this level of respect.

What does Firefox think of your name?