So what is the real harm?

The most obvious, and urgent harm, comes from exposing your website to potential comprimise. If a vulnerability is found in a WordPress plugin it can take just a few seconds to find a host of web sites  to attack. Using Google’s inurl command a simple search of inurl:wp-content/plugins returns more than 8 million results for a cracker to start his or her search for likely targets.

A dedicated cracker will comprimise your site, but there is no reason to make it easy for them.

Another less obvious problem is created by Google itself. In just this one simple search we’ve seen more than 8 million web pages that have no reason to be in the index. They serve no useful purpose other than to show how invasive Google can be with it’s crawler. It also demonstrates a duplicate content issue that needs to be addressed.

The real problem, however, is the harm this can cause each website this happens to.

It is known that Google may not index all of the pages in a website for various reasons. Assume you have a website with 100 pages. Yet Google decided to index 30 pages of your /wp-content or wp-admin/ folders. You have lost the postential for 30% of your pages to be indexed in favor of pages that should never have been in the index at all. I have seen sites with more than 50% of their indexed pages coming from the back end of WordPress.

What can you do about it?

There are two things that you should do to help secure your site from search engines exploring where they don’t belong.

1. Robots.txt: With every WordPress install I do these days I add this to my robots.txt file.

User-agent: *
Disallow: /blog/wp-admin/
Disallow: /blog/wp-content/
Disallow: /blog/wp-includes/

Be sure to adjust the URL for your site’s install folders.

2. Google’s Webmaster Tools: If you find these pages indexed for your site first install the robots.txt file. Once that is done you should enter your GWT account and remove those pages from the index. Once removed the robots.txt should keep them from being re-indexed.

Unfortunately, from then on you will see and error message in your GWT account. You can ignore this error.

What does all of this tell us? The biggest thing it tells us is that the Google spiders are not as smart as everyone, including Google, would like us to believe. Indexing these pages serves no purpose, and it shows that the bots can and will go to places that they really should not be in and you must be proactive in protecting your website from them. A person would know that there is no reason to index more than 8 million of the exact same pages. An algorithm cannot make that decision.

First of all there are two types of sitemaps. One sitemap for users and one for Search engines. A sitemap for users may be beneficial in that it allows you to put every page on your web site just one click away for end users. Fewer clicks tend to help conversions, and on large web sites they may provide for an easy way for users to find exactly what they are looking for with the fewest clicks.

Sitemaps for search engines are another matter entirely. Originally the xml sitemap, as proposed by Google, was designed to help their spider crawl complex URLs that might have been missed by their spiders. These URLs tended to be in shopping cart systems and database driven sites developed by programmers that had little regard for the real world needs of a modern web site. Urls like domain.com/id7734&ask?whyam-i:like&this made deep crawls by spiders nearly impossible.

Since then having an xml sitemap has been trotted out as the solution to a long list of problems with websites when it comes to them not ranking well, or not having as many pages in the Google index . And for the vast majority of sites this isn’t true.

One of the deep indexing factors that Google does rely on is linking. On Matt Cutt’s blog in 2006 a discussion was raging about the supplemental index and Matt mentioned that the number of links pointing toward a site was a factor in how often it was crawled and how deeply the site was indexed. No mention of the, by then, 1 year old sitemap.xml.

Google itself even says “We don’t guarantee that we’ll crawl or index all of your URLs¹” and that the sitemap is just used to learn about your site’s structure¹.

One of the major reasons that I recommend against xml sitemaps is that they require constant updating every time you update your website. And while there are tools available to help automate the process², that in itself is reason enough not to do it, more work for questionable benefits. The only time an xml sitemap might be justified is when your site navigation is less than useful and is a challenge for search engine spiders to crawl. Unfortunately this may mean that your site navigation is a challenge to follow for end users as well, so it should be dealt with by fixing the navigation, not slapping a sitemap on it and calling it “fixed”.

Another reason that I don’t recommend xml sitemaps, in most cases, is that Google has had a long standing policy that we should build web sites for our users and not for the search engines. Unless their spider isn’t advanced enough to do the job, then webmasters are expected to step up and make things easier for them. Xml sitemaps and nofollow are just a couple of examples of building something for search engines only that has no benefit as all for the user viewing your website.

So to answer the question, Do I really need a sitemap?, no, you don’t. You need clean navigation on a well structured web site. However, if you have complex URLs on an incredibly large web site with lousy navigation, it can’t hurt, but it also probably won’t help much either.

1. From Will Google crawl and index all of the URLs in my Sitemap?
2. Free Sitemap Generators

Normally I stay “at” the casino because I don’t really gamble all that much and love the ease of going up to the room and catching a nap. But the only rooms available were $350/night.. I’m not exactly cheap, but $350/night for a casino room? I can go to Vegas for that much money. So the quest was on to find a nice place to stay, for a fair price, close to New Buffalo. Who knew that it would be such a challenge?

If you are running just about anything in New Buffalo local search is wide open for you. It took far too long to find a nice bed and breakfast reasonably close to the casino, well, not really, but if it weren’t for the new maps in Google we would have never found the place we wound up staying. Sans Souci Euro Inn and Cottages (yeah, it’s their title and it’s on every page). Angie and Sue run an amazing little operation on 50 acres with a small private lake just full of fish. It’s quiet, it’s well furnished, it’s 3 miles from the casino, and it’s incredibly reasonably priced. We already have reservations for a full week later in the summer.

So what’s the point of this long post? Am I just trying to rub in the fact that I found a great place to stay? Well, yeah, sorta. But more importantly it’s about how few people are taking advantage of promoting their local business on the internet. And more specifically, on Google. A simple search for new buffalo michigan bed and breakfast on Google returns just 85,000 results, and barring the map, and just “one” real bed and breakfast, ranked #7. There is a B&B that is closed, it even says so on their website, that ranks #5. If it weren’t for the fact that Angie is looking to retire I’d offer to rebuild her site for her. I may still make an offer, but it might be to just buy the place and change careers!

While looking for something to do other than the casino I did a search for charter fishing new buffalo michigan. Not a single charter boat based in New Buffalo has a tag on the Google local map. Not one. Any SEO in New Buffalo Michigan may want to take a drive around the docks and look for a few new clients. Or maybe not, I may be up there in August fishing and could use the work.

To recap,

• Casino getaway = great!
• Quiet B&B to relax at = fabulous!
• Have a website for a business and not taking advantage of local search = epic fail

Some of my favorite shows are Danny Sullivan’s Daily Searchcast, Strike Point with Dave Naylor (DaveN) and Mikkel deMib (Mikkel), and SEO 101 with Carolyn Shelby (cshel), Brian Mark and David Brown (Ne0).

You can listen to the live stream for free, and even some of the not so old episodes for free, but to get to all of the back issue stuff, and it is worth reading, you need to sign up.. Hence the affiliate link here. Of course you can go there for free, but hey, momma needs new shoes so a little affiliate love for a good resource shouldn’t be too painful, should it? Go give Webmaster Radio a listen, you are bound to learn something.

IRM’s website is pure graphic. Not one single word of text on the page anywhere to be found. It boggles the mind that there are still websites. It doesn’t get any better on page two. More images with no text, no alt text, no title text, not even images with decent names.

Then we have the “Market Research” showing just how much internet radio has grown over the last three years. Unfortunately the years tracked are 2000 to 2003, not exactly current information. Maybe the site hasn’t been touched since it was registered and created in 2003. Maybe Ruben is spending most of his time on BoomerRadio. I don’t know, but putting up a site like IRM is worse than doing nothing at all.

Some things to remember when building a website.

1. Don’t build a website if you don’t intend to maintain it. (I think I might have a few of those)
2. Don’t build a website using nothing but images. (Pretty sure I don’t have any of those)

The client side:

No one wants to see their competitors gain an equal footing with them. So having your Internet Marketing company help your nearest competitor may not be very high on your list of things you would like to hear about. After all, your marketing company’s goal is to help you rank just as high as your website can, and how can they do that if they are helping someone else to the same thing?

The marketing side:

As marketers we spend a lot of time and resources developing contacts, relationships, and knowledge about very specific markets. To the point where it is not uncommon to see companies specialize in very specific niches. This goes for web developers, advertising companies, architects and auto mechanics. Specializing breeds excellence in a very narrow category. Having more than one client in a specific industry can be seen as a very good thing.

The non-compete:

Non-competes are generally thought of when dealing with an employer / employee relationship. But can also apply in a contractor situation. Basically it is a clause in a contract, you do work with contracts don’t you?, that defines who the contractor can work with and in what areas they can work when their actions may compete with the primary client.

What to do?

You can either take additional work in the same field or not. But how do you decide which is the right option for you? When we were doing software development we actually had a tiered pricing structure that I think may be appropriate in this situation.

Our clients had three options:

1. Open development. The clients paid a set fee and the work belonged to us and they simply licensed the end product.
2. Non-compete. The client paid a higher fee and we developed the software for them and did not sell it to other potential clients.
3. Total ownership. The client owned the software, the source code, and were allowed to do anything at all that they wanted to do with the finished product. And, of course, this was the most expensive option and rarely used.

There is no reason that an SEO company can’t build on this same model. Especially one focusing on local search. Offer your potential client the option, up front, to lock you in as their SEO and internet marketing company to the exclusion of all others, for a fair price. This protects them from having to contend with a new company coming in and climbing the rankings right behind them and it allows the marketing company the opportunity to receive a higher billable rate in compensation for reducing it’s potential client base.

Be pro-active on this subject. Approach your clients and let them know that they have the opportunity to lock you in and guarantee that your specialized services will be dedicated solely to them. And for the businesses out there, if you are concerned about your marketing company helping someone else in your market, call them, see what you can do to tie them to your company exclusively.

Non-competes aren’t for everyone, but you need to decide what your response will be before you are placed in the position of having to decide whether or not to turn down offers of new work.