Keeping Google out of the WordPress backend

Over the past few months we’ve noticed that more and more pages from inside the WordPress backend are finding their way in to the Google index. This has always been a problem, but as Google seems to index more useless pages, and crackers get more sophisticated at finding vulnerabilities in WordPress modules, it is important to protect your site from both the crackers and Google.

So what is the real harm?

The most obvious, and urgent harm, comes from exposing your website to potential comprimise. If a vulnerability is found in a WordPress plugin it can take just a few seconds to find a host of web sites  to attack. Using Google’s inurl command a simple search of inurl:wp-content/plugins returns more than 8 million results for a cracker to start his or her search for likely targets.

A dedicated cracker will comprimise your site, but there is no reason to make it easy for them.

Another less obvious problem is created by Google itself. In just this one simple search we’ve seen more than 8 million web pages that have no reason to be in the index. They serve no useful purpose other than to show how invasive Google can be with it’s crawler. It also demonstrates a duplicate content issue that needs to be addressed.

The real problem, however, is the harm this can cause each website this happens to.

It is known that Google may not index all of the pages in a website for various reasons. Assume you have a website with 100 pages. Yet Google decided to index 30 pages of your /wp-content or wp-admin/ folders. You have lost the postential for 30% of your pages to be indexed in favor of pages that should never have been in the index at all. I have seen sites with more than 50% of their indexed pages coming from the back end of WordPress.

What can you do about it?

There are two things that you should do to help secure your site from search engines exploring where they don’t belong.

1. Robots.txt: With every WordPress install I do these days I add this to my robots.txt file.

User-agent: *
Disallow: /blog/wp-admin/
Disallow: /blog/wp-content/
Disallow: /blog/wp-includes/

Be sure to adjust the URL for your site’s install folders.

2. Google’s Webmaster Tools: If you find these pages indexed for your site first install the robots.txt file. Once that is done you should enter your GWT account and remove those pages from the index. Once removed the robots.txt should keep them from being re-indexed.

Unfortunately, from then on you will see and error message in your GWT account. You can ignore this error.

What does all of this tell us? The biggest thing it tells us is that the Google spiders are not as smart as everyone, including Google, would like us to believe. Indexing these pages serves no purpose, and it shows that the bots can and will go to places that they really should not be in and you must be proactive in protecting your website from them. A person would know that there is no reason to index more than 8 million of the exact same pages. An algorithm cannot make that decision.

Posted in SEO, Search Results, Wordpress | 2 Comments »

Do I really need a sitemap?

Do I really need a sitemap? This question has been asked for a long time, and in mid 2005 Google made it even more complicated by introducing a sitemap format just for their spiders.

First of all there are two types of sitemaps. One sitemap for users and one for Search engines. A sitemap for users may be beneficial in that it allows you to put every page on your web site just one click away for end users. Fewer clicks tend to help conversions, and on large web sites they may provide for an easy way for users to find exactly what they are looking for with the fewest clicks.

Sitemaps for search engines are another matter entirely. Originally the xml sitemap, as proposed by Google, was designed to help their spider crawl complex URLs that might have been missed by their spiders. These URLs tended to be in shopping cart systems and database driven sites developed by programmers that had little regard for the real world needs of a modern web site. Urls like domain.com/id7734&ask?whyam-i:like&this made deep crawls by spiders nearly impossible.

Since then having an xml sitemap has been trotted out as the solution to a long list of problems with websites when it comes to them not ranking well, or not having as many pages in the Google index . And for the vast majority of sites this isn’t true.

One of the deep indexing factors that Google does rely on is linking. On Matt Cutt’s blog in 2006 a discussion was raging about the supplemental index and Matt mentioned that the number of links pointing toward a site was a factor in how often it was crawled and how deeply the site was indexed. No mention of the, by then, 1 year old sitemap.xml.

Google itself even says “We don’t guarantee that we’ll crawl or index all of your URLs¹” and that the sitemap is just used to learn about your site’s structure¹.

One of the major reasons that I recommend against xml sitemaps is that they require constant updating every time you update your website. And while there are tools available to help automate the process², that in itself is reason enough not to do it, more work for questionable benefits. The only time an xml sitemap might be justified is when your site navigation is less than useful and is a challenge for search engine spiders to crawl. Unfortunately this may mean that your site navigation is a challenge to follow for end users as well, so it should be dealt with by fixing the navigation, not slapping a sitemap on it and calling it “fixed”.

Another reason that I don’t recommend xml sitemaps, in most cases, is that Google has had a long standing policy that we should build web sites for our users and not for the search engines. Unless their spider isn’t advanced enough to do the job, then webmasters are expected to step up and make things easier for them. Xml sitemaps and nofollow are just a couple of examples of building something for search engines only that has no benefit as all for the user viewing your website.

So to answer the question, Do I really need a sitemap?, no, you don’t. You need clean navigation on a well structured web site. However, if you have complex URLs on an incredibly large web site with lousy navigation, it can’t hurt, but it also probably won’t help much either.

1. From Will Google crawl and index all of the URLs in my Sitemap?
2. Free Sitemap Generators

Posted in SEO, Search Results, Web Development | 1 Comment »

Attack of the 50 Foot Corporate Anthem!

They’re baaaaaaaaaack. In this month’s edition of The Attack of the 50 Ft. Corporate Anthem we get a look at Honeywell, an even more countryfied version of the DRUPA theme song, and some very strange off off off Broadway musical rendition about Glaucoma.

You can blame my wife for this batch of Corporate Anthems. She didn’t believe that there was a Honeywell anthem, and I had seen it referenced online, so the quest began. After hacking through the jungles of Borneo, climbing the heights of the Himalayas, and fighting off the giant tarantulas in the desert of New Mexico, I found it. The Quest for Quality! I think the struggle was worth it, my wife thinks I’m going to get her fired.

In either case, enjoy the new anthems and be sure to check out some of the old ones on the Corporate Anthems page. And as always, if you know of any anthems hiding out in the wild, let me know and I’ll go catch them and add them to the zoo.

Posted in Don't Do This, Old School Business | What Do You Think? »

Customer service: You have email, use it

Let me tell you a few stories about companies and the way they use email.

This quote appeared on one of the jewelry forums I follow.

Hi I ordered a custom (ring) from …. and last contact with him was Sept. 21 stating payment recieved will be in touch soon….It’s been almost a month now and I have sent a few emails to him with no response. Anyone have any ideas how I should proceed…all I want is an update with an estimation of completion…I feel funny to email him again I don’t want him to think I’m some kind of crazy stalker…hehehe….Just worried the ring won’t be here in time for my wedding which is Nov. 12.

This lady simply wants some reassurance that her ring will be ready for one of the biggest days of her life. We later found out that the jeweler was having some personal issues, which happen, but a 1 minute email would have gone a long way toward assuring his customer that she would be taken care of. In the mean time this thread went on for more than 3 weeks with his reputation being heavily damaged in the process.

Sticking his head in the sand and pretending that there wasn’t a problem didn’t make the problem go away.

Closely related to this is a company that answers it’s email when they get around to it. This is probably the single biggest problem I hear from our customers at Images Jewelers. Most people will send out several requests for more information about having a piece of handcrafted jewelry made to several jewelers. We constantly hear about how fast we are to respond, or how they had emailed someone else a couple weeks ago and never heard from them, or the best ones, when we are nearly finished creating the piece and the other jeweler has finally responded to the first request for information.

The number of jobs we land simply because we answered our email quickly at close to 10% of our total sales. It’s such an easy way to increase conversions but so few people really put an effort in to their email support that it continues to surprise me.

And then we have people like the folks at Fatheadz.

I had bought a pair of sunglasses from them and after about one month they exploded in to 4 or 5 pieces. They weren’t terribly expensive, about $100, but I had never seen an injection molded piece explode like that. So I sent their customer service an email letting them know what happened and what we could do about it.

About 30 minutes later I got an email from them apologizing for the problem and a solution. Send them the sunglasses back and pick any pair that I wanted from their website to replace them. No hassles, no questions, just an immediate response to the problem with a solution.

So many companies drop the ball when it comes to the basic things like responding to email in a timely manner. It shouldn’t be that way. Not every company can afford to have someone dedicated to answering email the instant it comes in, even if the answer is a yes, we got your email but I need to ask a few people to get an answer for you.

Make answering your email a part of your daily routine. And not once a day, several times a day. Do it first thing in the morning, immediately after lunch, and again about an hour before you go home for the day. Do it and watch your conversion and retention rate jump dramatically.

And for the record, I love my new Fatheadz Sunglasses!

Posted in Business Issues | 1 Comment »

Epic Fail == Me

Yes, I can admit when I make a mistake. In some cases even pretty earth shatteringly large mistakes. This time I pulled the trigger without doing enough due diligence and research, something I constantly tell people to do before they jump in to the deep end.

So, what did I fail at? A couple people knew that this was coming, but I bet that most haven’t noticed that this blog is now running on a new domain name and under a new operating name. Yup, I picked a business name without doing enough research, I let myself make an assumption that could have potentially turned in to wee bit ‘o drama in the future.

The name I had originally chosen was Large Orange Pop. I have a thing for the 50s even though I was born in the 60s, and the idea of looking for a name that had more potential as a brand than as a description of some services I may offer appealed to me. It appealed to me so much that I simply assumed that I was the only one to think of it. As usual, assumptions can go wrong. There is another company out there that had a similar idea, Orange Soda, only they had it several years ago.

So what’s a guy to do? The obvious, move on quickly and put it behind me. I found a new name that I like that also lends itself toward branding, and when I finally have the time to work on it, a new look and feel featuring my favorite 50s villains and heroes, robots. Hopefully the rebuild won’t take too long, but fortunately I am heavy with client work at the moment that may keep me from working on my own site.

Takeaway?

Do your research! You may have a great idea that you think is completely unique, but odds are you would be wrong.

Posted in Administrivia, Business Issues, Don't Do This, Linkerbation, Web Development, Wordpress | 2 Comments »

It’s hard to find a great place to sleep

Four Winds Casino, in New Buffalo, Michigan, is a new Indian Casino, can I say that or should I say Native American Casino?, and it seemed a great chance to go see what it was like.

Normally I stay “at” the casino because I don’t really gamble all that much and love the ease of going up to the room and catching a nap. But the only rooms available were $350/night.. I’m not exactly cheap, but $350/night for a casino room? I can go to Vegas for that much money. So the quest was on to find a nice place to stay, for a fair price, close to New Buffalo. Who knew that it would be such a challenge?

If you are running just about anything in New Buffalo local search is wide open for you. It took far too long to find a nice bed and breakfast reasonably close to the casino, well, not really, but if it weren’t for the new maps in Google we would have never found the place we wound up staying. Sans Souci Euro Inn and Cottages (yeah, it’s their title and it’s on every page). Angie and Sue run an amazing little operation on 50 acres with a small private lake just full of fish. It’s quiet, it’s well furnished, it’s 3 miles from the casino, and it’s incredibly reasonably priced. We already have reservations for a full week later in the summer.

So what’s the point of this long post? Am I just trying to rub in the fact that I found a great place to stay? Well, yeah, sorta. But more importantly it’s about how few people are taking advantage of promoting their local business on the internet. And more specifically, on Google. A simple search for new buffalo michigan bed and breakfast on Google returns just 85,000 results, and barring the map, and just “one” real bed and breakfast, ranked #7. There is a B&B that is closed, it even says so on their website, that ranks #5. If it weren’t for the fact that Angie is looking to retire I’d offer to rebuild her site for her. I may still make an offer, but it might be to just buy the place and change careers!

While looking for something to do other than the casino I did a search for charter fishing new buffalo michigan. Not a single charter boat based in New Buffalo has a tag on the Google local map. Not one. Any SEO in New Buffalo Michigan may want to take a drive around the docks and look for a few new clients. Or maybe not, I may be up there in August fishing and could use the work.

To recap,

• Casino getaway = great!
• Quiet B&B to relax at = fabulous!
• Have a website for a business and not taking advantage of local search = epic fail

Posted in Business Issues, Life In The Fat Lane, Local Search, Marketing, SEO, Web Development | 1 Comment »

More Corporate Anthems

I love them, God help me I do love them so.

I’ve come across a few more Corporate Anthems that I just had to add to my collection.

The latest additions include the theme song for the Drupa International Printing and Paper Fair another great tune for the fine folks at IBM a major change from the music of the 50s , and a great after school special ditty from KPMG .

But the biggest score this month is this cool video of Richard Stallman singing about free software. Not really a “corporate” anthem, but it’s just too good to not include in this list.

Check out the complete collection over on my Corporate Anthems page.

Posted in Don't Do This, Old School Business | What Do You Think? »

PayPal alternative, finally

I have made no secret of my dislike of PayPal. There has always been something loan sharkish about them. Often there is no recourse when they do something against you, they have become a haven for people of questionable ethics, and they are totally unregulated yet pretend they are a bank. I like my banks with rules I can count on.

Enter Revolution Money Exchange. Based in St. Petersburg, FL, RevolutionMoneyExchange is actually using accounts issued by a real bank with real rules! First Bank & Trust, Brookings, SD, is a real bank, a member of the FDIC, and is part of the Fishback Financial Corporation.

RevolutionMoneyExchange isn’t some basement operation either. Started by Steve Case, co-founder of AOL, it has received venture capital funding from Citi, Morgan Stanley, and Deutsche Bank.

Just to make their startup spectacular RevolutionMoneyExchange is giving everyone that signs up between now and April 15th gets $25, FREE! It also sends me $10 for the affiliate code used in these links, so please, use my affiliate code. Even if you don’t use my code, sign up, take the free money. And let’s start putting the hurt on PayPal. Maybe this will spur them in to adopting some real oversight. But I doubt it.

Posted in Affiliate Stuff, Business Issues | 2 Comments »

Forum spam gets organized

Spammers are a crafty bunch, a crafty bunch that may need to be crushed under heel at some point, but crafty non-the-less.

It all started with a simple need. Getting links. At first it was about getting friends to link to you, or joining a webring. Then it was trading links with other people whether you liked them or not. That was soon followed by submitting your website to hundreds of directories, free and paid. But what happened when you needed thousands of links submitted? Simple, you outsource it like everything else. $50 gets you 1000 link submissions!

Well, it seems like making money submitting websites to directories isn’t as lucrative as it once was. Take a look at this email Dave over at ChainzOnline got, I assume by mistake.

Date: Mon, 7 Apr 2008 23:01:53 +0300
From: SEO Company <info@seo4experts.com>
To:
Subject: Looking for Link Builder. Job Offer.

Hi,

We are a SEO company. We are looking to hire several link builders.

We want you to build links from forums. NOTE: we do NOT need any other type of links. Please do NOT reply to this email with your offers of other kinds of links. Thanks.

We’ve got a list of ~1,000 computer & software related forums. We want you to make 3 posts on each forum regularly. You will need to put a link in your signature. We want to hire you for a long period of time (month, year, several years++).

For how much are you ready to do this kind of work? How much do you want per forum? [UTF-8?](You’ll need to make 3 posts on each forum.)

We are sending out this email to several hundreds of Indian link builders in order to find the cheapest price. So please make as low quote as you can.

Who we want to find:

a) responsible person who wants to work
b) a person with good English
c) a person who knows how to make posts on forums

Please reply to this email if you are interested.

Thanks,
SEO Company

Here is a company spamming to find spammers that we can thank for a lot of the new spam heading our way. No link for you ( http://www.forumlinkbuilding.com/view.php/forum_link_building_service ). Take a look at this quote from their page:

Unique: We will build links to your website using forums. We will create 3 posts on each forum linking to your website from 3 different pages. We never spam forums.

They “never” spam forums yet they create accounts specifically to get three posts with links in them? Isn’t this spam? I’m sure we can all go on for hours about these types of “people”, but they won’t care. So what to do about it?

Over at WebWorkshop we set it up so that you need 50 posts to get any live links or sigs at all. And I talked with a mod at WebProWorld and they also instituted a min post count to get sig links. And I suspect that most other forums will do the same to help stem the flood of low income workers spamming our forums with “I like what you said” or “Hi, I’m from New York” via an India IP address posts.

I think my new hobby will be contacting all these companies that hire these guys and ask them why they are spamming our forums. Hmmmm, possible client recruitment potential. Forget that last part, you don’t need those clients.

Posted in Don't Do This, Linkerbation, Spammage | What Do You Think? »

Feed the pig – Really?

I’m sure that you have all seen this public service announcement on TV,

Maybe I’m wired wrong, but every time I see that PSA the only thing running through my head is, how is that poor salesman supposed to feed his kids? As a bit of marketing it does seem to grab your attention, but it does a terrible job at actually providing a solution. Buy me, don’t buy me, why should I want this? It doesn’t trigger a call to action response like it should.

Americans have been notorious for not saving money over the last couple of decades, the current economic issues lend a lot of support to this. The me economy of get it now pay for it later has hurt a lot of people. People that should have known better. If you make $30,000 a year you should not be buying a $600,000 house no matter how much you want it. The same can be said for many businesses out there.

Everyone wants to be the next Blue Nile or Amazon. And they seem to approach it from one of two directions.

1. They open up their wallets, dig a hole, and start filling the hole until they get what they think they want. This approach never works because you are competing with other companies with bigger wallets, bigger holes, and are just as crazy as you are.

2. They hire little Jimmy, their golf buddies high school kid that “builds web sites” for $10 an hour and then wonder why the world isn’t beating a path to their door.

Just like the Feed The Pig PSA’s, there never really seems to be a good plan in place for getting where they want to be, a stable economy with financially responsible people helping it recover. They have an idea, think it’s a great idea, and charge full speed ahead without taking the time to really think over their options and possible solutions and problems that the plan can introduce. The problem with the Feed the Pig commercials is that they say “Don’t spend money” without explaining why, or how to be more financially responsible. There is no plan, no solution, just a dressed up pig.

There is another commercial out there where the family decides to buy a new TV and you see the husband looking at the mega screen plasma super TVs, yet when they get home they have a reasonably sized TV that we assume they can afford comfortably. Reasoned and responsible spending.

What about your web site? Has it be reasoned out? Is there a plan, a goal, a solution? Or is it just a dressed up pig and you are hoping that your viewers “get it”??

Posted in Business Issues, Old School Business, Web Development | What Do You Think? »