Blog

As I said in my last post – I use www.GoDaddy.com for all of my web site hosting now. Some may say that they are a ‘budget web host’, but I’ve been developing an designing web sites for 10 years now, and I have no problems with them at all. In fact they are one of the best web hosts I’ve ever had. I don’t know where else I would be able to host 50 sites with mySQL databases for just $14.99! One of the things I like is the unique set of tools they have available (like podcasting accounts)…and when I was doing some administration in my hosting control panel the other day I noticed an option to “use my $25 credit for opening up a new Microsoft AdCenter account”.

I recently opened up a Google adWords account due to some of the postings over at Sabahan.com about Gaman’s recent experiences with adWords. I haven’t started a campaign in adWords just yet, because I wasn’t sure I wanted to spend the money. Something about MS AdCenter intrigued me though, probably because I thought that less people would be competing for keywords in this newer service than in google adWords. So I signed up for a free account and used my coupon code for a free $25 credit – why not?

I found AdCenter very easy to use and intuitive. I setup a new campaign, and typed in some keywords I thought would work, and it came back with how many times those words were searched in the last month (very handy). It asked me for a maximum “per click” I would pay each time my ad was clicked – so I put in 5 cents. I figured at this rate that I should get about 500 visits for my free $25.00 credit. I have a joke web site Top Jokes that isn’t getting much activity just yet, so it was perfect for the test!

Out of the keywords I looked at, the phrase “funny stuff” seemed to get about 5,000 searches per month through the Microsoft Live search engine. When I signed up – it said that MS Live gets about 99 million hits per month. I setup the campaign to end when my $25.00 was spent. For some reason, I thought in my head that the money would be used up anywhere from 3 or 4 days to a week. I should’ve gotten the calculator out, because 5,000 searches is only about 166 per day, and even with a click through rate of 5% (which is very high), I would only get about 32 clicks per day visiting my site.

Now, normally this joke site only gets like a dozen visits per day. The first day of my ad campaign I only got like 3 clicks, but I did get like 25 visits to my site. The second day I got 7 clicks, but 50 visits to my site. The third day was about 12 clicks and 100 visits to my site. This is the fourth day of the campaign and we’ll probably surpass that. because I’ve already had about 150 visits.

In the past, all my of site traffic (for all my sites) has always been purely organic. I’ve never used paid advertising of any kind. When I set this ad up, since I hadn’t done this before, it didn’t occur to me that the traditional rules of advertising totally apply in the online world in many ways.

Let’s reflect for a minute here:

• The ad campaign started out slow, but every day the number of clicks have grown
• The number of paid clicks vs. the number of new visitors is nearly double

On the web during the last so many years there has been much talk about “the long tail”. I think that’s just a very old marketing concept rehashed for the web, but this is definitely the long tail at work.

• I’ve received 30 paid clicks, but 425 new visits to the site
• I half a half dozen new rss subscribers (there were none)
• bsuite reports show new visitors arriving from email sent from our feedburner “email to a friend” links
• bsuite reports also show incoming searches from social sites like sphereIt and stumbleUpon

That is truly the long tail at work, and I’m now exited that my 500 paid clicks may turn into 1,500+ new visitors, new subscribers, and more “social” visitor activity. In addition there is the “name recognition” value of having an ad – even when people don’t click it. I don’t have the numbers (I wasn’t a marketing major), but it seems that I read years ago the average person has to see a billboard at least 3-4 times before they pay attention to what it says. If that’s the case, someone may have to see the ad for my site 3-4 times before they click.

Exposure is good! My latest report I just ran in AdCenter says I’ve had 806 impressions and 34 clicks. Evertime someone sees your ad (whether they click or not) is a good thing. I’ll post a follow up article once this campaign has finished to see if this trend (of the long tail) continues – and we’ll see what value I really got for my $25.00 credit! So in this post we learned that my latest blogging mistake was not to use any paid promotion for my web sites. I never would have dreamed I could have gotten this much activity for such a low price! If I’d just paid $25.00 – $50.00 per month who knows how many subscribers I could have by now!

I had a very large web site (30,000+ pages) that I wanted to move from one web host to another. Normally I would use phpMyAdmin to export tables or the entire database. This mySQL database was of course too large for that and phpMyAdmin would timeout trying to export, even if I exported only one table at a time (one table had 1.5 million records). It’s very easy to quickly export and import very, very large mySQL databases with even hundreds of thousands of records.

To accomplish this we can’t do it via web page tools or methods. So, we have to use the command line in UNIX. This will require a telnet (preferably secure telnet) account and a basic understanding on simple UNIX commands. You will also enough disk space free to extract your database. If you don’t have telnet access to your web site, you can always submit a support ticket telling them what to do based on these instructions. You will need to know in advance your database name, your database username and password, and whether or not your server is ‘localhost’ or a server name (and what it is). This information is normally already entered in your site’s database config or settings file. If you are using WordPress it’s in your /wp-config.php file in the root of your site, if you’re using Drupal it’s stored in your /sites/default/settings.php file.

Here are the instructions to export your mySQL database into one file on the UNIX command line:

• login to your web site using telnet
• navigate to the directory where you want to store your extracted database file
• run the following on the command line:
  mysqldump -a -u USERNAME -p DATABASE > FILENAME.mysql
• check the size of your file (command: ls -al)
• gzip your file to make the download faster using the command: gzip FILENAME.mysql
• download your file in ftp to your local pc to upload and import into your new database

Again, if you don’t have telnet access you can probably submit a ticket at your web host and they will extract the file for you so you can download it. Once downloaded you need to upload the file to your new web host so you can import it into a fresh database. First, I want to make you aware of a PHP script called Big Dump that does a ‘staggered import’ of mysql files into a new database. Even though it’s staggered (importing in chunks) it has it’s limits. I have successfully used it many times on databases under 50MB. It has also failed miserably for me on a 191MB database. If your database is too big to import using phpMyAdmin but under 50MB – give it a try first. Otherwise you’ll have to import on the command. If you use Big Dump you just enter your mysql server settings into the file itself, and I have had the best results uploading the mysql file to the server (same dir you place bigdump.php in) and hardcode the filename into the file before trying the import.

Now, if your file is >50MB or bigdump.php doesn’t work – I’m going to show you how to do a manual import of your mysql file:

• upload your mysql file to your new web site
• create a new blank database using myPHPAdmin
• note your database username and password, and your mysql servername
• unzip your database file with this command: gzip -d FILENAME.mysql
• run the following on the command line to import:
  mysql -u USERNAME -p DATABASE < FILENAME.mysql
• check your database in myPHPAdmin to make sure all the tables imported properly

That’s it! You’ve now learned how to successfully export and import large mySQL databases when myPHPAdmin or other tools won’t work! Again, if you don’t have telnet access, you can always ask your (new) web host to import the file for you. I did when I migrated from a host I used for 9 years to goDaddy. They don’t offer telnet access, so I uploaded the 191MB mysql file and gave them a call to submit a ticket. In less than 2 hours the import was done and I was very happy!

I’ve had a lot of people ask my why I use goDaddy, and I have to tell you…I have hosted with dozens of companies. I actually setup a small account with goDaddy 6 months ago just to try it out (it was only $3.99 per month). Now, 6 months later I’ve moved everything to them. All of my domain registrations, and all of my sites are completely hosted with goDaddy as of today (that big web site was the last one!). I was paying $29.99 per month for a professional webmaster account. I get the exact same thing (and more) with goDaddy for $14.99 per month. They have deals all the time where I pickup new domains for $1.99, and they had a special just today (when I called about the database import) where in July all web hosting was 25% off. It’s not just 25% off for new customers – it’s 25% off for ALL customers, whether you upgrade or just want to pay ahead. So I paid up a few months in advance to save even more money. That’s my shameless self-promotion for the day – use the host I use: www.GoDaddy.com

*UPDATE*

I almost forgot to add why this was a ‘blogging mistake’! I was going to migrate all of my web sites to goDaddy last month. I setup a new account a moved all but 2 or 3 sites – mainly because I couldn’t figure out how to get those huge databases off my old server and into my goDaddy account. Because I couldn’t figure it out my monthly hosting renewal came up, and I ended up paying for another month. Since I had a dedicated server (that I was grossly under utilizing), that blogging mistake ended up costing me $214!! Then I finally did a quick google search for “export large mysql database” and figured it out in an hour. My mistake was not doing enough research to fix the problem – it’s not like I was tackling something others hadn’t been through (and documented) before. Google is your friend – always search for solutions to your problems!

You know, I see a lot of “top wordpress plugin” lists on all kinds of blogs (including mine)…but I never seem to see a “wordpress plugin wishlist”. I got to thinking, if I had the money to pay someone, or could tell a really good coder the exact kinds of things I would like to be able to do in WordPress – what would they be? And from that – I’ve written my list of top wordpress plugins I would like to see created!

1. eBay Affilliate WordPress Plugin: For me, this would be the end-all be-all plugin to have. The holy grail of monetization for a blog. It probably wouldn’t be the top one for everyone, but look at the success of Auction Ads. They’ve now surpassed 25,000 publishers in just 6 months. That tells you that people were very hungry for an eBay affilliate tool (beyond what eBay provides). I mean, who wants to visit eBay and use the editor kit every time you want to add some relevant auctions to your posts and pages? I’ve done it many times and it’s a pain in the ass. I’ve tried auction ads, but all you can do is select blocks (like adsense sizes) to display auctions in dynamically using keywords. I want to have more control, and be able to drag and drop into posts and pages. Now then – if you haven’t already seen it, let me introduce you to the wp-amazon wordpress plugin. This is probably one of the best designed and easiest used wordpress plugin I’ve ever seen. Once installed, you just click on the amazon “a” in the right hand side of the page anytime your writing a post and a sidebar pops out with a search box. You enter some keywords, and results from amazon appear. You have a choice of small or large pic, and you drag and drop items (with your affiliate link included) into your blog post. You can drag pics, and you can drag (description) links. I suggested to the author that he take the exact same plugin and retrofit it to connect with and grab eBay items with your affiliate ID attached, but I haven’t hear much other than a comment that said “that would be cool”.Whoever creates this will be an instant wordpress plugin superstar, and probably get “dugg” to death literally over night.
2. Feedburner Admin WordPress Plugin: I have many sites and blogs, and many feedburner feeds. Even if I only had one site, I would want this plugin. There already a Feedburner Feedsmith WordPress Plugin, but I want a Feedburner plugin that will give me admin stats in my blog about rss subscribers, email subscribers, etc – right from my WP Dashboard (or sub-page). It would also be nice if this plugin would allow me to add other blogs I own, and view all my stats for different domains from one WP Dashboard – so I don’t have to visit every blog every day
3. Multi-Domain WP-Stat Dashboard WordPress Plugin: Piggy-backing on the last idea, I would like a WP plugin for my Dashboard where I could enter other blogs or sites I own, and the login info, and it would grab stats from them and display it on one page nicely formatted! Google analytics is nice and all, but a WP Dashboard page like this would show me at a glance where I had comments to approve, which sites were getting the most activity, the last time I posted, kind of like what bSuite does, but for multiple WP sites. Wow, would this one be a timesaver (for me)
4. Social Site Submission Plugin for WordPress: There a bunch of WP plugins that will put social site (digg, reddit, stumble, etc.) icons and links at the bottom of your posts. But I’ve yet to see one that will help you actually submit your posts to these sites. This would be another admin only plugin, but would let you enter your login info for digg, or reddit, or other sites, and submit a URL and a description. With an open URL text field, you could even submit other sites if you wanted from your admin panel.
5. Copyright Plugin for WordPress: I haven’t avidly searched for this, nor have I come across a plugin specifically for copyright issues. I would like a WP plugin that would handle all of your copyright needs for a blog. Like adding a copyright notice (with dynamic date) to the end of every post, and even in the footer. Who wants to copy and paste this every time or hardcode it in their theme? A plugin makes it easier to update and manage, and it will cover every post you’ve ever written (and your rss feed).
6. “FrontPage Designer” Plugin for WordPresss: This one is a bit harder to explain, and it would take a reaaaaally good designer and coder to build a nifty UI to pull this off. I will mangle the code in my WP theme when I have to. I prefer not to. I plugin that keeps me from puttering around in the code is awesome! I can’t tell you how many times I wanted to change the front page (or other pages / posts) of my blog. Not the sidebars – the frontpage. Most blogs are standard format….post, date, permalink, comment count….post, date, permalink, comment count. I have used plugins that would insert any content you wanted after the title, after the post, ramdomly in the post, etc. What I really want is a plugin that will allow me to change my front page layout anyway I want to. Think of it like an old school (html) wysiwig table editor. You start with a box. Then you add a table header (headline row). Then you split the bottom remainder into 2 columns. Then you split one of those cells into 2 more cells. Then you specify where the “posts” will go (top left, right, bottom, wherever…). And then you can specify (in the remaining cells) items you want and place text, code, or images there. You’ve probably seen WP blogs that looked like a newsletter, or sites that were using WordPress and you didn’t even know it (because of the non-standard layout). What did they do? They custom coded or masterfully hacked a wordpress theme. Give the power to the people man!! A plugin allowing you to “design your own FrontPage” would keep a ton of people from trying to hack up themes, and give them bunches of time back to keep writing content.
7. Google Webmaster Console WordPress Plugin: How many of us have gone out there and used seo tools to try and figure out our pagerank, or number of pages indexed, checked out our backlinks, or fervishly try to figure out what’s wrong with our google sitemap? Google are you listening? Get someone to create a wordpress plugin (during the “summer of code” maybe? probably too late…) that gives webmasters relevant info at the fingertips from within their blogs. How many pages do I have with pagerank? How many pages do I have indexed? When was I last crawled? Some of the key functions of the console could be available within a very nice WP plugin…..and we would be very happy!
8. Affiliate Earnings Plugin for WordPress: This is a simple one….very simple. Well, simple in theory anyway. I want to know how much money I’ve made (today, for the week, month, etc) at a glance. But I want to login to my wordpress blog dashboard and see all the places that I use. I want one dashboard with what I made from Amazon, from AdSense, from Kontera, from Commission Junction, and as many others as they could stuff into it. In plugin setup I would check off the monetization services I use, and from the WP Dashboard I could see at a glance whether I was making money or not. This would save me a whoooole bunch of time every week!

Alright – I promised you a Top 10 List of WordPress Plugins I’d like to see created….but I only have 7. But, I’m going to publish this post anyway – and based on YOUR SUGGESTIONS in the comments (or any other hairbrained ideas I come up with), I’ll udpate this post and add the last 3.

Please COMMENT NOW and tell me what you think of my suggestions so far, and what new suggestions you have!!

**DISCLAIMER: I don’t code or create wordpress plugins – there are merely suggstions for ones I would looooove to see someone create!

I’m the most successful blogger in North America – well, probably the world. I have monetized my blog so well that I’ve quit my job and now I earn six figures while typing on my laptop in my living room watching TVLand re-runs in my underwear with a beer at my side! I get so my comments to my posts that I have to shut down comments after only 48 hours, and my site gets so many hits that I had to lease a quad-processor dedicated server with 8gb of ram. Jason Calcanis, John Battelle, and GigaOM have nothing on me! Have you ever heard of a site with over 2 million backlinks?

That’s what most of us wish for isn’t it? I wish I was the most successful blogger in the world. Then again that would be an awesome responsibility. To have that many backlinks would require significant content. To have that many readers would require building a huge community. To have that kind of monetization requires planning, marketing, and probably some very experienced help. I think that as we (bloggers) build our sites these are admirable goals to strive for. Honestly, the majority of us will never become one of those top bloggers – but what’s the worst that can happen if we aim that high? At the very least, we’re going to be more successful than we currently are!

None of the top bloggers got to where they are without a significant amount of work. And none of them just did that work by the seat of their pants, I’m certain that each and every one had a plan. If the only thing sitting between you and the blog superstars is a methodical plan, and a regular system of work – what are you waiting for? I mean, the roadmap is already out there isn’t it? Maybe that’s why I’m not a blogging superstar, there really isn’t a personal ‘blogging roadmap to success’. What is out there are fragments….nuggets of gold and wisdom….1,000′s of posts full of enough knowledge to catapult your blog into the top 100 in the world. If you just keep reading them and implement haphazardly what you find as you find it (like I have) you will be more successful. But – if you want to be “uber-successful” you’ll need to spend some time physically writing down your personal blogging plan for success. Even if you read an entire book on the subject, or took a course – you would need to apply your situation.

My Blogging Mistake #1: Not Having a Written Plan to Follow

So – what am I going to do about it? I’m going to spend time developing one. It’s not something I can (completely) do in a day, and maybe not even in a week. I’ve read a lot of “blogging success” articles, which I’ll be linking for you in future articles. Each has contributed to what I think I need to do to make this a successful and profitable blog. For now, I’m going to start at a very high level.

I think my plan must consist of three very basic things:

1. Setup (20%)
2. Content (30%)
3. Promotion (50%)

Blog Setup: In my best practices for wordpress blog post, I talked specifically about how I setup this (and other) wordpress sites. I think that parts of that are “pieces” of what I would consider the “Setup” portion of a plan for blogging success. I think a lot of the points could be elaborated on, so I’ll spend future posts on that. Setup to me just means getting the blog up and running, but I also consider it maybe ‘blog maintenance’ over time….adding and removing plugins, refining your theme, backing up your database, etc. I put 20% next to setup, because I think that setup and maintenance are never ending, and in the end it can consume all of your time if you aren’t careful. One fifth of your time seems about accurate as far as the amount of time you should be spending setting up your blog. Any more and you won’t be writing or promoting anything. Any less, and your blog won’t be user-friendly enough to keep people reading.

Blog Content: Content is of course king, and what in the end your visitors are actually coming for. 30% of your time may seem low, but honestly your blog has to be setup properly so users can easily subscribe, comment, search, read, etc. And promotion is 50% simply because the most important thing of all is driving traffic to the site and then monetizing it (if of course that is your goal). 30% is not low if you consistently have high quality content.

Blog Promotion: For the reasons I just stated – blog promotion is (to me) the most important thing. This would include things like seo work, building links, commenting on other blogs, building relationships with other sites, writing articles for link directory submissions, analyzing stats and visitor habits, and ads, affiliate links, or methods of monetization.

Ok – so far my grand “plan for blogging success” only consists of three bullet points. But now I have a plan…..er…an outline rather. I have a starting point from which to build on. But it was obviously a huge blogging mistake not to have a plan at all. Fleshing out the details of that plan will be the subject of many future posts (I’m sure). Do you have a plan for blogging success? Will you create one now? Will you come back here to see how my plan is working out? Please comment now below…!

If you like this article and want to read more and follow my progress (and mistakes) – please signup for email subscriptions at the top of the page, or use the rss icon to subscribe to the rss feed.

If you are not using 301 Redirects you are making a blogging mistake. 301 Redirects are easy to setup, save your google pagerank and link love, and save your visitors from a nasty “Error 404 – Not Found” page. What is a 301 redirect? It’s a single line you add to you “.htaccess” file (more on this later) that says ‘anyone that visits that URL – send them to that URL instead’. The redirect can happen witin your site, you can redirect to an external site – it’s basically from any URL to any URL.

Here are some scenarios where you would use a 301 redirect:

1. You’ve changed your permalink structure
2. You’ve accidentally deleted a post and need to recreate it
3. You wish to change the post slug (url) to make it more seo friendly
4. You have a bunch of 404′s for pages that don’t (and maybe never) existed
5. You recently moved from one blogging platform to another (drupal, blogger, movable type, etc., to wordpress)
6. You changed, recreated, or renamed your categories

If you’re using WordPress, you can try and install a module to add redirects directly from your admin panel. I personally have not had luck with this plugin, but some have – you’re welcome to try the Objection Redirection WordPress 301 Redirect Plugin and see if it works for you.

If it doesn’t (or if you’re not using WordPress), you’ll need to edit your .htaccess file yourself. If using WordPress, you should already have a file called “.htaccess” in the root of your web site. If not, you’ll need to create it. Open up any text editor (notepad) on your PC. You’ll need an FTP account and FTP access to your web site to do this. If using WordPress, download your .htaccess file. It should already look something like this:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

If you’re not using WordPress, download your .htaccess file and open it (or create a new one if you don’t have one already). Basically – you’re going to add one line for each redirect. It’s very simple:

Redirect 301 “/blog” http://www.yourwebsite.com
Redirect 301 “/page.html” http://www.yourwebsite.com/page.html
Redirect 301 “/subdir/oldpage.html” http://www.yourwebsite.com/subdir
Redirect 301 “/blog” http://www.anotherwebsite.com

I tried several ways to do this, and this is the one that works for me every time. I’m hosted on a Linux server running Apache. Every line starts with “Redirect 301 “. Then (always in quotes) is the path on your server to redirect. I always start it with a slash (that means from the root of your site). There is no need to write out your full domain name in the first part. You can redirect a directory, a page, a page in a sub-directory, etc. Then just write the full URL to the page you are redirecting to. You can redirect to a page in your site, a sub-directory, or a page in a sub-directory. You can even redirect from your site to another (external) web site.

That’s it! Save you file and FTP back to the root of your web site and you’re done!

The reason I had to setup some 301 Redirects is because I migrated one of my web sites from Drupal to WordPress. With the date based permalink structure in WordPress, the URL’s of all of my important pages with pagerank was screwed up. They say if you redirect a page that had google page rank (to a new page) the (old) page rank will be passed on to the new page. I can’t verify this myself, because my redirects haven’t been in place long enough to tell. I will update after they’ve been online awhile.

So in the last article I talked about the rapid growth of my web site, and left handing with going from 10-15,000 unique pageviews per day to just over 500. So how does a web site just “die on the vine” like that? I guess it’s pretty easy when google attacks!! What I mean is, I had described to you that a couple months into my web site traffic growth google had contacted me for an interview. They even gave me a nice $50 amazon.com certificate for my time! I now realize that basically google had noticed how rapidly my web site traffic had grown – and wanted to know all the details I would give up regarding how I did it……SO THEY COULD STOP IT FROM HAPPENNING!!

Now, I’m not bitter about this – in fact I’m glad it happenned. I entitled this blog “learning everything about blogging the hard way”, because I learned “how not to blog” first. I went for the quick buck, the fast traffic – and I got burned…but I deserved to be! What I take issue with is the fact that when I was interviewed by google (the adsense team) they acted as if I was a ‘great performer’, never indicating that what I had done was potentially bad or wrong. In their defense, the people who interviewed me may only been told to interview me…but I have to believe that they reason they wanted my blueprint for success was to analyze it (and if it was bad) throw a monkey wrench in it. Months later my site just died, and I (at first) didn’t know why.

Here’s how it went down. After 6 months of outstanding traffic growth, I get an email letter from google saying I wasn’t complying with their (adsense) terms of service and had five days to comply. I emailed back, complaining that I didn’t know what to comply with (could they give me a clue). Their reply (that I was surprised to get) said that basically I had too many “adult content” links and posts on my site, which was not allowed, and I would need to remove them asap. They gave me 2 URL’s as examples, and told me to remove the adsense ads from the pages with questionable content. I hadn’t mentioned yet that one of the highest trafficed pages on that web site was one about a certain celebrity named “britney” and her getting out of a car showing her mommy bits to the world. Remember, I was crudely tracking popular keywords, phrases, and stories, and celeb scandals are always top of the flavor of the day. I wrote a short article about the debaucle, pointed where to find the pics, and that was that. Little did I know that one page would garner almost 1 million unique pageviews in only 4-5 months. That one page probably generated half of my traffic, and half of my revenue. It probably is also the entire reason I was contacted by the adsense team regarding the TOS violation. I’m sure that those kinds of traffic numbers, combined with those kinds of adsense earnings raised all kinds of flags.

Now, I didn’t really want to completely remove the pages with the content – because (being greedy) I wanted to continue generating the traffic. So I went through my site with a fine-toothed comb and removed the adsense ads from every single page I could find with even remotely racy or adult content (just like they asked). REPEAT – I REMOVED the adsense ads from all these pages. I had planned to just monetize them in some other way later on. I also removed all the links to these pages from my sidebars, etc. I replied to the adsense team and told them that I complied – but apparently that wasn’t good enough for them, 24 hours later I received an email stating that the site was no longer part of the adsense program due to TOS violations. It also stated that my account was still open, and I could continue to participate with my other sites, but THAT site could not use adsense ad all.

So that was that….since adsense was my primary stream of income on that site, removal from the adsense program pretty much killed it. I began to do reading on various blogs about the subject. I also did some digging on my site, and found that all of my search engine listings were (I thought at first) non-existent. It turns out that they seem to have been hit by the now infamous “google penalty”. What is a google penalty? There are many speculations, but the most talked about are the “-30 penalty” and the “-950″ penalty. There is no official documentation on these, but they are much talked about in various webmaster forums, including the google webmaster groups. Basically, if your site has done something google believes is some type of “cloaking”, “spamming”, “paid link exchange”, “excessive affiliate linking”, or other unethical means to boost your site in search rankings – you will receive a “google penalty” in their search index. The -30 penalty is the lighter of the two, where you site is supposedly knocked down 30 results (so you won’t appear in the top page or two). The -950 penalty is of course the worse, where you are still listed in the index, but relegated so far back in the results pages that no one will ever find you. I found that my site was still indexed, but the pages were just about dead last in the results pages, so I assume that I received the -950 penalty.

I can’t say that the 2 events went entirely hand-in-hand (the -950 penalty and removal from adsense, all within 24 hours) – but they were most certainly related. I’m certain that they was some sort of report or hand-off of information from the adsense team to the google index team to get them removal and the penalty all at once. As I said – I began doing some reading on blogging, seo, google penalties, indexing, and related topics. This is when I first learned (believe it or not) about Matt Cutts and other experts in the field of SEO. Many will not believe I didn’t know who he was before, but since my site was doing so well, I never really had to look into SEO or read much about successful blogging. I thought I was a successful blogger, when in fact I was an accidental successful splogger.

So that’s it, this is the end of my 4 Part series of “How I started blogging” (be sure to read the first 3 posts if you haven’t already). I learned how to do everything the wrong way, my “spammy” sites have been killed off (traffic wise), and I’m left to regroup and begin again. I have decided to be an ethical blogger. I’ve been reading all the pro sites, from problogger to shoemoney, and I’m excited about beginning to build quality sites from scratch. This site is the first one of many (I’ll be building other niche sites on topics I’m passionate and knowledgeable about), and I’ve already written several other posts about best practices getting this one setup. This site will consist of posts telling the tales of my success (and failures) as a blogger. Maybe someday I too will be a ‘pro blogger’.

What happenned to the web site that died? I had more than 43,000 pages when I stopped it from aggregating. I have been removing the pages by hand, and know I’ve got it down to about 10,000 pages. Once I finish removing all the aggregated posts it will probably only have under one to two hundred original content pages. I’m also only keeping only the original pages with truly useful content. The ones written just for the purpose of generating traffic are or will be removed for good. Getting that (and a few other) sites back in google’s good graces and full of original content will surely be the subject of future posts.

So in the last post I talked about finding an aggregator plugin for Drupal that aggregated rss feeds and posted them to the front page. As I said in the last post – this would change everything. When I enabled the plugin, I found that it would publish each item in an rss feed and turn it into a distinct page, with a distinct URL, and link back the original article. I quickly setup some tech feeds to see how it would work. In the beginning, as I remember, there were a few issues getting it to work properly. I think once the plugin had an update available – it worked fine. And in no time, I had turned my site into a new portal of sorts. I put in some of my favorite feeds and the amount of pages in my site began to grow, and at it’s peak at the rate of 100-200 posts per day.

I commented on the posts that I found interesting, and continued to write original content articles both in articles, and on a ‘blog’ page. I began to see the traffic on my site rapidly grow from a couple hundred unique views per day to a couple thousand. Then within months it was 5,000, then 10,000, and it topped out about 30,000 unique pageviews per day about 6 months later. The traffic was so great I had to change servers 3 times. My adsense revenue was growing as well in spurts…$100 one month, then $300, then $600, then $900, then $1,200 – I think the highest it ever got was $2,000 in one month at the peak (6 months later). I also signed up for bunches of affiliate links and added them wherever I could find a relevant connection or post. Oddly enough, the adsense was so targeted that click-through rates were high, but even despite loads of traffic my hand-picked affiliate links never really amounted to much. I signed up for reciprocal links exchanges on a few sites and added them to my high traffic pages with pagerank to try and getting even better PR.

When my traffic had first stared to spike (after the first 60-90 days or so) I was contacted via email by the google adsense team. They wanted to setup and interview over the phone to talk about my success! I was excited (to say the least), and felt like I must have achieved some kind of amazing feat (generating so much traffic in so little time) for them contact me directly. We setup an interview via phone, and I spilled my guts. I told them exactly what I had done, how I aggregated content and commented on the articles, how I picked keywords to craft specific stories to generate traffic, other ways I was trying to monetize besides adsense, and how I was trying to turn my site into a “supersite” news portal. Looking back on it, I was pretty nieve (more on this later…).

During the course of the user growth I began watching the things that people searched for. I viewed some lists I found with top search keywords. I aggregated some rss feeds that contained not only tech stories, but ones that were just plain popular (flavor of the day) from social sites. I began to craft ‘original content’ pages that centered on popular keywords and culture – and that was the beginning of the demise of my site. It was also the beginning of ‘greed’, and I was writing purely for traffic and profit – and not for people anymore. I fooled myself into believing that I had created a news portal, when in fact I turned it into a news scraper, spam, or splog (spam blog) as they’re sometimes called. Then it happenned, I was blindsided about 6 months after my growth began. I say ‘blindsided’, because I never saw it coming…I never thought my site (with 10-15,000 unique pageviews per day) would rapidly decline in about 10 days to just over 500 uniques per day. You’ll have to read the next post (Part 4) to see how it happenned and why!

I’m going to talk about setting up a robots.txt especially for your self hosted wordpress blog, to help the search engine crawlers to best index your site and help with with search engine optimization. Due to the recent content duplication rules in the google index, you want to make sure that you’re submitting one version of your posts/pages, and also that the crawler isn’t trying to index pages it really does need to at all. Pages like trackbacks, admin, includes, and your rss feed.

It seems from reading many blogs and postings that not everyone agrees about category pages. I’ve heard some say that they want their category pages indexed – and that helps them. I think it seems to depend on the site, and how you have been tagging things. Sometimes on some of my sites I go overboard on tagging, so I end up with a ton of category pages. And also, many times I tag things in many different categories. Having a post have it’s own page, be listed on the front page, and 5 category pages wouldn’t seem to be a very good plan for good seo and an obvious setup for content duplication (in my eyes). So just to be safe, I filter out my category pages too in my robots.txt.

First, I read over or Lorelle on WordPress (link in sidebar) that now google has sitemap inclusion, and you can add this line to your

robots.txt file:
User-agent: *
Sitemap: http://www.jtpratt.com/sitemap.xml

and you no longer have to submit your sitemap (the crawler will know what to do with it). So this is a new entry for me. I also read that you can tell the google image crawler where to (and not to) go in your site, so I added this:

# The Googlebot-Image is the image bot for google
User-agent: Googlebot-Image
# Allow Everything
Allow: /*

I also saw that can do the same for the adsense crawler, which has nothing to do with indexing, but if you use adsense it would be smart to have this as well:

# This is the ad bot for google
User-agent: Mediapartners-Google*
# Allow Everything
Allow: /*

So these are all new entries for me. Now daily blog tips (link in sidebar) has a quick, down and dirty post on a robots.txt file for wordpress. It’s pretty simple:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

I kinda like that, but it doesn’t seem to cover everything. Fili’s Tech has an article on wordpress seo for wordpress too, and I like his ideas. So I ended up with something like this:

# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/

# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$

# Disallow parsing individual post feeds, categories and trackbacks..
Disallow: /trackback/
Disallow: /feed/
Disallow: /category/

For right or wrong, I have one section for:

User-agent: Googlebot

and another section for:

User-agent: ia_archiver
User-agent: Scooter

User-agent: Atomz
User-agent: FAST-WebCrawler
User-agent: ArchitextSpider
User-agent: Googlebot
User-agent: Slurp.so/1.0
User-agent: Slurp/2.0j
User-agent: Slurp/2.0-KiteHourly
User-agent: Slurp/2.0-OwlWeekly
User-agent: Slurp/3.0-AU

User-agent: UltraSeek
User-agent: MantraAgent
User-agent: Lycos_Spider_(T-Rex)
User-agent: MSNBOT/0.1
User-agent: Gulliver
User-agent: Scrubby/
User-agent: ZyBorg

If you have any comments, improvements, or suggestions – please comment now!

Related:

• Best Practices for Setting Up a New WordPress blog in 60 Minutes or less

June 8th, 2007 | Posted in wordpress seo, wordpress | No Comments

So in part 1 I talked about going from a static site to a dynamic one, and then reading a blog post about making a little commission from an Amazon Associates link. I had started to think for the first time about monetizing my site. Never having had ads of any kind on any of my sites – there looked like there could be a little potential here. First I figured Amazon sold books – and all geeks read books, especially technical ones. Why couldn’t I add links to Linux books at the end of my Ubuntu posts? So I did. I had written one very lengthy article about my adventures converting video files in Linux on my Ubuntu box. An O’Reilly book about hacking multimedia in Linux had helped me a lot, so I linked it at the end of the aritcle. I ended up not only selling a few books per month, but sometimes when people clicked the book link – they bought other things as well. I didn’t make a lot, but it amounted to maybe $25-$30 every 3 months. Not bad for a single page.

A few months after that I wrote the article about my Sprint A920 cell phone and learned about Commission Junction. Commission Junction (CJ) has thousands of separate companies you can link to and sell products for. I had wanted a car charger for my phone, and purchased one online that had free shipping. I saw an “affiliates” link at the bottom of their page, and when I clicked it said to sign up at CJ to promote and sell their merchandise. So buying that charger actually led me to find about about CJ.

I waited until I got the charger, was pleased, and then signed up as an affiliate with CJ. Then, on my cell phone article page I linked to the chargers and other accessories for the phone. I had a really good article about my cell phone, but wasn’t really getting any traffic from search engines or anything. All the tips and tricks I learned in cell phone forums, and after awhile I became knowledgeable enough to be able to answer questions. I began posting answers and help in some of the more popular forums, and after I answered I would post something like – ‘check out my help page here’. Pretty soon I was getting some action on my cell phone page. It started at 50-100 unique pageviews per day, and within a month was seeing 1,000 visits per day. For awhile (when the phone was popular) I was making $30-$50 per month commission from the cell phone accessories on that one single page. Now I had 2 distinct pages that were driving people to my web site. I wrote some more how-to articles about Ubuntu Linux and linked more books. With the additional traffic I was making $50 or more per month consistently now. It was about this time I learned of a new aggregator plugin for Drupal that would suck in rss feeds and post them to the front page on the web site automatically. This would change everything (more to come in part 3)…

I’m going to talk about a lot of things in this blog about blogging, but let’s start with the basics – how I got started. I am a web developer and I build and maintain web sites by trade, and I’ve been doing that in the corporate environment since 1996′ish. I stopped building web sites (on the side) for the public Internet probably in 2002, but kept a domain out there for myself (www.smorgasbord.net) in various states of disrepair.

About two years ago I was introduced to drupal, which is a content management system or CMS for web sites. Before Drupal my site was a mixture of hand coded PHP and html. When I deleted the last hand coded version of my site almost 2 years ago and installed drupal I thought it was going to be much easier to maintain my web site. I was tired of hand coding html, and I was tired of programming custom PHP. I thought it was a good choice at the time, but drupal was as mature as I thought it was. Drupal has a learning curve that’s a bit steep – as far as figuring out how to customize it to your liking and how to present it the same way a static site would be.

At the time I first started using it – an ‘aggregator module’ hand been added and for the first time you could give it rss feeds and it would grab headlines for you and publish them in your site. I’ve been using bloglines for years to read feeds, and the thought of being able to add my favorite feeds into my own web site, and comment on them was appealing. I added in a few to test it out. All you could really do was add the latest 10 to the sidebar or view the aggregator page itself to view them. You couldn’t really comment on them like I wanted to either. At the time I also started using Ubuntu Linux, so I started writing a few articles about that and also started a ‘book’ in my site about things newbies to Ubuntu should learn.

So basically now I had switched from a traditional web site – and I was blogging. Quickly posting information, some short funny videos, informative articles…whatever came to mind. About six months went by, and I wrote an article about a new cell phone I got. It was a very long post, and I kept adding to it all the new features I was learning about the phone, and all kinds of tips and tricks. About this time I read a post by Jason Kottke about how he had unexpected earned some money with his blog by using the Amazon Associates program. I think he had some links to books, and someone did some shopping for a very expensive printer after clicking on the link to amazon from his site – which earned him a nice little commission. This got me thinking about my site in a whole different way (to be continued in part 2)…