We've had two long, frustrating years of servers going down.
It's been painful as Brian and I have had heated discussions (we don't really fight) over and over again about it. It's been the biggest bone of contention for me and a major stress point for our business--we do everything else perfect, why can't our servers be perfect?!!?!?!?
The truth is Brian and I have just never been good at keeping servers up for whatever reason. Part of this was we used Microsoft server software not Linux (we've since made the switch), and of course the fact that Mac folks are total freaks who sit there and hit the refresh key 10x a minute during the Steve Job keynote doesn't help.
Last week we were up and running for all of CES (unlike last year as some of you old-school folks remember), and today--just hours ago--we passed the second test: the Steve Jobs Keynote. With hundreds of thousands of page views over an hour our servers looked like we were about
This is all thanks to the hard work of Brian's team--including Gavin, Alex, and 394 people at AOL's server farm. They really did it.... we stayed up, and we were fast--like Google fast!
One of the best feelings in business/life is to turn your biggest flaw into your biggest strength. That's one of the things I love to do... figure out what we really suck at and figure out how to make that the thing we're best at (I'm going to come up with a term for it). We just did that: three months ago our server skills were the worst in the industry, now our server skills are second to none.
OK, enough kudos... let's fix the next thing! :-)
We did it! (or Turning your biggest weakness into your greatest strength)
Reader Comments
(Page 1 of 1)2. Keeping it up and running! Well - it arent that easy to keep a big network up and running. I've tried running one of my countries biggest community sites - keeping it alive on hardware for under $5k and still /spitting/ 1,5 million dynamic pages out + 20 million static requests. It was fun to try to optimize that - both databases, httpd servers, hardware. It was actually 2 httpd servers, 1 database server and 1 old machine for the banners. I just love to get the most out of standard hardware. I learned a lot of tricks - f.ex. memcache Jason mentioned in a posting before today. ==Stefan
Posted at 3:32PM on Jan 10th 2006 by Stefan Andersen
3. You should consider having a live events system. Besides being easier on the server if the page is static (you can serve a TON of static traffic off a single server or two) it's much easier on the reader. It was hard to follow Engadget's coverage because updates were in the middle of the page--below all the old stuff and above all the comments. In a perfect world there would have been something like most of the Mac sites--a simplified page that listed updates from newest to oldest and no comments (at least on the main page, break those off to another section). Or you could also do something clever like MacRumors and have the updates come in via AJAX. No need to F5 it :).
5. I loved Engadets coverage last year, but this year Macrumors had far superior coverage with their simplified page and clever use of AJAX. At one stage Engadgets coverage was lagging poorly behind. Not knocking the guys because I'm a regular reader and liked seeing pics from the keynote. Just pointing out that there was better delivery elsewhere, lessons to be learned. C'mon now, with the Intel rumours nobody could guess that they'd be a huge load? ;) btw never used F5 on the macrumors site once.
6. If you guys would list keynote play-by-play from newest to oldest, it would really help! You have to scroll to the bottom of the page every time you refresh. I agree that MacRumors handling of the Jobs keynote was excellent, and their use of a "countdown to update" cut down on my freakish manual refreshing. =)
Posted at 5:48PM on Jan 10th 2006 by Colin
7. I'll probably end up doing a case study for microsoft this year... 3 servers 1. Mail server sends out 700,000 emails a day and handles 70+ million pageviews/day, 5.5 million pageviews/hour at peak from Instant messenger polling, tiny code but huge traffic. (2 CPU's) 2. Webserver handles 12 million pageviews day at 50% CPU and 750k pages/hour at peak. Intensive code... (2 CPU's) 3. DB server to handel the database stuff, its around 40% usuage. Dual core Quad Opteron. There isn't a single competitor of mine my size that doesn't have over 100 servers, some of them have over 300 and they are smaller! What i've learned is that website optimization begins and ends at the database, if you can keep all your queries executing under 10 milliseconds you can scale a single server to 10's of millions of pageviews. Lets not forget that 12 million pageviews a day is only 230 pages/sec during peak which is peanuts for todays machines.
8. When I switched from Windows to Linux server it was a big headache, luckily I had had made a hard copies of of all data before doing the transfer. It all cached big time on me, the backups saved my day! Today I do daily automatic backups just to be on the safe side.
Add your comments
Please keep your comments relevant to this blog entry: inappropriate or purely promotional comments may be removed. Email addresses are never displayed, but they are required to confirm your comments. To create a live link, simply type the URL (including http://) or email address and we will make it a live link for you. You can put up to 3 URLs in your comments. Line breaks and paragraphs are automatically converted — no need to use <p> or <br> tags.
1. Switching from Windows to Linux is half the battle. Then there is all the load balancing stuff, which I admit I've never done, but can't be that complicated. I manage many linux servers with only of them being a Windows box. Would you like to take a guess at which is the only one that has ever crashed? :) Congrats, on keeping engadget up, the Engadget servers seemed very hard to get too when the keynote first started, but it sooned cleared up for me. The engadget team has been doing an amazing job covering CES and Mac World, nobody even comes close to providing the coverage they are. Jeff O'Hara http://blog.zemote.com
Posted at 3:04PM on Jan 10th 2006 by Jeff O'Hara