June 08, 2007

The story of the Kelsy Smith website

We built a site that went from launch to the #5 search term on Google and then to obsolete in three days. Here's the (updated) story ...

The first version of the Kelsey Smith website, made by a family friend on Monday and hosted at sitesled.com, was barely more than a web version of the "have you seen this person?" poster. As the friend added pictures to the site, it eventually did not perform well, even though it had only light traffic at the time.

Around 2:00 pm Monday my wife called me from the Smith home asking for help. We jumped right in and bought the domain findkelsey.com, quickly built a static HTML site (see mirror), and put it up on four servers at Vine Hosting, using DNS round robin for load balancing. The site went live around 5:00 pm and averaged 20+ page views per minute through the remaining waking hours of Monday. Not bad ... and we were grateful to have the opportunity to help with the search for Kelsey in this small way. Little did we know we were about to experience an exponential explosion in site traffic unlike anything any of us had ever seen.

Through the day on Tuesday the traffic increased by a factor of 7 to an average of 140+ page views per minute. We got as many page views on findkelsey.com Tuesday, 160,000, as we get in an average month at www.cor.org. Our infrastructure held up very well and we felt that this level of interest in the site represented a tremendous return on our investment in ministry.

Then Wednesday morning traffic increased by another factor of 7 to 1,000+ page views per minute and we had a serious problem on our hands. As I mentioned, the traffic growth was so rapid, at first Vine Hosting thought we were experiencing a DDOS attack aimed at findkelsey.com. Every site they host was being slowed to a crawl. We quickly became concerned that the bandwidth consumption could result in a huge bill for overage charges. Looking back now, we think it's likely the servers were just fine but Vine's WAN link was saturated (which, by the way, seriously hampered our efforts to fix the problem as you can imagine). We discovered later that "findkelsey.com" had become the #5 search term on Google and that the site had 11,000+ other pages linking to it.

As a practical matter, findkelsey.com was very slow to display for most site visitors, if it displayed at all. Vine was in serious trouble and most people weren't getting to the site anyway. Consequently, I felt the only good option was to take the site down from our servers at Vine and move it to Blogger, which is owned by Google. Only Google had the capacity to handle the traffic while being affordable (free). While we were moving the site, between 2:00 and 5:00 pm Wednesday, it was mostly inaccessible. Meanwhile, Vine had to do a great deal of work to restore full service for its customers while traffic slowly faded away as the DNS changes propagated. In an urgent situation like this, DNS propagation delays are terribly frustrating. There's nothing like taking down a site, rebuilding it, and moving it WHILE it is getting bombarded with thousands of page requests per minute. It's truly a mind boggling challenge.

During the three hours the site had been (mostly) down, Kelsey's body was discovered and the news quickly spread through the media. Around 5:00 pm when we had the site rebuilt and the DNS redirected, immediately we again experienced the 1,000+ page views per minute we had seen in the morning. Just as immediately, expressions of love and support began pouring in through Blogger's comment feature at the rate of 1 every 10 seconds. People came to the site from across the country and around the world to express their love, comfort, sorrow, and outrage. Those comments continued through Thursday and gradually tapered off to less than 1 every 5 minutes late Thursday night. In 30 hours the site received a total of 2,700+ comments that we hope will be a great encouragement to the family when they eventually have time to read them and reflect.

As the comments came flooding in, we realized that our ministry was not only to the Smith family to aid in the search and the immediate aftermath, but also to the public to provide a way for people to express themselves to the family. Other than the 1% we had to remove because they were inappropriate, the comments were overwhelmingly supportive. (Understandably, some expressed uncensored anger towards the perpetrator. Those we didn't remove.)

Thursday afternoon we got word from the family that the term "Find Kelsey" no longer fit their situation. The volunteers during the search had nicknamed themselves "Kelsey's Army" and that was the domain name the family wanted to use going forward. In light of the major challenges we had experienced with traffic, I advised against redirecting findkelsey.com to the new kelseysarmy.com. Instead, I posted a message at the top of the page referring people to the new site and indicating that comments would be closed.

And that's how it ended. The site went live on Monday around 5:00 pm and was mothballed late Thursday night. It was in production just over three days, and yet the ministry value it created in that short time was immeasurable.

No comments: