The story of the Kelsy Smith website

We built a site that went from launch to the #5 search term on Google and then to obsolete in three days. Here's the (updated) story ...

The first version of the Kelsey Smith website, made by a family friend on Monday and hosted at sitesled.com, was barely more than a web version of the "have you seen this person?" poster. As the friend added pictures to the site, it eventually did not perform well, even though it had only light traffic at the time.

Around 2:00 pm Monday my wife called me from the Smith home asking for help. We jumped right in and bought the domain findkelsey.com, quickly built a static HTML site (see mirror), and put it up on four servers at Vine Hosting, using DNS round robin for load balancing. The site went live around 5:00 pm and averaged 20+ page views per minute through the remaining waking hours of Monday. Not bad ... and we were grateful to have the opportunity to help with the search for Kelsey in this small way. Little did we know we were about to experience an exponential explosion in site traffic unlike anything any of us had ever seen.

Through the day on Tuesday the traffic increased by a factor of 7 to an average of 140+ page views per minute. We got as many page views on findkelsey.com Tuesday, 160,000, as we get in an average month at www.cor.org. Our infrastructure held up very well and we felt that this level of interest in the site represented a tremendous return on our investment in ministry.

Then Wednesday morning traffic increased by another factor of 7 to 1,000+ page views per minute and we had a serious problem on our hands. As I mentioned, the traffic growth was so rapid, at first Vine Hosting thought we were experiencing a DDOS attack aimed at findkelsey.com. Every site they host was being slowed to a crawl. We quickly became concerned that the bandwidth consumption could result in a huge bill for overage charges. Looking back now, we think it's likely the servers were just fine but Vine's WAN link was saturated (which, by the way, seriously hampered our efforts to fix the problem as you can imagine). We discovered later that "findkelsey.com" had become the #5 search term on Google and that the site had 11,000+ other pages linking to it.

As a practical matter, findkelsey.com was very slow to display for most site visitors, if it displayed at all. Vine was in serious trouble and most people weren't getting to the site anyway. Consequently, I felt the only good option was to take the site down from our servers at Vine and move it to Blogger, which is owned by Google. Only Google had the capacity to handle the traffic while being affordable (free). While we were moving the site, between 2:00 and 5:00 pm Wednesday, it was mostly inaccessible. Meanwhile, Vine had to do a great deal of work to restore full service for its customers while traffic slowly faded away as the DNS changes propagated. In an urgent situation like this, DNS propagation delays are terribly frustrating. There's nothing like taking down a site, rebuilding it, and moving it WHILE it is getting bombarded with thousands of page requests per minute. It's truly a mind boggling challenge.

During the three hours the site had been (mostly) down, Kelsey's body was discovered and the news quickly spread through the media. Around 5:00 pm when we had the site rebuilt and the DNS redirected, immediately we again experienced the 1,000+ page views per minute we had seen in the morning. Just as immediately, expressions of love and support began pouring in through Blogger's comment feature at the rate of 1 every 10 seconds. People came to the site from across the country and around the world to express their love, comfort, sorrow, and outrage. Those comments continued through Thursday and gradually tapered off to less than 1 every 5 minutes late Thursday night. In 30 hours the site received a total of 2,700+ comments that we hope will be a great encouragement to the family when they eventually have time to read them and reflect.

As the comments came flooding in, we realized that our ministry was not only to the Smith family to aid in the search and the immediate aftermath, but also to the public to provide a way for people to express themselves to the family. Other than the 1% we had to remove because they were inappropriate, the comments were overwhelmingly supportive. (Understandably, some expressed uncensored anger towards the perpetrator. Those we didn't remove.)

Thursday afternoon we got word from the family that the term "Find Kelsey" no longer fit their situation. The volunteers during the search had nicknamed themselves "Kelsey's Army" and that was the domain name the family wanted to use going forward. In light of the major challenges we had experienced with traffic, I advised against redirecting findkelsey.com to the new kelseysarmy.com. Instead, I posted a message at the top of the page referring people to the new site and indicating that comments would be closed.

And that's how it ended. The site went live on Monday around 5:00 pm and was mothballed late Thursday night. It was in production just over three days, and yet the ministry value it created in that short time was immeasurable.

Kelsey update

Where do I begin?

Many of you no doubt have already heard that Kelsey Smith was found murdered today. My wife, Laura, was at the Smith home when the detectives came with the news that a body had been found and it was likely Kelsey. The identification was later confirmed. Laura and my daughter, Beth, are attending a prayer service at Hillcrest Covenant Church as I write this. Prayers are flooding in from across the country and around the globe. Please add your prayers to them. I will have much more to say about this in the coming days.

Quite aside from the unspeakable human tragedy, my technical team and I learned a great deal through this. The traffic to www.findkelsey.com (see mirror of original HTML site) was 160,000 page views yesterday and grew exponentially this morning. By 10:00 am we had serious performance impact, essentially taking down Vine Hosting, the ISP where our web servers are located. Searches for findkelsey.com reached #5 on Google as reported by Google Trends. You read that right. Our little hand-built quickie site was the #5 search on Google today. (Right now "Kelsey Smith" is #3, "find Kelsey" is # 7, and "findkelsey.com" is #9.) The flood of legitimate traffic was so great, Glenn - the director of operations for Vine Hosting - at first thought it was a DDOS. In reality, it was all legit traffic. Too much traffic. Glenn is still dealing with the aftermath.

The only people with enough resources to absorb this kind of traffic is, that's right, Google. Realizing the disaster we were causing for Vine, I made the command decision to totally take down all traffic related to findkelsey.com at Vine, even the DNS. Then, as rapidly as possible, we created a new site on Blogger (which is owned by Google), changed to a high-volume DNS host, and pointed the DNS at Blogger. As soon as the DNS change propagated, we started getting comments on our posts. In fact, my first indication that we finally had things working was when a comment appeared in my inbox. It took almost another hour to figure out how to get most traffic coming to us as it should. In the following 2.5 hours, 754 comments have come in from around the country, directed to Kelsey's family. That's one comment every 10-15 seconds. Unbelievable.

Today I had the experience of building one of the highest traffic sites on the Internet. I was building it WHILE it was being hit.

Find Kelsey Smith

My daughter, Beth, was in band at Shawnee Mission West High School with Kelsey Smith, the 18 year-old girl apparently abducted from a Target parking lot here in Overland Park on Saturday night. The band has a very active parents' organization called Band Aides West. Within 12 hours of Kelsey's disappearance, Band Aids was taking the lead to organize the search and support the family. Band Aids asked my wife, Laura, to go to the Smith home yesterday and take a shift answering the phone. After arriving there, Laura became aware of a need for a web site about Kelsey that could handle the load coming from national news exposure, including Greta Van Susteren and Nancy Grace, as well as extensive coverage from the local media.

Through the intense efforts of Brian, Leo, and Chuck http://www.findkelsey.com/ (see mirror of original HTML site) went live yesterday around 5 pm. Glenn Kelly and Vine Hosting are providing hosting at no charge. Glenn even missed part of his daughter's second birthday party to help set up the site on multiple servers to handle the load.

The site is linked from the home page of the Kansas City Star, the story about Kelsey on Fox News, and many other sites. It is also on flyers being posted all over town. The site is now averaging 2.6+ page views per second during waking hours.

Please stop what you are doing right now and pray for Kelsey, for her family, for the police, and for all of the volunteers who are supporting them and helping search.

KC RoundTable update

The KC-area CITRT met for lunch today. It was a blessed time with Mike, Ron, Jason, Jim, Warren, and Dwight, along with my Resurrection staff. I reviewed with them our plans for the Fall RoundTable and got their feedback. We are VERY excited about what God is doing in the church IT world and what will happen here at Resurrection the first week of October. I can't wait!

Our next meeting will be at Indian Creek Community Church on June 28 at 12:30 pm. All those interested in attending should RSVP to Jason Reynolds, IT Director at Indian Creek. Geeks for Jesus!

ESX update

Our VMware ESX project is well underway. Ian, our server/network admin, and I have been spending a great deal of time communicating with our ESX consultants and documenting both our existing configuration and our intended configuration when the project is complete. A detailed plan is beginning to emerge.

While we're at it, we decided to upgrade the power distribution and core network switching in our data center. We have temporarily stacked two APC 2200 UPSes, the new Dell/EMC AX150i SAN with its dedicated UPS, and both the existing and the new Dell 2950 servers. All of this will be permanently installed in our new Dell cabinet when it arrives this week. We also installed a second HP 4104 switch to provide redundancy for both the SAN and the primary network.

Our consultants have loaded ESX Server onto the new 2950 server. The next step is to install Virtual Center onto a desktop computer that will be used for management. Once that is done, we will avail ourselves of Dell's remote install assistance they bundle with every AX150 sold.

Our consultants will be in all day Friday and Saturday to complete as many of the remaining tasks as possible. Our goal is to have at least one or two virtual servers running on the new 2950 by Saturday.

Weigh in on Clif's smartphone decision

It's finally time for me to upgrade to a smartphone. I've never had a smartphone before. I used a Palm Pilot for almost four years until it died a couple of months ago and my cell is a 5 year-old workhorse that's nothing more than a phone. I definitely need the phone to have EVDO Rev. A for use as a modem for my laptop. I use Sprint and I'm not interested in Blackberry. I've read a lot of reviews, narrowing my short list down to the Treo 755p, the Motorola Q, or possibly even the PPC 6700.

I need real-world advice from smartphone users. Should I assimilate into to the Treo club? Should I go sleek with the Q? Or should I put a priority on WiFi and choose the PPC 6700? Should I wait for Windows Mobile 6? Should I stick with Palm OS? Help!

EVDO rocks

Many of you know that my wife is a church planter. I've been struggling with finding a good way to get high-speed Internet service at the church since we moved into the building more than two years ago. DSL service is not available in that semi-rural location and cable is too expensive.

In late January Sprint completed their roll out of EVDO Rev. A service in the Kansas City area. The first day it was available I got a Pantech card and started testing/using the service everywhere I went, including the church. Bottom line: it rocks. It gives a user experience very similar to DSL or even cable for everything except large file downloads. I'm seeing speeds in the neighborhood of 750+ Kb/s down and 250+ Kb/s up (though it does vary somewhat by location and time).

So having passed the performance and reliability test, I decided it would work well for the church until a faster, affordable land-line service becomes available. Yesterday I got a Linksys WRT54G3G-ST router that accepts an EVDO card and/or a wired broadband connection, routes, has a 4-port switch, and does WiFi. In other words, it's a WiFi router that can use EVDO to get to the Internet. This thing is very cool and it works quite well, but it was a pain to set up. Turns out you need to upgrade the router firmware before it can see/use the Pantech card, but finding the upgraded firmware on the Linksys site is nearly impossible. (By the way, it's here.)

Now ... think about how this could fit in to your disaster recovery planning. This little guy is a completely mobile WiFi hotspot. All you need is electrical power and EVDO coverage. (If it can't find a Rev A tower it will drop back to the the older EVDO Rev 0 or 1xRtt.) Admittedly, the bandwidth is pretty minimal for a subtantial number of users trying to share an Internet connection, but it could be set up instantly almost anywhere. One of these and some computers could provide an instant office with zero lead time to order DSL or T1 service to a location.

In the event of disaster, you could go to your local Sprint or Verizon store, sign up for service, and walk out within an hour with an activated card. Getting an EVDO router could be a bit tougher to find on zero notice. Stores like CompUSA are carrying some of these, but not all are compatible with all EVDO cards and services. You need to verify compatibility prior to purchase. We tried a D-Link router we bought at retail with no success and had to take it back.

Also ... think about how this could be used to provide backup for a critical Internet link in a remote location such as a second campus. I configured the Linksys to give priority to the wired WAN link and connected that link to my cable modem at home. Then I connected my laptop to the router via WiFi and I was on the Internet through the cable. To test the failover, I pulled the plug on the cable. Instantly the Linksys connected to EVDO and continued my sessions without interruption. Neither my Skype nor my MSN IM connection dropped. Sweet!

Adam Hamilton at the RoundTable!

I'm very excited to announce that our senior pastor, Adam Hamilton, will be coming to the Fall RoundTable for a 30-minute Q&A session. Adam is an amazing man who started this church in 1990 and has seen it grow to more than 12,000 adult members and average worship attendance in the 7,000 range. I am grateful he has accepted my invitation to spend some time with us during one of the busiest weeks of his year. It will be very interesting to get Adam's insight into the value of IT in the church and how we IT people can better serve the mission, minisry, and senior leadership of our churches.

KC-area church IT lunch

The next KC-area church IT lunch will be at Chili's in the Merriam Town Center mall at Antioch and Johnson Drive in Merriam, Kansas on Thursday, May 31, at 12:30 pm. See you there!

Include more people and keep it small

Tony Dye hit the nail on the head when he said that the consistent sentiment of Church IT RoundTable participants is that we should "include more people and keep it small." That paradox still makes me chuckle every time I think about it, but it does set out a vision that I believe (all humor aside) we should pursue. How could we do it?

In my original invitation to come to Church of the Resurrection for the Fall RoundTable, I explained a half-baked idea about how the RoundTable can grow while retaining the best aspects of being small. The essence of the idea is to have roundtables in cities across the country and then link them technologically a couple of times each year for a national roundtable of roundtables. Now I'd like to update you on how that half-baked idea is maturing into something that could be truly workable.

Our plan for the Fall RoundTable is to try this once with everyone located in our building. We think that is a good idea because it will allow us to test our facilitation technique and technology with all of us physically together. (Prototype? Cool!) If something goes horribly wrong, we can scramble and put everyone together in one large room and salvage the event. (Contingency plan? Cool!)

We are going to break up the large group into four separate classrooms, each of which will accommodate 20 participants and a few observers. With this idea we will be able to handle as many as 80 participants and 20 vendors/observers before we have to close registration. (I've been told this is similar to the approach taken by Leadership Network, but I've never been to their events so I have to take the word of others on that.)

Each classroom will have two projectors: one showing content from the conversation in that room, and the other showing content from all of the other rooms. Each participant will use their own laptop to join an Adobe Connect meeting with everyone else in their room. Through Connect, anyone in the room will be able to show content (web sites, pictures, diagrams, PowerPoints, etc.) on the primary projector. United Methodist Communications (UMCOM) has agreed to let us use their Adobe Connect system at no charge, and Sean McAtee from UMCOM Tech Shop will be here to assist us with his deep expertise in Connect.

So that we are not limited to interacting only with the participants in our own room, the secondary projector will show content from the conversations happening in the other rooms. We're still in the process of figuring out exactly how this will work. We will run some tests within the next couple of weeks to evaluate various ideas. I'll post once we have a clear picture of this in order to solicit your feedback.

When I suggested this idea to Tony, he said, "Sounds exciting, crazy, and even a bit prone to potential failure. Perfect! :-)" Tony's right. This is both exciting and risky. But I figure, if a bunch of techno-geeks can't make something like this work, who can?

Fall Roundtable schedule overview

I'm very excited that so many of you have commented on my invitation post indicating your interest in coming to the Fall Roundtable, October 3-4, 2007. We're actively working on detailed plans so we can open registration.

We will start on Wednesday around 9:00 am and end Thursday at 5:00 pm. We're also planning a get-together for Tuesday (October 2) afternoon around 3:00 pm for those who are flying in on Tuesday and would like to get a facilities/IT tour of Church of the Resurrection. After the roundtable is Resurrection's annual Leadership Institute, October 5-6. Leadership Institute is our biggest conference of the year. If you are able to stay, I know it will be a blessing to you as well.

For any vendors that would like to attend, sponsorship opportunities are available. Please e-mail me for info (clif.guy at cor dot org).

Watch this blog for more deets to follow soon. It's going to be an AWESOME event!

ESX, here we come

We're currently running the free VMware server using Windows 2003 Server as the host operating system on our original Dell 2950 server. We have 8-10 virtual servers running on that box now, none of which are demanding resource-wise (gratefully).

Three weeks ago we bought a second 2950 like the one we bought last year, a Dell/EMC AX150i iSCSI SAN, and a bunch of expensive ESX 3 software from VMware. We're going to hook the two 2950s to the SAN, load ESX on them, and convert all of our existing virtual servers as well as 3 older physical servers to the new virtual environment. We have hired some consultants to help us and hope to have it in production by June 4. We'll keep you posted ...

Sitting in church

Miles McPherson from The Rock Church in San Diego presented this week at the Beeson Institute. My favorite line from his talk: "There is no spiritual gift called 'sitting in church.'"

The Roundtable is an "Unconference"

BusinessWeek has a nice overview of the unconference idea that we've stumbled into for the church IT community. "Roundtable" is a much better word, in my opinion, than "unconference" or "camp" (as in BarCamp or PodCamp). I'm glad we're doing an unconference and calling it a roundtable - that's the best of both!

Mike Breaux at Resurrection

The Beeson Institute for Advanced Congregational Leadership is having one of its sessions at Church of the Resurrection this week in our Student Center. Mike Breaux, teaching pastor at Willow Creek, presented today.

Mike says our churches should have signs like:
"Come As You Are"
"Grace Happens"
"You Matter to God"

Goodbye Annex

Demolition Day has arrived for the humble annex. Turns out you can use a backhoe to bring down a building.

In the upper left corner you can see our camera crew recording the whole thing from the roof of our West Building. I'm sure they'll find some creative use for the video!

Vista when?

The IT organizations of Intel and Dell are waiting for SP1 to upgrade to Vista. I know we'll run Vista eventually, but I also know it won't be in 2007. We'll look closely at it in November when we need to decide whether to budget for it in 2008. If it doesn't seem like a good move, we'll wait and look at it again in November 2008.

What are your Vista roll-out plans?

KC lunch meeting?

I'd like to get together for lunch with my KC-area church IT homies sometime in the next couple of weeks. How about the 10th, 17th, or 31st? Leave a comment ...

Silverlight?

Who out there is paying attention to Microsoft's Silverlight technology announcements? Is it as big a deal as Scoble suggests?

Spring Church IT Round Table - Round Table Day

I've been quite the slacker in blogging this, but better late than never eh?

The round table was a great experience. In short it was about 30 guys seated around folding tables, and discuss we did. The dynamic of the round table went like this. A topic was thrown out and an expert board was formed on the fly. Expert board = the guys and gals that had worked around that topic a great deal. People would ask the board questions, the board would question or concur with each other, and great value was gained by everyone in the room. I thought it was a well-balanced event between free-flow conversation and a structured discussion.

The best analysis I've seen, gathered from feedback of the event, was Tony's summary of what everyone wants. As Resurrection is hosting the Fall event this year, I had walked around the crowd during breaks and sampled feedback on what we should do next - I came to the exact same conclusion as Tony. "Keep it small, but include more people." Sorry boys and girls - some of you are gonna have to take one for the team. :) In all seriousness, "taking one for the team" is exactly what we want to avoid.

Not to beat a dead horse, but if you haven't given feedback on our idea of how to truly tackle "small but more people," take a look at the invite Clif posted and let us know.

Overall, the spring round table was an unparalleled experience. Every individual was great, and they inspired me. The conversations in the hall, and the more structured times blended well. I would hope that if you are in the ministry of church I.T., yes it is a ministry, you have the opportunity to make one of the future round tables and be fed, have your tank filled, or other appropriate metaphor. ;)

Props to all my church I.T. brethren, it was a pleasure and will be again.