This week we finished the process of converting our year-old Dell 2950 server previously running Win2K3 x64 to our new VMware ESX insfrastructure. We had SQL Server 2005 on that server and virtualized it. Unfortunately, we saw a Shelby performance drop around 50% when we moved the database to the new virtualized server running SQL. So now we have some tuning to do. Other than that, this has been the smoothest major server upgrade project I've ever seen. Many thanks to our ESX consultant, Cameron Shove, for making that possible.
The next step for us is to virtualize three older physical servers. These are old enough and/or have significant enough issues that we aren't going to try P2V. Instead, we are building virtual servers from scratch and migrating the applications and services. We have a good start on virtualizing a terminal server and an Exchange server. Last to get virtualized will be our main file server.
When the project is complete by the middle of July, we will have only one remaining physical Windows server in our central campus data center - a server that will run ARCserve with disk and tape backup storage. All of our production will be on approximately 13 virtual servers running on the two physical Dell 2950s sharing the Dell/EMC AX150i iSCSI SAN. It's going to be sweet!
June 23, 2007
June 20, 2007
Technology Vision (reponding to Jim Walton)
Jim Walton posted this morning about technology vision for the church.
My thinking on this rests on two important principles:
1. Mission/Vision alignment is more important than any clever statement you might create.
2. Too often, mission/vision/purpose statements are "we" centric. They talk more about "us" and what "we" do than they do about those we serve.
To illustrate, Church of the Resurrection's purpose is: "To build a Christian community where non-religious and nominally-religious people are becoming deeply committed Christians."
Correspondingly, our IT Department's purpose is: "To build a Christian community where non-religious and nominally-religious people are becoming deeply committed Christians. We do everything possible to fulfill this purpose directly and to help Resurrection's other ministries make effective use of technology as they also strive toward this purpose. Our users' mission is our mission. Additionally, we take very seriously our responsibilities for information security and high availability of all of the mission-critical services we provide."
This purpose is perfectly aligned with the church's purpose and it puts the priority on those we serve. Our vision for technology is to be excellent at fulfilling this purpose.
For church IT, I believe in serving first and leading second. If we start by earning a reputation for excellent, high-touch service, then when we are called upon to lead, we will be much more persuasive and effective.
My thinking on this rests on two important principles:
1. Mission/Vision alignment is more important than any clever statement you might create.
2. Too often, mission/vision/purpose statements are "we" centric. They talk more about "us" and what "we" do than they do about those we serve.
To illustrate, Church of the Resurrection's purpose is: "To build a Christian community where non-religious and nominally-religious people are becoming deeply committed Christians."
Correspondingly, our IT Department's purpose is: "To build a Christian community where non-religious and nominally-religious people are becoming deeply committed Christians. We do everything possible to fulfill this purpose directly and to help Resurrection's other ministries make effective use of technology as they also strive toward this purpose. Our users' mission is our mission. Additionally, we take very seriously our responsibilities for information security and high availability of all of the mission-critical services we provide."
This purpose is perfectly aligned with the church's purpose and it puts the priority on those we serve. Our vision for technology is to be excellent at fulfilling this purpose.
For church IT, I believe in serving first and leading second. If we start by earning a reputation for excellent, high-touch service, then when we are called upon to lead, we will be much more persuasive and effective.
June 19, 2007
What drives your church?
Seth Godin says most successful organizations are driven by something. Then he goes on to list a number of drivers. So I'm reading the list and nodding, when suddenly, this jumps out:
SUBSCRIPTION DRIVEN: How do we transform a stranger into someone who uses what we do, all the time. Intuit, certainly.
(He means that the company Intuit is an example.) This seems very much like things I've heard from Adam Hamilton, our senior pastor. To help people become deeply committed Christians, first we need to get them to try us and then we need to get them to become a regular. Spiritual growth comes out of habitual practice of the spiritual disciplines that we offer in our churches. We need to convert people from occasionally picking up a copy from the news stand into every week subscribers.
Bowling nerds!
My boss took me and my senior staff peers in the Operations Division bowling a couple of weeks ago. What a nerdy looking group! Turns out, Kelly was a bowler in high school, so she smoked us. I broke 100 both games, which isn't bad considering it's been a few years since I last bowled.
Left to right: Dick Cooper, Director of Facilities; yours truly; Susan Bell, Director of Guest Services; Kelly Williams, Director of Finance; Brent Messick, Managing Executive Director and CFO (our boss); Lisa Viehweg, Human Resources Director, Nancy Hines, Director of Resurrection Resources (book store manager); Travis Morgan, Director of Operations for Resurrection West
Left to right: Dick Cooper, Director of Facilities; yours truly; Susan Bell, Director of Guest Services; Kelly Williams, Director of Finance; Brent Messick, Managing Executive Director and CFO (our boss); Lisa Viehweg, Human Resources Director, Nancy Hines, Director of Resurrection Resources (book store manager); Travis Morgan, Director of Operations for Resurrection West
June 18, 2007
Smartphone advice
I got some good smartphone advice when I asked for it a few weeks ago. Thanks to all of you who commented. Despite that good advice, I am still undecided. Here are my priorities (in this order):
1. Must work with Sprint service
2. Must be able to do EVDO Rev. A
3. Must be able to do phone-as-modem, preferably via Bluetooth
4. Must be easy to use as a phone
5. Must be easy to synch with Outlook and relatively straightforward to use for PDA functions
6. Must be easy to carry, preferaby in my pocket (as small and light as possible)
E-mail is fine, but it's not the killer app for the way I use/intend to use my phone. I'm sure it will be cool to have a camera in my phone, but I don't expect to use it much. WiFi is nifty, but not a requirement. Ability to run various 3rd party applications could be cool too, but again not a requirement.
With all of that in mind, should I buy the new Mogul (story here, details here)?
June 14, 2007
Shooting yourself in the foot
Er, I would have gotten your e-mail about SonicWALL acquiring Aventail, except my SonicWALL anti-spam software (MailFrontier) junked it!
June 12, 2007
The media at Kelsey's memorial service
The memorial service for Kelsey Smith was this afternoon.
Nick Kastelan, one of the guys in our media department (Saving Grace Productions - SGP), had worked in TV news in Kansas City before joining our staff. He and Peter Metz, our Director of Communications, handled media relations for the visitation last night and the service today. Working with our facilities and security teams, they did an awesome job with the media.
They selected a perfect location where the media could have the access they needed without being in the way. They blocked off the street between our two main buildings, making it available exclusively for the TV trucks and other media vehicles. This street is conveniently located outside our backstage entrance, right next to our video control room.
SGP generated a live audio/video feed from the output of our audio mixer and video switcher and hooked it to a distribution box in the control room. Each of the TV stations ran their own heavy-duty cables from their truck, along the ground, under the door and into the control room. The complete service video is here. You can see news reports using our video here, here, and here. The family allowed one local reporter inside the building. You can see his report here.
SGP also set up a media room backstage where print media people could watch the service on video monitors. We provided power strips and they had access to our guest WiFi.
This was an excellent way to handle a very delicate situation. The media were able to get the content they needed without being visible to mourners inside the building. I attended the service myself and never felt I was on display because the only video available to the media was determined by our video director in our control room.
Today I saw a very special example of the use of technology in ministry. Props to SGP!
Nick Kastelan, one of the guys in our media department (Saving Grace Productions - SGP), had worked in TV news in Kansas City before joining our staff. He and Peter Metz, our Director of Communications, handled media relations for the visitation last night and the service today. Working with our facilities and security teams, they did an awesome job with the media.
They selected a perfect location where the media could have the access they needed without being in the way. They blocked off the street between our two main buildings, making it available exclusively for the TV trucks and other media vehicles. This street is conveniently located outside our backstage entrance, right next to our video control room.
SGP generated a live audio/video feed from the output of our audio mixer and video switcher and hooked it to a distribution box in the control room. Each of the TV stations ran their own heavy-duty cables from their truck, along the ground, under the door and into the control room. The complete service video is here. You can see news reports using our video here, here, and here. The family allowed one local reporter inside the building. You can see his report here.
SGP also set up a media room backstage where print media people could watch the service on video monitors. We provided power strips and they had access to our guest WiFi.
This was an excellent way to handle a very delicate situation. The media were able to get the content they needed without being visible to mourners inside the building. I attended the service myself and never felt I was on display because the only video available to the media was determined by our video director in our control room.
Today I saw a very special example of the use of technology in ministry. Props to SGP!
June 09, 2007
Kelsey's service will be at Resurrection
I mentioned that my wife, Laura, was the only pastor present on Wednesday afternoon when Kelsey's family learned that she had been found murdered. A few minutes later, pastors from the Smith's church, Hillcrest Covenant, arrived. Laura quietly suggested to them that when it came time to begin discussing memorial service plans, the family should consider having the service at Church of the Resurrection. At the risk of sounding immodest, I truly believe Resurrection is the only church in the area with the capability to handle a funeral like this. Our main sanctuary seats 3,000+ and we have extensive video production capabilities to be able to generate a pool feed for the media while keeping them mostly out of sight.
Wednesday afternoon I let our upper management know that the suggestion had been made. On Thursday morning several of our staff came to me and said that all of the teams required to support the service were on board and eager to help, if asked.
Then Friday the word came that the family had decided to take our suggestion. Normally, the church has fees for outside funerals, but they were completely waived in this case, even though a huge effort will be required. This will be a major event in the middle of the work day (2 pm Tuesday) when volunteer availability will be low.
The planning began immediately Friday, showing great foresight and attention to detail. Our experience with the Ali Kemp funeral five years ago has been invaluable. From traffic, to parking, hospitality, facilities, security, and A/V we have vast expertise and resources to handle every detail so the family and friends can simply be in the moment and celebrate Kelsey's life. This is the Body of Christ functioning as should (but too rarely does) - joining together across congregational and denominational boundaries to serve in a very special way in a time of great need. Although I will not have a personal role in any of this, I have never been prouder of Church of the Resurrection than I am right now. It is a special place to serve.
Wednesday afternoon I let our upper management know that the suggestion had been made. On Thursday morning several of our staff came to me and said that all of the teams required to support the service were on board and eager to help, if asked.
Then Friday the word came that the family had decided to take our suggestion. Normally, the church has fees for outside funerals, but they were completely waived in this case, even though a huge effort will be required. This will be a major event in the middle of the work day (2 pm Tuesday) when volunteer availability will be low.
The planning began immediately Friday, showing great foresight and attention to detail. Our experience with the Ali Kemp funeral five years ago has been invaluable. From traffic, to parking, hospitality, facilities, security, and A/V we have vast expertise and resources to handle every detail so the family and friends can simply be in the moment and celebrate Kelsey's life. This is the Body of Christ functioning as should (but too rarely does) - joining together across congregational and denominational boundaries to serve in a very special way in a time of great need. Although I will not have a personal role in any of this, I have never been prouder of Church of the Resurrection than I am right now. It is a special place to serve.
June 08, 2007
The story of the Kelsy Smith website
We built a site that went from launch to the #5 search term on Google and then to obsolete in three days. Here's the (updated) story ...
The first version of the Kelsey Smith website, made by a family friend on Monday and hosted at sitesled.com, was barely more than a web version of the "have you seen this person?" poster. As the friend added pictures to the site, it eventually did not perform well, even though it had only light traffic at the time.
Around 2:00 pm Monday my wife called me from the Smith home asking for help. We jumped right in and bought the domain findkelsey.com, quickly built a static HTML site (see mirror), and put it up on four servers at Vine Hosting, using DNS round robin for load balancing. The site went live around 5:00 pm and averaged 20+ page views per minute through the remaining waking hours of Monday. Not bad ... and we were grateful to have the opportunity to help with the search for Kelsey in this small way. Little did we know we were about to experience an exponential explosion in site traffic unlike anything any of us had ever seen.
Through the day on Tuesday the traffic increased by a factor of 7 to an average of 140+ page views per minute. We got as many page views on findkelsey.com Tuesday, 160,000, as we get in an average month at www.cor.org. Our infrastructure held up very well and we felt that this level of interest in the site represented a tremendous return on our investment in ministry.
Then Wednesday morning traffic increased by another factor of 7 to 1,000+ page views per minute and we had a serious problem on our hands. As I mentioned, the traffic growth was so rapid, at first Vine Hosting thought we were experiencing a DDOS attack aimed at findkelsey.com. Every site they host was being slowed to a crawl. We quickly became concerned that the bandwidth consumption could result in a huge bill for overage charges. Looking back now, we think it's likely the servers were just fine but Vine's WAN link was saturated (which, by the way, seriously hampered our efforts to fix the problem as you can imagine). We discovered later that "findkelsey.com" had become the #5 search term on Google and that the site had 11,000+ other pages linking to it.
As a practical matter, findkelsey.com was very slow to display for most site visitors, if it displayed at all. Vine was in serious trouble and most people weren't getting to the site anyway. Consequently, I felt the only good option was to take the site down from our servers at Vine and move it to Blogger, which is owned by Google. Only Google had the capacity to handle the traffic while being affordable (free). While we were moving the site, between 2:00 and 5:00 pm Wednesday, it was mostly inaccessible. Meanwhile, Vine had to do a great deal of work to restore full service for its customers while traffic slowly faded away as the DNS changes propagated. In an urgent situation like this, DNS propagation delays are terribly frustrating. There's nothing like taking down a site, rebuilding it, and moving it WHILE it is getting bombarded with thousands of page requests per minute. It's truly a mind boggling challenge.
During the three hours the site had been (mostly) down, Kelsey's body was discovered and the news quickly spread through the media. Around 5:00 pm when we had the site rebuilt and the DNS redirected, immediately we again experienced the 1,000+ page views per minute we had seen in the morning. Just as immediately, expressions of love and support began pouring in through Blogger's comment feature at the rate of 1 every 10 seconds. People came to the site from across the country and around the world to express their love, comfort, sorrow, and outrage. Those comments continued through Thursday and gradually tapered off to less than 1 every 5 minutes late Thursday night. In 30 hours the site received a total of 2,700+ comments that we hope will be a great encouragement to the family when they eventually have time to read them and reflect.
As the comments came flooding in, we realized that our ministry was not only to the Smith family to aid in the search and the immediate aftermath, but also to the public to provide a way for people to express themselves to the family. Other than the 1% we had to remove because they were inappropriate, the comments were overwhelmingly supportive. (Understandably, some expressed uncensored anger towards the perpetrator. Those we didn't remove.)
Thursday afternoon we got word from the family that the term "Find Kelsey" no longer fit their situation. The volunteers during the search had nicknamed themselves "Kelsey's Army" and that was the domain name the family wanted to use going forward. In light of the major challenges we had experienced with traffic, I advised against redirecting findkelsey.com to the new kelseysarmy.com. Instead, I posted a message at the top of the page referring people to the new site and indicating that comments would be closed.
And that's how it ended. The site went live on Monday around 5:00 pm and was mothballed late Thursday night. It was in production just over three days, and yet the ministry value it created in that short time was immeasurable.
The first version of the Kelsey Smith website, made by a family friend on Monday and hosted at sitesled.com, was barely more than a web version of the "have you seen this person?" poster. As the friend added pictures to the site, it eventually did not perform well, even though it had only light traffic at the time.
Around 2:00 pm Monday my wife called me from the Smith home asking for help. We jumped right in and bought the domain findkelsey.com, quickly built a static HTML site (see mirror), and put it up on four servers at Vine Hosting, using DNS round robin for load balancing. The site went live around 5:00 pm and averaged 20+ page views per minute through the remaining waking hours of Monday. Not bad ... and we were grateful to have the opportunity to help with the search for Kelsey in this small way. Little did we know we were about to experience an exponential explosion in site traffic unlike anything any of us had ever seen.
Through the day on Tuesday the traffic increased by a factor of 7 to an average of 140+ page views per minute. We got as many page views on findkelsey.com Tuesday, 160,000, as we get in an average month at www.cor.org. Our infrastructure held up very well and we felt that this level of interest in the site represented a tremendous return on our investment in ministry.
Then Wednesday morning traffic increased by another factor of 7 to 1,000+ page views per minute and we had a serious problem on our hands. As I mentioned, the traffic growth was so rapid, at first Vine Hosting thought we were experiencing a DDOS attack aimed at findkelsey.com. Every site they host was being slowed to a crawl. We quickly became concerned that the bandwidth consumption could result in a huge bill for overage charges. Looking back now, we think it's likely the servers were just fine but Vine's WAN link was saturated (which, by the way, seriously hampered our efforts to fix the problem as you can imagine). We discovered later that "findkelsey.com" had become the #5 search term on Google and that the site had 11,000+ other pages linking to it.
As a practical matter, findkelsey.com was very slow to display for most site visitors, if it displayed at all. Vine was in serious trouble and most people weren't getting to the site anyway. Consequently, I felt the only good option was to take the site down from our servers at Vine and move it to Blogger, which is owned by Google. Only Google had the capacity to handle the traffic while being affordable (free). While we were moving the site, between 2:00 and 5:00 pm Wednesday, it was mostly inaccessible. Meanwhile, Vine had to do a great deal of work to restore full service for its customers while traffic slowly faded away as the DNS changes propagated. In an urgent situation like this, DNS propagation delays are terribly frustrating. There's nothing like taking down a site, rebuilding it, and moving it WHILE it is getting bombarded with thousands of page requests per minute. It's truly a mind boggling challenge.
During the three hours the site had been (mostly) down, Kelsey's body was discovered and the news quickly spread through the media. Around 5:00 pm when we had the site rebuilt and the DNS redirected, immediately we again experienced the 1,000+ page views per minute we had seen in the morning. Just as immediately, expressions of love and support began pouring in through Blogger's comment feature at the rate of 1 every 10 seconds. People came to the site from across the country and around the world to express their love, comfort, sorrow, and outrage. Those comments continued through Thursday and gradually tapered off to less than 1 every 5 minutes late Thursday night. In 30 hours the site received a total of 2,700+ comments that we hope will be a great encouragement to the family when they eventually have time to read them and reflect.
As the comments came flooding in, we realized that our ministry was not only to the Smith family to aid in the search and the immediate aftermath, but also to the public to provide a way for people to express themselves to the family. Other than the 1% we had to remove because they were inappropriate, the comments were overwhelmingly supportive. (Understandably, some expressed uncensored anger towards the perpetrator. Those we didn't remove.)
Thursday afternoon we got word from the family that the term "Find Kelsey" no longer fit their situation. The volunteers during the search had nicknamed themselves "Kelsey's Army" and that was the domain name the family wanted to use going forward. In light of the major challenges we had experienced with traffic, I advised against redirecting findkelsey.com to the new kelseysarmy.com. Instead, I posted a message at the top of the page referring people to the new site and indicating that comments would be closed.
And that's how it ended. The site went live on Monday around 5:00 pm and was mothballed late Thursday night. It was in production just over three days, and yet the ministry value it created in that short time was immeasurable.
June 06, 2007
Kelsey update
Where do I begin?
Many of you no doubt have already heard that Kelsey Smith was found murdered today. My wife, Laura, was at the Smith home when the detectives came with the news that a body had been found and it was likely Kelsey. The identification was later confirmed. Laura and my daughter, Beth, are attending a prayer service at Hillcrest Covenant Church as I write this. Prayers are flooding in from across the country and around the globe. Please add your prayers to them. I will have much more to say about this in the coming days.
Quite aside from the unspeakable human tragedy, my technical team and I learned a great deal through this. The traffic to www.findkelsey.com (see mirror of original HTML site) was 160,000 page views yesterday and grew exponentially this morning. By 10:00 am we had serious performance impact, essentially taking down Vine Hosting, the ISP where our web servers are located. Searches for findkelsey.com reached #5 on Google as reported by Google Trends. You read that right. Our little hand-built quickie site was the #5 search on Google today. (Right now "Kelsey Smith" is #3, "find Kelsey" is # 7, and "findkelsey.com" is #9.) The flood of legitimate traffic was so great, Glenn - the director of operations for Vine Hosting - at first thought it was a DDOS. In reality, it was all legit traffic. Too much traffic. Glenn is still dealing with the aftermath.
The only people with enough resources to absorb this kind of traffic is, that's right, Google. Realizing the disaster we were causing for Vine, I made the command decision to totally take down all traffic related to findkelsey.com at Vine, even the DNS. Then, as rapidly as possible, we created a new site on Blogger (which is owned by Google), changed to a high-volume DNS host, and pointed the DNS at Blogger. As soon as the DNS change propagated, we started getting comments on our posts. In fact, my first indication that we finally had things working was when a comment appeared in my inbox. It took almost another hour to figure out how to get most traffic coming to us as it should. In the following 2.5 hours, 754 comments have come in from around the country, directed to Kelsey's family. That's one comment every 10-15 seconds. Unbelievable.
Today I had the experience of building one of the highest traffic sites on the Internet. I was building it WHILE it was being hit.
Many of you no doubt have already heard that Kelsey Smith was found murdered today. My wife, Laura, was at the Smith home when the detectives came with the news that a body had been found and it was likely Kelsey. The identification was later confirmed. Laura and my daughter, Beth, are attending a prayer service at Hillcrest Covenant Church as I write this. Prayers are flooding in from across the country and around the globe. Please add your prayers to them. I will have much more to say about this in the coming days.
Quite aside from the unspeakable human tragedy, my technical team and I learned a great deal through this. The traffic to www.findkelsey.com (see mirror of original HTML site) was 160,000 page views yesterday and grew exponentially this morning. By 10:00 am we had serious performance impact, essentially taking down Vine Hosting, the ISP where our web servers are located. Searches for findkelsey.com reached #5 on Google as reported by Google Trends. You read that right. Our little hand-built quickie site was the #5 search on Google today. (Right now "Kelsey Smith" is #3, "find Kelsey" is # 7, and "findkelsey.com" is #9.) The flood of legitimate traffic was so great, Glenn - the director of operations for Vine Hosting - at first thought it was a DDOS. In reality, it was all legit traffic. Too much traffic. Glenn is still dealing with the aftermath.
The only people with enough resources to absorb this kind of traffic is, that's right, Google. Realizing the disaster we were causing for Vine, I made the command decision to totally take down all traffic related to findkelsey.com at Vine, even the DNS. Then, as rapidly as possible, we created a new site on Blogger (which is owned by Google), changed to a high-volume DNS host, and pointed the DNS at Blogger. As soon as the DNS change propagated, we started getting comments on our posts. In fact, my first indication that we finally had things working was when a comment appeared in my inbox. It took almost another hour to figure out how to get most traffic coming to us as it should. In the following 2.5 hours, 754 comments have come in from around the country, directed to Kelsey's family. That's one comment every 10-15 seconds. Unbelievable.
Today I had the experience of building one of the highest traffic sites on the Internet. I was building it WHILE it was being hit.
June 05, 2007
Find Kelsey Smith
My daughter, Beth, was in band at Shawnee Mission West High School with Kelsey Smith, the 18 year-old girl apparently abducted from a Target parking lot here in Overland Park on Saturday night. The band has a very active parents' organization called Band Aides West. Within 12 hours of Kelsey's disappearance, Band Aids was taking the lead to organize the search and support the family. Band Aids asked my wife, Laura, to go to the Smith home yesterday and take a shift answering the phone. After arriving there, Laura became aware of a need for a web site about Kelsey that could handle the load coming from national news exposure, including Greta Van Susteren and Nancy Grace, as well as extensive coverage from the local media.
Through the intense efforts of Brian, Leo, and Chuck http://www.findkelsey.com/ (see mirror of original HTML site) went live yesterday around 5 pm. Glenn Kelly and Vine Hosting are providing hosting at no charge. Glenn even missed part of his daughter's second birthday party to help set up the site on multiple servers to handle the load.
The site is linked from the home page of the Kansas City Star, the story about Kelsey on Fox News, and many other sites. It is also on flyers being posted all over town. The site is now averaging 2.6+ page views per second during waking hours.
Please stop what you are doing right now and pray for Kelsey, for her family, for the police, and for all of the volunteers who are supporting them and helping search.
Through the intense efforts of Brian, Leo, and Chuck http://www.findkelsey.com/ (see mirror of original HTML site) went live yesterday around 5 pm. Glenn Kelly and Vine Hosting are providing hosting at no charge. Glenn even missed part of his daughter's second birthday party to help set up the site on multiple servers to handle the load.
The site is linked from the home page of the Kansas City Star, the story about Kelsey on Fox News, and many other sites. It is also on flyers being posted all over town. The site is now averaging 2.6+ page views per second during waking hours.
Please stop what you are doing right now and pray for Kelsey, for her family, for the police, and for all of the volunteers who are supporting them and helping search.
Subscribe to:
Posts (Atom)
