General Assembly is a family affair

General Assembly of the Christian Church is a family affair for us. My dad was a DOC pastor and his sister Marje also married a DOC pastor, Mickey. Uncle Mickey also had brothers who were DOC pastors. Each of them had children that are DOC pastors: my brother, Greg, and his wife, Karen; my sister, Mary; my wife, Laura; my cousin, Michael and his wife, Jennifer; my cousin Marci married a DOC pastor, Dan; and my 2nd cousin Shannon married a DOC pastor, Kevin. Is that ridiculous, or what???

At the closing worship service, the following members of my extended family were in the house.

l to r: niece Elizabeth, daughter Beth, niece Judith,
wife Laura, son Rob, brother Greg, niece Kimberly

Kevin and my 2nd cousin, Shannon

Jennifer and my cousin, Michael

Celebrating New Churches at General Assembly

At the end of the evening worship service on Tuesday, there was a brief celebration of the new Disciples of Christ congregations established since 2001 and a prayer for the planters.

Those planters who were available that night went up on the stage with Rick Morse, the head of our new church organization. To Rick's right is Sharon Watkins, the head of our denomination. If you look closely, you'll see that the third person to the right of Sharon is Laura in her Living Water shirt!

2020 Vision

On July 15 I preached at Living Water because Laura was out on a mission trip. I talked about the new church movement within Disciples of Christ. We have the most active and dynamic new church initiative of all mainline denominations. In the sermon, I mentioned two key people in our new church movement: Dick Hamm, the former head of our denomination who cast the 2020 Vision which included the goal of 1,000 new congregations by the year 2020; and Rick Morse, the wild-eyed spiritual entrepreneur who took on the challenge of making this vision become a reality and leads our New Church Ministry Team. Read a recap of my sermon or listen to the MP3 here.

Little did I know when preaching that five days later at the National Evangelism Workshop, a sort of pre-conference prior to General Assembly, I would be in a breakout session with Dick, Rick, and about 10 other people. Very cool. After the session, I invited them to come to the Living Water site and listen to me talk about them!

State of the Church

Sunday night, the head of our denomination (we use the fancy title "General Minister and President" or GMP for short), Sharon Watkins, spoke on the State of the Church. Though flawed in some ways (for example, it was too long and too repetitive), her message was very powerful and motivating. She showed great leadership by pointing out many positive accomplishments of the last two years, including an impressive response to the Katrina disaster which occurred just a few weeks after our last General Assembly. Her theme was that Disciples are "choosing life" - a reference to Deut. 30:19-20. It was evident that she had spent weeks preparing her message of hope and encouragement and the response she received was enthusiastic. Way to go Sharon!

After the State of the Church, those of us who hung around enjoyed a brief concert featuring outstanding Disciple musicians from around the country. We heard some terrific original music from these artists in a wide variety of styles. It was a wonderful celebration of musical creativity and spiritual vitality. I believe they made a CD of the concert. If so, we should get it!

Sunday at General Assembly

Sunday morning we attended the youth worship service in the main auditorium at the Fort Worth Convention Center. It was the best worship service we've ever experienced at General Assembly. It was so good, in fact, that Laura bought a DVD of the service so we can show it to Living Water when we get home.

Keith McAliley was masterful leading worship from the piano

Stacey Spencer was off the hook in his message on 2 Kings 7:3-20
Read the report in Disciples World here

After the service ran into some of Rob's and Beth's homies from
Kansas City outside the main entrance to the Convention Center

Sunday evening Laura organized a dinner for the Bethany Fellowships at one of Fort Worth's most famous restaurants, Joe T. Garcia's. The food was even better than I remembered. The service was outstanding. And though it was a little hot to be seated out on the patio, a grand time was had by all. And it didn't rain until dinner was over!

Bethany Fellowships dinner at Joe T. Garcia's

In Fort Worth for General Assembly

Our family has been in Fort Worth since last Wednesday for the Christian Church (Disciples of Christ) bi-annual General Assembly. I'm a life-long Disciple. My dad was a Disciple pastor for 40+ years; my sister, brother, and sister-in-law are DOC pastors; and now my wife is a DOC church planter. Yes, I'm a church nerd even when I'm on vacation!

On Thursday we're driving down to the Big Bend area. Will be there for 3 nights. Then San Antonio for 2 nights and Galveston for 2 nights before returning to KC. It's a driving tour of Texas in the hottest weeks of the summer. Are we nuts? Actually, the weather hasn't been bad at all compared to other Dallas-Fort Worth summers I've experienced. I hope it holds up.

l to r: my son Rob, age 15; my wife Laura;

my brother Greg (also a DOC pastor); my daughter Beth, age 18

Selling an idea in your church

A few days ago Guy Kawasaki reviewed the top 9 stories that people like to talk about as explained in a new book by Lois Kelly. The idea is to use these types of stories whenever you're trying to persuade people to believe something or do something. In my experience, we IT people aren't especially good at marketing and selling our ideas. I'm going to learn from this post and keep these stories in mind as I have opportunities to persuade. I think particularly powerful in a church setting are #1 - stories about aspirations and beliefs, and #6 - personal stories.

Best Christian Workplaces

Church of the Resurrection got a very nice write up in the July 2007 issue of Church Executive magazine regarding the Best Christian Workplaces survey and what we've been doing over the last few years to improve our human resources process. Our own Dan Entwistle (one of our top two executives reporting to our senior pastor) was featured with a big photo on the cover. Go Dan!

Dare I say this is going well?

This week we finished the process of converting our year-old Dell 2950 server previously running Win2K3 x64 to our new VMware ESX insfrastructure. We had SQL Server 2005 on that server and virtualized it. Unfortunately, we saw a Shelby performance drop around 50% when we moved the database to the new virtualized server running SQL. So now we have some tuning to do. Other than that, this has been the smoothest major server upgrade project I've ever seen. Many thanks to our ESX consultant, Cameron Shove, for making that possible.

The next step for us is to virtualize three older physical servers. These are old enough and/or have significant enough issues that we aren't going to try P2V. Instead, we are building virtual servers from scratch and migrating the applications and services. We have a good start on virtualizing a terminal server and an Exchange server. Last to get virtualized will be our main file server.

When the project is complete by the middle of July, we will have only one remaining physical Windows server in our central campus data center - a server that will run ARCserve with disk and tape backup storage. All of our production will be on approximately 13 virtual servers running on the two physical Dell 2950s sharing the Dell/EMC AX150i iSCSI SAN. It's going to be sweet!

Technology Vision (reponding to Jim Walton)

Jim Walton posted this morning about technology vision for the church.

My thinking on this rests on two important principles:

1. Mission/Vision alignment is more important than any clever statement you might create.

2. Too often, mission/vision/purpose statements are "we" centric. They talk more about "us" and what "we" do than they do about those we serve.

To illustrate, Church of the Resurrection's purpose is: "To build a Christian community where non-religious and nominally-religious people are becoming deeply committed Christians."

Correspondingly, our IT Department's purpose is: "To build a Christian community where non-religious and nominally-religious people are becoming deeply committed Christians. We do everything possible to fulfill this purpose directly and to help Resurrection's other ministries make effective use of technology as they also strive toward this purpose. Our users' mission is our mission. Additionally, we take very seriously our responsibilities for information security and high availability of all of the mission-critical services we provide."

This purpose is perfectly aligned with the church's purpose and it puts the priority on those we serve. Our vision for technology is to be excellent at fulfilling this purpose.

For church IT, I believe in serving first and leading second. If we start by earning a reputation for excellent, high-touch service, then when we are called upon to lead, we will be much more persuasive and effective.

What drives your church?

Seth Godin says most successful organizations are driven by something. Then he goes on to list a number of drivers. So I'm reading the list and nodding, when suddenly, this jumps out:

SUBSCRIPTION DRIVEN: How do we transform a stranger into someone who uses what we do, all the time. Intuit, certainly.

(He means that the company Intuit is an example.) This seems very much like things I've heard from Adam Hamilton, our senior pastor. To help people become deeply committed Christians, first we need to get them to try us and then we need to get them to become a regular. Spiritual growth comes out of habitual practice of the spiritual disciplines that we offer in our churches. We need to convert people from occasionally picking up a copy from the news stand into every week subscribers.

Bowling nerds!

My boss took me and my senior staff peers in the Operations Division bowling a couple of weeks ago. What a nerdy looking group! Turns out, Kelly was a bowler in high school, so she smoked us. I broke 100 both games, which isn't bad considering it's been a few years since I last bowled.

Left to right: Dick Cooper, Director of Facilities; yours truly; Susan Bell, Director of Guest Services; Kelly Williams, Director of Finance; Brent Messick, Managing Executive Director and CFO (our boss); Lisa Viehweg, Human Resources Director, Nancy Hines, Director of Resurrection Resources (book store manager); Travis Morgan, Director of Operations for Resurrection West

Smartphone advice

I got some good smartphone advice when I asked for it a few weeks ago. Thanks to all of you who commented. Despite that good advice, I am still undecided. Here are my priorities (in this order):

1. Must work with Sprint service

2. Must be able to do EVDO Rev. A

3. Must be able to do phone-as-modem, preferably via Bluetooth

4. Must be easy to use as a phone

5. Must be easy to synch with Outlook and relatively straightforward to use for PDA functions

6. Must be easy to carry, preferaby in my pocket (as small and light as possible)

E-mail is fine, but it's not the killer app for the way I use/intend to use my phone. I'm sure it will be cool to have a camera in my phone, but I don't expect to use it much. WiFi is nifty, but not a requirement. Ability to run various 3rd party applications could be cool too, but again not a requirement.

With all of that in mind, should I buy the new Mogul (story here, details here)?

Shooting yourself in the foot

Er, I would have gotten your e-mail about SonicWALL acquiring Aventail, except my SonicWALL anti-spam software (MailFrontier) junked it!

The media at Kelsey's memorial service

The memorial service for Kelsey Smith was this afternoon.

Nick Kastelan, one of the guys in our media department (Saving Grace Productions - SGP), had worked in TV news in Kansas City before joining our staff. He and Peter Metz, our Director of Communications, handled media relations for the visitation last night and the service today. Working with our facilities and security teams, they did an awesome job with the media.

They selected a perfect location where the media could have the access they needed without being in the way. They blocked off the street between our two main buildings, making it available exclusively for the TV trucks and other media vehicles. This street is conveniently located outside our backstage entrance, right next to our video control room.


SGP generated a live audio/video feed from the output of our audio mixer and video switcher and hooked it to a distribution box in the control room. Each of the TV stations ran their own heavy-duty cables from their truck, along the ground, under the door and into the control room. The complete service video is here. You can see news reports using our video here, here, and here. The family allowed one local reporter inside the building. You can see his report here.

SGP also set up a media room backstage where print media people could watch the service on video monitors. We provided power strips and they had access to our guest WiFi.

This was an excellent way to handle a very delicate situation. The media were able to get the content they needed without being visible to mourners inside the building. I attended the service myself and never felt I was on display because the only video available to the media was determined by our video director in our control room.

Today I saw a very special example of the use of technology in ministry. Props to SGP!

Kelsey's service will be at Resurrection

I mentioned that my wife, Laura, was the only pastor present on Wednesday afternoon when Kelsey's family learned that she had been found murdered. A few minutes later, pastors from the Smith's church, Hillcrest Covenant, arrived. Laura quietly suggested to them that when it came time to begin discussing memorial service plans, the family should consider having the service at Church of the Resurrection. At the risk of sounding immodest, I truly believe Resurrection is the only church in the area with the capability to handle a funeral like this. Our main sanctuary seats 3,000+ and we have extensive video production capabilities to be able to generate a pool feed for the media while keeping them mostly out of sight.

Wednesday afternoon I let our upper management know that the suggestion had been made. On Thursday morning several of our staff came to me and said that all of the teams required to support the service were on board and eager to help, if asked.

Then Friday the word came that the family had decided to take our suggestion. Normally, the church has fees for outside funerals, but they were completely waived in this case, even though a huge effort will be required. This will be a major event in the middle of the work day (2 pm Tuesday) when volunteer availability will be low.

The planning began immediately Friday, showing great foresight and attention to detail. Our experience with the Ali Kemp funeral five years ago has been invaluable. From traffic, to parking, hospitality, facilities, security, and A/V we have vast expertise and resources to handle every detail so the family and friends can simply be in the moment and celebrate Kelsey's life. This is the Body of Christ functioning as should (but too rarely does) - joining together across congregational and denominational boundaries to serve in a very special way in a time of great need. Although I will not have a personal role in any of this, I have never been prouder of Church of the Resurrection than I am right now. It is a special place to serve.

The story of the Kelsy Smith website

We built a site that went from launch to the #5 search term on Google and then to obsolete in three days. Here's the (updated) story ...

The first version of the Kelsey Smith website, made by a family friend on Monday and hosted at sitesled.com, was barely more than a web version of the "have you seen this person?" poster. As the friend added pictures to the site, it eventually did not perform well, even though it had only light traffic at the time.

Around 2:00 pm Monday my wife called me from the Smith home asking for help. We jumped right in and bought the domain findkelsey.com, quickly built a static HTML site (see mirror), and put it up on four servers at Vine Hosting, using DNS round robin for load balancing. The site went live around 5:00 pm and averaged 20+ page views per minute through the remaining waking hours of Monday. Not bad ... and we were grateful to have the opportunity to help with the search for Kelsey in this small way. Little did we know we were about to experience an exponential explosion in site traffic unlike anything any of us had ever seen.

Through the day on Tuesday the traffic increased by a factor of 7 to an average of 140+ page views per minute. We got as many page views on findkelsey.com Tuesday, 160,000, as we get in an average month at www.cor.org. Our infrastructure held up very well and we felt that this level of interest in the site represented a tremendous return on our investment in ministry.

Then Wednesday morning traffic increased by another factor of 7 to 1,000+ page views per minute and we had a serious problem on our hands. As I mentioned, the traffic growth was so rapid, at first Vine Hosting thought we were experiencing a DDOS attack aimed at findkelsey.com. Every site they host was being slowed to a crawl. We quickly became concerned that the bandwidth consumption could result in a huge bill for overage charges. Looking back now, we think it's likely the servers were just fine but Vine's WAN link was saturated (which, by the way, seriously hampered our efforts to fix the problem as you can imagine). We discovered later that "findkelsey.com" had become the #5 search term on Google and that the site had 11,000+ other pages linking to it.

As a practical matter, findkelsey.com was very slow to display for most site visitors, if it displayed at all. Vine was in serious trouble and most people weren't getting to the site anyway. Consequently, I felt the only good option was to take the site down from our servers at Vine and move it to Blogger, which is owned by Google. Only Google had the capacity to handle the traffic while being affordable (free). While we were moving the site, between 2:00 and 5:00 pm Wednesday, it was mostly inaccessible. Meanwhile, Vine had to do a great deal of work to restore full service for its customers while traffic slowly faded away as the DNS changes propagated. In an urgent situation like this, DNS propagation delays are terribly frustrating. There's nothing like taking down a site, rebuilding it, and moving it WHILE it is getting bombarded with thousands of page requests per minute. It's truly a mind boggling challenge.

During the three hours the site had been (mostly) down, Kelsey's body was discovered and the news quickly spread through the media. Around 5:00 pm when we had the site rebuilt and the DNS redirected, immediately we again experienced the 1,000+ page views per minute we had seen in the morning. Just as immediately, expressions of love and support began pouring in through Blogger's comment feature at the rate of 1 every 10 seconds. People came to the site from across the country and around the world to express their love, comfort, sorrow, and outrage. Those comments continued through Thursday and gradually tapered off to less than 1 every 5 minutes late Thursday night. In 30 hours the site received a total of 2,700+ comments that we hope will be a great encouragement to the family when they eventually have time to read them and reflect.

As the comments came flooding in, we realized that our ministry was not only to the Smith family to aid in the search and the immediate aftermath, but also to the public to provide a way for people to express themselves to the family. Other than the 1% we had to remove because they were inappropriate, the comments were overwhelmingly supportive. (Understandably, some expressed uncensored anger towards the perpetrator. Those we didn't remove.)

Thursday afternoon we got word from the family that the term "Find Kelsey" no longer fit their situation. The volunteers during the search had nicknamed themselves "Kelsey's Army" and that was the domain name the family wanted to use going forward. In light of the major challenges we had experienced with traffic, I advised against redirecting findkelsey.com to the new kelseysarmy.com. Instead, I posted a message at the top of the page referring people to the new site and indicating that comments would be closed.

And that's how it ended. The site went live on Monday around 5:00 pm and was mothballed late Thursday night. It was in production just over three days, and yet the ministry value it created in that short time was immeasurable.

Kelsey update

Where do I begin?

Many of you no doubt have already heard that Kelsey Smith was found murdered today. My wife, Laura, was at the Smith home when the detectives came with the news that a body had been found and it was likely Kelsey. The identification was later confirmed. Laura and my daughter, Beth, are attending a prayer service at Hillcrest Covenant Church as I write this. Prayers are flooding in from across the country and around the globe. Please add your prayers to them. I will have much more to say about this in the coming days.

Quite aside from the unspeakable human tragedy, my technical team and I learned a great deal through this. The traffic to www.findkelsey.com (see mirror of original HTML site) was 160,000 page views yesterday and grew exponentially this morning. By 10:00 am we had serious performance impact, essentially taking down Vine Hosting, the ISP where our web servers are located. Searches for findkelsey.com reached #5 on Google as reported by Google Trends. You read that right. Our little hand-built quickie site was the #5 search on Google today. (Right now "Kelsey Smith" is #3, "find Kelsey" is # 7, and "findkelsey.com" is #9.) The flood of legitimate traffic was so great, Glenn - the director of operations for Vine Hosting - at first thought it was a DDOS. In reality, it was all legit traffic. Too much traffic. Glenn is still dealing with the aftermath.

The only people with enough resources to absorb this kind of traffic is, that's right, Google. Realizing the disaster we were causing for Vine, I made the command decision to totally take down all traffic related to findkelsey.com at Vine, even the DNS. Then, as rapidly as possible, we created a new site on Blogger (which is owned by Google), changed to a high-volume DNS host, and pointed the DNS at Blogger. As soon as the DNS change propagated, we started getting comments on our posts. In fact, my first indication that we finally had things working was when a comment appeared in my inbox. It took almost another hour to figure out how to get most traffic coming to us as it should. In the following 2.5 hours, 754 comments have come in from around the country, directed to Kelsey's family. That's one comment every 10-15 seconds. Unbelievable.

Today I had the experience of building one of the highest traffic sites on the Internet. I was building it WHILE it was being hit.

Find Kelsey Smith

My daughter, Beth, was in band at Shawnee Mission West High School with Kelsey Smith, the 18 year-old girl apparently abducted from a Target parking lot here in Overland Park on Saturday night. The band has a very active parents' organization called Band Aides West. Within 12 hours of Kelsey's disappearance, Band Aids was taking the lead to organize the search and support the family. Band Aids asked my wife, Laura, to go to the Smith home yesterday and take a shift answering the phone. After arriving there, Laura became aware of a need for a web site about Kelsey that could handle the load coming from national news exposure, including Greta Van Susteren and Nancy Grace, as well as extensive coverage from the local media.

Through the intense efforts of Brian, Leo, and Chuck http://www.findkelsey.com/ (see mirror of original HTML site) went live yesterday around 5 pm. Glenn Kelly and Vine Hosting are providing hosting at no charge. Glenn even missed part of his daughter's second birthday party to help set up the site on multiple servers to handle the load.

The site is linked from the home page of the Kansas City Star, the story about Kelsey on Fox News, and many other sites. It is also on flyers being posted all over town. The site is now averaging 2.6+ page views per second during waking hours.

Please stop what you are doing right now and pray for Kelsey, for her family, for the police, and for all of the volunteers who are supporting them and helping search.

KC RoundTable update

The KC-area CITRT met for lunch today. It was a blessed time with Mike, Ron, Jason, Jim, Warren, and Dwight, along with my Resurrection staff. I reviewed with them our plans for the Fall RoundTable and got their feedback. We are VERY excited about what God is doing in the church IT world and what will happen here at Resurrection the first week of October. I can't wait!

Our next meeting will be at Indian Creek Community Church on June 28 at 12:30 pm. All those interested in attending should RSVP to Jason Reynolds, IT Director at Indian Creek. Geeks for Jesus!

ESX update

Our VMware ESX project is well underway. Ian, our server/network admin, and I have been spending a great deal of time communicating with our ESX consultants and documenting both our existing configuration and our intended configuration when the project is complete. A detailed plan is beginning to emerge.

While we're at it, we decided to upgrade the power distribution and core network switching in our data center. We have temporarily stacked two APC 2200 UPSes, the new Dell/EMC AX150i SAN with its dedicated UPS, and both the existing and the new Dell 2950 servers. All of this will be permanently installed in our new Dell cabinet when it arrives this week. We also installed a second HP 4104 switch to provide redundancy for both the SAN and the primary network.

Our consultants have loaded ESX Server onto the new 2950 server. The next step is to install Virtual Center onto a desktop computer that will be used for management. Once that is done, we will avail ourselves of Dell's remote install assistance they bundle with every AX150 sold.

Our consultants will be in all day Friday and Saturday to complete as many of the remaining tasks as possible. Our goal is to have at least one or two virtual servers running on the new 2950 by Saturday.

Weigh in on Clif's smartphone decision

It's finally time for me to upgrade to a smartphone. I've never had a smartphone before. I used a Palm Pilot for almost four years until it died a couple of months ago and my cell is a 5 year-old workhorse that's nothing more than a phone. I definitely need the phone to have EVDO Rev. A for use as a modem for my laptop. I use Sprint and I'm not interested in Blackberry. I've read a lot of reviews, narrowing my short list down to the Treo 755p, the Motorola Q, or possibly even the PPC 6700.

I need real-world advice from smartphone users. Should I assimilate into to the Treo club? Should I go sleek with the Q? Or should I put a priority on WiFi and choose the PPC 6700? Should I wait for Windows Mobile 6? Should I stick with Palm OS? Help!

EVDO rocks

Many of you know that my wife is a church planter. I've been struggling with finding a good way to get high-speed Internet service at the church since we moved into the building more than two years ago. DSL service is not available in that semi-rural location and cable is too expensive.

In late January Sprint completed their roll out of EVDO Rev. A service in the Kansas City area. The first day it was available I got a Pantech card and started testing/using the service everywhere I went, including the church. Bottom line: it rocks. It gives a user experience very similar to DSL or even cable for everything except large file downloads. I'm seeing speeds in the neighborhood of 750+ Kb/s down and 250+ Kb/s up (though it does vary somewhat by location and time).

So having passed the performance and reliability test, I decided it would work well for the church until a faster, affordable land-line service becomes available. Yesterday I got a Linksys WRT54G3G-ST router that accepts an EVDO card and/or a wired broadband connection, routes, has a 4-port switch, and does WiFi. In other words, it's a WiFi router that can use EVDO to get to the Internet. This thing is very cool and it works quite well, but it was a pain to set up. Turns out you need to upgrade the router firmware before it can see/use the Pantech card, but finding the upgraded firmware on the Linksys site is nearly impossible. (By the way, it's here.)

Now ... think about how this could fit in to your disaster recovery planning. This little guy is a completely mobile WiFi hotspot. All you need is electrical power and EVDO coverage. (If it can't find a Rev A tower it will drop back to the the older EVDO Rev 0 or 1xRtt.) Admittedly, the bandwidth is pretty minimal for a subtantial number of users trying to share an Internet connection, but it could be set up instantly almost anywhere. One of these and some computers could provide an instant office with zero lead time to order DSL or T1 service to a location.

In the event of disaster, you could go to your local Sprint or Verizon store, sign up for service, and walk out within an hour with an activated card. Getting an EVDO router could be a bit tougher to find on zero notice. Stores like CompUSA are carrying some of these, but not all are compatible with all EVDO cards and services. You need to verify compatibility prior to purchase. We tried a D-Link router we bought at retail with no success and had to take it back.

Also ... think about how this could be used to provide backup for a critical Internet link in a remote location such as a second campus. I configured the Linksys to give priority to the wired WAN link and connected that link to my cable modem at home. Then I connected my laptop to the router via WiFi and I was on the Internet through the cable. To test the failover, I pulled the plug on the cable. Instantly the Linksys connected to EVDO and continued my sessions without interruption. Neither my Skype nor my MSN IM connection dropped. Sweet!

Adam Hamilton at the RoundTable!

I'm very excited to announce that our senior pastor, Adam Hamilton, will be coming to the Fall RoundTable for a 30-minute Q&A session. Adam is an amazing man who started this church in 1990 and has seen it grow to more than 12,000 adult members and average worship attendance in the 7,000 range. I am grateful he has accepted my invitation to spend some time with us during one of the busiest weeks of his year. It will be very interesting to get Adam's insight into the value of IT in the church and how we IT people can better serve the mission, minisry, and senior leadership of our churches.

KC-area church IT lunch

The next KC-area church IT lunch will be at Chili's in the Merriam Town Center mall at Antioch and Johnson Drive in Merriam, Kansas on Thursday, May 31, at 12:30 pm. See you there!

Include more people and keep it small

Tony Dye hit the nail on the head when he said that the consistent sentiment of Church IT RoundTable participants is that we should "include more people and keep it small." That paradox still makes me chuckle every time I think about it, but it does set out a vision that I believe (all humor aside) we should pursue. How could we do it?

In my original invitation to come to Church of the Resurrection for the Fall RoundTable, I explained a half-baked idea about how the RoundTable can grow while retaining the best aspects of being small. The essence of the idea is to have roundtables in cities across the country and then link them technologically a couple of times each year for a national roundtable of roundtables. Now I'd like to update you on how that half-baked idea is maturing into something that could be truly workable.

Our plan for the Fall RoundTable is to try this once with everyone located in our building. We think that is a good idea because it will allow us to test our facilitation technique and technology with all of us physically together. (Prototype? Cool!) If something goes horribly wrong, we can scramble and put everyone together in one large room and salvage the event. (Contingency plan? Cool!)

We are going to break up the large group into four separate classrooms, each of which will accommodate 20 participants and a few observers. With this idea we will be able to handle as many as 80 participants and 20 vendors/observers before we have to close registration. (I've been told this is similar to the approach taken by Leadership Network, but I've never been to their events so I have to take the word of others on that.)

Each classroom will have two projectors: one showing content from the conversation in that room, and the other showing content from all of the other rooms. Each participant will use their own laptop to join an Adobe Connect meeting with everyone else in their room. Through Connect, anyone in the room will be able to show content (web sites, pictures, diagrams, PowerPoints, etc.) on the primary projector. United Methodist Communications (UMCOM) has agreed to let us use their Adobe Connect system at no charge, and Sean McAtee from UMCOM Tech Shop will be here to assist us with his deep expertise in Connect.

So that we are not limited to interacting only with the participants in our own room, the secondary projector will show content from the conversations happening in the other rooms. We're still in the process of figuring out exactly how this will work. We will run some tests within the next couple of weeks to evaluate various ideas. I'll post once we have a clear picture of this in order to solicit your feedback.

When I suggested this idea to Tony, he said, "Sounds exciting, crazy, and even a bit prone to potential failure. Perfect! :-)" Tony's right. This is both exciting and risky. But I figure, if a bunch of techno-geeks can't make something like this work, who can?

Fall Roundtable schedule overview

I'm very excited that so many of you have commented on my invitation post indicating your interest in coming to the Fall Roundtable, October 3-4, 2007. We're actively working on detailed plans so we can open registration.

We will start on Wednesday around 9:00 am and end Thursday at 5:00 pm. We're also planning a get-together for Tuesday (October 2) afternoon around 3:00 pm for those who are flying in on Tuesday and would like to get a facilities/IT tour of Church of the Resurrection. After the roundtable is Resurrection's annual Leadership Institute, October 5-6. Leadership Institute is our biggest conference of the year. If you are able to stay, I know it will be a blessing to you as well.

For any vendors that would like to attend, sponsorship opportunities are available. Please e-mail me for info (clif.guy at cor dot org).

Watch this blog for more deets to follow soon. It's going to be an AWESOME event!

ESX, here we come

We're currently running the free VMware server using Windows 2003 Server as the host operating system on our original Dell 2950 server. We have 8-10 virtual servers running on that box now, none of which are demanding resource-wise (gratefully).

Three weeks ago we bought a second 2950 like the one we bought last year, a Dell/EMC AX150i iSCSI SAN, and a bunch of expensive ESX 3 software from VMware. We're going to hook the two 2950s to the SAN, load ESX on them, and convert all of our existing virtual servers as well as 3 older physical servers to the new virtual environment. We have hired some consultants to help us and hope to have it in production by June 4. We'll keep you posted ...

Sitting in church

Miles McPherson from The Rock Church in San Diego presented this week at the Beeson Institute. My favorite line from his talk: "There is no spiritual gift called 'sitting in church.'"

The Roundtable is an "Unconference"

BusinessWeek has a nice overview of the unconference idea that we've stumbled into for the church IT community. "Roundtable" is a much better word, in my opinion, than "unconference" or "camp" (as in BarCamp or PodCamp). I'm glad we're doing an unconference and calling it a roundtable - that's the best of both!

Mike Breaux at Resurrection

The Beeson Institute for Advanced Congregational Leadership is having one of its sessions at Church of the Resurrection this week in our Student Center. Mike Breaux, teaching pastor at Willow Creek, presented today.

Mike says our churches should have signs like:
"Come As You Are"
"Grace Happens"
"You Matter to God"

Goodbye Annex

Demolition Day has arrived for the humble annex. Turns out you can use a backhoe to bring down a building.

In the upper left corner you can see our camera crew recording the whole thing from the roof of our West Building. I'm sure they'll find some creative use for the video!

Vista when?

The IT organizations of Intel and Dell are waiting for SP1 to upgrade to Vista. I know we'll run Vista eventually, but I also know it won't be in 2007. We'll look closely at it in November when we need to decide whether to budget for it in 2008. If it doesn't seem like a good move, we'll wait and look at it again in November 2008.

What are your Vista roll-out plans?

KC lunch meeting?

I'd like to get together for lunch with my KC-area church IT homies sometime in the next couple of weeks. How about the 10th, 17th, or 31st? Leave a comment ...

Silverlight?

Who out there is paying attention to Microsoft's Silverlight technology announcements? Is it as big a deal as Scoble suggests?

Spring Church IT Round Table - Round Table Day

I've been quite the slacker in blogging this, but better late than never eh?

The round table was a great experience. In short it was about 30 guys seated around folding tables, and discuss we did. The dynamic of the round table went like this. A topic was thrown out and an expert board was formed on the fly. Expert board = the guys and gals that had worked around that topic a great deal. People would ask the board questions, the board would question or concur with each other, and great value was gained by everyone in the room. I thought it was a well-balanced event between free-flow conversation and a structured discussion.

The best analysis I've seen, gathered from feedback of the event, was Tony's summary of what everyone wants. As Resurrection is hosting the Fall event this year, I had walked around the crowd during breaks and sampled feedback on what we should do next - I came to the exact same conclusion as Tony. "Keep it small, but include more people." Sorry boys and girls - some of you are gonna have to take one for the team. :) In all seriousness, "taking one for the team" is exactly what we want to avoid.

Not to beat a dead horse, but if you haven't given feedback on our idea of how to truly tackle "small but more people," take a look at the invite Clif posted and let us know.

Overall, the spring round table was an unparalleled experience. Every individual was great, and they inspired me. The conversations in the hall, and the more structured times blended well. I would hope that if you are in the ministry of church I.T., yes it is a ministry, you have the opportunity to make one of the future round tables and be fed, have your tank filled, or other appropriate metaphor. ;)

Props to all my church I.T. brethren, it was a pleasure and will be again.

Church IT Blogroll

Today I realized I had never posted a roll of the church IT blogs I read. So I went through and cleaned up the subscription list in my feed reader (IE 7), exported the list as OPML, and then imported it into blogrolling.com in order to quickly create the new blogroll you see on the right.

Then I realized, this could be a really cool way to link church IT blogs, using the same approach as the Methodist Blogroll. So, here goes ...

Church IT Blogroll Instructions
Benefits:

• We will maintain the blogroll and update it regularly
• As links are added, updated, and deleted, it will update on your blog automatically
• A great way to keep track of the church IT blogging world and build the fledgling Church IT Roundtable/Church IT Association

Requirements:

• Your blog is focused on church IT and/or church web sites
• You update it regularly (if we see no posts for a few months, we'll take you off)
• You provide an RSS feed of your posts (because the feed is what we'll actually put on the blogroll)

To get on the blogroll:

• E-mail me (clif.guy at cor.org)
• We'll check out your blog and add you if you meet the requirements

To add the blogroll to your own site, paste the following code in your blog template:

<script language="javascript" type="text/javascript" src="http://rpc.blogrolling.com/display.php
?r=9407d5ed585629e658efc2318e1add48"></script>

FBI update

In my wee-hours haze last night I linked something totally unrelated to Andrew's video. It's here.

The FBI pays us a visit

The Annex is going away. It's worth less than it would cost to move, so what to do? I know! Blow it up, real good! Our Facilities folks invited the local FBI SWAT team to use it for practicing a hostage situation. Andrew took a bit of video showing the guys in full battle gear assaulting the Annex. And here are some pictures of the damage they left in their wake.

Annex main entrance with shattered glass everywhere:

Glass from the back door lays shattered on the floor:

A spent shotgun shell (they used shotguns to shoot open locked doors):

Smashed door hinge:

This door latch was blown off by a shotgun blast. It threw the door knob into the sheetrock wall, making a nice hole:

The black spot on the floor is from a "flashbang" - a non-lethal grenade that makes a big flash and a loud bang to stun anyone in the vicinity. You can see damage on the adjacent partition. The white powder on the floor is fire extinguisher residue:

Somehow they knocked a hole in the outside wall. Yes, that's daylight you can see through there!

Lesson: don't be in the way when your friendly, local FBI SWAT team storms the building!

The Leawood Police are coming in tomorrow for more training exercises. There's nothing left to smash. ;-)

Seth's "Memo to the very small"

Those of you who subscribe to Seth Godin's blog have already seen his post a week ago titled "Memo to the very small." Interesting that he mentioned churches in the opening paragraph as possible users of his method for small organizations to use the web.

People from small or technically-unsophisticated churches frequently ask us for advice about how to build or rebuild their church web site. Seth describes these people perfectly when he says:

These are businesses that have trouble dealing with the Yellow Pages. Too much trouble, too much time, way too expensive. So, should local micro-businesses just ignore the web? Or should they become experts in the art of building and maintaining a website?

His suggestion is to use Typepad with a standard template, a Squidoo lens, and a set of pictures on Flickr. Those of you who work with small churches as a volunteer or consultant, does this sound like a good recommendation?

Neat Patch?

We don't need no stinking Neat Patch. (Sorry, Jason, couldn't resist.)

This is one of the racks for our new office at Southcreek. It has two 48-port HP switches and fewer than 10 ports are free since we completed our moves yesterday.

On the shelf below the patch panel is our SonicWall firewall and DSL modem. These provide a backup (a VERY slow backup) for our wireless LAN bridge.

Moving Day 2

Yesterday was the last big moving day in a sequence of moves we've been doing for more than a week.

Below is the humble Annex, a temporary building obtained for our rapidly-expanding Children's Ministry back in the mid 1990s. For the last several years it has been the central church office, housing our receptionists, mail room, main copy/work room, most of our executive management, and numerous other staff. It's hot in the summer, cold in the winter, and frequently smells of animals that have crawled underneath to die! Next week it will be used by local law enforcement for training exercises before it is finally dismantled and hauled off.

Front door to the Annex with the East Bldg reflected in the glass. The sign directs visitors to the new reception desk in the East Bldg.

"We are moving! Please pardon our mess while we pack"

Load 'em up!

And move 'em out!

Spring Church IT Roundtable

I neglected to mention in my first post that today we are being hosted by Sugar Creek Baptist Church, http://www.sugarcreek.net/, at the LYF youth center. Our hosts were very gracious, and I want to thank them for opening up their facility and catering to us. (Image to right is an X-Box array for youth.)

Today we heard from many Vendors and consultants ranging from VMWare, ScriptLogic, DataCore, Service-U and Web Empowered Church.

I was familiar with VMWare and Service-U, and obviously Web Empowered Church, but the other two I'd never heard of before. DataCore is a mind expanding virtualization software that virtualizes your SAN or slack disk space across multiple servers, and synchronizing between virtual storage. I'd have to think about that more, but one thing that is potentially big for churches is synchronizing virtual storage between geographically distributed locations. What we were told was that you can synchronize between 8 different locations. So if multiple churches are using DataCore and have unused disk space, they can use each other as back up locations. Potentially pretty cool.

ScriptLogic produces a toolset of automation and management applications that makes nearly everything in Windows system administration easier. Active Directory users, GPOs auditing and alerting, patching, spyware, file permissions and ACLs, USB port security, MSI packaging, system imaging and software distribution, and file system level auditing. I'm excited about the potential of using ScriptLogic apps in our environment because it is easily powerful enough to make up for a half time network admin guy or more, and our network guy would salivate all over his shirt to have these tools at his disposal. The tools are not only cool, but can replace other tools like Symantec Ghost and MS SMS, saving you some money nad giving you one software manufacturer to yell at when things go wrong. :)

All in all, a very cool pre-roundtable day and I enjoyed meeting the vendors, including Tim Whitehorn, CEO and original creator of Service-U. Resurrection is a long time user of Service-U, and I have always suggested it to churches looking for a resource management and calendaring solution.

Looking forward to tomorrow, but first - a good nights sleep. Aaaaah.

And Here We Go!

I'm here at the church IT event organized by Tony Dye and friends. Our first day is vendor presentations - but first we started off with a short worship service. It was amazingly high energy for a ~30 minute service. In short, they rocked, we worshiped, and when they finished everyone simply stood in awe and silence for many seconds until someone started that quiet "is it ok yet?" clapping.

Thus far I've been thrilled to meet some of the guys I know only by blogs and e-mail addresses. I expect shock and awe over the next few days. Geekfest. :)

Moving Day 1

We're working closely with our Facilities Department to move 50 staff people, starting yesterday through the end of next week. Yesterday was Move Day 1 when we moved 21 people from Central Campus to the leased office at Southcreek. Next Wednesday and Thursday we'll be moving 11 people from one place to another within campus (including Andrew). And then Friday we'll move another 18 people from campus to Southcreek. Whew!

Philip (foreground), Jeremy, and Brian working among the cubicles:

Movers from Fry-Wagner:

Dick Cooper, our Director of Facilities, holding forth:

Jeremy prepping phone cords:

Ian, concentrating on settting up phone extensions, amid the chaos:

Kelly Williams (center) and some of her staff from the Finance Department surprised us with lunch (what a blessing!). Uber voluteer Doug Blackwood, retired from management of the help desk at HP, is on the far right.

Outfitting the new office space

Here's some of the work we've been doing over the last three weeks to prepare for 40 staff moving into the 9000 sq. ft. of office space we leased to replace our 13 year-old temporary building.

In the process of installing data and phone cables:
Ian and Jeremy begin installing equipment in the phone/network room:


Jerry and Kevin from WKT (our phone vendor) installing the phone system:

Ian installing a repurposed Dell 2650 for use as a file/print server (it does DFS replication with a server in our data center on Central Campus):

Should we say "no"?

Another answer to Tony's question comes from Joel Dehlin:

I.T. should not be saying no. You should give the customer(s) information about what resources you have available, and they should (collectively) make decisions about what to do with the resource. You can cajole, influence, or merely suggest, but the customers should be making these calls.

Rain fade update

I posted previously about the wireless LAN bridge we installed from our Central Campus to our new leased office space, and I mentioned the problems we've had with rain fade.

Since my last post I learned that the radios nominally use the 80 GHz band, but since it's a full-duplex link, there are two diffferent frequencies in use: around 80 GHz in one direction and around 72 GHz in the other direction. Consequently, the signal strength in one direction isn't the same as the signal strength in the other direction. The graph I posted previously was for the stronger of the two.

Since my last post our vendor has repositioned the radios again and wrung out another 3 dB, which is great. Unfortunately, the link is dropping at -66 dBm, whereas the spec says it won't drop until somewhere in the -69 to -72 range. And according to Bridgewave, our radios tested to -69 in the factory before they were shipped. So we still have a problem.

Here is the signal strength graph from yesterday when in rained starting around noon and continued at varying intensities for the next 24 hours:

In all of that we had only one drop lasting around 30 seconds at 3:39 pm. You can't see the excursion below -65 because the monitoring system is on the other end of the link. (The monitor needs the link to be up to grab the signal strength from the other end.)

Now, one 30 second drop doesn't sound to bad considering it's been raining for 24 hours. But the problem is, if we had staff working here yesterday, every phone call in progress at 3:39 pm would have been dropped. Most likely every Shelby session would also have been dropped. That doesn't seem good enough to me. What do you think?

Three values

In reponse to Tony's post about how much to serve users ...

This is good stuff, Tony, but twelve values is probably too many. I try to simplify things down to three if at all possible.

1. Our default answer is "yes". We always hope to find a way to meet the user's need, even if it isn't in quite the way the user expressed the need or quite the solution the user proposed. See: http://appianway.blogspot.com/2006/01/our-default-answer-is-yes.html

2. Our mission is our user's mission. That is, when we're working with Children's Ministry, our mission is to teach kids about Jesus; when we're working with Congregational Care, our mission is to visit people in the hospital; etc. Rather than focusing on our own mission, we're focused on the mission of those we serve.

3. The one mission that is uniquely our own is information security. If we don't look after information security, no one will. We define information security as providing or denying information access as appropriate and ensuring that critical information can't be lost. If #1 or #2 above conflict with security, then we make a tradeoff.

We don't always achieve our internal IT Department goals (roll out the software update by Feb. 1, upgrade environmental monitoring in the data center by the end of the month, etc.), but we always get high marks from our user community. As far as I'm concerned that's the correct priority. And of course, we have to work very hard behind the scenes to make sure we don't work exclusively on short-term or user-visible things. Sometimes the best thing we can do for our user community is to work on something that takes a lot of resources and users never know about. But that's nothing fancy, it's just good, disciplined execution.

Fall Roundtable update

From the number of comments on my Fall IT Roundtable Invitation, it seems like we'll have a great group here in Kansas City for the Fall Roundtable. For now I just wanted to acknowledge all the responses. There will be a lot more information and discussion about the Fall event during the Spring Roundtable in Houston. We want to get feedback from the Spring Roundtable before we start posting more specifics.

Have a blessed Good Friday and Easter.

Rain fade

I posted earlier about the wireless LAN bridge we installed from our campus to our new leased office a couple of miles away.

These radios operate at such a high frequency (80 GHz) that nearly everything -- a tree, a dense flock of birds, an errant golf ball (okay, I just made that one up) -- is opaque to the signal. Consequently, they suffer from signal attenuation due to rain. This is a well-known phenomenon called "rain fade" that the RF engineers have to take into account in their designs.

We're having some trouble getting sufficient path clearance (getting the base signal high enough) to avoid a link drop during periods of intense rain. Right now the base level is approx. -44 dBm. We were expecting rain fade of approx. 16 dBm, which should make it bottom out around -63 dBm. According to Bridgewave, these radios are supposed to be able to maintain the 100 Mb/s link down to -72 dBm. Therefore, we should have approximately 9 dBm of safety margin.

But we have two problems. First, we're seeing rain fade of 22 dBm. Second, we're seeing the link drop when the signal goes below -65 dBm. With 6 dBm more rain fade than expected and the link dropping 7 dBm before it should, our 9 dBm of safety margin is shattered.

Here is a graph showing two times when we had brief periods of intense rain. Each time the link dropped for a few minutes and then came back up after the intense rain passed:

After the the second drop you can see other periods of rain fade that weren't intense enough to cause a drop. (The graph shows signal level in 100ths of a dBm. i.e. -4400 is -44 dBm.)

I gave a requirement of four 9s of availability (99.99% up time). Clearly, with the current base signal level, the amount of fade we're experiencing, and the link drop level, we're not going to be close to four 9s. Thankfully, we're installing this well ahead of staff moving into the office so we have time to improve the situation. Also, I'm grateful that we're installing in the Spring when we have rain every couple of days. Each time we make a change, we don't have to wait too long before a real-world test presents itself.

We're out there on the bleeding edge with this one! Please pray with us that this wireless LAN bridge story has a happy ending.