May 31, 2008

Our new web home

Our Resurrection web sites are currently hosted on a pair of dedicated servers at Vine Hosting in Philadelphia.  (Vine Hosting is affiliated with Web Empowered Church, which has received major funding from the Methodist Foundation for Evangelism.)  Vine Hosting's data center is colocated with XO Communications in a massive carrier hotel at 401 N. Broad.  From there they have direct access to bandwidth from XO, Verizon, Level 3, and Internap.  This is a true, world-class data center facility.  I don't recall having a single outage or service interruption with Vine since they moved into the carrier hotel. 

Through our time with Vine Hosting, Glenn Kelley, its founder and high-energy leader, has become a trusted friend and partner in ministry.  No one is more passionate about sharing the gospel through the use of technology than Glenn.

So why are we moving?

Before I settled on Vine in 2005, I did an exhaustive search of hosting facilities in Kansas City.  I found a couple of decent ones, but nothing at the level I was seeking.  So we eventually hooked up with Vine and the ensuing partnership has been excellent as I mentioned.

Three years later, the situation has changed.  Kansas City now has a world-class carrier hotel of its own.  The 70 year-old, 26 story Bryant Building at 1102 Grand sits right on AFS's KC metro fiber ring.  All of the major telecom companies in KC are connected to the metro fiber ring and all of them except Sprint have a presence in 1102 Grand.  Fiber comes into the building via diverse underground vaults.  The building gets power from two separate KCP&L substations, has a 2 MW diesel generator in the basement, and provides centralized UPS power for those tenants that want it.  To keep everything cool, the building has 1100 tons of cooling.  Physical security is provided by card and code access and monitored video surveillance.


Ian and I got a tour in March and knew right away we were seeing something special.  Our wheels began turning.  We immediately recognized the potential for 1102 Grand to be the hub of a regional network connecting all of our church facilities.  We already have staff located in Leawood, Overland Park, and Olathe.  Our next campus will be in downtown KC, just two blocks from 1102 Grand.  We have land in Cass County for a retreat center.  And we're beginning to dream about a campus several miles south of our Leawood location.  Over the next few years we will need to come up with an elegant, cost-effective way to connect all of these locations.

Next, we thought about 1102 Grand as a potential disaster recovery site.  Business interruption/disaster recovery planning has been on my radar since I joined the staff in 2003.  Each year we have made incremental improvements in our DR posture.  For example, we now use Iron Mountain for offsite backup tape storage.  But I could never find an affordable, nearby DR site.  Until now.

The more we thought about it, this opportunity was simply too compelling to pass up even though we are extremely happy with Vine Hosting.

Move in

Yesterday we moved in.  Below is Ian setting up his web cam before starting the install.  The cage behind him is part of the carrier-neutral Meet Me room operated by the building owner where tenants, including carriers, ISPs and end users like us, connect with each other.  Our provider is KCNAP.  They have a patch panel in the Meet Me room.  From there it's just a short cat 5 cable drop into our cabinet.  We have the bottom third of the cabinet in the background (they charge less for the bottom third and we're very price-sensitive!).

Ian preparing for install

Below is the front of our cabinet.  Note the broom in Ian's hand to sweep out the dust and debris from the floor before installing any equipment.

Resurrection's cabinet

Below is the back of our cabinet.  A 3U-tall power distribution/fan unit is at the top.  Fully redundant power (two city power grids, diesel generator, and UPS) comes into the cabinet in the outlet box at the left.

Resurrection's cabinet

Below is what it looked like after physically installing the four 1U servers, network switch, KVM switch, and a flat-panel monitor.  The rack rails we got from Dell won't work in this cabinet, so we'll have to get the right ones in order to permanently install the Dells.  You can see them temporarily sitting on top of an Appro server donated to us by Tradebot Systems.  Note the jack hanging down in the upper right corner of the picture.  That's the drop from KCNAP.  They are multihomed to multiple tier 1 providers via BGP.

Resurrection's cabinet with servers installed

We have some high-quality Cat 6 patch cords that are too short for our data center on campus so we brought them along.  Too short for our main data center and too long for this cabinet!  So Ian did what all good techs do: he broke out the tie wraps!

Resurrection's cabinet with servers and interconnect cables

Although the physical installation went very smoothly (except the Dell rack rails), we had to leave yesterday afternoon before we could establish network connectivity.  Ian was having trouble with the pfSense firewall/router he configured.

We have until the end of June to move all of our existing sites and services from Vine into the new data center. We're very excited about the new capability this represents for us.

Technorati Tags:

May 29, 2008

Not keeping up with the price of gas

I bought gas at the Phillips 66 station in Parkville last night.  The van tank holds approximately 23 gallons and it was nearly empty.  Price was $3.799, so that's an $85 fill up.  Towards the end of the fill up, I noticed the flow slow and then stop exactly on $75. Apparently the pump had authorized my card for $75 and it wouldn't allow me to pump any more.  I had to complete that transaction and then initiate a second one to pump the last $10.

At $2/gal, $75 would fill any tank except the largest commercial vehicles.  At $3.799, $75 won't fill an ordinary Ford Freestar.  You think they should change the pump software to authorize a higher amount?

May 23, 2008

Plan the celebration too

The next time you have a large project that takes months to plan and execute, be sure to plan the celebration too. For the Arena implementation project we recently completed, we planned two celebration events.

First, we invited all staff to an ice cream social in the Student Center on the afternoon of go live. We took Shelby down starting at noon. At 2:30 pm we brought Shelby back up (now in a limited role), turned on Arena for all staff, and started the party. At the party we used the Internet cafe computers for staff to log in and try a few simple things in Arena. The timing turned out to be really cool. By having the party exactly as the system was going live, we weren't buried in help desk calls and it was too soon for anyone to have had a negative experience. Bonus!

For the two weeks immediately after go-live, we beefed up our help desk by increasing the hours of our contractors, Philip and Leo. This allowed us to have a "dispatcher" physically at the help desk at all times to take calls, e-mails, and work orders.

When this period was over, we had a second celebration. This time it was just the IT Department (Ian's daughters crashed the party). Six months ago Brian moved to a house just 3 miles from the church. We decided to seize the opportunity to invite ourselves over to sit on his back deck and just relax. We all brought something to put on the barbecue grill. God provided a spectacular spring day and a good time was had by all. Here are a few pictures.




Yours truly










Philip and Jeremy

Technorati Tags:

May 20, 2008

Fall RoundTable, yeah baby!


Attention all church IT people.  The Fall 2008 Church IT RoundTable will be hosted by Trace Pupke at Seacoast Church in Mount Pleasant, South Carolina (Charleston area) October 8-10!

Seacoast is among the largest, fastest growing, and most innovative churches in the country. Don't miss the opportunity to see and learn about this awesome church while connecting with your fellow church IT people.

I will be there along with Brian and Jeremy from my team at ResurrectionI strongly urge you to join us in Charleston to meet a bunch of amazing people and find out if this for you.  We learn a lot from each other and, more importantly, we draw inspiration from each other.  To understand what I mean by that, read my posts after the Fall 2007 RoundTable and the Spring 2008 RoundTable.

Trace has set up a separate blog just for the Fall 2008 CITRT. Check it out here and subscribe to the feed in order to keep up to date with all the details as they come out.

I'm jazzed!  Are you?

Technorati Tags:

May 19, 2008

IDF rewiring project

We've been working with our Facilities Dept. since March on a new office suite for our pastors of caring ministries who handle the traditional "pastoral" functions such as sacraments, weddings, funerals, hospital visits, counseling, and worship leadership.  Over the last 5 years this group outgrew one office and spilled over into two additional offices in separate parts of the building.  We hired a contractor to come in and remodel a set of classrooms into an office suite large enough to consolidate all three groups of staff. 

The part of the building where the new suite is located is served by IDF 2E. Like the other IDFs in the older East Bldg., 2E was originally wired by well-meaning volunteers who had no concept of professional cable installation.  Everything was a tangled, unlabeled, undocumented mess.  There was a wall-mounted half rack for the data patch panel and a traditional telecom mounting board with a 66 block for the voice.  There was no way to expand the existing design to accommodate the additional 40+ voice/data pairs for the new suite.  Plus, the long-term plan is to convert that entire wing of the building from classrooms into offices.  So it made sense to completely rebuild the IDF to make it neat, well-documented, and expandable.

The electrical contractor for the new wing installed all the jacks and pulled the new cable to the IDF.  They installed a standard 2-post rack and dropped the cable through the ceiling in a pair of large sleeves.

Unfortunately, I didn't think of taking pictures until halfway through the project.  Here is a typical tangle of four different colors of existing voice wires in the ceiling above the IDF.  This is actually much better than it started because by the time I took this we had already cleaned up all the data cables.


Here is Ian working on the new rack.  All of the old data lines have been moved and he is almost finished with the new data lines.  You can see the fiber tray at the top, a rack-mounted power strip, two HP 2650 switches, and four rows of patch panels.  All the new cabling is Commscope UltraMedia Cat 6 - blue for data and white for voice.  The unterminated bundle of white voice lines is at the right.


We used the beautiful (but expensive) Panduit Cat 6 RJ45 jacks and their companion patch panels.  You can see how Ian had carefully labeled each wire with the new numbering scheme we will use throughout the wing served by this IDF.


The Panduit patch panel allows you to connect a jack to each wire separately and then snap the jacks into plastic brackets holding 4 jacks each.


This is the almost finished rack.  All of the new voice lines have been terminated.  We still need to move all of the old voice lines.  Notice the really cool floor plan pinned to the wall showing every jack location and its number.



Technorati Tags:

May 16, 2008

Bill Hybels breakfast

Yesterday was one of those days when I pinch myself to make sure I'm not dreaming that I'm on staff at Church of the Resurrection.

We had 300+ church leaders from 70 area churches in to hear Bill Hybels and to promote the upcoming Leadership Summit.  After a delicious hot breakfast, our senior pastor, Adam Hamilton, introduced Bill who gave a great talk on the Reveal study and the implications for every church.  Can't wait for Leadership Summit 2008!

My friend, Mark Baltzley, who earns his living taking pictures, brought his camera along and took more than 160 shots.  Here are a few of my favorites.  (All are copyright 2008, Mark Baltzley Photography and Design.)

Adam Hamilton at Bill Hybels breakfast

Bill Hybels in the Resurrection Student Center

Bill and his 4 circles

Bill Hybels in front of white board

Bill Hybels

May 09, 2008

Arena communications plan

Subtitle: "Another lesson in how difficult it is to communicate effectively"

Regular readers of this blog know that Church of the Resurrection went live on the Arena church management system earlier this week.  Travis (the project manager) and I knew that frequent and effective communication would be critical for the success of such a large project impacting all staff.  We developed and executed what I think is a model plan not only for selecting and implementing a church management system, but also for keeping the staff informed throughout the process.  The below outline shows the many ways, times, and occasions on which we communicated to large numbers of staff regarding the project and the status.

  • 8/23/07 - Project kick-off meeting.  Key staff from every department present.  All executive management present and supportive.
  • 8/24 - 9/21/07 - Requirements gathering meetings held with each department.
  • 9/6/07 - Project overview presentation at quarterly all staff lunch meeting.
  • 11/15/07 - Selection of Arena announced in Staff Chapel and subsequent all staff e-mail.
  • 1/10/08 - Implementation kick-off meeting.  All staff invited.  Arena demonstrated.  Go live date of 5/6/08 announced.
  • 1/29 - 2/14/08 - Arena functionality and design review meetings held with each department.
  • 2/7/08 - Arena demo presentation at quarterly all staff lunch meeting.  Go live date of 5/6/08 announced.
  • 3/13/08 - Arena training plan announced at monthly senior staff meeting. 50 training classes to be held over a 5-week period immediately prior to go-live.
  • 3/21/08 - All staff e-mail stating that all Shelby users need to take Arena training, with a link to review class schedules and sign up.
  • 3/17 - 4/28/08 - Once per week all staff announcement promoting Arena with reminders about training classes and go live date.
  • 4/10/08 - Staff Chapel and all staff e-mail announcement reminding people to sign up for training.
  • 5/1/08 - Staff Chapel and all staff e-mail announcement about the scheduled Shelby outage, Arena go-live, and go-live party on 5/6/08.
  • 5/6/08 - All staff e-mail announcement that Arena is live.  First sentence of second paragraph: "Starting now, you will use Shelby only for Financials, Check-in, and Contributions (Check-in and Contributions will be moved to Arena later this year)."

Looks like a solid plan, huh?  Help desk ticket received this afternoon:

I am not able to access ShelbyEZ. I don't know if I have forgotten my login name and password or what? I think it use to automatically connect..? Could you please let me know what my information is to access it? Thanks.

In response I sent the following e-mail to this person's supervisor:

Apparently after 9+ months of heavy communication about the Next Generation Church Management System and Arena, [this manager-level staff person] still isn’t aware that we aren’t using Shelby any more. 

What lessons do you draw from this?

Technorati Tags: ,

May 08, 2008

Chris Randolph joins the BBQ tour


Chris Randolph, lead support tech for Arena joined us yesterday and today for the immediate post go-live support period.  In accordance with our Shelby-Resurrection tradition, we had to take him for barbecue.  (We also had to memorialize it with the traditional bad cell phone picture taken by a well-meaning waitress with an unsteady hand!)  This time it was the original Jack Stack in Martin City.  Those of you who came to the Fall 2007 Church IT RoundTable will remember Jack Stack from our Wednesday night banquet.  Good stuff, even for Memphis boy Chris.

This Arena thing has been cool.  Every time we meet with one of their team, we eat meat.  When we think about Arena, we start salivating.  Pavlov's dog had nothing on us.

Technorati Tags: ,

May 07, 2008

Breakfast with Bill Hybels

Calling all my KC-area church IT homies. 


Bill Hybels, senior pastor of Willow Creek Community Church, will be here at Resurrection next Thursday morning, May 15, for a breakfast meeting to talk about leadership and promote the Willow Creek Leadership Summit coming up in August. 

All church IT leaders are invited to come to the breakfast as my guest.  Please drop me a comment, an e-mail to clif dot guy at cor dot org, or a tweet.  I have to turn in a count for the breakfast by Tuesday morning, May 13.  Help a brotha out - I am required to have at least one guest or I can't go myself!

May 06, 2008

Arena go-live party


Jason Gant, Director of Student Ministries (youth pastor) picks his toppings


Brent Messick, Managing Executive Director of Operations (my boss)


Debi Nixon, Executive Director of Adult Discipleship is EXCITED!


Specially trained "Arena Facilitators" each received a fuzzy monkey to make it easy for their fellow staff to identify them


Andrew Conard, Congregational Care Pastor (aka "Nerd Pastor") gives Arena the thumbs up


 Adam Hamilton, Senior Pastor, gets his first chance to use Arena

Technorati Tags: ,

Arena go-live

Today is the day we go live on Arena.  We've been planning and preparing for this day since August 2007, which is 9+ months.  To track our progress, check out:

Arena go-live war room

Time line starting now:

  • Shelby V5 goes down
  • Backup V5 database
  • Change permissions in V5 to limit what users can do in V5 so they will be required to use Arena
  • Install 2-way triggers
  • Start Arena agents
  • Test
  • Shelby V5 comes back up
  • Arena is live
  • Ice cream party to celebrate Arena go-live
Technorati Tags: ,

May 05, 2008

Next stop on the Shelby-Resurrection BBQ Tour

Ben Lane from Shelby is here for our Arena go-live tomorrow. Continuing our long-standing tradition, we had KC barbecue for lunch. Despite the crude and blurry cell phone pictures, I think you can get an idea of the experience. Yum!

President's Platter from Gates


L to R: Linda Ronsick (data quality), Ian Beyer (cyberentomology), Matt Bradshaw (bit shepherd), Ben Lane (disc golf connoisseur and Arena trainer), Leo Johns (consultant, pro keyboard player, amateur disc golfer), Travis Morgan (MBA and disc golfer), Doug Blackwood (uber volunteer), Jeremy Grabrian (a man confident enough to wear shorts), Brian Slezak (curmudgeon)

Ian and Matt are carrying boxes of leftovers. Yes, we had so much food that even with 10 people eating we couldn't finish it. God provided abundantly for us.

Go live is tomorrow. I hope we're ready!

Technorati Tags: ,

May 02, 2008

How to bring your powerful database server to its knees

Don't let the mild-mannered appearance of this group fool you.



We're in our last day of Arena training before go live on Tuesday next week. Chuck is teaching the Report Builder in SQL Reporting Services.  With these eager students, his first exercise was to simply select first and last name and then run the report.  This resulted in 9 simultaneous attempts to report all 60,000 people in our database.  It soon became clear we had a problem.

We're running SQL Server in an ESX guest VM.  This has been our configuration for 9 months with no issues.  The CPU on the host Dell 2950 server (total of 8 cores) spiked for a minute or two but then settled back down.  Yet the server was still whimpering in the corner.  No one could do anything.  We had a class full of students trying to use a server that was dead for all practical purposes.

Turns out SQL Server itself was okay.  IIS (running on the same ESX guest VM with SQL Server) was gulping enormous slurps of RAM to generate the multiple thousand-page reports.  Windows responded by trying to expand its page file.  Under ESX, this is not a good scenario.  We had an unscheduled outage on our main database server in the middle of the work day.  Clif not happy.  The good thing is, you add RAM to a guest VM with just a few mouse clicks and a reboot, which Ian did.  After a 20 minute delay, the class resumed.  From then on, Chuck made sure to run the examples himself with everyone else just watching.

With the number of guest VMs we have running on that host, Ian thinks we need to add to the 16 GB of physical RAM we already have on that box.  This is why IT is so FUN!