Moving Day 1

We're working closely with our Facilities Department to move 50 staff people, starting yesterday through the end of next week. Yesterday was Move Day 1 when we moved 21 people from Central Campus to the leased office at Southcreek. Next Wednesday and Thursday we'll be moving 11 people from one place to another within campus (including Andrew). And then Friday we'll move another 18 people from campus to Southcreek. Whew!

Philip (foreground), Jeremy, and Brian working among the cubicles:

Movers from Fry-Wagner:

Dick Cooper, our Director of Facilities, holding forth:

Jeremy prepping phone cords:

Ian, concentrating on settting up phone extensions, amid the chaos:

Kelly Williams (center) and some of her staff from the Finance Department surprised us with lunch (what a blessing!). Uber voluteer Doug Blackwood, retired from management of the help desk at HP, is on the far right.

Outfitting the new office space

Here's some of the work we've been doing over the last three weeks to prepare for 40 staff moving into the 9000 sq. ft. of office space we leased to replace our 13 year-old temporary building.

In the process of installing data and phone cables:
Ian and Jeremy begin installing equipment in the phone/network room:


Jerry and Kevin from WKT (our phone vendor) installing the phone system:

Ian installing a repurposed Dell 2650 for use as a file/print server (it does DFS replication with a server in our data center on Central Campus):

Should we say "no"?

Another answer to Tony's question comes from Joel Dehlin:

I.T. should not be saying no. You should give the customer(s) information about what resources you have available, and they should (collectively) make decisions about what to do with the resource. You can cajole, influence, or merely suggest, but the customers should be making these calls.

Rain fade update

I posted previously about the wireless LAN bridge we installed from our Central Campus to our new leased office space, and I mentioned the problems we've had with rain fade.

Since my last post I learned that the radios nominally use the 80 GHz band, but since it's a full-duplex link, there are two diffferent frequencies in use: around 80 GHz in one direction and around 72 GHz in the other direction. Consequently, the signal strength in one direction isn't the same as the signal strength in the other direction. The graph I posted previously was for the stronger of the two.

Since my last post our vendor has repositioned the radios again and wrung out another 3 dB, which is great. Unfortunately, the link is dropping at -66 dBm, whereas the spec says it won't drop until somewhere in the -69 to -72 range. And according to Bridgewave, our radios tested to -69 in the factory before they were shipped. So we still have a problem.

Here is the signal strength graph from yesterday when in rained starting around noon and continued at varying intensities for the next 24 hours:

In all of that we had only one drop lasting around 30 seconds at 3:39 pm. You can't see the excursion below -65 because the monitoring system is on the other end of the link. (The monitor needs the link to be up to grab the signal strength from the other end.)

Now, one 30 second drop doesn't sound to bad considering it's been raining for 24 hours. But the problem is, if we had staff working here yesterday, every phone call in progress at 3:39 pm would have been dropped. Most likely every Shelby session would also have been dropped. That doesn't seem good enough to me. What do you think?

Three values

In reponse to Tony's post about how much to serve users ...

This is good stuff, Tony, but twelve values is probably too many. I try to simplify things down to three if at all possible.

1. Our default answer is "yes". We always hope to find a way to meet the user's need, even if it isn't in quite the way the user expressed the need or quite the solution the user proposed. See: http://appianway.blogspot.com/2006/01/our-default-answer-is-yes.html

2. Our mission is our user's mission. That is, when we're working with Children's Ministry, our mission is to teach kids about Jesus; when we're working with Congregational Care, our mission is to visit people in the hospital; etc. Rather than focusing on our own mission, we're focused on the mission of those we serve.

3. The one mission that is uniquely our own is information security. If we don't look after information security, no one will. We define information security as providing or denying information access as appropriate and ensuring that critical information can't be lost. If #1 or #2 above conflict with security, then we make a tradeoff.

We don't always achieve our internal IT Department goals (roll out the software update by Feb. 1, upgrade environmental monitoring in the data center by the end of the month, etc.), but we always get high marks from our user community. As far as I'm concerned that's the correct priority. And of course, we have to work very hard behind the scenes to make sure we don't work exclusively on short-term or user-visible things. Sometimes the best thing we can do for our user community is to work on something that takes a lot of resources and users never know about. But that's nothing fancy, it's just good, disciplined execution.

Fall Roundtable update

From the number of comments on my Fall IT Roundtable Invitation, it seems like we'll have a great group here in Kansas City for the Fall Roundtable. For now I just wanted to acknowledge all the responses. There will be a lot more information and discussion about the Fall event during the Spring Roundtable in Houston. We want to get feedback from the Spring Roundtable before we start posting more specifics.

Have a blessed Good Friday and Easter.

Rain fade

I posted earlier about the wireless LAN bridge we installed from our campus to our new leased office a couple of miles away.

These radios operate at such a high frequency (80 GHz) that nearly everything -- a tree, a dense flock of birds, an errant golf ball (okay, I just made that one up) -- is opaque to the signal. Consequently, they suffer from signal attenuation due to rain. This is a well-known phenomenon called "rain fade" that the RF engineers have to take into account in their designs.

We're having some trouble getting sufficient path clearance (getting the base signal high enough) to avoid a link drop during periods of intense rain. Right now the base level is approx. -44 dBm. We were expecting rain fade of approx. 16 dBm, which should make it bottom out around -63 dBm. According to Bridgewave, these radios are supposed to be able to maintain the 100 Mb/s link down to -72 dBm. Therefore, we should have approximately 9 dBm of safety margin.

But we have two problems. First, we're seeing rain fade of 22 dBm. Second, we're seeing the link drop when the signal goes below -65 dBm. With 6 dBm more rain fade than expected and the link dropping 7 dBm before it should, our 9 dBm of safety margin is shattered.

Here is a graph showing two times when we had brief periods of intense rain. Each time the link dropped for a few minutes and then came back up after the intense rain passed:

After the the second drop you can see other periods of rain fade that weren't intense enough to cause a drop. (The graph shows signal level in 100ths of a dBm. i.e. -4400 is -44 dBm.)

I gave a requirement of four 9s of availability (99.99% up time). Clearly, with the current base signal level, the amount of fade we're experiencing, and the link drop level, we're not going to be close to four 9s. Thankfully, we're installing this well ahead of staff moving into the office so we have time to improve the situation. Also, I'm grateful that we're installing in the Spring when we have rain every couple of days. Each time we make a change, we don't have to wait too long before a real-world test presents itself.

We're out there on the bleeding edge with this one! Please pray with us that this wireless LAN bridge story has a happy ending.

GodTube mocked in Wired

Did anyone else notice this Wired blog post mocking GodTube?

There are many things swirling in my mind about this:

1. Does it make sense to have "Christian" versions of popular, secular web sites, or does it make more sense to proclaim the good news on the secular sites themselves? John Brownlee, the writer of the post, gets at question when he says, "What is disturbing about GodTube is that it is an observable microcosm of the way that fundamentalist Christians have shut themselves off from any outside perspective." Where is the line between being "in" the world but not "of" it?

2. John does us a favor by saying what many others also must be thinking. Doesn't this help us gain insight into the mindset of secular humanists and atheists? We can't possibly present the good news persuasively without knowing the views of our hearers.

3. Also facinating are the comments at the end of the post. Clearly John struck a nerve because many people wanted to react to what he said -- both echoes of John's ideas as well as thoughtful Christian responses.

Network and server monitoring

We are in the process of taking our network management up to the next level. Our server and network infrastructure has gotten sufficiently large and complex that we need to keep a much closer eye on what is happening. We need to be able to monitor resources, applications, and errors. We need a way to alarm, for example, when a server, switch, or application is down or when a resource gets to a critical range.

We're currently evaluating What's Up Gold. I also recently heard about Splunk. Do any of you have any thoughts on either of these or something else we should evalutate?

Wireless LAN bridge

We are preparing to move 40 staff to an office building a couple of miles from the church. For the last several years those staff have worked out of a temporary building for which the temporary use permit is now expiring. So how do you connect 40 remote staff to your LAN? If you have a clear line-of-sight, a great option is to use a wireless LAN bridge.

Here are pictures taken yesterday as our vendor performed alignment of the integrated radio/antenna units. These are 80 GHz (millimeter wave) radios that are capable of 1 Gb/s full-duplex, but we got the version that is rate-limted to 100 Mb/s.

Radio on roof of office building, pointed towards the church. The steel wall in front is a "penthouse" that hides roof-top air handlers an other equipment. The mast is mounted to steel girders supporting the penthouse.

Close up of back of radio with volt meter showing signal strength.

Looking over the top of the radio toward the church building, 1.7 miles away.

Close up taken from the office building with black arrow showing the location of the radio on the roof of the church building.

Radio on roof of church building. The mast is held by a 4-point mount and weighed down by concrete blocks so roof penetration isn't required. This location was chosen for maximum wind protection (air handling unit to the right; roof parapet to the left). It also happens to be right next to a roof access hatch.

Ian (our network admin) with his shiny new radio!

Jay, the lead installer for our vendor, tweaks the alignment by turning nuts on the mount while looking at his volt meter.

Jay relays meter readings to his co-worker on the roof of the other building.

On this end the radio is connected by fiber.

Close up taken from the church building with white arrow showing the location of the radio on the roof of the office building.

Is redundant more reliable?

Pete Whiting, in a guest post on Joel Dehlin's blog, has some insights into the perils of complexity. This is good food for thought as we plan our server virtualization project at Resurrection, which we have scheduled for implementation in May. I particularly agree with Pete's point about isolating/avoiding tight coupling between services. For example, it's much better if an application is dependent only on one server, than if it requires two (or more) servers to be able to operate. With virtualization, we are making it a priority to gather everything a particular application needs onto a single virtual server, which will be able to run on either of two redundant physical servers. This architecture provides the benefits of redundancy without introducing unnecessary interdependence.

A Long Overdue Thank-You

Thanks Tony for your outstanding hospitality during my visit to Perimeter Church back in December with Mark Stephenson. Please offer my long overdue thanks to the rest of your team as well. Our tour of Perimeter Church's facilities was amazing. God has blessed the church with an innovative spirit that is never satisfied with past accomplishments. The oldest part of the building is only 15 years old, but already many parts have been purposed and repurposed, designed and redesigned, modeled and remodeled numerous times as the church’s needs have grown and changed. That kind of passionate pursuit of ministry warms my heart and re-energizes me.

Speaking of open source

Scoble posted today about the Viacom-Google lawsuit. He says Viacom loses by winning. YouTube, which is owned by Google, wins by losing. It will be a few years before we know, but I hope he's right.

Open Source Kingdom

Mark Batterson posted recently about the kingdom impact of freely sharing creative ideas. Here's hoping Mark's vision spreads to decision-makers in other congregations. The Church should be the greatest open source community in the world.

The DST Saga Continues (with a happy ending, we think?)

Tony has been blogging about various Daylight Savings Time issues at Perimeter Church here, here, and here. We, too, have had some serious headaches with this. I commented on one of Tony's posts saying that we couldn't get the Exchange Calendar Update tool to work. Tony generously put the question out to IT Discuss and we got a response from Jason Hand of Walk Through the Bible. Thanks Jason, but unfortunately it didn't help.

So ... Yesterday, when everything was going horribly wrong ;-) Ian, our network administrator, found someone at Microsoft who could help us. He posted a full explanation of what happend here. Turns out even the developer at Microsoft who wrote the tool couldn't get it to work in our case. In the end he threw up his hands and had Ian do a manual workaround to the front-end of the process that allowed the rest of it to run successfully.

Things have finally settled down here. We have only a few, minor issues being reported today. DST - was it fun for you?

Fall IT Roundtable Invitation

I was part of a group of IT leaders from very large churches that met a couple of times a year for three years or so. The group (which never had a name other than “megachurch IT group”) included Mike Gold of Willow Creek, Terry Chapman of Fellowship Church, Vic Mollett of Southland Christian Church, Eric Busby of Saddleback Church, and others. When last we met in May of 2005 I told the group I would host the next meeting here at Church of the Resurrection in the Kansas City area in 2006. Unfortunately, I got slammed last year with workload and staff turnover issues so I wasn't able to make it happen. I recently exchanged e-mail with Mike Gold about renewing my invitation to host the group here and suggested merging the group with the Church IT Roundtable. He was very supportive and hoped to be able to attend and invite the old gang to join the Roundtable.

Accordingly, we would like to invite you to come to Kansas City for the Fall Roundtable October 3-4, which is right before Leadership Institute, our biggest conference of the year. After the Roundtable you could stay for Leadership Institute if you’re so inclined. I spoke with Tony Dye about getting this on the agenda for discussion at the Spring Roundtable to determine the level of interest. To get the conversation started, please leave a comment here and let me know if this sounds good.

National Church IT Association?

If you read Jason Powell’s blog, you already know about his hope to create some sort of national church IT association. (Tony has arranged for Phill Martin of NACBA to come to the Spring Roundtable to speak about this.) Now as for me, I strongly prefer the roundtable format for meeting with and learning from other church IT leaders. I worry that formalizing and scaling up what we’re doing into a national association would result in losing the powerful, peer-to-peer sharing of the roundtable format. I also worry that we’ll discover even the 25-30 people expected to attend the Spring Roundtable will be too many for everyone to fully participate. So what to do?

Fall Roundtable Format

Now here’s where it gets really exciting (we think?!) or maybe even crazy. For the last couple of years I have been meeting informally with church IT leaders here in the Kansas City area such as Mike Mayfield of Pleasant Valley Baptist Church. I suspect many of you have had similar meetings with other church IT leaders in your own cities. Perhaps some of you have already held roundtables in your cities. This makes me wonder if the best way for us to scale up would be to have roundtables in cities across the country and then link them technologically.

What if we made a first effort at this idea for the Fall Roundtable? What if we limit attendance here in Kansas City to 15-20 people and find some great way to link in roundtable groups from other cities? This idea is partially inspired by the Willow Creek Leadership Summit that brings together church leaders from around the world. We’re imagining some kind of “un-conference” that retains the strengths of peer-to-peer sharing but is scalable to involve many more people than you can fit around one table. We would like for everyone to be able to see/hear each other and use their own laptops to annotate/live blog the discussion and have side conversations. We’re not even close to figuring out how to do this from either a facilitation or technical standpoint, but one idea is to use Adobe Connect (formerly Macromedia Breeze). Whatever the mechanism is, we would need to fully test both the technology and the method of facilitation prior to the meeting. And, no doubt, we’d learn a great deal from doing this once. At that point we would know either a) this works; b) this works but needs to be tweaked/improved; or c) it doesn’t/can’t work.

Please comment and let me know your thoughts or post on your own blog and link back to this post.

Don't you be doing that church hop

Perry Noble points us to Steven Furtick blasting church hoppers/church shoppers like I've never seen it done before. I was fired up by his comments. On the other hand, I have good friends who have struggled with finding the right place where God is calling them to worship, grow, and serve. There's a line between a genuine spiritual struggle and a crassly consumerist attitude toward church.

The best approach for churches is to reach out and welcome everyone with all of their experiences, sins, mistakes, bad hair, doubts, fears, body odor, struggles, biases, broken relationships, addictions, questions, emotions, mental illness, inappropriate behavior ... and yes, even warped attitudes towards church. We start with people wherever they are. And then, pay close attention here, we must rely on the power of the Holy Spirit to work through our teaching, worship, service, and deep Christian community to transform them. No doubt some of those people have been church goers for years, yet they haven't learned even the basics of Christian discipleship. Instead of giving up and calling them out, let's call them up to a richer, fuller, deeper understanding of what Jesus asks of them. To be clear, I'm talking about starting with people where they are, but not being complacent about letting them stay there.

Web Software Developer position open

We're seeking a highly-skilled web software developer for a challenging and fulfilling position at The Church of the Resurrection (Kansas City area). Work in a high-energy, innovative environment with other talented technologists. BS in computer science or related field or equivalent experience required. Minimum of 2 years professional experience with the LAMP platform and 2 years professional experience with the .NET platform. Reports to the Director of Information Technology. Don’t miss this unique opportunity to serve Christ with your software development skills. To apply, please e-mail Cork Kurlbaum (our uber-volunteer/professional IT recruiter) for immediate consideration.

Five hot technologies for 2007

Yesterday Computerworld published an interesting article predicting five technologies will be hot over the next 12 months:

1. Ruby on Rails - web application development framework

2. NAND Flash drives - solid state mass storage with big enough capacity to substitute for small hard drives and 100x the performance

3. Ultra-Wideband (UWB) - short range wireless at 200x the speed of Bluetooth

4. Grid Computing - rent a data center by the gigabyte such as Amazon's S3 service

5. Advanced CPU architectures on higher-density chips

Y2K7

I just spent all day yesterday, along with a co-worker, tracking down one of the more heinous time calculation bugs I've ever seen. If you're a programmer, you probably already know about the Daylight Savings Time change that is occurring in this year of 2007. This is sending a shock wave of problems through the IT industry on a scale of which we probably haven't seen since Y2k. Everyone is having to update code that calculates when DST begins and ends.

In short, Microsoft is either in a very tight spot, or they screwed up big time. With the new calculation of DST, any year previous to 2007 now has the potential to show the wrong time! Here is a quick example.

This page is using JavaScript within the viewers browser to calculate dates. You will notice that if you are viewing this page from Vista, or a Linux based OS, that all the dates are displayed as expected. If you're on a Windows XP or earlier OS, the middle two dates will show as being in DST when they should not be:

Wed Mar 15 00:00:00 CDT 2006 <- Um, not quite yet actually. CDT doesn't start until April 02, 2006 02:00.

This is because MS OS' previous to Vista can only store one set of DST rules. So when you installed that cool new patch that fixes 2007 and future dates, you break every year previous to it. Nice. Granted this only crops up in past dates between the previous DST starting time, and the new one. I.e. between 2007 and 2006, this is between March 11 and April 2. See another conversation on this.

This is going to cause problems until it is fixed. Hopefully MS can get us something quickly, but depending on how deeply the OS is affected by this - they could have a serious issue on their hands.