The Appian Way

November 28, 2006

MailFrontier is struggling

We have been happy users of spam blocking software MailFrontier since 2004. MailFrontier was purchased by SonicWALL in February. When we first heard about the purchase, we weren't concerned because we respect SonicWALL and are happy users of their 2040 firewall with web content filtering.

Over the last few months, our satisfaction with MailFrontier (now called SonicWALL Email Security) is declining. We don't know if the issues have anything to do with SonicWALL taking over. We only know that spam blocking effectiveness is going down while total volume of inbound spam is increasing. Last week we saw a huge spike in inbound spam (Tony noticed this too) that coincided with most of our staff being out of the office for the Thanksgiving holiday. When they returned to their Outlook yesterday, they found large amounts of spam that got through the filter. With one voice they rose up in open revolt and stormed our IT office with pitchforks and torches.

Ahh the life of an IT Director. Now we either have to make MailFrontier work or start scrambling for other solutions. SonicWALL are you listening?

November 22, 2006

Mark's questions for staff evaluations

I will be conduction staff evaluations in January/February. This year I'm going to use Mark's questions.

November 21, 2006

Visual communication

At Church of the Resurrection we're working to get better at visual communication. Visual communication is much more effective than communication with words alone. The problem is, most of us (me included) spent the majority of our education years learning how to communicate with words. And most of us (me included) have spent the majority of our careers communicating with words -- memos, e-mails, talking with co-workers, speaking in front of groups, etc. We need to learn a new skill: visual communication.

Kathy Sierra is one of my favorite visual communicators. Her recent post on creating graphics illustrates why. Kathy, thanks for helping us with this. (Notice that even this post on visual communication has no visuals. See? I need to learn it too.)

Perry Noble provokes his fellow pastors

Most, if not all, pastors I know have thought the same things Perry shares in his post today. They agree with Perry. Acting on those ideas is what they find difficult. (Everyone in my family is a pastor except me, so I have a lot of opportunity to know what's on their minds.) Thanks, Perry.

November 13, 2006

Shelby is stable! Hallelujah!

In case the suspense has been killing you, yes the awful problems we had with SQL Server and our Shelby database are finally over. We're new now running Shelby 5.6.2009 on SQL Server 2005. It's rock stable and noticeably faster (though still not fast enough by a long shot). Praise the Lord!

Of course, we have had a number of issues with 5.6.2000 and 5.6.2009, which is hardly surprising given Shelby's track record, but all of that seems trivial compared to the database pain that has been relieved. Can I get an amen?

November 09, 2006

How NOT to move to a new database server

We bought a new server (a shiny Dell 2950 with Windows 2003 x64 and SQL Server 2005) back in May with the idea of moving Shelby and our other SQL databases to it in a planned, smooth way in June or July. We needed to free up a server for our new Rez West office so we figured it was a great opportunity to upgrade the server hardware and maybe speed up Shelby a good bit.

Due to a number of delays in the Rez West office completion, we had plenty of time to play with the new server. As June melted into July, we systematically moved our non-Shelby databases to the new server and started configuring it for Shelby.

When we updated to Shelby 5.6 in early August, massive performance problems followed. After some investigation, we came to the conclusion that 5.6 placed enough additional resource demands on the server that the old guy, a Dell 2500, wasn't getting the job done any more.

Before we could plan a cut over, the real adventure began. The old server crashed on a Wednesday morning, a time of the week when Shelby is typically under its heaviest load. In response, I divided our IT staff into two teams: one team would work on repairing the old server, while the second team would work on moving to the new server. We would go with whichever team finished first. In the course of that, we discovered that Shelby 5.6 still wasn't compatible with SQL Server 2005, despite the fact that we had been told that the May 2006 update would be compatible. So we finished restoring the old server (the crash had been caused by Windows registry corruption) and after a 6-hour outage we had no choice but to stay there - as painfully slow as it was.

By early October, we were faced with a dilemma. We had to move off of the old server to free it up for Rez West (not to mention the fact that it was so slow it was nearly unusable), but we still didn't have a SQL 2005-compatible version from Shelby and we already had databases running on SQL 2005 - there was no going back. That's when we got the idea to use the free version of VMware to create a virtual server for Shelby that would be configured with SQL 2000 and run on the new 2950 server. Our thought was to run this configuration temporarily until Shelby's October update, which was promised to be SQL 2005-compatible. Seemed like a brilliant idea at the time. Little did we know.

We have been on that configuration for the last five weeks, suffering from massive instability the entire time, including another server crash causing a 12-hour unplanned outage two weeks ago. The source of our grief was SQL 2000, not Shelby, but the user experience was that Shelby would randomly get disconnected from its database and start throwing errors. Shelby error handling is an awful mess of infinite loops, so the user's only recourse was to bring up Task Manager, shut down the client, and restart it. We spent hours on the phone with Microsoft tech support. They had us apply all kinds of updates and patches, including an unreleased hotfix. We also tried a number of changes to our VMware configuration, some of which actually made the problem worse. Each time we and our user community hoped that stability would finally be achieved, only to be bitterly disappointed. Not only was this frustrating for our users, but it was embarrassing for us. Though they were characteristically gracious, the users must have been thinking, "Does our IT Dept. know what it's doing?"

Our goal for this time was simply to hang on until we could install the SQL 2005-compatible version of Shelby, which was released last week. Every day the database was flaky and users were inconvenienced. Then we had something go bad last week in Bank Reconciliation, preventing our Finance folks from reconciling October. Shelby tech support concluded there was a damaged software component, which would be reinstalled by updating to the latest software version. So we scheduled the update for Sunday afternoon. Unfortunately, the Shelby update procedure didn't work. We fought that all day Monday and finally got all desktops updated.

The last step was to install new backup software because the version of Veritas we were running doesn't support Windows 2003 Server x64. To address that, we installed a trial version of ARCserv yesterday.

We finally had all the pieces in place to cut over to SQL Server 2005, running directly on our 2950. Then today the Bank Rec thing started crashing SQL 2000 every time we ran it. Since it was crashing and has been completely unstable for five weeks, we figured we would go ahead and cut over with yet another unplanned outage.

So there you have a tale of how NOT to move to a new database server. By the way, so far it seems stable and faster by 50-60%. Now we're praying it's really over.

SQL Server woes

Our Shelby system is down right now because of ongoing woes with SQL Server. When you are responsible for IT in a very large church that uses Shelby for every aspect of church operations, "Shelby down" is not a happy thing. This is an unplanned outage made necessary by the fact that every time we ran a bank reconcilliation this afternoon, it blew up SQL Server 2000 (errors in the log and the need to manually restart SQL Server every time).

In response, we're moving our database to a new server running SQL Server 2005. This is something we've been working on for months. The final step before moving to 2005 was to install Shelby 5.6.2000, which we did earlier this week. (Since we're down, we're installing the 5.6.2009 brand new patch right now.)

We're now 105 minutes into an unplanned 30 minute outage. Remind me again why I have made a 22 year career in IT???

November 01, 2006

ChMS budget, timeline, and short list

I spent 75 minutes on the phone with Tony yesterday, which is great for me because I'm always smarter after I've talked with Tony than I was before. Perimeter is just ahead of us in the ChMS selection process. He has a deadline of this month to make a final decision. For us, our only looming deadline is to put in our budget request for 2007 this week. I met with our executive management on Monday. Based on that meeting, it seems very likely that our budget request for ChMS will be approved. If so, 2007 will be the year we make a change, but we still have a couple of months to decide what to do.

Here's our short list:

MSCRM 3.0 plus add-ons from Microsoft CRM partners
Fellowship One
Shelby Arena
Build-our-own open source system in partnership with WEC

Interestingly, two of my finalists overlap with Tony's three finalists.

Tony's third finalist is Blackbaud. I'm intrigued by that and enjoyed the chance earlier this year to discuss it in detail with Shelley Hildebrand of Tony's staff at Perimeter. Unfortunately, the cost of Blackbaud is up there near what it would cost to build our own, so that's not a feasible option for us.

Also like Tony, I know there's a chance our short list will get longer before it gets shorter. Did I mention this decision is risky and complicated?

October 31, 2006

Network World article on church IT

Jason mentioned that Network World featured him extensively in an article on church IT. Jason also gave the reporter my name and I was quoted too (on pages 2 and 4). This is the first time I've been quoted in a national trade publication. Thankfully, I have Brian to keep me humble!

October 29, 2006

Thanks Frank, Alfred, Steve, Joe, and Mark

I had the privilege of spending the day Friday with Alfred Johnson, Steve Pruitt, and their teams at Shelby Systems. Frank Canady also joined us for lunch. We had deep discussions about the Church Management System (ChMS) market, and specifically the soon-to-be-released Arena ChMS.

I first met Alfred at the IT Roundtable. He arranged the day and introduced me to the senior management team of Shelby. Alfred, you are a gentleman and a great host. The warmth of your hospitality made my time there special.

Shelby has been around a long time and has been a leader in the ChMS market for many years. During my time there, I gained a lot of insight into Shelby's history, how they see the marketplace, and their product plans going forward. Some of those things I was asked to keep to myself, but many of them are open and I will be commenting on them in the coming days.

We had a frank discussion about our concerns as a Shelby customer, and everyone was gracious enough to really listen and take my concerns seriously. At the same time, I was surprised to learn that my impressions of Shelby's customer base and target market were incorrect in several respects. Like the Fellowship Tech team before them, they also indulged me an extensive discussion of strategic considerations without insisting on jumping right to a product demo. There will be plenty of time for that.

The Shelby senior management team is seasoned and humble. They have a good understanding of what the marketplace is asking of them. Regardless of whether we become an Arena customer, I'm glad to say I have new friends in Memphis who care about the local church as much as I do.

I will continue to post about my trip and my impressions as we make budget plans for 2007 over the next few weeks.

October 25, 2006

Thanks Jeff, Jeff, Curtis, Curtis, and Scott

No, it's not an obscure law firm. I'm referring to Jeff Hook and his team at Fellowship Technologies.

Today I flew to Dallas for a tour and discussions to help me decide whether Fellowship One will be our next church management system. Jeff, Jeff, Curtis, Curtis, and Scott were awesome hosts and I enjoyed my time with them very much. The team is impressive (even those whose names aren't Jeff or Curtis!) and their passion for the local church came through loud and clear. I particularly appreciated the openness of our conversation. Church management system selection can be complex (it certainly is in our case) and the issues often can't be articulated in bullet points or sound bites. So they indulged me by taking the time to fully air the issues and explore them from multiple angles. It definitely sharpened my thinking. I love serious, deep conversations with smart people.

Thanks again Jeff, Jeff, Curtis, Curtis, and Scott!

October 18, 2006

Friends don't let friends upgrade to iTunes 7.0

You've been warned. Thanks Jason!

October 15, 2006

Rez West office

Chuck posted back in August regarding the launch of our first satellite campus, Resurrection West (affectionately known as "Rez West"). The staff of Rez West includes a campus pastor, campus associate pastor, operations director, children's director, music people, etc. - 7 or 8 people altogether. They have been officing in a temporary space on the central campus since June.

Meanwhile, our Facilities folks found some space for lease near the junior high school where Rez West worships. For the last couple of months they have been working on tenant finish to turn it into usable office space plus a large general purpose meeting room and a garage for the A/V equipment trailer. Last week, while the tenant finish was nearing completion, it was our turn to come in and start assembling the technology: phone system, voice mail, Internet connection, firewall, network switch, file and print server, printers, and computers.

Everything but the printers and computers are being installed into one cabinet. Here's Ian terminating the voice station wires into the patch panel (the guy on the right is our new desktop support guy, Jeremy, who doesn't even start until November 13 but came out to help):

Me with Ian and Jeremy:

Me testing the jacks by plugging in my laptop. (Meter? We don't need no stinking meter!)

October 13, 2006

Thanks for coming

A big "thank you" to everyone who came to Church of the Resurrection for Leadership Institute and/or any of the other conferences we had going on last week. My friend Jim posted his final thoughts here.

October 12, 2006

Leadership Institute blog posts

September 30, 2006

ChMS: Buy it, build it, or integrate it?

If you have spent a lot of time thinking about how to improve the Church Management System (ChMS) function in your church, you probably have reservations about buying one of the existing ChMS products (“buy it”) but at the same time realize you can’t afford to build your own system (“build it”). I want a third alternative: “integrate it.”

With the right technical and non-technical factors in place, it should be possible for your team to integrate technology from multiple commercial and open source providers, along with in-house developed modules, to achieve the range and degree of automation that makes sense for your unique ministry.

In my last post, I listed three technical factors that would need to be in place in order for an open ChMS marketplace to come into existence:

1. A common schema for data exchange. (The best technology for this right now is XML. See ebXML for an example.)

2. An architecture for generating and handling data change events (such as a Web Services API).

3. An architecture for single sign-on.

This list comes out of our experience integrating multiple web applications that weren’t designed to work together. Perhaps there are one or two additional technical requirements I haven’t thought of yet. This brings me to the second part of my question ...

What non-technical (business) factors would need to be in place to foster development of an open ChMS marketplace?

I'd like to suggest we work toward the following:

1. Backing of a few influential churches and/or para-church organizations plus one or more open source or commercial ChMS vendors.

2. A funded standards organization.

3. A Kingdom-minded spirit of unity and cooperation across denominational and geographical boundaries as well as among both non-profit and for-profit organizations.

I’ve said before that we have differing styles, mission fields, and theological points of view, but we have one Lord and we proclaim one gospel. We're all on the same team. Could we begin to move together in a direction that honors God and helps the Church pursue the Great Commission?

I look forward to your comments.

September 26, 2006

Example of how a Standard and an Open Marketplace changed one industry...

I'm on the part-time IT staff at COR, and I get to talk to Clif directly. I agree with his perspectives on ChMS. We've argued out lots of these points over lunches and Skype chats.

I'm also a worship musician. And I was there in 1983 when the music industry figured out something critical: that a common standard could help *everybody*. Even though the manufacturers were scared and skeptical at first, when they finally dived in THE WORLD CHANGED.

In the 70s and 80s, keyboard players (like me; hey it was 1983!) spent thousands of dollars on instruments: music synthesizers with names like Moog and Yamaha. Learning to play them was cumbersome, and each was a completely independent experience.

In 1982 (still pretty much before PC's) my 3-man company became an original member of the old IMA, right along side giants like Yamaha and Sequential. The objective was to develop Dave Smith's idea for a Musical Instrument Digital Interface (MIDI), which was/is merely a language that all manufacturers could use to translate Black & White key presses into music notes for the sound generator of a synthesizer. If there was a *standard* language for this (the unlikely reasoning went), then instruments could communicate, new grass-roots development from outside would introduce hybrid vigor, and everyone would benefit. But, it was a hard sell: most companies had lots of money tied up in their own systems, and saw things through Old-World colored glasses, hoping for customers to stay "locked in" to the company after learning one instrument's subtleties.

AND THEN: "In a rare example of insight and cooperation, U.S. and overseas companies began working together..." (see Mix Mag's interview with Dave Smith) - - and the 2 companies that spent huge $$ to technically design and implement MIDI gave the technology away to everyone and told *them* how to implement it. (read about the birth of MIDI)

So, in January 1983, at the Anaheim NAMM Trade Show, one guy carried one synth to another synth company's booth, one tiny little MIDI cable linked 2 totally different instruments, and The World Changed. We all quickly discovered digital control, computer mixing, new soundscapes, and a slew of eager new companies hatching new strategies to exploit the newly expanding market. Practically every bit of the worship music I listen to (and create) every day depends on MIDI - a standard that everyone bought into when no one was sure anyone would.

Guess what? Yamaha, and Korg, and the Big Boys are still around, leading their industries, and happy that MIDI came along. The MIDI specification, now nearly 25 years old, transformed the creation of contemporary music. Every church I know of uses it every week, and you've benefitted from it a hundred thousand times. You probably even have .mid files on your PC.

The lesson (of course): working to develop or promote standard schemas will help *your company*, *your church*, and *God's Kingdom*.

More about MIDI at these links:
* Wikipedia - -
* MIDI.org - -

Towards an open ChMS marketplace

See previous posts in this series:
The ChMS conversation we started (but know we’ll never finish)
Choosing a ChMS – the risk calculation

Also earlier posts on related subjects:
Integrating CRM and the Web
Open source Church Management System (ChMS)

No customer lock-in
So what could ChMS vendors do to help me with my conundrum and reduce the risk that I’ll make a bad choice? First, let me suggest a non-technical answer: they need to reject the tempting strategy of achieving and maintaining customer lock-in. They need to go out of their way to make it easy for me to change in the future if I so desire. In other words, they need to set aside their interests and focus on the interests of the Kingdom (and in so doing, they’ll find that their interests are served as well). Instead of competing primarily on a feature list, they need to compete on ease of integration, openness, and customer services. If the vendor can achieve lock-in, they don’t have to be great at these things because they’ll have acceptable rates of customer retention regardless, at least in the short term. On the other hand, if the ChMS vendor rejects the lock-in strategy, my risk as the buyer goes down because I know the vendor will have to continue to earn my business long after the initial sale and implementation.

Second (and this is much more difficult), ChMS suppliers need to find their way towards an open ChMS marketplace. By “open” I mean one in which innovation can originate from many places and come together to meet the needs of a particular church at a particular time. For example, wouldn’t it be great if Granger could run Saddleback’s small groups system with Fellowship’s check-in and WEC’s content management? Wouldn’t it be awesome if Perimeter could run Shelby with Resurrection’s missions opportunity registration system? Wouldn’t it be excellent if Resurrection could run iModules’ online community tools with Perimeter’s Elder system and a web store like OS Commerce? I’m envisioning a marketplace in which there could be multiple for-profit companies contributing technology and services alongside churches, para-church organizations, and open source communities giving away software that automates their process innovations. Wouldn’t that be a major advancement for the Kingdom?

What is stopping us from doing this right now? Well that’s easy enough to answer. The systems I mentioned were built at different times on different platforms by for-profit and non-profit organizations without ever considering the possibility of interoperability. So that’s a big technical barrier. Also, the ChMS vendors don’t see an economic incentive to cooperate with their competitors, which is a big non-technical barrier. Could we overcome these barriers going forward?

To overcome the technical barriers, it seems to me we would need three things:
1. A common schema for data exchange. (The best technology for this right now is XML.)
2. An architecture for generating and handling data change events (such as a Web Services API).
3. An architecture for single sign-on.

Tony posted a comment in which he used the term “platform” to describe what we need. He also quoted me in the ChMS Google group, which attracted a number of comments exactly along the lines I’m thinking.

More in the next post ...

September 25, 2006

Worship attendance

At the Roundtable we had an extensive discussion of approaches to automate worship attendance. Today Kevin McCord weighs in on the same subject. He thinks printing out a nametag will be enough incentive to get many/most worship attendees to stop at a kiosk and check in.

September 22, 2006

Choosing a ChMS – the risk calculation

As I alluded in the previous post, a huge consideration for me in choosing our next Church Management Systems (ChMS) is minimizing the risk of making a bad choice. The question of ChMS is the most strategically complex and the riskiest decision I’ve faced since becoming IT Director at Church of the Resurrection three years ago. Why do I say that? What makes this decision especially risky?

The risk is high mainly because Church Management Systems automate core processes. In business or church, whenever you automate core processes you set in motion a complex interplay between your operations and your technology. You adapt the automation system as best you can to fit your processes and you inevitably adapt some of your processes to fit the automation system. You train your staff on the system and over time they develop some deep expertise in its quirks. You purchase and integrate other technology with the system. Before long, the thought of changing the system becomes nearly unthinkable. Companies that build and sell systems that automate core processes understand this very well. Once they get a customer fully on board with the system, they know it will be very difficult for the customer to go another direction. The customer is “locked in” and they're happy (that is, the vendor is happy).

Going forward from the initial implementation, a ChMS is likely to become a two-edged sword. While your operations become more efficient, operational innovation can be held back if the automation technology can’t keep pace. Curtis Harris of Fellowship Tech asked at the Roundtable if it wasn’t possible to have a set of best practices that the system would automate, benefiting all churches that use it. I don’t believe that’s 100% the case (although we all can and should be learning from each other). The best senior pastors are the innovators. They zig when everyone else is zagging. So there’s bound to be some mismatches between what any particular system does and my particular church’s unique way of doing ministry. I strongly suspect that will be the case for other churches too.

Right now Resurrection is too dependent on Shelby. If Shelby doesn’t keep pace with new technology and new ministry models, there’s not a lot we can do because we’re “locked in.” If we go out and find a new system that “stinks the least” and change to that system, we might be better off than we are now, but fundamentally it’s the same strategy. Our risk of Shelby not performing simply changes to a risk of the new vendor not performing. And ultimately, I don’t believe it will be possible for Shelby or any vendor to perfectly, exactly meet our needs.

So what could ChMS vendors do to help me with this conundrum and reduce the risk that I’ll make a bad choice? I’ll share some thoughts on that in my next post.

November 28, 2006

November 22, 2006

November 21, 2006

November 13, 2006

November 09, 2006

November 01, 2006

October 31, 2006

October 29, 2006

October 25, 2006

October 18, 2006

October 15, 2006

October 13, 2006

October 12, 2006

September 30, 2006

September 26, 2006

September 25, 2006

September 22, 2006

Church IT RoundTable

About

Contributors

Subscribe

Clif's Tweets

Blog Archive

Links

Church IT Blogroll

The Methodist Blogroll