Nerds on retreat!

We got out of the office for a day-long retreat yesterday. Most of my IT team was there:




Clockwise starting from left (goofy guy with Texas Tech t-shirt):
Chuck Russell - Internet Communications Director
Jeremy Grabrian - Desktop Support Specialist
Ian Beyer - Network Administrator
George Smiley - Desktop Support Technician (part time)
Brian Slezak - Software Developer
Leo Johns - web consultant (part time)

Oh yeah, and the end of the table with a laptop, a flat-panel monitor, and no one sitting there - that's me. ;-)

Fun With Statistics

So I am perusing the new Umc.org and looking at findachurch - which has been vastly improved - see Resurrections Page here. I decided to play around with the Advanced Search functions - mostly because I couldn't type in 66062 - my zip code, and see Resurrection come up - that's crazy do you know how many resurrection people don't live in the Church's zip code - I'll tell you - most of them, anyway tangent averted back to my point I found some interesting stuff when i searched for churches by size. Here are my results (More than 1000 means between 1000 and 2000 etc)

33 United Methodist Churches with more than 2000 in attendance
136 United Methodist Churches with more than 1000 in attendance
649 United Methodist Churches with more than 500 in attendance
3093 United Methodist Churches with more than 200 in attendance
5348 United Methodist Churches with more than 500 in attendance
8746 United Methodist Churches with more than 50 in attendance
10905 United Methodist Churches with more than 20 in attendance
4648 United Methodist Churches with less than 20 in attendance

So 818 who have made it to 500 + and 3900 approximately who have made it over the 200 barrier. Interesting stuff. Of course since our worship attendance is over 8000 now and they have us listed at 6300 (This is GCFA's fault not UMC.org's by the way) - I dont know how accurate those counts are but there probably pretty close.

Chuck

Who is using the new Blogger?

Who out there is using the upgraded version of Blogger? Is it cool? Should we upgrade?

MailFrontier is nearly perfect again

Last week we finally had time to focus on our spam problem. With some assistance from our friendly SonicWall tech support person, we installed a software update and tweaked our configuration. Turns out that at some point we white-listed everything from cor.org, no doubt to address a false positive situation, but that was a bad idea - too much spam is spoofed to come from your own domain. With the new software and the tweaks, we're back to nearly perfect, and just in time for the long Christmas-New Year's break when many of our staff take vacation. Almost no spam is getting through to our users' inboxes. Wee hoo!

SonicWall, I owe you an apology. When properly configured, SonicWall Mail Security is every bit as good as the spam filter built-in to Outlook 2003.

Guy Kawasaki talks about evangelism - the Christian kind

In case you missed it, I found it very interesting to see Guy Kawasaki reveal a bit of his personal faith in this recent blog post. In the post he includes this link to presentations at a church leadership conference in Hawaii this Fall. I'm glad to know Guy is interested in the original meaning of evangelism and not just his borrowed use of the term.

WEC Update

Now that Web Empowered Church (WEC) has been underway for a couple of years, Mark Stephenson has come to the belief that it won’t reach its full potential without Church Management System (ChMS) functionality, or at least deep integration with an existing ChMS. I totally agree with that assessment. So Mark and I thought it would be a great time to get together with Rubin Perry to discuss how ChMS fits with WEC’s vision and strategy. Like my meetings with Fellowship Technologies and Shelby Systems, our conversation was detailed, candid, and thought-provoking. While I’m not free to disclose everything we discussed, I can say that WEC has deep pockets, a passion for helping local churches use the Internet effectively, and a long-term view.

I’ve noted previously that combining content management (CoMS) functionality with ChMS and Constituent Relationship Management (CRM) functionality in a single, integrated system would be ideal. Most forward-thinking church IT leaders I know agree with this. WEC is looking for churches and church IT leaders who will partner with them to build just such a system based on the LAMP stack and TYPO3. As I mentioned, another possibility would be to do a deep integration of a market-leading ChMS with TYPO3. We’re in the process of discerning whether it makes sense for Church of the Resurrection to be one of those partners.

WEC is fully committed to open source software. By that they mean basing the software on the LAMP stack, licensing the software they develop using the GPL, and employing open source development methodologies. The advantages of this include risk reduction, ease of integration, competitive diversity, and distributed innovation. Already WEC is being adopted by churches around the world that have the technical ability to make use of complex technology. Such churches appreciate the fact that open source means WEC doesn’t have the licensing cost barrier associated with commercial software.

For churches that lack technical skills, however, adoption is challenging. TYPO3 is more than powerful enough for even the largest churches, but it is quite complex, it is difficult to learn, and it suffers from poor usability in some areas. Accordingly, addressing these issues represent top priorities for WEC.

1. WEC is actively working with the TYPO3 Human-Computer Interaction (HCI) team to improve usability.
2. WEC is developing TYPO3 training videos.
3. WEC is working on improved processes for providing tech support.

So that's a summary of how WEC fits into our ChMS analysis. I'm praying for clarity.

Ben Hill UMC

On Wednesday I had the pleasure of visiting Ben Hill United Methodist Church, just a few miles from the airport in south Atlanta. The Foundation for Evangelism’s technology leader, Rubin Perry, is a member of that storied church and one of its hardest-working unpaid servants. After 100 years of ministry, Ben Hill is still learning, growing, dreaming, taking risks, and serving. It has purchased the former headquarters site of Delta Airlines, a large acreage with a number of corporate-style buildings that have been sitting vacant for 20+ years. The site, adjacent to the Greenbriar Mall in Atlanta, has huge potential but carries many challenges. With Rubin helping to manage the project, the church is in the process of remodeling and renovating the first building, which will be used as a worship center when completed. The Holy Spirit is moving at Ben Hill and for that I say, “yeah God!”

The Spam Wars – What Does Microsoft know that SonicWALL doesn’t know?

The New York Times confirms that spam is getting worse, due primarily to two factors: 1) spammers are using farms of spambots running on insecure computers, which are abundant in homes with cable modems, making it ineffective to identify spam by originating IP address. Distributed security threats are always more difficult to block than localized threats. 2) A new type of spam with the message in an image that’s changed slightly each time it’s sent, making it impossible to recognize spam by a checksum on the contents.

I mentioned that the effectiveness of our spam filter, SonicWALL Email Security (formerly MailFrontier), is declining. On the other hand, it seems that Outlook 2003’s junk filter is keeping up. Since I began my experiment the week of Thanksgiving, I'm seeing 94-99% daily effectiveness from Outlook 2003 alone (with MailFrontier disabled on my mailbox). Yesterday it was 100% effective. Other staff we put on Outlook 2003 for our test have given similar reports.

What does Microsoft know that SonicWALL doesn't know? Why would we pay SonicWALL thousands of dollars per year for filtering that is LESS effective than the filter built-in to Outlook? Jerry, my SonicWALL reseller, has tried to get their attention. I also e-mailed one of their product managers nearly two weeks ago, with no reply at all so far. Jerry says we should hear something on Monday. Grrr.

Flurry of posts

Look out for a flurry of posts from me. It's amazing how much writing you can do in an airport and on a plane with no Internet. ;-)

Speaking of the airport, this is what greeted me when I tried to check the price to get WiFi today in Atlanta ...







Note: Because you are using an outdated browser, you may only view the raw textual content of this site. In order to view, use, and enjoy this site to the fullest at the maximum security level, please refer to our Browser Upgrade page to view a list of browsers that support web standards.

Let me see if I have this right, Sprint. I’m using an OUTDATED browser (IE 7) that should be UPGRADED to something that supports WEB STANDARDS (such as IE 6)??? You guys have had months to fix this page. Doesn’t anyone at Sprint use Sprint’s WiFi in airports? Haven’t at least some of them upgraded to IE 7? Even if they were asleep, it’s difficult to understand how they could have missed this. (Yes, Sprint is our friendly, neighborhood, telecom company located only two miles from Church of the Resurrection.)

Which ring makes the most sense?

Here's how I see the Church Management System (ChMS) marketplace. We're confronting the question of which ring of the market makes the most sense for us.


As you move in toward the center, the functionality is a closer and closer match to our requirements, but fewer and fewer organizations will be running it. On the outer rings, more organizations are innovating and influencing the feature set. On the inner rings, fewer organizations are innovating, but the features are more likely to be directly applicable to us.

Atlanta, here I come

This week I'm going to Atlanta to meet with Tony Dye as well as Mark Stephenson and Rubin Perry. Tony has graciously agreed to show me around Perimeter Church, introduce me to his A-team (A is for "awesome"), and talk about Church Management Systems (ChMS). Mark is joining in on the fun. Separately, Mark and I are going to meet with Rubin at Ben Hill United Methodist Church to talk about WEC's fledgling ChMS project.

This is my second trip exploring ChMS options, following my trips in October to Fellowship Technologies and Shelby. I should know by the end of next week whether my budget request to replace Shelby V5 has been approved for 2007.

Point, click, make a product to sell to the world

My friend Jay points us to Amazon's grand new scheme. Now let's see, how do we point, click, and plant a new church? Or, better yet, point, click, and lead people into a deeper relationship with Jesus? Hmmm ...

Birthplace of the web

Wired posted some pictures of the new particle accelerator under construction at CERN. At the end is this picture of CERN's "Internet Exchange Point," birthplace of the web. I'm feeling all weepy!

MailFrontier is struggling

We have been happy users of spam blocking software MailFrontier since 2004. MailFrontier was purchased by SonicWALL in February. When we first heard about the purchase, we weren't concerned because we respect SonicWALL and are happy users of their 2040 firewall with web content filtering.

Over the last few months, our satisfaction with MailFrontier (now called SonicWALL Email Security) is declining. We don't know if the issues have anything to do with SonicWALL taking over. We only know that spam blocking effectiveness is going down while total volume of inbound spam is increasing. Last week we saw a huge spike in inbound spam (Tony noticed this too) that coincided with most of our staff being out of the office for the Thanksgiving holiday. When they returned to their Outlook yesterday, they found large amounts of spam that got through the filter. With one voice they rose up in open revolt and stormed our IT office with pitchforks and torches.

Ahh the life of an IT Director. Now we either have to make MailFrontier work or start scrambling for other solutions. SonicWALL are you listening?

Mark's questions for staff evaluations

I will be conduction staff evaluations in January/February. This year I'm going to use Mark's questions.

Visual communication

At Church of the Resurrection we're working to get better at visual communication. Visual communication is much more effective than communication with words alone. The problem is, most of us (me included) spent the majority of our education years learning how to communicate with words. And most of us (me included) have spent the majority of our careers communicating with words -- memos, e-mails, talking with co-workers, speaking in front of groups, etc. We need to learn a new skill: visual communication.

Kathy Sierra is one of my favorite visual communicators. Her recent post on creating graphics illustrates why. Kathy, thanks for helping us with this. (Notice that even this post on visual communication has no visuals. See? I need to learn it too.)

Perry Noble provokes his fellow pastors

Most, if not all, pastors I know have thought the same things Perry shares in his post today. They agree with Perry. Acting on those ideas is what they find difficult. (Everyone in my family is a pastor except me, so I have a lot of opportunity to know what's on their minds.) Thanks, Perry.

Shelby is stable! Hallelujah!

In case the suspense has been killing you, yes the awful problems we had with SQL Server and our Shelby database are finally over. We're new now running Shelby 5.6.2009 on SQL Server 2005. It's rock stable and noticeably faster (though still not fast enough by a long shot). Praise the Lord!

Of course, we have had a number of issues with 5.6.2000 and 5.6.2009, which is hardly surprising given Shelby's track record, but all of that seems trivial compared to the database pain that has been relieved. Can I get an amen?

How NOT to move to a new database server

We bought a new server (a shiny Dell 2950 with Windows 2003 x64 and SQL Server 2005) back in May with the idea of moving Shelby and our other SQL databases to it in a planned, smooth way in June or July. We needed to free up a server for our new Rez West office so we figured it was a great opportunity to upgrade the server hardware and maybe speed up Shelby a good bit.

Due to a number of delays in the Rez West office completion, we had plenty of time to play with the new server. As June melted into July, we systematically moved our non-Shelby databases to the new server and started configuring it for Shelby.

When we updated to Shelby 5.6 in early August, massive performance problems followed. After some investigation, we came to the conclusion that 5.6 placed enough additional resource demands on the server that the old guy, a Dell 2500, wasn't getting the job done any more.

Before we could plan a cut over, the real adventure began. The old server crashed on a Wednesday morning, a time of the week when Shelby is typically under its heaviest load. In response, I divided our IT staff into two teams: one team would work on repairing the old server, while the second team would work on moving to the new server. We would go with whichever team finished first. In the course of that, we discovered that Shelby 5.6 still wasn't compatible with SQL Server 2005, despite the fact that we had been told that the May 2006 update would be compatible. So we finished restoring the old server (the crash had been caused by Windows registry corruption) and after a 6-hour outage we had no choice but to stay there - as painfully slow as it was.

By early October, we were faced with a dilemma. We had to move off of the old server to free it up for Rez West (not to mention the fact that it was so slow it was nearly unusable), but we still didn't have a SQL 2005-compatible version from Shelby and we already had databases running on SQL 2005 - there was no going back. That's when we got the idea to use the free version of VMware to create a virtual server for Shelby that would be configured with SQL 2000 and run on the new 2950 server. Our thought was to run this configuration temporarily until Shelby's October update, which was promised to be SQL 2005-compatible. Seemed like a brilliant idea at the time. Little did we know.

We have been on that configuration for the last five weeks, suffering from massive instability the entire time, including another server crash causing a 12-hour unplanned outage two weeks ago. The source of our grief was SQL 2000, not Shelby, but the user experience was that Shelby would randomly get disconnected from its database and start throwing errors. Shelby error handling is an awful mess of infinite loops, so the user's only recourse was to bring up Task Manager, shut down the client, and restart it. We spent hours on the phone with Microsoft tech support. They had us apply all kinds of updates and patches, including an unreleased hotfix. We also tried a number of changes to our VMware configuration, some of which actually made the problem worse. Each time we and our user community hoped that stability would finally be achieved, only to be bitterly disappointed. Not only was this frustrating for our users, but it was embarrassing for us. Though they were characteristically gracious, the users must have been thinking, "Does our IT Dept. know what it's doing?"

Our goal for this time was simply to hang on until we could install the SQL 2005-compatible version of Shelby, which was released last week. Every day the database was flaky and users were inconvenienced. Then we had something go bad last week in Bank Reconciliation, preventing our Finance folks from reconciling October. Shelby tech support concluded there was a damaged software component, which would be reinstalled by updating to the latest software version. So we scheduled the update for Sunday afternoon. Unfortunately, the Shelby update procedure didn't work. We fought that all day Monday and finally got all desktops updated.

The last step was to install new backup software because the version of Veritas we were running doesn't support Windows 2003 Server x64. To address that, we installed a trial version of ARCserv yesterday.

We finally had all the pieces in place to cut over to SQL Server 2005, running directly on our 2950. Then today the Bank Rec thing started crashing SQL 2000 every time we ran it. Since it was crashing and has been completely unstable for five weeks, we figured we would go ahead and cut over with yet another unplanned outage.

So there you have a tale of how NOT to move to a new database server. By the way, so far it seems stable and faster by 50-60%. Now we're praying it's really over.

SQL Server woes

Our Shelby system is down right now because of ongoing woes with SQL Server. When you are responsible for IT in a very large church that uses Shelby for every aspect of church operations, "Shelby down" is not a happy thing. This is an unplanned outage made necessary by the fact that every time we ran a bank reconcilliation this afternoon, it blew up SQL Server 2000 (errors in the log and the need to manually restart SQL Server every time).

In response, we're moving our database to a new server running SQL Server 2005. This is something we've been working on for months. The final step before moving to 2005 was to install Shelby 5.6.2000, which we did earlier this week. (Since we're down, we're installing the 5.6.2009 brand new patch right now.)

We're now 105 minutes into an unplanned 30 minute outage. Remind me again why I have made a 22 year career in IT???