We have been happy users of spam blocking software MailFrontier since 2004. MailFrontier was purchased by SonicWALL in February. When we first heard about the purchase, we weren't concerned because we respect SonicWALL and are happy users of their 2040 firewall with web content filtering.
Over the last few months, our satisfaction with MailFrontier (now called SonicWALL Email Security) is declining. We don't know if the issues have anything to do with SonicWALL taking over. We only know that spam blocking effectiveness is going down while total volume of inbound spam is increasing. Last week we saw a huge spike in inbound spam (Tony noticed this too) that coincided with most of our staff being out of the office for the Thanksgiving holiday. When they returned to their Outlook yesterday, they found large amounts of spam that got through the filter. With one voice they rose up in open revolt and stormed our IT office with pitchforks and torches.
Ahh the life of an IT Director. Now we either have to make MailFrontier work or start scrambling for other solutions. SonicWALL are you listening?
November 28, 2006
November 22, 2006
Mark's questions for staff evaluations
I will be conduction staff evaluations in January/February. This year I'm going to use Mark's questions.
November 21, 2006
Visual communication
At Church of the Resurrection we're working to get better at visual communication. Visual communication is much more effective than communication with words alone. The problem is, most of us (me included) spent the majority of our education years learning how to communicate with words. And most of us (me included) have spent the majority of our careers communicating with words -- memos, e-mails, talking with co-workers, speaking in front of groups, etc. We need to learn a new skill: visual communication.
Kathy Sierra is one of my favorite visual communicators. Her recent post on creating graphics illustrates why. Kathy, thanks for helping us with this. (Notice that even this post on visual communication has no visuals. See? I need to learn it too.)
Kathy Sierra is one of my favorite visual communicators. Her recent post on creating graphics illustrates why. Kathy, thanks for helping us with this. (Notice that even this post on visual communication has no visuals. See? I need to learn it too.)
Perry Noble provokes his fellow pastors
Most, if not all, pastors I know have thought the same things Perry shares in his post today. They agree with Perry. Acting on those ideas is what they find difficult. (Everyone in my family is a pastor except me, so I have a lot of opportunity to know what's on their minds.) Thanks, Perry.
November 13, 2006
Shelby is stable! Hallelujah!
In case the suspense has been killing you, yes the awful problems we had with SQL Server and our Shelby database are finally over. We're new now running Shelby 5.6.2009 on SQL Server 2005. It's rock stable and noticeably faster (though still not fast enough by a long shot). Praise the Lord!
Of course, we have had a number of issues with 5.6.2000 and 5.6.2009, which is hardly surprising given Shelby's track record, but all of that seems trivial compared to the database pain that has been relieved. Can I get an amen?
Of course, we have had a number of issues with 5.6.2000 and 5.6.2009, which is hardly surprising given Shelby's track record, but all of that seems trivial compared to the database pain that has been relieved. Can I get an amen?
November 09, 2006
How NOT to move to a new database server
We bought a new server (a shiny Dell 2950 with Windows 2003 x64 and SQL Server 2005) back in May with the idea of moving Shelby and our other SQL databases to it in a planned, smooth way in June or July. We needed to free up a server for our new Rez West office so we figured it was a great opportunity to upgrade the server hardware and maybe speed up Shelby a good bit.
Due to a number of delays in the Rez West office completion, we had plenty of time to play with the new server. As June melted into July, we systematically moved our non-Shelby databases to the new server and started configuring it for Shelby.
When we updated to Shelby 5.6 in early August, massive performance problems followed. After some investigation, we came to the conclusion that 5.6 placed enough additional resource demands on the server that the old guy, a Dell 2500, wasn't getting the job done any more.
Before we could plan a cut over, the real adventure began. The old server crashed on a Wednesday morning, a time of the week when Shelby is typically under its heaviest load. In response, I divided our IT staff into two teams: one team would work on repairing the old server, while the second team would work on moving to the new server. We would go with whichever team finished first. In the course of that, we discovered that Shelby 5.6 still wasn't compatible with SQL Server 2005, despite the fact that we had been told that the May 2006 update would be compatible. So we finished restoring the old server (the crash had been caused by Windows registry corruption) and after a 6-hour outage we had no choice but to stay there - as painfully slow as it was.
By early October, we were faced with a dilemma. We had to move off of the old server to free it up for Rez West (not to mention the fact that it was so slow it was nearly unusable), but we still didn't have a SQL 2005-compatible version from Shelby and we already had databases running on SQL 2005 - there was no going back. That's when we got the idea to use the free version of VMware to create a virtual server for Shelby that would be configured with SQL 2000 and run on the new 2950 server. Our thought was to run this configuration temporarily until Shelby's October update, which was promised to be SQL 2005-compatible. Seemed like a brilliant idea at the time. Little did we know.
We have been on that configuration for the last five weeks, suffering from massive instability the entire time, including another server crash causing a 12-hour unplanned outage two weeks ago. The source of our grief was SQL 2000, not Shelby, but the user experience was that Shelby would randomly get disconnected from its database and start throwing errors. Shelby error handling is an awful mess of infinite loops, so the user's only recourse was to bring up Task Manager, shut down the client, and restart it. We spent hours on the phone with Microsoft tech support. They had us apply all kinds of updates and patches, including an unreleased hotfix. We also tried a number of changes to our VMware configuration, some of which actually made the problem worse. Each time we and our user community hoped that stability would finally be achieved, only to be bitterly disappointed. Not only was this frustrating for our users, but it was embarrassing for us. Though they were characteristically gracious, the users must have been thinking, "Does our IT Dept. know what it's doing?"
Our goal for this time was simply to hang on until we could install the SQL 2005-compatible version of Shelby, which was released last week. Every day the database was flaky and users were inconvenienced. Then we had something go bad last week in Bank Reconciliation, preventing our Finance folks from reconciling October. Shelby tech support concluded there was a damaged software component, which would be reinstalled by updating to the latest software version. So we scheduled the update for Sunday afternoon. Unfortunately, the Shelby update procedure didn't work. We fought that all day Monday and finally got all desktops updated.
The last step was to install new backup software because the version of Veritas we were running doesn't support Windows 2003 Server x64. To address that, we installed a trial version of ARCserv yesterday.
We finally had all the pieces in place to cut over to SQL Server 2005, running directly on our 2950. Then today the Bank Rec thing started crashing SQL 2000 every time we ran it. Since it was crashing and has been completely unstable for five weeks, we figured we would go ahead and cut over with yet another unplanned outage.
So there you have a tale of how NOT to move to a new database server. By the way, so far it seems stable and faster by 50-60%. Now we're praying it's really over.
Due to a number of delays in the Rez West office completion, we had plenty of time to play with the new server. As June melted into July, we systematically moved our non-Shelby databases to the new server and started configuring it for Shelby.
When we updated to Shelby 5.6 in early August, massive performance problems followed. After some investigation, we came to the conclusion that 5.6 placed enough additional resource demands on the server that the old guy, a Dell 2500, wasn't getting the job done any more.
Before we could plan a cut over, the real adventure began. The old server crashed on a Wednesday morning, a time of the week when Shelby is typically under its heaviest load. In response, I divided our IT staff into two teams: one team would work on repairing the old server, while the second team would work on moving to the new server. We would go with whichever team finished first. In the course of that, we discovered that Shelby 5.6 still wasn't compatible with SQL Server 2005, despite the fact that we had been told that the May 2006 update would be compatible. So we finished restoring the old server (the crash had been caused by Windows registry corruption) and after a 6-hour outage we had no choice but to stay there - as painfully slow as it was.
By early October, we were faced with a dilemma. We had to move off of the old server to free it up for Rez West (not to mention the fact that it was so slow it was nearly unusable), but we still didn't have a SQL 2005-compatible version from Shelby and we already had databases running on SQL 2005 - there was no going back. That's when we got the idea to use the free version of VMware to create a virtual server for Shelby that would be configured with SQL 2000 and run on the new 2950 server. Our thought was to run this configuration temporarily until Shelby's October update, which was promised to be SQL 2005-compatible. Seemed like a brilliant idea at the time. Little did we know.
We have been on that configuration for the last five weeks, suffering from massive instability the entire time, including another server crash causing a 12-hour unplanned outage two weeks ago. The source of our grief was SQL 2000, not Shelby, but the user experience was that Shelby would randomly get disconnected from its database and start throwing errors. Shelby error handling is an awful mess of infinite loops, so the user's only recourse was to bring up Task Manager, shut down the client, and restart it. We spent hours on the phone with Microsoft tech support. They had us apply all kinds of updates and patches, including an unreleased hotfix. We also tried a number of changes to our VMware configuration, some of which actually made the problem worse. Each time we and our user community hoped that stability would finally be achieved, only to be bitterly disappointed. Not only was this frustrating for our users, but it was embarrassing for us. Though they were characteristically gracious, the users must have been thinking, "Does our IT Dept. know what it's doing?"
Our goal for this time was simply to hang on until we could install the SQL 2005-compatible version of Shelby, which was released last week. Every day the database was flaky and users were inconvenienced. Then we had something go bad last week in Bank Reconciliation, preventing our Finance folks from reconciling October. Shelby tech support concluded there was a damaged software component, which would be reinstalled by updating to the latest software version. So we scheduled the update for Sunday afternoon. Unfortunately, the Shelby update procedure didn't work. We fought that all day Monday and finally got all desktops updated.
The last step was to install new backup software because the version of Veritas we were running doesn't support Windows 2003 Server x64. To address that, we installed a trial version of ARCserv yesterday.
We finally had all the pieces in place to cut over to SQL Server 2005, running directly on our 2950. Then today the Bank Rec thing started crashing SQL 2000 every time we ran it. Since it was crashing and has been completely unstable for five weeks, we figured we would go ahead and cut over with yet another unplanned outage.
So there you have a tale of how NOT to move to a new database server. By the way, so far it seems stable and faster by 50-60%. Now we're praying it's really over.
SQL Server woes
Our Shelby system is down right now because of ongoing woes with SQL Server. When you are responsible for IT in a very large church that uses Shelby for every aspect of church operations, "Shelby down" is not a happy thing. This is an unplanned outage made necessary by the fact that every time we ran a bank reconcilliation this afternoon, it blew up SQL Server 2000 (errors in the log and the need to manually restart SQL Server every time).
In response, we're moving our database to a new server running SQL Server 2005. This is something we've been working on for months. The final step before moving to 2005 was to install Shelby 5.6.2000, which we did earlier this week. (Since we're down, we're installing the 5.6.2009 brand new patch right now.)
We're now 105 minutes into an unplanned 30 minute outage. Remind me again why I have made a 22 year career in IT???
In response, we're moving our database to a new server running SQL Server 2005. This is something we've been working on for months. The final step before moving to 2005 was to install Shelby 5.6.2000, which we did earlier this week. (Since we're down, we're installing the 5.6.2009 brand new patch right now.)
We're now 105 minutes into an unplanned 30 minute outage. Remind me again why I have made a 22 year career in IT???
November 01, 2006
ChMS budget, timeline, and short list
I spent 75 minutes on the phone with Tony yesterday, which is great for me because I'm always smarter after I've talked with Tony than I was before. Perimeter is just ahead of us in the ChMS selection process. He has a deadline of this month to make a final decision. For us, our only looming deadline is to put in our budget request for 2007 this week. I met with our executive management on Monday. Based on that meeting, it seems very likely that our budget request for ChMS will be approved. If so, 2007 will be the year we make a change, but we still have a couple of months to decide what to do.
Here's our short list:
MSCRM 3.0 plus add-ons from Microsoft CRM partners
Fellowship One
Shelby Arena
Build-our-own open source system in partnership with WEC
Interestingly, two of my finalists overlap with Tony's three finalists.
Tony's third finalist is Blackbaud. I'm intrigued by that and enjoyed the chance earlier this year to discuss it in detail with Shelley Hildebrand of Tony's staff at Perimeter. Unfortunately, the cost of Blackbaud is up there near what it would cost to build our own, so that's not a feasible option for us.
Also like Tony, I know there's a chance our short list will get longer before it gets shorter. Did I mention this decision is risky and complicated?
Here's our short list:
MSCRM 3.0 plus add-ons from Microsoft CRM partners
Fellowship One
Shelby Arena
Build-our-own open source system in partnership with WEC
Interestingly, two of my finalists overlap with Tony's three finalists.
Tony's third finalist is Blackbaud. I'm intrigued by that and enjoyed the chance earlier this year to discuss it in detail with Shelley Hildebrand of Tony's staff at Perimeter. Unfortunately, the cost of Blackbaud is up there near what it would cost to build our own, so that's not a feasible option for us.
Also like Tony, I know there's a chance our short list will get longer before it gets shorter. Did I mention this decision is risky and complicated?
Subscribe to:
Posts (Atom)
