CGI/ssh Upgrade

This is just a short note to reflect a couple of upgrades.

First, we have upgraded the ssh environment to more powerful hardware in order to allow for continued growth and to make sure scheduled tasks (still coming soon) will have enough resources to run without driving anybody’s site into the ground.

Second, we have updated Perl in the CGI/ssh environment from 5.8.8 to 5.8.9. Despite being a minor-version upgrade, this required the rebuild of nearly 1,000 CPAN modules and dozens of supporting libraries. Thus, we wanted to let people know to be on the lookout for resulting weirdness or incompatibilities. A few CPAN modules bought the farm on this upgrade, and we’ll list those below.
Continue reading CGI/ssh Upgrade…

Email forwarding follow-up

Our email forwarding upgrade has been completed! In general, it went very smoothly, with (of course) a couple of exceptions. The disruption was minimal, significantly less than expected; we were able to do a live migration of our UI to use the new backend without having to take it offline at all, and nobody was unable to manage their email forwarding for more than a few minutes. The new servers are running very well and the email processing scheme we have implemented seems to be working out.
Continue reading Email forwarding follow-up…

Email forwarding changes and UI downtime

Over the next few days, we will be making major changes to our email forwarding service in order to make sure that we are providing a consistent, high-quality forwarding service that meets our members’ expectations. This is something we have wanted to do for awhile, but recent events have forced us to pursue a change strategy that might not have been our first choice.
Continue reading Email forwarding changes and UI downtime…

Friday night maintenance window for MySQL

As previously announced, we are proceeding with a maintenance window on Friday night between 10pm and 1am Arizona/Pacific time (1am and 4am Eastern, 5am and 8am UTC).

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 – 45 minutes.
Continue reading Friday night maintenance window for MySQL…

MySQL FreeBSD 7 ZFS Beta update & MySQL upgrades

Some time ago, we announced a beta test of MySQL running on FreeBSD 7 using the experimental ZFS filesystem. We’ve now concluded that beta test, and I wanted to let you know the results.

Here’s the executive summary: FreeBSD 7 is, by and large, great for us and we will be aggressively deploying it throughout our network in the near future. ZFS is also great, but while it definitely has a future on our network someday, this is not that day.
Continue reading MySQL FreeBSD 7 ZFS Beta update & MySQL upgrades…

Scheduled Maintenance: IP Renumbering In Progress

We are in the process of renumbering out of our current IP addresses and into a larger block of addresses. The effect of this change will be better routing performance and more room for us to grow and expand our service.
Continue reading Scheduled Maintenance: IP Renumbering In Progress…

New offsite dead-drop backup service

It’s often hard to think about disaster planning. The thing about all disasters is that they’re really unlikely, but the consequences of winning the disaster lotto are, well, disastrous.

We don’t want anything horrible to happen to our service, we don’t expect anything horrible to happen, and (for the paranoid among us) aren’t aware of any horrible about to happen. Prudence simply dictates that we acknowledge that disasters are possible and take reasonable precautions to ensure that, were our disaster ticket to get punched that we would be eventually able to recover. (We’re talking about large scale the-entire-datacenter-is-permanently-gone-or-unusable disasters here.)

For us, this means keeping heavily encrypted offsite copies of our key databases, all of our custom server source code, and a lot of configuration information. That’s all the stuff we would need to rebuild our service, member and account balance records from scratch. What it does not include is offsite copies of our members’ content. While we would love to be able to do that automatically, there’s so much of it that the expense would be considerable. We’ve chosen instead to allow people to choose for themselves whether they feel that level of additional protection (and cost) is justified. In a lot of cases, it probably won’t be, but it’s something we need to do for our data so we want you to have the option to do it for yours as well.

Therefore, we’ve entered into a relationship with highly-regarded backup provider rsync.net to offer an innovative (we think) kind of offsite backup service for hosted sites and MySQL processes.
Continue reading New offsite dead-drop backup service…

Public and member UI sites scheduled downtime today

We will need to take some of our internal servers down briefly today: a couple of times for about 10 minutes each between 6pm and 10pm Arizona time (1am and 5am UTC).
Continue reading Public and member UI sites scheduled downtime today…

Surprise WordPress Upgrade

We received a note from Technorati today about a serious security problem with old versions of WordPress, including the version we were running, that is now being exploited on a widespread scale. We’ve thus hastily upgraded to WordPress 2.5. That did cause a brief bit of disruption to the “News & Announcements” portion of our member site, which is now resolved.

If you want to run WordPress, you too may want to check whether you’re running the most current version with the latest patches. Better safe than sorry!

Contest to redesign our public website!

If you’ve ever seen our public website, you may have notice that it… (cough) blows. We’d like to fix that but, well, we don’t exactly have an impressive track record of award-winning web design. We’re also chronically short on available time. Thus, we’ve decided to throw a redesign out there as a contest for anyone who thinks they can do a better job. (And if you don’t think you can do a better job, please consider self-esteem classes.)
Continue reading Contest to redesign our public website!…