Email forwarding follow-up

Our email forwarding upgrade has been completed! In general, it went very smoothly, with (of course) a couple of exceptions. The disruption was minimal, significantly less than expected; we were able to do a live migration of our UI to use the new backend without having to take it offline at all, and nobody was unable to manage their email forwarding for more than a few minutes. The new servers are running very well and the email processing scheme we have implemented seems to be working out.
Continue reading Email forwarding follow-up…

Email forwarding changes and UI downtime

Over the next few days, we will be making major changes to our email forwarding service in order to make sure that we are providing a consistent, high-quality forwarding service that meets our members’ expectations. This is something we have wanted to do for awhile, but recent events have forced us to pursue a change strategy that might not have been our first choice.
Continue reading Email forwarding changes and UI downtime…

Friday night maintenance window for MySQL

As previously announced, we are proceeding with a maintenance window on Friday night between 10pm and 1am Arizona/Pacific time (1am and 4am Eastern, 5am and 8am UTC).

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 – 45 minutes.
Continue reading Friday night maintenance window for MySQL…

MySQL FreeBSD 7 ZFS Beta update & MySQL upgrades

Some time ago, we announced a beta test of MySQL running on FreeBSD 7 using the experimental ZFS filesystem. We’ve now concluded that beta test, and I wanted to let you know the results.

Here’s the executive summary: FreeBSD 7 is, by and large, great for us and we will be aggressively deploying it throughout our network in the near future. ZFS is also great, but while it definitely has a future on our network someday, this is not that day.
Continue reading MySQL FreeBSD 7 ZFS Beta update & MySQL upgrades…

UPDATED: Domain registration price increases September 30th

It’s that time of year again. The leaves start to turn. The nights get a little cooler. And the domain registry operators gratuitously raise their prices, because ICANN apparently sent Homer Simpson to negotiate their contracts. We received notice of the specific effects on us today.

Effective September 30th, 2008, our prices for domain registrations, transfers, and renewals in our supported TLDs (com, net, org, biz, info & name) will increase from $7.99 to $8.59 per domain per year.
Continue reading UPDATED: Domain registration price increases September 30th…

Scheduled Maintenance: IP Renumbering In Progress

We are in the process of renumbering out of our current IP addresses and into a larger block of addresses. The effect of this change will be better routing performance and more room for us to grow and expand our service.
Continue reading Scheduled Maintenance: IP Renumbering In Progress…

SSH RSA host key changed

After a review of our security practices brought on by the recent Debian ssh key security issue (which does not affect our FreeBSD-based service), we’ve decided to upgrade the strength of our RSA ssh host key from 1024 bits to the same length that we recommend you use, 4096 bits.

The correct/current RSA host key:

fc:89:a1:64:74:70:a4:82:58:c1:73:4e:72:59:63:56

Continue reading SSH RSA host key changed…

Experimental FreeBSD 7 + ZFS + MySQL technology trial

For most of our production servers, we run FreeBSD 6.3, which is a well-tested, stable, and excellent-performing release. However, the FreeBSD world moves on and FreeBSD 7.0 was released earlier this year. The primary benefit to the new version is supposed to be vastly improved performance, ranging from 350% to 1500% faster, under heavy workloads.

Hey, we have some heavy workloads…
Continue reading Experimental FreeBSD 7 + ZFS + MySQL technology trial…

New offsite dead-drop backup service

It’s often hard to think about disaster planning. The thing about all disasters is that they’re really unlikely, but the consequences of winning the disaster lotto are, well, disastrous.

We don’t want anything horrible to happen to our service, we don’t expect anything horrible to happen, and (for the paranoid among us) aren’t aware of any horrible about to happen. Prudence simply dictates that we acknowledge that disasters are possible and take reasonable precautions to ensure that, were our disaster ticket to get punched that we would be eventually able to recover. (We’re talking about large scale the-entire-datacenter-is-permanently-gone-or-unusable disasters here.)

For us, this means keeping heavily encrypted offsite copies of our key databases, all of our custom server source code, and a lot of configuration information. That’s all the stuff we would need to rebuild our service, member and account balance records from scratch. What it does not include is offsite copies of our members’ content. While we would love to be able to do that automatically, there’s so much of it that the expense would be considerable. We’ve chosen instead to allow people to choose for themselves whether they feel that level of additional protection (and cost) is justified. In a lot of cases, it probably won’t be, but it’s something we need to do for our data so we want you to have the option to do it for yours as well.

Therefore, we’ve entered into a relationship with highly-regarded backup provider rsync.net to offer an innovative (we think) kind of offsite backup service for hosted sites and MySQL processes.
Continue reading New offsite dead-drop backup service…

Public and member UI sites scheduled downtime today

We will need to take some of our internal servers down briefly today: a couple of times for about 10 minutes each between 6pm and 10pm Arizona time (1am and 5am UTC).
Continue reading Public and member UI sites scheduled downtime today…