Brief Network Maintenance July 20-22

This is just a quick announcement about some upcoming network maintenance.

Due to our load-balancing capabilities, most of this will be done with no disruption to our services. There are a few exceptions, though. We’ll be doing maintenance over the next few days in the early morning hours (between 1am and 5am US Eastern time — 5am and 9am UTC), the following services will be briefly disrupted (should be about 5-10 minutes each):
Continue reading Brief Network Maintenance July 20-22…

File server “f1” replacement

Our venerable old file server “f1” had some problems last month that left us with some doubt as to the viability of its redundant power supplies over the long term. Since then, we’ve been planning and preparing to migrate all the sites it handles to other, newer file servers.

That’s all been prepped now, and what we’re going to do is automatically migrate everyone during the month of April. If you have affected sites, you can get a specific time for each site from our member interface, and the main sites page will star any site scheduled for an upgrade on your list of sites so you can see at a glance which sites are affected.
Continue reading File server “f1” replacement…

Scheduled Downtime for Friday, November 20

We have some facilities maintenance scheduled for this Friday. As part of this maintenance, we will need to physically move a handful of critical file and database servers between racks in our Phoenix datacenter. Since that equipment forms the heart of our hosting service, we’ll need to shut almost everything down briefly, just long enough to move it.

The maintenance window will be from 10am to 4pm MST (5pm to 11pm UTC) on Friday, November 20th, 2009.
Continue reading Scheduled Downtime for Friday, November 20…

Email forwarding follow-up

Our email forwarding upgrade has been completed! In general, it went very smoothly, with (of course) a couple of exceptions. The disruption was minimal, significantly less than expected; we were able to do a live migration of our UI to use the new backend without having to take it offline at all, and nobody was unable to manage their email forwarding for more than a few minutes. The new servers are running very well and the email processing scheme we have implemented seems to be working out.
Continue reading Email forwarding follow-up…

Email forwarding changes and UI downtime

Over the next few days, we will be making major changes to our email forwarding service in order to make sure that we are providing a consistent, high-quality forwarding service that meets our members’ expectations. This is something we have wanted to do for awhile, but recent events have forced us to pursue a change strategy that might not have been our first choice.
Continue reading Email forwarding changes and UI downtime…

Friday night maintenance window for MySQL

As previously announced, we are proceeding with a maintenance window on Friday night between 10pm and 1am Arizona/Pacific time (1am and 4am Eastern, 5am and 8am UTC).

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 – 45 minutes.
Continue reading Friday night maintenance window for MySQL…

Scheduled Maintenance: IP Renumbering In Progress

We are in the process of renumbering out of our current IP addresses and into a larger block of addresses. The effect of this change will be better routing performance and more room for us to grow and expand our service.
Continue reading Scheduled Maintenance: IP Renumbering In Progress…

SSH RSA host key changed

After a review of our security practices brought on by the recent Debian ssh key security issue (which does not affect our FreeBSD-based service), we’ve decided to upgrade the strength of our RSA ssh host key from 1024 bits to the same length that we recommend you use, 4096 bits.

The correct/current RSA host key:

fc:89:a1:64:74:70:a4:82:58:c1:73:4e:72:59:63:56

Continue reading SSH RSA host key changed…

Public and member UI sites scheduled downtime today

We will need to take some of our internal servers down briefly today: a couple of times for about 10 minutes each between 6pm and 10pm Arizona time (1am and 5am UTC).
Continue reading Public and member UI sites scheduled downtime today…

Facility move post-mortem analysis

It has been a little over two weeks since our facility move. I believe at some point we promised some post-mortem analysis of what went right and what went wrong, and here it is.

The outcome of the move is a success. All of our core hardware is in one place, it is all working, and we are in a better position to grow and expand than we have ever been.
Continue reading Facility move post-mortem analysis…