Scheduled Downtime for Friday, November 20

We have some facilities maintenance scheduled for this Friday. As part of this maintenance, we will need to physically move a handful of critical file and database servers between racks in our Phoenix datacenter. Since that equipment forms the heart of our hosting service, we’ll need to shut almost everything down briefly, just long enough to move it.

The maintenance window will be from 10am to 4pm MST (5pm to 11pm UTC) on Friday, November 20th, 2009.
Continue reading Scheduled Downtime for Friday, November 20…

Email forwarding follow-up

Our email forwarding upgrade has been completed! In general, it went very smoothly, with (of course) a couple of exceptions. The disruption was minimal, significantly less than expected; we were able to do a live migration of our UI to use the new backend without having to take it offline at all, and nobody was unable to manage their email forwarding for more than a few minutes. The new servers are running very well and the email processing scheme we have implemented seems to be working out.
Continue reading Email forwarding follow-up…

Email forwarding changes and UI downtime

Over the next few days, we will be making major changes to our email forwarding service in order to make sure that we are providing a consistent, high-quality forwarding service that meets our members’ expectations. This is something we have wanted to do for awhile, but recent events have forced us to pursue a change strategy that might not have been our first choice.
Continue reading Email forwarding changes and UI downtime…

Friday night maintenance window for MySQL

As previously announced, we are proceeding with a maintenance window on Friday night between 10pm and 1am Arizona/Pacific time (1am and 4am Eastern, 5am and 8am UTC).

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 – 45 minutes.
Continue reading Friday night maintenance window for MySQL…

Scheduled Maintenance: IP Renumbering In Progress

We are in the process of renumbering out of our current IP addresses and into a larger block of addresses. The effect of this change will be better routing performance and more room for us to grow and expand our service.
Continue reading Scheduled Maintenance: IP Renumbering In Progress…

SSH RSA host key changed

After a review of our security practices brought on by the recent Debian ssh key security issue (which does not affect our FreeBSD-based service), we’ve decided to upgrade the strength of our RSA ssh host key from 1024 bits to the same length that we recommend you use, 4096 bits.

The correct/current RSA host key:

fc:89:a1:64:74:70:a4:82:58:c1:73:4e:72:59:63:56

Continue reading SSH RSA host key changed…

Public and member UI sites scheduled downtime today

We will need to take some of our internal servers down briefly today: a couple of times for about 10 minutes each between 6pm and 10pm Arizona time (1am and 5am UTC).
Continue reading Public and member UI sites scheduled downtime today…

Facility move post-mortem analysis

It has been a little over two weeks since our facility move. I believe at some point we promised some post-mortem analysis of what went right and what went wrong, and here it is.

The outcome of the move is a success. All of our core hardware is in one place, it is all working, and we are in a better position to grow and expand than we have ever been.
Continue reading Facility move post-mortem analysis…

Planned downtime for Monday, September 10

On Monday, September 10, 2007 at around noon Arizona time (3pm Eastern, 7pm UTC), we will temporarily shut down our entire network to complete a migration of our equipment to a new datacenter. We anticipate that it will take four to eight hours to complete the move. All our services will be offline during that time.
Continue reading Planned downtime for Monday, September 10…

Offsite Network Status Page

We know that when there is a problem with our services, one of the most important things for us to do is communicate effectively with you about what the problem is and when we expect it to be fixed. Nothing sucks more than being left in the dark.

Ordinarily, we use System Problem Reports on the Support panel in our member interface to let you know about problems with the service. In most cases, this works well.

However, certain categories of outages, like upstream network failures, can render us unable to post such reports, or leave you unable to read them until after the problem is resolved. While such events should be rare, that’s the most important time for us to be letting you know what’s going on.

To that end, we’ve created an offsite network status page at http://status.nearlyfreespeech.net/. This site runs on one of our servers located in Dallas. That server doesn’t depend on our Phoenix hosting cluster at all, and will continue to operate even if everything else is offline.
Continue reading Offsite Network Status Page…