Maintenance for Christmas

Christmas Eve and Christmas Day are the lowest-usage days of the year (both in terms of member activity and in terms of visits to member sites), so we are going to roll out some core system upgrades over the next 36 hours. These updates relate mostly to file servers.

Despite having no single point of failure from the hardware perspective, each site’s content is still backed by a single system image (necessary for coherency), so these updates may cause some temporary disruptions to affected sites. We will do our best to minimize that.

We do also plan to upgrade our core database servers. These are fully redundant, so we do not anticipate disruption, but the possibility does exist. We hope this upgrade will resolve an issue that mainly manifests as intermittent errors in our member interface early in the morning (UTC) on Sundays.

Upcoming updates, upgrades, and maintenance

We have accumulated some housekeeping tasks that we’ll be taking care of over the next couple of months. They’re all necessary things to make sure our service keeps running at its best, and though we work hard to prevent these types of things from impacting services, occasionally they do intrude. As a result, we want to let everyone know what we’re up to and what the effects will be.
Continue reading Upcoming updates, upgrades, and maintenance…

Automatic file server upgrades

As most of our members are aware, one of our older file servers, f5, has been causing intermittent problems. The time has come to move the sites still using it to newer, faster, more reliable equipment. The ability to do that manually has been available in our UI for about a week now, and it has not surprisingly been pretty popular. But after that server caused additional downtime this past week, we’re moving to the next phase: moving sites automatically.
Continue reading Automatic file server upgrades…

Post-mortem report of Saturday’s file server failure

On Saturday, March 29 at about 4pm US Eastern time, we rebooted one of our file servers that hosts content for member sites. It experienced a critical hardware failure and did not come back online. It took about 28 hours to get things back into service. We’re going to talk briefly about why that happened, and what we’ll be doing differently in the future.
Continue reading Post-mortem report of Saturday’s file server failure…

IPv6, SSL, scheduled tasks, storage discounts & bulk bandwidth

We have quite a few feature announcements to bring you this holiday season. We’ve added support for several features, some of which have been requested for years like IPv6, SSL, and scheduled tasks (AKA cron jobs). We’ve also introduced new billing options that make our service pricing fairer and more scalable; these options will help a broad variety of our members save money.
Continue reading IPv6, SSL, scheduled tasks, storage discounts & bulk bandwidth…

Security flaw with login corrected

One of our members informed us a couple of days ago that due to a strange combination of actions and circumstances, he hit a flaw in our login system that enabled him to access the membership of another member with a similar name.

Of course we promptly investigated; the problem has been permanently fixed.
Continue reading Security flaw with login corrected…

Scheduled maintenance November 22 and December 15

We are scheduling two maintenance windows in the next month to move some equipment:

Date: November 22nd, 2010
Window: 9am to noon UTC (4am to 7am US Eastern, 1am to 4am US Pacific)
Affecting: MySQL nodes m2, m3, and m21

Date: December 15th, 2010
Window: 8am to 1pm UTC (3am to 8am US Eastern, midnight to 5am US Pacific)
Affecting: File servers f2 and f5

Each server should be offline for about one hour, not the whole window. This will cause some downtime. While the MySQL nodes are offline, those MySQL processes hosted on them will be unreachable. While the file servers are offline, sites hosted by those file servers will show an official maintenance page.
Continue reading Scheduled maintenance November 22 and December 15…

Removing deprecated IP block

Many years ago, we were assigned the IP address block 64.238.220.0/23 by one of our upstream network providers. We officially deprecated the use of that block way back in 2008, and we will be returning it on December 1st, 2010, so it will not work after that point.
Continue reading Removing deprecated IP block…

Brief Network Maintenance July 20-22

This is just a quick announcement about some upcoming network maintenance.

Due to our load-balancing capabilities, most of this will be done with no disruption to our services. There are a few exceptions, though. We’ll be doing maintenance over the next few days in the early morning hours (between 1am and 5am US Eastern time — 5am and 9am UTC), the following services will be briefly disrupted (should be about 5-10 minutes each):
Continue reading Brief Network Maintenance July 20-22…

File server “f1” replacement

Our venerable old file server “f1” had some problems last month that left us with some doubt as to the viability of its redundant power supplies over the long term. Since then, we’ve been planning and preparing to migrate all the sites it handles to other, newer file servers.

That’s all been prepped now, and what we’re going to do is automatically migrate everyone during the month of April. If you have affected sites, you can get a specific time for each site from our member interface, and the main sites page will star any site scheduled for an upgrade on your list of sites so you can see at a glance which sites are affected.
Continue reading File server “f1” replacement…