Automatic file server upgrades

As most of our members are aware, one of our older file servers, f5, has been causing intermittent problems. The time has come to move the sites still using it to newer, faster, more reliable equipment. The ability to do that manually has been available in our UI for about a week now, and it has not surprisingly been pretty popular. But after that server caused additional downtime this past week, we’re moving to the next phase: moving sites automatically.

We’ve been testing the replacement file servers for some time now, with hundreds of test sites and various use cases, and they have done very well. Naturally, we’re still paranoid that something will go wrong, but in addition to the testing we have an aggressive backup and replication schedule. So it’s time to move ahead.

Beginning August 4th and continuing through the end of the month, we will start automatically migrating affected sites. If you have any, they are marked with an asterisk on the Sites tab in our UI, with more details on the Site Info Panel for each affected site. The Site Info Panel will also let you adjust the scheduled upgraded, allowing you to migrate a site at any time or (to an extent) postpone an upgrade that is scheduled at a bad time for you.

Most sites don’t need to make any changes as a result of this migration. Based on our testing and the sites that have voluntarily migrated thus far, less than 1% of sites need anything modified to continue working after the upgrade. These changes are related to hardcoding absolute paths that won’t be valid after the migration. I.e. anything starting with /f5/sitename/. These fall into two broad categories.

First, .htaccess files. If you’re using HTTP basic authentication or something similar in your .htaccess file that uses absolute pathnames, those will have to be changed after the migration. You’ll be able to get the new path to use from your site info panel after the migration.

Second, if you’re still using PHP 5.3 Fast and you have hardcoded paths in your PHP code, those will also need to be updated. Using hardcoded paths in this situation was never recommended; it’s always preferable to use a preset variable like $_SERVER[‘DOCUMENT_ROOT’] or $_SERVER[‘NFSN_SITE_ROOT’] if at all possible. PHP 5.3 has also been obsolete for a long time. So if you find yourself in that situation, this is a great time to upgrade that from our UI as well. You’ll still have to change the paths, but this will be the last time. All the currently-supported versions of PHP (5.4 and later) use /home-based paths, just like CGI and ssh, and those never change.

To help you find out if your site needs to be modified, we’ve developed a scan which is run during the migration. When the migration is finished, it will email you to let you know it’s done and whether or not it found any potential problems. It may not catch every possible issue, but it does a very very good job.

Once f5 is no longer in use, it’ll be tempting to give it the full Office Space treatment due to the problems it has caused, but the truth is that it served us incredibly well for a long time, so giving it a salute as it is ejected into space in a decaying orbit into the sun would better fit the totality of its service. (Although that’s admittedly not in the budget, so recycling is a more likely outcome unless the console prints “Will I dream?” as we shut it down for the last time, in which case we probably won’t have the heart.)

Although only a tiny fraction of our members will have even minor problems with this upgrade, each and every one of our members and each and every one of their sites is important to us. If you do run into any snags related to migrated sites (or, really, anything else), please feel free to post on our forum and we’ll do what we can to help you out. (But please don’t post about them here; blog comments are a terrible venue for providing support, second only to Twitter in sheer awfulness and unnecessary difficulty.)

10 Comments

RSS feed for comments on this post.

1. “As most of our members are aware…” How exactly would we be aware of this? I certainly had no inkling, either of problems with the servers hosting my sites, or of the impending migration, and it’s only pure chance that I logged into my site panel tonight to discover there was a planned upgrade.

   Would have been really nice if you’d reached out to affected folks over email about *before* this happened.

   Comment by rp — August 11, 2014 #

2. While I understand that you personally might prefer we send out regular mass email updates about the status of our service, and that you’re probably not alone in that wish, that is not our practice. The majority of our members prefer that we not do so, so the supported ways to get routine information about updates are through our blog, our Twitter feed, and our offsite status page. Most of our members are aware of this either through those, or because it has caused visible downtime on two separate occasions this year.

   As outlined in the post, we are sending email messages upon completion of the migration, since our system is able to analyze the site during the migration and try to spot trouble. Thus that’s the point where our system can generate messages for people that contain specific information relevant to them. What we are finding with that approach is that the number of sites for which this migration requires any action at all isn’t just less than 1%, it’s running about 1 in 1000. And in some of those cases, the action needed was due to a (now fixed) glitch on our part with the migration. So we do feel the decision not to send out a mass email is the correct one, as in almost all cases it would be repeating information that people already know about something they don’t need to do anything about.

   In any case, I understand your point of view and I’m sorry we aren’t able to accommodate everyone’s individual contact preferences during unique situations like this. In the future, we would like to offer a more general option to receive blog posts by email, but I don’t have any timeline for when that might be implemented.

   -jdw

   Comment by jdw — August 11, 2014 #

3. I certainly appreciate that you do not send out emails all the time, but I do believe it might be helpful to allow people to opt-in to certain updates. Even a subscribe to blog signup would be nice and it would be completely opt in. There are lots of providers that could do it for you, I’ve used MailChimp. For smaller lists, you could even do it for free.

   Comment by Adam — August 13, 2014 #

4. First I want to reiterate from my previous comment: “In the future, we would like to offer a more general option to receive blog posts by email, but I don’t have any timeline for when that might be implemented.”

   However, I also think the three comments requesting similar-but-not-identical things illustrate the fundamental problem with broadcast emails. Each of these three messages want to receive email updates, but each wants email updates about a different subset of things. Two say they think it should be opt-in. One sounds like it should be opt-out. No one solution meets the criteria of all three people, and that’s just three people.

   The other problem is that every email sent that a person doesn’t care about reduces the amount of attention they’ll pay to future emails. We purposefully don’t email people very much so that when we do, they know it’s important.

   So this is one of those rough areas where there’s nothing we can do that will satisfy everyone, and I’m sorry for that. So in the future we’ll do something basic and middle-of-the-road, like offer a more general option to receive blog posts by email. (Actually, probably notification of the posts.) But I don’t have any timeline for when that might be implemented. 🙂

   -jdw

   Comment by jdw — August 13, 2014 #

5. To put a different spin on this, I think if we could poll all of our members (you know, somehow, without emailing them) with a question like “Do you think we should send out email updates about things that are important to you?” we would see a response of something like 90% yes, 10% no. But if we asked a more specific question like, “Should we send out email updates in situation _____?” or “Should these messages be opt-in or opt-out?” the answer rates would be a lot closer to 50/50.

   So, say we had one button to opt-in to messages sent out in two very specific situations. If each one of those three points got a 50/50 response, already 87.5% percent of people are at least somewhat dissatisfied with the implementation. It’s hard to call that anything but a failure.

   Confronted with math like that, the natural instinct is to shut up and keep your head down.

   (Keep in mind that most of the specific situations that would poll really well are individualized ones that we already notify about: your balance is low or zero, your domain is about to expire, we detected somebody trying to hack your WordPress blog, etc. Generic broadcast messages are much more balanced, as evidenced by the person who says they’re on freebsd6; quite a lot of people don’t care what we do as long as it don’t break their stuff.)

   -jdw

   Comment by jdw — August 13, 2014 #

6. Put me in the don’t care what you do if it doesn’t break my stuff camp. I’m not a huge user, if i was I would pay more attention to this stuff.

   Comment by pnutjam — August 15, 2014 #

7. Why don’t you have a Twitter account and keep your customers updated with any maintenance or updates? I find it rather bizarre that you have no Twitter account in this day and age.

   Comment by ayo — August 18, 2014 #

8. We have used our Twitter account to keep our members informed about maintenance and updates since 2010. -jdw

   Comment by jdw — August 18, 2014 #

9. Lol. Thank you guys for an entertaining read. Being a fan of the original Hitchhiker’s Guide series, I can just see that super cool black spaceship now while Disaster Area plays one of their songs about boy-being meeting girl-being… as they speed headlong into a blazing sun…

   NFS rocks!

   Comment by JASON — August 19, 2014 #

10. I also enjoyed a lol, thanks to your humor-spiced message.

    For those of us who manage to learn enough of the gobbldygook of coding websites in order to launch and maintain a site here, it’s hard to imagine a better place than NFS.

    I admire the way you make decisions, limit emails to essentials, and explain processes for anyone to see if we simply login and update ourselves occasionally.

    Gratefully,
    James T. Laffrey

    PS. You have the best “Leave A Comment” intro: “This blog is our free speech platform, not necessarily yours.”

    Comment by JamesLaffrey — August 26, 2014 #

Sorry, the comment form is closed at this time.