Scheduled Downtime for Friday, November 20

We have some facilities maintenance scheduled for this Friday. As part of this maintenance, we will need to physically move a handful of critical file and database servers between racks in our Phoenix datacenter. Since that equipment forms the heart of our hosting service, we’ll need to shut almost everything down briefly, just long enough to move it.

The maintenance window will be from 10am to 4pm MST (5pm to 11pm UTC) on Friday, November 20th, 2009.

This will affect web hosting, email forwarding, and “support services” (i.e. our sites, SSH, FTP) for all members.

We don’t expect the actual downtime to be anything close to the whole time. In an ideal case, it would take us about an hour to prep and and hour to take everything down, move it, and bring it back up. However, this is the real world, not the ideal one, so we’re giving ourselves some additional room to maneuver.

I’m sure there’s someone out there for whom this is a spectacularly inconvenient time, and to them we sincerely apologize. Any time we picked for maintenance of this sort would be bad for somebody. We did strive to pick a low-usage time when we could guarantee the manpower we needed.

We also would have liked to provide more notice, but up until this evening any announcement would have been pretty much content-free. (“We will be scheduling a maintenance downtime of unknown length at an unknown point in the future.”) As of right now, we have a schedule we believe can be met (the original proposed date has already passed, so any earlier specific announcement would have turned out to be wrong), and so we’re bringing it to you as quickly as we can.

We apologize again for the inconvenience. On Friday, as every day, we’ll all be working hard to bring you the best, most reliable hosting service we can. “NEITHER RAIN NOR SNOW NOR GLOM OF NIT.”

12 Comments

RSS feed for comments on this post.

1. So, who will this affect? Everyone? Every NFS-hosted site will be offline for possibly six hours?

   As indicated in the blog post you’re replying to, we expect that the downtime will last about an hour plus however long it takes to fix whatever goes wrong. That’s a pessimistic attitude, but as with skinny cooks, never trust an optimistic system administrator. The purpose of scheduling a six-hour window is to allow extra time for things to go wrong, even though we hope they don’t, and to allow some flexibility as far as when we start in case prep work takes longer than expected. Naturally, we won’t keep anything down for even a moment longer than absolutely necessary. -jdw

   Comment by AmbroseChapel — November 18, 2009 #

2. Will DNS be affected as well, or just web, file, and database hosting?

   Great question. We do not anticipate that DNS will be affected. -jdw

   Comment by David Mandelberg — November 18, 2009 #

3. Thanks for letting us know. I really appreciate it.

   Comment by Eddy — November 19, 2009 #

4. I was caught totally unprepared by this.

   Announcing only on your blog that services will be down for a couple of hours in the middle of the day is not the greatest form of communication; I would have appreciated a short mass-email.

   Believe it or not, I don’t usually have the time to visit the blogs of all my various service providers and product vendors every day to keep track of their announcements.

   I recommend using an RSS feed aggregator or an RSS-to-email gateway on our “News & Announcements” feed. It’s very low-noise and will keep you abreast of important issues without getting us spam blacklisted for sending out mass emails. -jdw

   Comment by Rob — November 20, 2009 #

5. Would have been nice to have more notice. 2 days just isn’t acceptable, regardless of your barely specified excuses.

   No matter how notice there is; more would always be better. “As much as possible” is the best we can do. -jdw

   Comment by Bob — November 20, 2009 #

6. Is there any chance we could set up our own “we’re down for maintenance” pages in future? I’d prefer to brand ours with our own content rather than with NearlyFreeSpeech.NET’s content — this confused some of our readers.

   It would be nice if that were possible, but in a case such as this, anything like that would also have been affected. We were pretty much stuck with a one-size-fits-all message. Sorry for any confusion that resulted! -jdw

   Comment by Graham — November 20, 2009 #

7. I concur with the previous comments regarding an emailed announcement. An RSS subscription (requiring me to be proactive) is no substitute for good customer communication.

   I would add further, that every other vendor that i use schedules maintenance like this for an overnight window, and I would have (and still do) expect the same from NFS.

   The nature of a global Internet is that somebody’s overnight is somebody else’s midday. We picked the right time based on all the factors at play, including historic network activity which is far more important than time of day. I’m sorry if it didn’t suit your particular situation. -jdw

   Comment by Chris — November 20, 2009 #

8. I recommend using an RSS feed aggregator or an RSS-to-email gateway on our “News & Announcements” feed.

   Could you please create and use a new category: “Maintenance Pending”, and removed the category from any blog post once the maintenance is complete.

   I could then add an RSS sidebar widget to my blog attached to that feed. Currently I can attach to the “Network Status” feed, but then stale announcements hang around forever.

   It’s very low-noise and will keep you abreast of important issues without getting us spam blacklisted for sending out mass emails. -jdw

   Every other business I deal with on the internet sends me mass emails no problem. (It’s not spam if you have a prior business arrangement.) I don’t understand why you’re different. (And it’s not like you’ve never sent mass emails before.)

   Comment by Daran — November 22, 2009 #

9. Copying over a message posted in a related discussion in our forum. Anyone who has suggestions, please bring them in over there. As I’ve opined before, I don’t feel blog comments are a good place for a threaded discussion.

   -jdw

   If someone wants to suggest a reasonable mechanism for doing this, we’d certainly consider it.

   Every time we’ve been down that road, we wound up regretting it.

   “Just send the messages” isn’t an answer. It results in us spending days getting off of spam blacklists.

   We also aren’t quite large enough, nor do we send enough correspondence to justify using one of the big legitimate bulk email firms. They’re really expensive because they want you to be sending out regular newsletters, something we’d never do.

   It’s a situation where the solution sounds easy, but actually isn’t.

   Comment by jdw — November 23, 2009 #

10. As many have noted above, email or some form of “reach out and touch” us communication for this sort of announcement…. twitter seems perfect for this sort of thing? and you don’t have to worry about mass-mail blacklists…. anyways, just my 2 cents.

    Comment by Ben — November 26, 2009 #

11. Just a quick comment to say that this is exactly what I expect from NFSN (in a good way). I’m glad you don’t email me, I’d rather take the RSS feed. It’s a shame there’s got to be downtime, but no amount of notice would have made it better so no problems there either.

    Once again, I admire the comparative honesty and frankness that we get from NFSN.

    We should remember that we’re not paying for some kind of 99.999% uptime SLA. Just like we’re not paying for poor jdw to stay up all night moving servers around. At least we’re not paying during these outages (like pay monthly customers would).

    Comment by Mike — November 27, 2009 #

12. I feel that NFSN did an excellent job with this maintenance period. They are open and forthcoming, much more than I get with other service providers. As JDW mentioned, maintenance is mandatory. Fortunately, they were able to predict and meet their targets. A very creditable job in my opinion.

    Thanks, Jeff

    Comment by Jeff — November 29, 2009 #

Sorry, the comment form is closed at this time.