Comments on: Post-mortem report of Saturday’s file server failure

By: philodygmn

philodygmn — Thu, 15 May 2014 05:33:04 +0000

It’s clear from your account that you were, are, and will continue to do everything in your power to bring us the very best of the very most affordable possible web hosting. Thanks for all your hard work, dedication, and especially your savvy and independent-mindedness. I most appreciate your pay-per-use option for static sites, and am exceedingly grateful you offer it.

By: Chris

Chris — Tue, 13 May 2014 11:45:20 +0000

Like many other posters, I’d like to thank you for the explanation. Other companies would ignore the problem, deny it, blame somebody else, and MAYBE eventually say “sorry” and leave it at that. Your blog during downtime, and this explanation afterwards, are very refreshing.

You’re good to me, and I’m good to you. I’ve asked for your help twice (once as a prospective customer and once while setting up DNS) and you’ve been polite, very quick, and absurdly helpful. Complaining about this downtime would be like complaining that my mother didn’t bring me breakfast in bed when she had the flu!

Keep up the good work, and thanks for the years of service!

By: Jon

Jon — Sat, 10 May 2014 14:37:32 +0000

I love that when something /does/ go wrong at NSF, we get a page’s worth of explanation from you guys, rather than the “it looks fine from our end, try it again in… uh… three, maybe three and a half hours” explanation I’ve had from other companies.

By: Mark Preston

Mark Preston — Wed, 07 May 2014 00:15:05 +0000

You guys are terrific. I will never leave NFS while you are around.

By: Ingbert

Ingbert — Wed, 30 Apr 2014 15:28:59 +0000

Thank you so much for all of your service. I also really appreciate how open and honest and straightforward you are when things go wrong. I have had nothing but good experiences with NFS, and I regularly recommend it to my students when they are asking for good web hosts. Your integrity is why I keep using this service.

By: lostnbronx

lostnbronx — Mon, 21 Apr 2014 07:41:58 +0000

Add my voice to the “Well done!” crowd. You guys did an amazing job getting the problem fixed…and an amazing job reporting on it. NFS is truly one of a kind.

By: Aaron Mason

Aaron Mason — Sat, 12 Apr 2014 22:42:12 +0000

I don’t drink the flavor aid that suggests that failures can somehow be avoided entirely. As I’m sure jdw is fully aware, failures can happen and it’s often the difference between being dead and being inconvenienced. One builds to mitigate the effects of failure, not avoid it entirely, the latter often resulting in a horrendously complicated solution that fails spectacularly in a fashion you didn’t see coming.

By: Phil

Phil — Sat, 12 Apr 2014 08:41:17 +0000

There is only one thing rarer than a no-fail server…

Honesty.

🙂

Thanks lots.

By: Olivier

Olivier — Fri, 11 Apr 2014 18:36:24 +0000

Thanks for feedback, this outage happened at a really bad timing for my projects here, had to switch server last minute before a live streaming event! Was a bit of hell for me. But 15 minute solution made me up again. Compared to your marathon it’s much less hassle.

You guys evaluated GlusterFS for clustering file servers?

Yes. -jdw

By: Aaron Mason

Aaron Mason — Fri, 11 Apr 2014 05:50:22 +0000

Your statement that it was occurring on old versions of FreeBSD is comforting.

I’m planning an upgrade to my file server involving a Supermicro Avoton-based motherboard, 3TB drives for z2, 1TB drives for hardware RAID10 on a Dell PERC5/i card and FreeNAS (because EON doesn’t support the PERC, needed for iSCSI exports to two Sun V20Z servers in a Hyper-V cluster) and what you said, knowing how extensively you guys use FreeBSD, was concerning.