Due to our load-balancing capabilities, most of this will be done with no disruption to our services. There are a few exceptions, though. We’ll be doing maintenance over the next few days in the early morning hours (between 1am and 5am US Eastern time — 5am and 9am UTC), the following services will be briefly disrupted (should be about 5-10 minutes each):

Tuesday Morning
- www.nearlyfreespeech.net
- members.nearlyfreespeech.net
- various error pages
- Member MySQL Node m9

Wednesday Morning
- Our email (e.g. support@nearlyfreespeech.net)
- Outbound mail sent from CGI scripts
- Member ssh & FTP
- phpMyAdmin server
- Email forwarding
- Member MySQL Node m4
- Member MySQL Node m6
- Member MySQL Node m7
- Member MySQL Node m8
- Member MySQL Node m23
- Member MySQL Node m25
- Most pools (a beta feature in limited testing — you know if you have one)

Thursday Morning
- Member MySQL Node m1
- Member MySQL Node m10
- Member MySQL Node m11
- Member MySQL Node m12
- Member MySQL Node m13
- Member MySQL Node m14
- Member MySQL Node m15
- Member MySQL Node m16
- Member MySQL Node m17
- Member MySQL Node m18
- Member MySQL Node m19
- Member MySQL Node m20
- Member MySQL Node m22
- Member MySQL Node m24
- Member MySQL Node m26

To reiterate, these disruptions will be very short (<10 minutes); you probably won’t even notice them, but we wanted to let you know what’s going on anyway. Although it’s only a few minutes, we’ll be changing a lot of stuff under the hood, which is an important step in helping us build a more reliable network.

We wish it were possible to build a network with 100% availability, but we’re still working on that. In the meantime, this scheduled maintenance should definitely help us get a little closer.

Thanks!

That’s all been prepped now, and what we’re going to do is automatically migrate everyone during the month of April. If you have affected sites, you can get a specific time for each site from our member interface, and the main sites page will star any site scheduled for an upgrade on your list of sites so you can see at a glance which sites are affected.

There are two drawbacks to this upgrade.

The first is that the upgrade involves minor downtime since the files need to be transferred from one file server to the other, which requires log files and any dynamic data files be closed and held inactive while they’re moved. The downtime will be roughly proportional to the size of the site, so this is a great time to peek in and check for any files you’re not using or log files you want to trim; it’ll make the upgrade go faster and you’ll save money on storage.

In addition, while most sites will be moved transparently, sites that use absolute paths in .htaccess files (or in PHP scripts, although we don’t advise that) will need to be updated to work properly after the move. If this is applicable at all (I personally moved all my sites without needing to change anything), it should just take a few minutes to update.

For those two reasons, we understand that you may want some control over upgrade schedule. To that end, you can log in and upgrade any of your eligible sites at any time between now and its scheduled date. You can also postpone a site’s upgrade if the scheduled time doesn’t work for you, within reason. You can’t postpone an upgrade past April 20, and postponements are for random intervals just so we don’t end up with an unmanageable number of sites to move on April 30th at 11pm.

The target file servers for this upgrade, f2 and f5, are a lot faster, have SSD-based L2 caches for busy files, superior network performance, and never require painfully long filesystem checks after an unclean shutdown, so we think people will be really happy with the change. Oh, and they have much newer and better redundant power supply setups!

We hope that the preparations we’ve made will make the transition as easy and painless as possible, and that everyone affected will agree that the end results are well worth having.

The maintenance window will be from 10am to 4pm MST (5pm to 11pm UTC) on Friday, November 20th, 2009.

This will affect web hosting, email forwarding, and “support services” (i.e. our sites, SSH, FTP) for all members.

We don’t expect the actual downtime to be anything close to the whole time. In an ideal case, it would take us about an hour to prep and and hour to take everything down, move it, and bring it back up. However, this is the real world, not the ideal one, so we’re giving ourselves some additional room to maneuver.

I’m sure there’s someone out there for whom this is a spectacularly inconvenient time, and to them we sincerely apologize. Any time we picked for maintenance of this sort would be bad for somebody. We did strive to pick a low-usage time when we could guarantee the manpower we needed.

We also would have liked to provide more notice, but up until this evening any announcement would have been pretty much content-free. (“We will be scheduling a maintenance downtime of unknown length at an unknown point in the future.”) As of right now, we have a schedule we believe can be met (the original proposed date has already passed, so any earlier specific announcement would have turned out to be wrong), and so we’re bringing it to you as quickly as we can.

We apologize again for the inconvenience. On Friday, as every day, we’ll all be working hard to bring you the best, most reliable hosting service we can. “NEITHER RAIN NOR SNOW NOR GLOM OF NIT.”

We did use that situation as an opportunity to make some big improvements to the quarantine feature that make dealing with bounce messages more palatable, including changing the “forward” option to just re-send the original message (where possible, since sometimes bounce messages from other servers don’t include the whole message) once the problem is fixed.

The quarantine is now also paginated, so you can see the whole thing if you have more than one page of quarantined messages. It also has an option to decode properly-formatted bounce messages, so you don’t have to stare at a list of 30 identical MAILER-DAEMON “Returned Mail” messages trying to figure out which is which. That option is resource-intensive, though, so it cuts the number of messages displayed from 50 to 10 to keep page load times down.

With these changes, the quarantine is starting to look and feel a little like a rudimentary webmail client. I want to remind everyone that, although it’s essentially now possible (if not pleasant) to read quarantined mail as if it were a mailbox, that’s not what it’s for. It is a temporary, volatile holding tank for messages that can’t or shouldn’t be forwarded and can’t or shouldn’t be returned. Our policy is to provide at least 10 megabytes per domain for quarantine. Currently, once a domain’s quarantine exceeds 12 megabytes, the system will expire out messages until it drops under 10. 10 megabytes can go faster than you think if something with a large attachment winds up in there, so don’t treat the quarantine like permanent storage.

The other glitch pertains to a handful of people who were participating in an email hosting trial we offered earlier this year using a reseller relationship that no longer exists. They were left in the lurch by this change, but we’ve contacted those people privately, and we’ll continue to do our best to help each of them make alternate arrangements.

Those two issues aside, I am very happy with the way this change managed, and super proud of our new system. Almost proud enough to make a foray of our own into email hosting. Almost. Maybe next year.

Since we felt we finally had the service we wanted to provide, we restarted the billing for email forwarding last night. Anyone using third-party DNS need not worry, we’ve pointed all our MX server names at the new email cluster.

We’ll be updating all of our documentation in the coming weeks to reflect the changes. In the future, we may offer the ability to search the quarantine; the new servers are so much faster than the old ones that such an option is now viable, where it wasn’t before.

But first we’ve got some other things to take care of, like the cron support we’re committed to releasing by the end of the year.

The older one is called Legacy Forwarding. It is a service we provide using our own servers, and adheres to our draconian email acceptance policy.

The newer one is simply called Email Forwarding. It is a service we provide in conjunction with a partnership with a third-party company. It is supposed to work a little differently; we call it a “hybrid” forwarding system because it combines certain aspects of a mailbox with certain aspects of email forwarding. Their servers are willing to accept virtually any email, but they then perform content-filtering checks and are supposed to divert suspect messages into a special “spam quarantine” accessible via our member interface.

I say “supposed to” because at some point in the recent past, without consulting with or notifying us, the third-party company unilaterally changed their system to discard all spam-suspect messages instead of placing them into the quarantine folder. At first we thought this was isolated to a few domains, but we have since discovered that the change was global. This means that for all of our members who have “Email Forwarding,” any messages you receive that are believed by their system to be spam, including false positives, are currently being silently discarded. This is absolutely not acceptable to us, but our efforts to redress this with them have not been successful.

Ultimately, our email forwarding members are paying us for email forwarding services, not a third party company, so it is our responsibility to make sure our members get the service they are paying for. To that end, we are taking the following steps:

• We will stop billing for all email forwarding domains until we feel things are working the way we intend.
• We are deploying a new hybrid email forwarding system, described below, which will be applied to all members’ email forwarding domains.
• Because we need to make changes, we will be temporarily disabling the email forwarding portions of our UI. We are not disabling email forwarding, just the UI that controls it.

The new email forwarding system will apply all of our existing email acceptance policy criteria, but it will do so a little differently. As with our legacy forwarding system, messages that meet all of our criteria will sail through the system with no disruption or delay, as will messages sent by domains with proper SPF configurations, even if they have other problems. Also as with the legacy system, messages from blacklisted servers will be immediately rejected.

For messages sent by servers with improper or no SPF and one or more other configuration problems, instead of bouncing them as the legacy forwarding system does, we will be trying a technique called greylisting, which tells the server sending the message to try again in a few hours. If the sending server does that, the message will be accepted.

We will also be adding a spam/virus checking component that will reject email containing viruses and quarantine egregious spam that somehow passes all of the above filters. (The current email forwarding scheme quarantines both spam and viruses.) Also new will be a feature that places post-forwarding bounce messages into the same quarantine, so if there are problems with delivering forwarded messages to you, you will be better able to detect and resolve them.

We feel that this system will make the best trade-offs between meeting our members’ need for reliable forwarding, keeping our forwarding mail servers off of global blacklists, and making life absolutely as miserable and unrewarding as we possibly can for spammers. It will also let us have one forwarding service for all members, and document it appropriately.

All of our email forwarding uses MX records in the nearlyfreespeech.net domain, so we retain full control over how they are handled. The above system is still in final development, but we believe it can be completed and deployed without any major disruption to anyone’s forwarded email. Due to the way Internet email works, even if we do wind up needing to take forwarding offline for a few hours in the middle of the night, messages will simply queue and redeliver when it comes back up.

We need to disable the member UI to work on it because we need the overall configuration to hold still long enough for us to update the UI code and move everyone over to the new system. It is our goal to have the entire transition completed by Saturday. Since we won’t be billing for email forwarding until this is resolved, we will be very highly motivated to make it happen as quickly and accurately as possible. If for any reason you do need to make an urgent email forwarding change that just can’t wait, or if you want to remove it altogether, just drop us a Secure Support Request and we’ll do our best to take care of you.

Since this is still an evolving situation, the above reflects what we want to do, and what we reasonably believe we can do. If and when circumstances change, we will update you accordingly. These changes will not affect the cost of email forwarding, and while they may ultimately put us in a more flexible position with respect to offering email hosting in the future, that is not a goal of this project and nothing is changing on that front at this time.

I apologize for the short notice and rushed character of this change. We spent a lot of effort (probably too much) trying to come to a less dramatic solution, and we were entirely unsuccessful.

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 – 45 minutes.

You can tell if you are likely to be affected if your MySQL @@hostname variable is “m2″ or “m3.” However, while we don’t plan to move any MySQL processes between now and then, we do reserve the right to move them at any time for any reason if appropriate to maintain network stability and performance.

This upgrade will increase total RAM available to member MySQL processes by 60%, with a corresponding increase in CPU power. After the upgrade is complete, we will be able to finish the migration to FreeBSD 7 and the combined result will be better MySQL performance for all.

At the same time, we will be adding RAM and CPU cores to the hosting cluster, but that should not cause downtime since we can rotate the affected servers out of production prior to the upgrade and redistribute the load; there is plenty of spare capacity at that particular time.

As always, we appreciate your patience as we update and upgrade our network to make your hosting better, faster, and more reliable.

We are pursuing this renumbering very aggressively. Partly this is because, although we waited a year to get these IP addresses, we have a very limited window in which to renumber out of the old ones. Such is the nature of bureaucracy. But we have accelerated even more because we have discovered routing issues with an upstream provider related to the IP block we are in, which we believe was responsible for the disruptions some people experienced last week. Unfortunately, we got the IP addresses we currently have from them, so we are not in a position to tell them what to do with them. The new IPs are a direct allocation from ARIN (the American Registry of Internet Numbers) to our network operator, and thus do not have competing interests.

This also means that, while we may add addresses (as conservatively as possible), there should be no reason we will ever need to renumber out of existing ones for the foreseeable future once this migration is complete.

Unfortunately, this change is not without downsides. As stated above, we are attempting to minimize disruption. However, “minimized disruption” is not the same as “no disruption.” This change requires us to make major configuration changes to each and every server, switch, router, and firewall on our network. When it comes to routing, even the tiniest glitch or hiccup can throw everything into disarray for several minutes or more. (Something of that nature happened earlier this morning, for which I apologize. I likewise acknowledge that the blog post about the scheduled maintenance is supposed to happen before the scheduled maintenance. My bad.) Since we are only human, and we are very much working without a net, we are announcing scheduled maintenance between 2am and 6am EDT (6am to 10am UTC) for the next seven days. This does not mean that our service will be down at those times, or even a fraction of them. It simply means we will be performing configuration changes during those times, leading to a significantly increased risk of network downtime, and we want to make you aware of this. These times were chosen by statistical measurement of our network usage; usage during this period is much lower than at any other equivalent period.

As a nerdy little side note, we now have native IPv6 available at the edge of our network and, as part of this change, we will be configuring our core network paths to support it. This doesn’t mean we’re about to offer IPv6 hosting, but it does mean we won’t have to make those changes later on.

And for those that want to know, NearlyFreeSpeech.NET’s new IP block is 208.94.116.0/23. Get used to it, you’ll be seeing it in traceroutes soon, and it’ll be around for a good long while.

This is only one of the projects currently underway to improve the core infrastructure of our service, with an emphasis on making it faster and more reliable for everyone (you may have noticed that new feature development is at a standstill while we do that). While we’re working on this project, we apologize for any inconvenience or downtime that results, we thank you for your patience, and we promise to do our best to make the migration as smooth and trouble-free as possible. We have one goal, and that’s to make our web hosting service amazingly good. This will help.

We will try to update this blog post as we go, so we can keep you posted on how far along we are and how it’s going. In the mean time, to preserve the signal-to-noise ratio, please keep comments on-topic and remember that due to all that’s going on, we’re super busy right now and may be somewhat slow to approve and respond to them.

Update 2008-09-14: We’re making good progress. Most of our firewall/edge network has been converted, and a lot of small subnets have been retired. We are on track to finish IPv4 renumbering on schedule. We have also, just as an oh-so-beta proof of concept, put up IPv6 versions of our two main sites: www.ipv6.nearlyfreespeech.net and members.ipv6.nearlyfreespeech.net. (You’ll get certificate warnings if you’re one of the lucky few who can access these at all.)

Update 2008-09-18: We continue to make progress. New sites started being created in the new IP range this morning, and DNS and SMTP servers have also been cut over. We will be migrating existing sites to the new range over the next day or two. This will not cause downtime, as both the old IPs and the new ones work for all sites during the transition. We are now running about two days behind because of the DDOS attack experienced earlier this week; one maintenance window was lost to dealing with the attack, and a second one due to recovering from dealing with the attack.

Update 2008-02-22: The first phase of this project did complete successfully. The second phase of the project will require us to detect and contact people who have hardcoded the old IP addresses. The second phase has no operational impact and will not require maintenance windows. We will add a new blog post or an update when the second phase is complete and we are ready to begin final turn-down of the old address ranges.

The correct/current RSA host key:

fc:89:a1:64:74:70:a4:82:58:c1:73:4e:72:59:63:56

This naturally changes the host key fingerprint, and is likely to cause warnings for people who have previously connected. This doesn’t indicate a security problem, but you should compare the new key against its fingerprint.

You can verify this on our secure site in the member FAQ.

We have not changed the DSA key, as limitations in the DSA algorithm render longer keys pointless. We discourage but will continue to allow the use of DSA keys.

We apologize to anyone alarmed by the ALL CAPITAL LETTERS!! warnings that ssh applications tend to produce when a host key changes. The new key length should guarantee that we don’t have to change it again for a good long time.

In other key-related news, we scanned all member ssh keys for the Debian weak keys and notified everyone who appeared to be affected. Since there are exploits in the wild, we will shortly be removing any remaining weak keys, and we will not accept any new weak keys for access to member sites.

This outage will not affect member services, only our own sites (www, blog, and members), FTP, and ssh will be affected during these brief downtimes.

We apologize for the short notice and hope this is not too disruptive to anyone’s site maintenance plans.

The outcome of the move is a success. All of our core hardware is in one place, it is all working, and we are in a better position to grow and expand than we have ever been.

Our network monitoring has been vastly improved, with more improvements on the way, which will allow us to be a lot more proactive about keeping your sites healthy in the future, rather than waiting for you to complain when there’s a problem.

Our network infrastructure has been greatly improved. We have incredibly more bandwidth available on our internal network than ever before, so much that we are still tuning the most effective way to utilize it all to serve your sites as quickly as possible. Our cluster technology has always had the ability to failover your site from one web server to another if there’s a problem, but we now have similar (but faster) hot failover technology for our routers, firewalls, and edge proxies.

Our network defenses have also received a sharp boost. We have more and better firewall capability, better inbound packet scrubbing, better backup access to our equipment during attacks, and the ability to selectively blackhole attackers based on complex criteria before they ever reach NearlyFreeSpeech.NET. Not only will this pay off big time in the event of future DDOS attacks, but it will manifest as better protection against everyday nonsense like large spam runs.

You’ll be hearing more about a concrete example of the effect of some of these improvements in a near-future blog post.

While the outcome is successful, the implementation of the move was… not a great success. While nothing specific went horribly wrong, a number of little factors conspired to make the downtime last longer than expected.

First, we estimated the time required to disconnect, transport, and install our equipment very accurately. However, we (I) drastically underestimated the amount of time required to cable the equipment at the equipment at the new location. We use a partial mesh cluster topology, so it is not a simple matter of plugging each server into the closest switch; servers have, on average, 4-6 connections each, and some of those connections required quite a bit of cable routing. As a result, cabling took several hours more than expected and became the single most time-consuming task.

Second, our “placeholder pages” that were intended to be up during the move did work sometimes, but they were dependent on equipment at the new location that wound up needing to be reconfigured once the equipment from the old location arrived. We did not spot the unresolved dependency that caused this during the planning stages.

Third, a couple of our new servers, which had been very thoroughly tested prior to the move, completely flipped out on us as soon as they were brought up in the live configuration. They were quickly stabilized but their timing absolutely sucked.

After the move was completed and things started working again, we encountered a number of additional problems, most notably errors from our network edge proxies: Connection Refused, Host not responding, Host is Down, and Unknown Site. Some of these were related to the new configuration, but several were related to previously unexercised (or undetected) bugs in our clustering software that have now been fixed.

Unfortunately, it is not really useful to say “We have learned (something) from this experience about what to do next time.” That is because, having attempted what we did, only a complete idiot would do it again.

Should we ever need to move facilities in the future, no matter how long it takes or how much it costs, we will just build out the new facility in its entirety, move all the services between the two live facilities, and then burn down the old one for the insurance money.*

We have also renumbered into new IP addresses. This is temporary, and we will renumber again once our network brings some additional carriers online, with the primary holdup being bureaucracy. (The only reason the Internet hasn’t run out of IP addresses already is that the regional Internet registries make the application process so very tedious and time-consuming that it discourages all but the most determined applicants, like us.)

At this point, although we’re still in the period where most people automatically assume most problems are move-related even though most aren’t, we do still have a few remaining issues we are looking into:

• We have scattered reports that DNS performance may not be satisfactory. Although there’s nothing specific to back that up, we are developing a way to instrument our DNS performance to measure speed and reliability, rather than blindly making changes we think might help and hoping for the best. In the mean time, if you have any problems like this please contact us, especially if you can document them.
• Although it was not directly move-related, some people have encountered a unintended change in the default behavior of our network when Apache issues an internal redirect, causing their sites to fall back on the example.nfshost.com alias if, for example, someone enters a directory name without the trailing slash. This change is actually the result of a bug fix and represents the correct behavior, but we understand that people find it annoying. There is an .htaccess workaround for enforcing canonical site URLs in our Member Wiki, but we are working on allowing you to specify the canonical name for your site from our member interface, which will then be used in all such redirects.
• The block of IP addresses that we are temporarily using is on a blacklist maintained by a secret cabal of British P2P users that apparently like to trade illegal files and don’t want to get caught. They originally had the range blacklisted “a lot” because of former users of the same IP addresses, but switched it to “just a little bit” (rather than removing us) because they were apparently worried that NearlyFreeSpeech.NET was really just a front for the RIAA, but subsequently blacklisted us “a lot” again for “not showing any gratitude” about being incorrectly blacklisted only a little bit. So we’re not just spies, we’re ungrateful spies. Seriously, I couldn’t make this stuff up if I tried. Fortunately, this has no effect at all unless:
  1. you are a NearlyFreeSpeech.NET member,
  2. you run a P2P blocking application called “Peer Guardian,”
  3. you edit your site with FTP.

  If all those are the case, when you access your site with FTP you will receive a warning that you can bypass by clicking on it. We have specific instructions available if you need them, but I think the handful of affected people have already contacted us.

  This has no effect on web access to your site, and will most likely not be an issue at all after we renumber our IP’s again. If you have any concerns about it, please feel free to contact us.

• The blog-to-email service we were using appears to have imploded. We’ve selected a replacement, and we’ll be setting that up shortly, so email notification of NearlyFreeSpeech.NET news & announcements will be possible soon.
• One thing I noticed during the downtime was how liberating it was to have an open issue and the offsite status page that let us give out real-time updates. I want to find some way for us to do that on an ongoing basis. Something less “significant” than a blog post; almost like an internal members-only Twitter that we could update multiple times a day so you can find out what’s going on right now and what we’ve been working on lately to make your service better. Provided we give you the tools to find what you want, we really can’t give you too much information.

Beyond those things, everything seems to be back to normal. We’ve still got a mountain of cleanup work, but give us a couple of weeks and we should be able to turn our focus back to the sort of new feature development that really differentiates NearlyFreeSpeech.NET from the mob of cookie cutter web hosts.

I want to take this opportunity to thank our members for all of their support, patience, and understanding during and after our move. We are so grateful for all of you. With that said, I don’t want to trivialize the downtime we experienced during the move (or the issues that led up to it). We are very sorry to everyone about the downtime caused by our move, and for the issues of the past couple of months that led up to such drastic action.

Naturally we did receive a few fairly snarky messages about the move and downtime, but that’s human nature, and frankly we got fewer than we expected. I’d like to offer a special apology to those few people for putting them in a position where they felt like that was their best recourse.

To close out the subject of the move, I would love to be able to state firmly, right here and now, “We will never do that again!” but the reality is that never is a really long time.

What I can say is that as a result of this move, we’ve gone from having almost no control over our network to having almost complete control and we’re chiseling away at the “almost” even now. We’re also now able to start building our own network, which will mean the ability to spread out to multiple locations. At that point, moving no longer affects everything we own. It just becomes a turn up in one location followed by a turn down in another. All the while providing the high quality of service you expect from NearlyFreeSpeech.NET.

Thanks again!

*Note: That was a joke. Kids! When it comes to arson, environmentally damaging disposal of old computer equipment, and insurance fraud: just say “No!”

(Just a reminder: our blog is not a venue for member support and we can’t discuss issues specific to your service with you in public due to our Privacy Policy. If you’re having a problem with your service, whether you think it’s move-related or not, please submit a secure support request so we can help you out.)