The older one is called Legacy Forwarding. It is a service we provide using our own servers, and adheres to our draconian email acceptance policy.

The newer one is simply called Email Forwarding. It is a service we provide in conjunction with a partnership with a third-party company. It is supposed to work a little differently; we call it a “hybrid” forwarding system because it combines certain aspects of a mailbox with certain aspects of email forwarding. Their servers are willing to accept virtually any email, but they then perform content-filtering checks and are supposed to divert suspect messages into a special “spam quarantine” accessible via our member interface.

I say “supposed to” because at some point in the recent past, without consulting with or notifying us, the third-party company unilaterally changed their system to discard all spam-suspect messages instead of placing them into the quarantine folder. At first we thought this was isolated to a few domains, but we have since discovered that the change was global. This means that for all of our members who have “Email Forwarding,” any messages you receive that are believed by their system to be spam, including false positives, are currently being silently discarded. This is absolutely not acceptable to us, but our efforts to redress this with them have not been successful.

Ultimately, our email forwarding members are paying us for email forwarding services, not a third party company, so it is our responsibility to make sure our members get the service they are paying for. To that end, we are taking the following steps:

• We will stop billing for all email forwarding domains until we feel things are working the way we intend.
• We are deploying a new hybrid email forwarding system, described below, which will be applied to all members’ email forwarding domains.
• Because we need to make changes, we will be temporarily disabling the email forwarding portions of our UI. We are not disabling email forwarding, just the UI that controls it.

The new email forwarding system will apply all of our existing email acceptance policy criteria, but it will do so a little differently. As with our legacy forwarding system, messages that meet all of our criteria will sail through the system with no disruption or delay, as will messages sent by domains with proper SPF configurations, even if they have other problems. Also as with the legacy system, messages from blacklisted servers will be immediately rejected.

For messages sent by servers with improper or no SPF and one or more other configuration problems, instead of bouncing them as the legacy forwarding system does, we will be trying a technique called greylisting, which tells the server sending the message to try again in a few hours. If the sending server does that, the message will be accepted.

We will also be adding a spam/virus checking component that will reject email containing viruses and quarantine egregious spam that somehow passes all of the above filters. (The current email forwarding scheme quarantines both spam and viruses.) Also new will be a feature that places post-forwarding bounce messages into the same quarantine, so if there are problems with delivering forwarded messages to you, you will be better able to detect and resolve them.

We feel that this system will make the best trade-offs between meeting our members’ need for reliable forwarding, keeping our forwarding mail servers off of global blacklists, and making life absolutely as miserable and unrewarding as we possibly can for spammers. It will also let us have one forwarding service for all members, and document it appropriately.

All of our email forwarding uses MX records in the nearlyfreespeech.net domain, so we retain full control over how they are handled. The above system is still in final development, but we believe it can be completed and deployed without any major disruption to anyone’s forwarded email. Due to the way Internet email works, even if we do wind up needing to take forwarding offline for a few hours in the middle of the night, messages will simply queue and redeliver when it comes back up.

We need to disable the member UI to work on it because we need the overall configuration to hold still long enough for us to update the UI code and move everyone over to the new system. It is our goal to have the entire transition completed by Saturday. Since we won’t be billing for email forwarding until this is resolved, we will be very highly motivated to make it happen as quickly and accurately as possible. If for any reason you do need to make an urgent email forwarding change that just can’t wait, or if you want to remove it altogether, just drop us a Secure Support Request and we’ll do our best to take care of you.

Since this is still an evolving situation, the above reflects what we want to do, and what we reasonably believe we can do. If and when circumstances change, we will update you accordingly. These changes will not affect the cost of email forwarding, and while they may ultimately put us in a more flexible position with respect to offering email hosting in the future, that is not a goal of this project and nothing is changing on that front at this time.

I apologize for the short notice and rushed character of this change. We spent a lot of effort (probably too much) trying to come to a less dramatic solution, and we were entirely unsuccessful.

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 - 45 minutes.

You can tell if you are likely to be affected if your MySQL @@hostname variable is “m2″ or “m3.” However, while we don’t plan to move any MySQL processes between now and then, we do reserve the right to move them at any time for any reason if appropriate to maintain network stability and performance.

This upgrade will increase total RAM available to member MySQL processes by 60%, with a corresponding increase in CPU power. After the upgrade is complete, we will be able to finish the migration to FreeBSD 7 and the combined result will be better MySQL performance for all.

At the same time, we will be adding RAM and CPU cores to the hosting cluster, but that should not cause downtime since we can rotate the affected servers out of production prior to the upgrade and redistribute the load; there is plenty of spare capacity at that particular time.

As always, we appreciate your patience as we update and upgrade our network to make your hosting better, faster, and more reliable.

We are pursuing this renumbering very aggressively. Partly this is because, although we waited a year to get these IP addresses, we have a very limited window in which to renumber out of the old ones. Such is the nature of bureaucracy. But we have accelerated even more because we have discovered routing issues with an upstream provider related to the IP block we are in, which we believe was responsible for the disruptions some people experienced last week. Unfortunately, we got the IP addresses we currently have from them, so we are not in a position to tell them what to do with them. The new IPs are a direct allocation from ARIN (the American Registry of Internet Numbers) to our network operator, and thus do not have competing interests.

This also means that, while we may add addresses (as conservatively as possible), there should be no reason we will ever need to renumber out of existing ones for the foreseeable future once this migration is complete.

Unfortunately, this change is not without downsides. As stated above, we are attempting to minimize disruption. However, “minimized disruption” is not the same as “no disruption.” This change requires us to make major configuration changes to each and every server, switch, router, and firewall on our network. When it comes to routing, even the tiniest glitch or hiccup can throw everything into disarray for several minutes or more. (Something of that nature happened earlier this morning, for which I apologize. I likewise acknowledge that the blog post about the scheduled maintenance is supposed to happen before the scheduled maintenance. My bad.) Since we are only human, and we are very much working without a net, we are announcing scheduled maintenance between 2am and 6am EDT (6am to 10am UTC) for the next seven days. This does not mean that our service will be down at those times, or even a fraction of them. It simply means we will be performing configuration changes during those times, leading to a significantly increased risk of network downtime, and we want to make you aware of this. These times were chosen by statistical measurement of our network usage; usage during this period is much lower than at any other equivalent period.

As a nerdy little side note, we now have native IPv6 available at the edge of our network and, as part of this change, we will be configuring our core network paths to support it. This doesn’t mean we’re about to offer IPv6 hosting, but it does mean we won’t have to make those changes later on.

And for those that want to know, NearlyFreeSpeech.NET’s new IP block is 208.94.116.0/23. Get used to it, you’ll be seeing it in traceroutes soon, and it’ll be around for a good long while.

This is only one of the projects currently underway to improve the core infrastructure of our service, with an emphasis on making it faster and more reliable for everyone (you may have noticed that new feature development is at a standstill while we do that). While we’re working on this project, we apologize for any inconvenience or downtime that results, we thank you for your patience, and we promise to do our best to make the migration as smooth and trouble-free as possible. We have one goal, and that’s to make our web hosting service amazingly good. This will help.

We will try to update this blog post as we go, so we can keep you posted on how far along we are and how it’s going. In the mean time, to preserve the signal-to-noise ratio, please keep comments on-topic and remember that due to all that’s going on, we’re super busy right now and may be somewhat slow to approve and respond to them.

Update 2008-09-14: We’re making good progress. Most of our firewall/edge network has been converted, and a lot of small subnets have been retired. We are on track to finish IPv4 renumbering on schedule. We have also, just as an oh-so-beta proof of concept, put up IPv6 versions of our two main sites: www.ipv6.nearlyfreespeech.net and members.ipv6.nearlyfreespeech.net. (You’ll get certificate warnings if you’re one of the lucky few who can access these at all.)

Update 2008-09-18: We continue to make progress. New sites started being created in the new IP range this morning, and DNS and SMTP servers have also been cut over. We will be migrating existing sites to the new range over the next day or two. This will not cause downtime, as both the old IPs and the new ones work for all sites during the transition. We are now running about two days behind because of the DDOS attack experienced earlier this week; one maintenance window was lost to dealing with the attack, and a second one due to recovering from dealing with the attack.

Update 2008-02-22: The first phase of this project did complete successfully. The second phase of the project will require us to detect and contact people who have hardcoded the old IP addresses. The second phase has no operational impact and will not require maintenance windows. We will add a new blog post or an update when the second phase is complete and we are ready to begin final turn-down of the old address ranges.

The correct/current RSA host key:

fc:89:a1:64:74:70:a4:82:58:c1:73:4e:72:59:63:56

This naturally changes the host key fingerprint, and is likely to cause warnings for people who have previously connected. This doesn’t indicate a security problem, but you should compare the new key against its fingerprint.

You can verify this on our secure site in the member FAQ.

We have not changed the DSA key, as limitations in the DSA algorithm render longer keys pointless. We discourage but will continue to allow the use of DSA keys.

We apologize to anyone alarmed by the ALL CAPITAL LETTERS!! warnings that ssh applications tend to produce when a host key changes. The new key length should guarantee that we don’t have to change it again for a good long time.

In other key-related news, we scanned all member ssh keys for the Debian weak keys and notified everyone who appeared to be affected. Since there are exploits in the wild, we will shortly be removing any remaining weak keys, and we will not accept any new weak keys for access to member sites.

This outage will not affect member services, only our own sites (www, blog, and members), FTP, and ssh will be affected during these brief downtimes.

We apologize for the short notice and hope this is not too disruptive to anyone’s site maintenance plans.

The outcome of the move is a success. All of our core hardware is in one place, it is all working, and we are in a better position to grow and expand than we have ever been.

Our network monitoring has been vastly improved, with more improvements on the way, which will allow us to be a lot more proactive about keeping your sites healthy in the future, rather than waiting for you to complain when there’s a problem.

Our network infrastructure has been greatly improved. We have incredibly more bandwidth available on our internal network than ever before, so much that we are still tuning the most effective way to utilize it all to serve your sites as quickly as possible. Our cluster technology has always had the ability to failover your site from one web server to another if there’s a problem, but we now have similar (but faster) hot failover technology for our routers, firewalls, and edge proxies.

Our network defenses have also received a sharp boost. We have more and better firewall capability, better inbound packet scrubbing, better backup access to our equipment during attacks, and the ability to selectively blackhole attackers based on complex criteria before they ever reach NearlyFreeSpeech.NET. Not only will this pay off big time in the event of future DDOS attacks, but it will manifest as better protection against everyday nonsense like large spam runs.

You’ll be hearing more about a concrete example of the effect of some of these improvements in a near-future blog post.

While the outcome is successful, the implementation of the move was… not a great success. While nothing specific went horribly wrong, a number of little factors conspired to make the downtime last longer than expected.

First, we estimated the time required to disconnect, transport, and install our equipment very accurately. However, we (I) drastically underestimated the amount of time required to cable the equipment at the equipment at the new location. We use a partial mesh cluster topology, so it is not a simple matter of plugging each server into the closest switch; servers have, on average, 4-6 connections each, and some of those connections required quite a bit of cable routing. As a result, cabling took several hours more than expected and became the single most time-consuming task.

Second, our “placeholder pages” that were intended to be up during the move did work sometimes, but they were dependent on equipment at the new location that wound up needing to be reconfigured once the equipment from the old location arrived. We did not spot the unresolved dependency that caused this during the planning stages.

Third, a couple of our new servers, which had been very thoroughly tested prior to the move, completely flipped out on us as soon as they were brought up in the live configuration. They were quickly stabilized but their timing absolutely sucked.

After the move was completed and things started working again, we encountered a number of additional problems, most notably errors from our network edge proxies: Connection Refused, Host not responding, Host is Down, and Unknown Site. Some of these were related to the new configuration, but several were related to previously unexercised (or undetected) bugs in our clustering software that have now been fixed.

Unfortunately, it is not really useful to say “We have learned (something) from this experience about what to do next time.” That is because, having attempted what we did, only a complete idiot would do it again.

Should we ever need to move facilities in the future, no matter how long it takes or how much it costs, we will just build out the new facility in its entirety, move all the services between the two live facilities, and then burn down the old one for the insurance money.*

We have also renumbered into new IP addresses. This is temporary, and we will renumber again once our network brings some additional carriers online, with the primary holdup being bureaucracy. (The only reason the Internet hasn’t run out of IP addresses already is that the regional Internet registries make the application process so very tedious and time-consuming that it discourages all but the most determined applicants, like us.)

At this point, although we’re still in the period where most people automatically assume most problems are move-related even though most aren’t, we do still have a few remaining issues we are looking into:

• We have scattered reports that DNS performance may not be satisfactory. Although there’s nothing specific to back that up, we are developing a way to instrument our DNS performance to measure speed and reliability, rather than blindly making changes we think might help and hoping for the best. In the mean time, if you have any problems like this please contact us, especially if you can document them.
• Although it was not directly move-related, some people have encountered a unintended change in the default behavior of our network when Apache issues an internal redirect, causing their sites to fall back on the example.nfshost.com alias if, for example, someone enters a directory name without the trailing slash. This change is actually the result of a bug fix and represents the correct behavior, but we understand that people find it annoying. There is an .htaccess workaround for enforcing canonical site URLs in our Member Wiki, but we are working on allowing you to specify the canonical name for your site from our member interface, which will then be used in all such redirects.
• The block of IP addresses that we are temporarily using is on a blacklist maintained by a secret cabal of British P2P users that apparently like to trade illegal files and don’t want to get caught. They originally had the range blacklisted “a lot” because of former users of the same IP addresses, but switched it to “just a little bit” (rather than removing us) because they were apparently worried that NearlyFreeSpeech.NET was really just a front for the RIAA, but subsequently blacklisted us “a lot” again for “not showing any gratitude” about being incorrectly blacklisted only a little bit. So we’re not just spies, we’re ungrateful spies. Seriously, I couldn’t make this stuff up if I tried. Fortunately, this has no effect at all unless:
  1. you are a NearlyFreeSpeech.NET member,
  2. you run a P2P blocking application called “Peer Guardian,”
  3. you edit your site with FTP.

  If all those are the case, when you access your site with FTP you will receive a warning that you can bypass by clicking on it. We have specific instructions available if you need them, but I think the handful of affected people have already contacted us.

  This has no effect on web access to your site, and will most likely not be an issue at all after we renumber our IP’s again. If you have any concerns about it, please feel free to contact us.

• The blog-to-email service we were using appears to have imploded. We’ve selected a replacement, and we’ll be setting that up shortly, so email notification of NearlyFreeSpeech.NET news & announcements will be possible soon.
• One thing I noticed during the downtime was how liberating it was to have an open issue and the offsite status page that let us give out real-time updates. I want to find some way for us to do that on an ongoing basis. Something less “significant” than a blog post; almost like an internal members-only Twitter that we could update multiple times a day so you can find out what’s going on right now and what we’ve been working on lately to make your service better. Provided we give you the tools to find what you want, we really can’t give you too much information.

Beyond those things, everything seems to be back to normal. We’ve still got a mountain of cleanup work, but give us a couple of weeks and we should be able to turn our focus back to the sort of new feature development that really differentiates NearlyFreeSpeech.NET from the mob of cookie cutter web hosts.

I want to take this opportunity to thank our members for all of their support, patience, and understanding during and after our move. We are so grateful for all of you. With that said, I don’t want to trivialize the downtime we experienced during the move (or the issues that led up to it). We are very sorry to everyone about the downtime caused by our move, and for the issues of the past couple of months that led up to such drastic action.

Naturally we did receive a few fairly snarky messages about the move and downtime, but that’s human nature, and frankly we got fewer than we expected. I’d like to offer a special apology to those few people for putting them in a position where they felt like that was their best recourse.

To close out the subject of the move, I would love to be able to state firmly, right here and now, “We will never do that again!” but the reality is that never is a really long time.

What I can say is that as a result of this move, we’ve gone from having almost no control over our network to having almost complete control and we’re chiseling away at the “almost” even now. We’re also now able to start building our own network, which will mean the ability to spread out to multiple locations. At that point, moving no longer affects everything we own. It just becomes a turn up in one location followed by a turn down in another. All the while providing the high quality of service you expect from NearlyFreeSpeech.NET.

Thanks again!

*Note: That was a joke. Kids! When it comes to arson, environmentally damaging disposal of old computer equipment, and insurance fraud: just say “No!”

(Just a reminder: our blog is not a venue for member support and we can’t discuss issues specific to your service with you in public due to our Privacy Policy. If you’re having a problem with your service, whether you think it’s move-related or not, please submit a secure support request so we can help you out.)

We have been collocated with Limelight Networks since our inception in 2002. While Limelight is an excellent network, and their newest datacenters are works of art, we have run into a number of issues that have conspired to push us in a different direction:

1) Limelight recently completed a successful IPO based on their highly successful and incredibly awesome CDN service, and as such they are now almost exclusively focused on that service. In fact, they no longer offer collocation as a product for companies like us. We do not want to find ourself in a position where we cannot expand because the facilities are no longer available.

2) At Limelight, they run the datacenter and they are our only Internet carrier. While that’s very convenient for us, it’s putting all our eggs in one basket, which is not good for redundancy. We need (and have obtained) a carrier-neutral facility that will enable us to connect directly to as many carriers as we like, improving our overall reliability and also empowering us to do things like cherry-pick carriers that will improve overseas connectivity.

3) With multiple carriers, we will be able to request our own IP assignments directly from ARIN, which will remove our exposure to certain types of problems caused by routing issues on other networks.

4) We will be behind dedicated hot-failover routers and firewalls under our control, which will improve our reliability and enable us to react much more quickly to DDOS attacks, eliminating an entire class of such attacks through proactive filtering, and mitigating the rest much more quickly because we will be able to do it ourselves without depending on any third party.

5) We will be retiring some legacy equipment that is constraining our network performance but is too critical and too deeply wired in to address any other way.

6) We will be deploying an infrastructure capable of scaling to at least 10Gbps.

As you can see, this move represents us stepping up and taking control over a lot of areas where we have previously been dependent on others.

Q. Why are you moving your datacenter now?

In July and August, we experienced an unacceptable number of reliability issues. While the causes of these issues were almost exclusively not “our fault” and factors beyond our control (DDOS attacks, routing problems at relatively distant carriers, etc.), at some point it becomes our fault for not taking control of those factors, which is what this move is doing for us.

Q. Why will the downtime be so long?

While we can’t say for certain how long it will take, here are some of the reasons we estimate it will take so long:

1) It’s a lot of equipment.

2) We are actually consolidating two datacenters into one.

3) Virtually all of our production infrastructure equipment (routers & switches) will have to be completely reconfigured on the fly.

4) We have to make two trips to ensure that we don’t have every copy of our members’ data in the same truck at the same time.

5) It’s much better to set a realistic, conservative timeline than try to rush things and wind up making a critical mistake.

Q. Why didn’t you find a way to do this without downtime?

Anticipating the need to expand, we spent several months trying to plan a way to do a move without affecting service. In fact, we have already moved roughly a third of our equipment with relatively little impact. (Example: our services are supposed to all be N+2 redundant, but right now all the +2’s are at the new building, leaving us not a lot of margin for error.)

In the end, a complete zero-downtime move proved impossible. At some point, your site’s data is on a disk array, and that array can only be in one place at a time. The same for your MySQL process. We tried MySQL replication between facilities on our own master database, and we were disappointed with the results (it blew up at least once a day). Similarly, the amount of data associated with many sites is just too much and changes too fast to keep it synchronized between two locations. We have experimented with all sorts of VPN-type solutions, hoping to facilitate “sneaking” equipment from one location to another, and they just aren’t reliable enough to depend on for the length of time it would take to migrate that much data.

Still, such a move is theoretically possible with enough planning and preparation. However, given the issues we’ve seen in the past few months, and our current vulnerabilities to those same sorts of issues reoccuring, it becomes a risk assessment question: will the unknown, unplanned downtime caused by external forces during the time it takes us to devise a zero-downtime move be worse for our members than a single, known, scheduled downtime? We just can’t take the chance that the answer is yes.

Q. What are you doing to minimize the effects of the downtime?

We have been migrating services and equipment to the new facility as much as we can without causing disruption, and will continue to do so right up until the last minute. Everything we can move before is something we don’t have to move during.

Also, rather than leave your web site completely unresponsive, we are going to have a “maintenance page” served for all requests indicating to people that your site is not gone or shut down, merely being serviced for a few hours, and asking them to check back later.

Q. Will you be changing your IP addresses during the move?

Yes. This move will entail a complete IP renumbering on our part. Actually, after it is done, we will have to renumber again a month or so later to satisfy some obscure bureaucratic requirments.

We will be leaving a handful of equipment at the old location to preserve the old IP addresses and catch people with hardcoded third-party DNS or domain registrations. Once the second renumber is complete, we will start contacting people using the old addresses to let them know what changes they need to make.

For those people who have their domain registrations and DNS with us, all your IP information will be updated automatically both times with no action required on your part.

Q. How can I stay up-to-date on the downtime?

As soon as the downtime arrives, we will begin posting updates on our offsite status page with our trademark transparency. We will keep it updated throughout the move, particularly if anything unforeseen happens.

Q. How are you making sure everyone is aware of this?

We know not everyone follows our blog, and that the third-party blog-to-email service we were using recently imploded. For these reasons, and because of the scope of the move, we will be taking the unusual step of sending out a mass-email to all of our members making them aware of the planned downtime.

Q. Are you sorry you have to do this?

Yes, we are really, really sorry we have to do this, and we hope you will agree that the long-term benefits overwhelm the short-term pain of this downtime.

Ordinarily, we use System Problem Reports on the Support panel in our member interface to let you know about problems with the service. In most cases, this works well.

However, certain categories of outages, like upstream network failures, can render us unable to post such reports, or leave you unable to read them until after the problem is resolved. While such events should be rare, that’s the most important time for us to be letting you know what’s going on.

To that end, we’ve created an offsite network status page at http://status.nearlyfreespeech.net/. This site runs on one of our servers located in Dallas. That server doesn’t depend on our Phoenix hosting cluster at all, and will continue to operate even if everything else is offline.

The Offsite Network Status Page collects information from three sources:

1) We can post information directly to that site in the event of a serious problem.

2) It attempts to download a list of System Problem Reports every few minutes.

3) It attempts to mirror recent blog posts every few minutes.

If either of the latter two automatic updates fail, that might indicate a problem, so the site will display a message to that effect, even if we haven’t been able to post any information manually yet.

This is a tool that we hope we don’t find ourselves using. However, it definitely falls into the category of things that is better to have and not need than to need and not have. In that spirit, we hope that you will add status.nearlyfreespeech.net as the bookmark you hope you never use.

If this is the only post you see in a category, it just means the category hasn’t been assigned any real posts yet.