Scheduled Maintenance: IP Renumbering In Progress

We are in the process of renumbering out of our current IP addresses and into a larger block of addresses. The effect of this change will be better routing performance and more room for us to grow and expand our service.

In order to minimize disruption, we plan to bring up the new IP addresses in parallel with the old ones and then, once everything is confirmed working, renumber all of our public-facing services. As part of that process, we will automatically renumber all relevant DNS records hosted on our service. If you use third party DNS, you will need to renumber manually in the near future. However, we will provide a transitional period for you to do so. During that period, we will automatically detect people using the old addresses and notify the site operators accordingly. Once the transitional period is complete, we will phase out the old addresses.

We are pursuing this renumbering very aggressively. Partly this is because, although we waited a year to get these IP addresses, we have a very limited window in which to renumber out of the old ones. Such is the nature of bureaucracy. But we have accelerated even more because we have discovered routing issues with an upstream provider related to the IP block we are in, which we believe was responsible for the disruptions some people experienced last week. Unfortunately, we got the IP addresses we currently have from them, so we are not in a position to tell them what to do with them. The new IPs are a direct allocation from ARIN (the American Registry of Internet Numbers) to our network operator, and thus do not have competing interests.

This also means that, while we may add addresses (as conservatively as possible), there should be no reason we will ever need to renumber out of existing ones for the foreseeable future once this migration is complete.

Unfortunately, this change is not without downsides. As stated above, we are attempting to minimize disruption. However, “minimized disruption” is not the same as “no disruption.” This change requires us to make major configuration changes to each and every server, switch, router, and firewall on our network. When it comes to routing, even the tiniest glitch or hiccup can throw everything into disarray for several minutes or more. (Something of that nature happened earlier this morning, for which I apologize. I likewise acknowledge that the blog post about the scheduled maintenance is supposed to happen before the scheduled maintenance. My bad.) Since we are only human, and we are very much working without a net, we are announcing scheduled maintenance between 2am and 6am EDT (6am to 10am UTC) for the next seven days. This does not mean that our service will be down at those times, or even a fraction of them. It simply means we will be performing configuration changes during those times, leading to a significantly increased risk of network downtime, and we want to make you aware of this. These times were chosen by statistical measurement of our network usage; usage during this period is much lower than at any other equivalent period.

As a nerdy little side note, we now have native IPv6 available at the edge of our network and, as part of this change, we will be configuring our core network paths to support it. This doesn’t mean we’re about to offer IPv6 hosting, but it does mean we won’t have to make those changes later on.

And for those that want to know, NearlyFreeSpeech.NET’s new IP block is 208.94.116.0/23. Get used to it, you’ll be seeing it in traceroutes soon, and it’ll be around for a good long while.

This is only one of the projects currently underway to improve the core infrastructure of our service, with an emphasis on making it faster and more reliable for everyone (you may have noticed that new feature development is at a standstill while we do that). While we’re working on this project, we apologize for any inconvenience or downtime that results, we thank you for your patience, and we promise to do our best to make the migration as smooth and trouble-free as possible. We have one goal, and that’s to make our web hosting service amazingly good. This will help.

We will try to update this blog post as we go, so we can keep you posted on how far along we are and how it’s going. In the mean time, to preserve the signal-to-noise ratio, please keep comments on-topic and remember that due to all that’s going on, we’re super busy right now and may be somewhat slow to approve and respond to them.

Update 2008-09-14: We’re making good progress. Most of our firewall/edge network has been converted, and a lot of small subnets have been retired. We are on track to finish IPv4 renumbering on schedule. We have also, just as an oh-so-beta proof of concept, put up IPv6 versions of our two main sites: www.ipv6.nearlyfreespeech.net and members.ipv6.nearlyfreespeech.net. (You’ll get certificate warnings if you’re one of the lucky few who can access these at all.)

Update 2008-09-18: We continue to make progress. New sites started being created in the new IP range this morning, and DNS and SMTP servers have also been cut over. We will be migrating existing sites to the new range over the next day or two. This will not cause downtime, as both the old IPs and the new ones work for all sites during the transition. We are now running about two days behind because of the DDOS attack experienced earlier this week; one maintenance window was lost to dealing with the attack, and a second one due to recovering from dealing with the attack.

Update 2008-02-22: The first phase of this project did complete successfully. The second phase of the project will require us to detect and contact people who have hardcoded the old IP addresses. The second phase has no operational impact and will not require maintenance windows. We will add a new blog post or an update when the second phase is complete and we are ready to begin final turn-down of the old address ranges.

6 Comments

RSS feed for comments on this post.

1. Thanks for keeping us updated. Keep up the good work!

   Comment by nfsn_user — September 12, 2008 #

2. Hi man, just to let you know I’m _very very_ happy with your service and I always recommend it
   =D

   Comment by g — September 13, 2008 #

3. I’d just want to point out that it’s currently impossible to log in over IPv6 because the cookie is set for the “members.nearlyfreespeech.net” domain:

   Set-Cookie: PHPSESSID=XXXXXXXXXXXXXXXXXXXXX; expires=Tue, 16 Sep 2008 20:42:23 GMT; path=/; domain=.members.nearlyfreespeech.net; secure; HttpOnly

   I’m pretty excited about this feature; as far as I can tell NFS would be one of the first serious web hosts to support IPv6! 🙂

   Comment by intgr — September 16, 2008 #

4. intgr, that’s a really good point. We were afraid to bring up IPv6 on the members.nearlyfreespeech.net name, because it would cause lousy performance for people on seedy tunnel brokers who pref IPv6 over IPv4 over the same name, which I think is most people at this stage of IPv6 deployment. (I think the same reasoning is behind ipv6.google.com.) I personally had quite a bit of trouble with exactly that at home using the Airport Extreme default IPv6 setup, but then I switched to the Hurricane Electric tunnel broker. Now my IPv6 at home flies and is totally usable. Highly recommended.

   I would like it if assigning an IPv6 address on a site-by-site or alias-by-alias basis was an option by the end of the year, but I’m certainly not going to guarantee that at this early stage.

   Also, just to let people know… As warned, I did have to delete a couple of comments that were about unrelated issues, like the recent DDOS attack. If you have questions about or problems with our service, please feel free to post in our member forums or submit a secure support request. We want to hear about problems, we’re always interested in feedback, and we’re not afraid of criticism as long as it’s venue-appropriate.

   -jdw

   Comment by jdw — September 18, 2008 #

5. Hopefully this is on topic…

   If we can have a IPV6 address on a site by site basis, would this mean that we would be able to have our very own SSL certs?

   It might someday mean that. -jdw

   Comment by Michael Lockyear — September 23, 2008 #

6. You could make it work by simply moving from “members.ipv6.nearlyfreespeech.net” to “ipv6.members.nearlyfreespeech.net”. 🙂

   Good idea. We’ll probably do that soon. -jdw

   Comment by intgr — September 24, 2008 #

Sorry, the comment form is closed at this time.