The older one is called Legacy Forwarding. It is a service we provide using our own servers, and adheres to our draconian email acceptance policy.

The newer one is simply called Email Forwarding. It is a service we provide in conjunction with a partnership with a third-party company. It is supposed to work a little differently; we call it a “hybrid” forwarding system because it combines certain aspects of a mailbox with certain aspects of email forwarding. Their servers are willing to accept virtually any email, but they then perform content-filtering checks and are supposed to divert suspect messages into a special “spam quarantine” accessible via our member interface.

I say “supposed to” because at some point in the recent past, without consulting with or notifying us, the third-party company unilaterally changed their system to discard all spam-suspect messages instead of placing them into the quarantine folder. At first we thought this was isolated to a few domains, but we have since discovered that the change was global. This means that for all of our members who have “Email Forwarding,” any messages you receive that are believed by their system to be spam, including false positives, are currently being silently discarded. This is absolutely not acceptable to us, but our efforts to redress this with them have not been successful.

Ultimately, our email forwarding members are paying us for email forwarding services, not a third party company, so it is our responsibility to make sure our members get the service they are paying for. To that end, we are taking the following steps:

• We will stop billing for all email forwarding domains until we feel things are working the way we intend.
• We are deploying a new hybrid email forwarding system, described below, which will be applied to all members’ email forwarding domains.
• Because we need to make changes, we will be temporarily disabling the email forwarding portions of our UI. We are not disabling email forwarding, just the UI that controls it.

The new email forwarding system will apply all of our existing email acceptance policy criteria, but it will do so a little differently. As with our legacy forwarding system, messages that meet all of our criteria will sail through the system with no disruption or delay, as will messages sent by domains with proper SPF configurations, even if they have other problems. Also as with the legacy system, messages from blacklisted servers will be immediately rejected.

For messages sent by servers with improper or no SPF and one or more other configuration problems, instead of bouncing them as the legacy forwarding system does, we will be trying a technique called greylisting, which tells the server sending the message to try again in a few hours. If the sending server does that, the message will be accepted.

We will also be adding a spam/virus checking component that will reject email containing viruses and quarantine egregious spam that somehow passes all of the above filters. (The current email forwarding scheme quarantines both spam and viruses.) Also new will be a feature that places post-forwarding bounce messages into the same quarantine, so if there are problems with delivering forwarded messages to you, you will be better able to detect and resolve them.

We feel that this system will make the best trade-offs between meeting our members’ need for reliable forwarding, keeping our forwarding mail servers off of global blacklists, and making life absolutely as miserable and unrewarding as we possibly can for spammers. It will also let us have one forwarding service for all members, and document it appropriately.

All of our email forwarding uses MX records in the nearlyfreespeech.net domain, so we retain full control over how they are handled. The above system is still in final development, but we believe it can be completed and deployed without any major disruption to anyone’s forwarded email. Due to the way Internet email works, even if we do wind up needing to take forwarding offline for a few hours in the middle of the night, messages will simply queue and redeliver when it comes back up.

We need to disable the member UI to work on it because we need the overall configuration to hold still long enough for us to update the UI code and move everyone over to the new system. It is our goal to have the entire transition completed by Saturday. Since we won’t be billing for email forwarding until this is resolved, we will be very highly motivated to make it happen as quickly and accurately as possible. If for any reason you do need to make an urgent email forwarding change that just can’t wait, or if you want to remove it altogether, just drop us a Secure Support Request and we’ll do our best to take care of you.

Since this is still an evolving situation, the above reflects what we want to do, and what we reasonably believe we can do. If and when circumstances change, we will update you accordingly. These changes will not affect the cost of email forwarding, and while they may ultimately put us in a more flexible position with respect to offering email hosting in the future, that is not a goal of this project and nothing is changing on that front at this time.

I apologize for the short notice and rushed character of this change. We spent a lot of effort (probably too much) trying to come to a less dramatic solution, and we were entirely unsuccessful.

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 - 45 minutes.

You can tell if you are likely to be affected if your MySQL @@hostname variable is “m2″ or “m3.” However, while we don’t plan to move any MySQL processes between now and then, we do reserve the right to move them at any time for any reason if appropriate to maintain network stability and performance.

This upgrade will increase total RAM available to member MySQL processes by 60%, with a corresponding increase in CPU power. After the upgrade is complete, we will be able to finish the migration to FreeBSD 7 and the combined result will be better MySQL performance for all.

At the same time, we will be adding RAM and CPU cores to the hosting cluster, but that should not cause downtime since we can rotate the affected servers out of production prior to the upgrade and redistribute the load; there is plenty of spare capacity at that particular time.

As always, we appreciate your patience as we update and upgrade our network to make your hosting better, faster, and more reliable.

Here’s the executive summary: FreeBSD 7 is, by and large, great for us and we will be aggressively deploying it throughout our network in the near future. ZFS is also great, but while it definitely has a future on our network someday, this is not that day.

FreeBSD 7 offers significant benefits for SMP scalability, which is pretty important to us since most MySQL processes and member sites run on eight-core servers. The performance benefits on MySQL are very real, particularly on CPU-intensive workloads (which tend to be common if your tables are indexed properly). The performance benefits for member sites are less pronounced for some pretty complicated reasons, but they’re still there and they’re still worth pursuing.

ZFS is a really neat technology. By and large, it works pretty well. It’s not a great match for a MySQL server, though, since it seems to want a lot of CPU and RAM at the same time MySQL wants lots of CPU and RAM, because it works hard when asked for data and MySQL asks for data when it’s working hard. Thus, we noticed a little bit of contention and we were concerned about what would happen in low memory conditions, such as if a bunch of MySQL processes happened to surge busily at once. ZFS seems best suited to sticking on super-tightly controlled file servers that have gigs of RAM free and don’t do anything else. Even so, we’re not 100% sold on its stability, so we’ll revisit it sometime in the future and do further testing before we start loading it up on mission-critical file servers.

All of our MySQL FreeBSD 7 ZFS beta volunteers have been migrated to production FreeBSD 7 (non-ZFS) MySQL servers. They’re just the vanguard, though; we will be aggressively upgrading almost all member MySQL processes over the next few days.

The actual process is a bit complicated, but here are the essentials: we’ve brought some new hardware online running the target environment (FreeBSD 7, MySQL 5.0.67). Call that “Server A.” We’ve moved all the MySQL processes from Server B (one of the older ones) onto server A. Then, we reinstall Server B. Then we move everyone on Server C to Server B and reinstall server C. And so on, down the line.

The upgrade requires no action on your part and entails MySQL downtime averaging less than five minutes; a significant percentage of our MySQL-using members will notice only one to two minutes of disruption, if they notice it at all. Even so, we’re limiting it to low-usage periods (certain times during weekends, late nights, and early mornings). Due to the number of processes being upgraded and our need to schedule upgrades based on specific server assignments, we will not be able to coordinate individual process upgrades.

We do still have some longtime members hanging around on MySQL 4.1 running on FreeBSD 4. Since MySQL 4.1 to MySQL 5.0 isn’t always a seamless migration, we’ll be contacting those affected to let them know their situation and try to coordinate their journey out of the stone age before we do anything there.

If you want to see what version of FreeBSD or MySQL is powering your MySQL process, you can use the SQL command “SHOW GLOBAL VARIABLES” and look at the “version” and “version_compile_os” variables, or access them in queries as “SELECT @@version” or “SELECT @@version_compile_os.” The former shows the MySQL server version (which is also displayed in phpMyAdmin and by the MySQL command line client), and the latter shows the OS version: “portbld-freebsd6.3″ for FreeBSD 6.3 and “portbld-freebsd7.0″ for FreeBSD 7.0.

We also have a significant RAM and CPU power upgrade planned for MySQL once this upgrade is complete. That will require a brief off-hours downtime for some MySQL processes, which we will schedule and announce at a future time.

Finally, though MySQL processes will never be able to flit around our network the way web requests do, we are taking baby steps in the direction of SAN-based MySQL storage in order to decrease the lock-tight correlation between MySQL processes and specific frontend hardware.

Effective September 30th, 2008, our prices for domain registrations, transfers, and renewals in our supported TLDs (com, net, org, biz, info & name) will increase from $7.99 to $8.59 per domain per year.

This gives you almost two weeks to renew existing domains at the old prices. Remember that domains can be renewed up to ten years in advance, so you can lock in present prices for an awfully long time if you choose.

I want to remind people of a little-known feature, the Renewal Monitor, which makes it easy to see what domains are up for renewal and to renew them. We also changed our system a while back so that it is no longer necessary to unlock domains before renewing them.

Update 2008-09-29: The original increase was to $8.89/year. Like many of our members, we felt this was excessive, and like our members, we weren’t hesitant to say so. I’m pleased to report that our view has apparently prevailed, because we have been informed by our registrar that some of the increase is being rolled back. The new price will be $8.59/year. A $0.60 increase is still more than the official $0.44 increase Verisign helped themselves to on the .COM TLD, or the $0.36 increase in .NET, but it’s in-line with the increase on .ORG, so I suspect we’ll all have to live with it. Maybe infinite gTLDs will bring some much-needed competition to the domain registration industry.

We are pursuing this renumbering very aggressively. Partly this is because, although we waited a year to get these IP addresses, we have a very limited window in which to renumber out of the old ones. Such is the nature of bureaucracy. But we have accelerated even more because we have discovered routing issues with an upstream provider related to the IP block we are in, which we believe was responsible for the disruptions some people experienced last week. Unfortunately, we got the IP addresses we currently have from them, so we are not in a position to tell them what to do with them. The new IPs are a direct allocation from ARIN (the American Registry of Internet Numbers) to our network operator, and thus do not have competing interests.

This also means that, while we may add addresses (as conservatively as possible), there should be no reason we will ever need to renumber out of existing ones for the foreseeable future once this migration is complete.

Unfortunately, this change is not without downsides. As stated above, we are attempting to minimize disruption. However, “minimized disruption” is not the same as “no disruption.” This change requires us to make major configuration changes to each and every server, switch, router, and firewall on our network. When it comes to routing, even the tiniest glitch or hiccup can throw everything into disarray for several minutes or more. (Something of that nature happened earlier this morning, for which I apologize. I likewise acknowledge that the blog post about the scheduled maintenance is supposed to happen before the scheduled maintenance. My bad.) Since we are only human, and we are very much working without a net, we are announcing scheduled maintenance between 2am and 6am EDT (6am to 10am UTC) for the next seven days. This does not mean that our service will be down at those times, or even a fraction of them. It simply means we will be performing configuration changes during those times, leading to a significantly increased risk of network downtime, and we want to make you aware of this. These times were chosen by statistical measurement of our network usage; usage during this period is much lower than at any other equivalent period.

As a nerdy little side note, we now have native IPv6 available at the edge of our network and, as part of this change, we will be configuring our core network paths to support it. This doesn’t mean we’re about to offer IPv6 hosting, but it does mean we won’t have to make those changes later on.

And for those that want to know, NearlyFreeSpeech.NET’s new IP block is 208.94.116.0/23. Get used to it, you’ll be seeing it in traceroutes soon, and it’ll be around for a good long while.

This is only one of the projects currently underway to improve the core infrastructure of our service, with an emphasis on making it faster and more reliable for everyone (you may have noticed that new feature development is at a standstill while we do that). While we’re working on this project, we apologize for any inconvenience or downtime that results, we thank you for your patience, and we promise to do our best to make the migration as smooth and trouble-free as possible. We have one goal, and that’s to make our web hosting service amazingly good. This will help.

We will try to update this blog post as we go, so we can keep you posted on how far along we are and how it’s going. In the mean time, to preserve the signal-to-noise ratio, please keep comments on-topic and remember that due to all that’s going on, we’re super busy right now and may be somewhat slow to approve and respond to them.

Update 2008-09-14: We’re making good progress. Most of our firewall/edge network has been converted, and a lot of small subnets have been retired. We are on track to finish IPv4 renumbering on schedule. We have also, just as an oh-so-beta proof of concept, put up IPv6 versions of our two main sites: www.ipv6.nearlyfreespeech.net and members.ipv6.nearlyfreespeech.net. (You’ll get certificate warnings if you’re one of the lucky few who can access these at all.)

Update 2008-09-18: We continue to make progress. New sites started being created in the new IP range this morning, and DNS and SMTP servers have also been cut over. We will be migrating existing sites to the new range over the next day or two. This will not cause downtime, as both the old IPs and the new ones work for all sites during the transition. We are now running about two days behind because of the DDOS attack experienced earlier this week; one maintenance window was lost to dealing with the attack, and a second one due to recovering from dealing with the attack.

Update 2008-02-22: The first phase of this project did complete successfully. The second phase of the project will require us to detect and contact people who have hardcoded the old IP addresses. The second phase has no operational impact and will not require maintenance windows. We will add a new blog post or an update when the second phase is complete and we are ready to begin final turn-down of the old address ranges.

The correct/current RSA host key:

fc:89:a1:64:74:70:a4:82:58:c1:73:4e:72:59:63:56

This naturally changes the host key fingerprint, and is likely to cause warnings for people who have previously connected. This doesn’t indicate a security problem, but you should compare the new key against its fingerprint.

You can verify this on our secure site in the member FAQ.

We have not changed the DSA key, as limitations in the DSA algorithm render longer keys pointless. We discourage but will continue to allow the use of DSA keys.

We apologize to anyone alarmed by the ALL CAPITAL LETTERS!! warnings that ssh applications tend to produce when a host key changes. The new key length should guarantee that we don’t have to change it again for a good long time.

In other key-related news, we scanned all member ssh keys for the Debian weak keys and notified everyone who appeared to be affected. Since there are exploits in the wild, we will shortly be removing any remaining weak keys, and we will not accept any new weak keys for access to member sites.

Hey, we have some heavy workloads…

However, as longtime members may know, when confronted with a choice between stability and performance, we tend to choose stability. In fact, that preference led us to skip FreeBSD 5 entirely while the FreeBSD team tirelessly worked out the changes needed to support modern SMP hardware. (I look back to the FreeBSD 4 days when we had to use single-core systems and I whimper.)

Another flagship technology in FreeBSD 7 is ZFS. The primary benefits of ZFS wouldn’t accrue directly to our members (although it might provide the underpinnings for some neat stuff), but rather it would make our lives a lot easier. The problem with that is that there’s some debate on whether or not ZFS is ready for prime-time usage. So far, it’s tested pretty well and we’ve done some fairly abusive things to it.

Even so, odds are that FreeBSD 7.0 will never see production in our web cluster; we will most likely give serious consideration to 7.1 or 7.2, depending on the results of our internal testing. That internal testing is (and has been) underway, and we’d like to offer the adventurous a chance to participate.

Putting FreeBSD 7.0 into our web cluster isn’t really a good option, because it would entail building FreeBSD 7 versions of a couple thousand software packages, CPAN modules, python eggs, and etc, and might cause problems for custom-built CGI binaries. Instead, we feel that heavy MySQL usage is most likely to show the benefits of improved SMP code in a controlled software environment.

Thus we have set up an experimental MySQL server running MySQL 5.0.51a on FreeBSD 7 with a ZFS filesystem. If you would like your MySQL process moved to this experimental server, please submit a secure support request indicating the name of your MySQL process. Migrating MySQL between servers is very fast, but it takes about 10 minutes for a moved process’s new IP address to propagate through our network.

The trade-offs should look like this:

• Your MySQL process will probably be a lot faster, especially if it’s busy.
• There might be an increased chance of MySQL downtime due to instability.

Naturally we won’t let stability issues get out of hand on that server. If ZFS presents a problem, we’ll blow it away and try FreeBSD 7 under UFS2, and if that’s also unstable, we’ll kill the test and try again at 7.1. We’d also move anyone back out of the test who encountered problems on request.

Give it some thought. If your MySQL process is not mission-critical and you’re interested in helping us plan the next generation of NearlyFreeSpeech.NET servers, please drop us a line.

We don’t want anything horrible to happen to our service, we don’t expect anything horrible to happen, and (for the paranoid among us) aren’t aware of any horrible about to happen. Prudence simply dictates that we acknowledge that disasters are possible and take reasonable precautions to ensure that, were our disaster ticket to get punched that we would be eventually able to recover. (We’re talking about large scale the-entire-datacenter-is-permanently-gone-or-unusable disasters here.)

For us, this means keeping heavily encrypted offsite copies of our key databases, all of our custom server source code, and a lot of configuration information. That’s all the stuff we would need to rebuild our service, member and account balance records from scratch. What it does not include is offsite copies of our members’ content. While we would love to be able to do that automatically, there’s so much of it that the expense would be considerable. We’ve chosen instead to allow people to choose for themselves whether they feel that level of additional protection (and cost) is justified. In a lot of cases, it probably won’t be, but it’s something we need to do for our data so we want you to have the option to do it for yours as well.

Therefore, we’ve entered into a relationship with highly-regarded backup provider rsync.net to offer an innovative (we think) kind of offsite backup service for hosted sites and MySQL processes.

Here’s how it works:

We set you up with a numbered “primary” user account on their system. This is a full-featured rsync.net account that you can use for whatever they normally allow. But our backup server gets a second “subaccount” that lives in a subdirectory of your rsync.net account. Our system then uses this subaccount to make automatic daily backups of sites or MySQL processes that you designate during our periods of lower bandwidth utilization. If you use the primary account for other purposes, you can set the permissions such that you can read the subaccount’s backup files, but that the subaccount can’t read yours.

Once you’re set up, we provide two different backup mechanisms. The recommended method uses a neat open-source tool called duplicity to make incremental backups. These backups are digitally signed by our backup server and we support (optional) encryption of the backup files using a GnuPG public key of your designation. You can retrieve the files we upload using your own account at any time without any dependency on us or our system. Since this is an incremental backup system, you can restore not only the most recent backup, but any successful backup from the preceding 14 days.

(If encrypted, the files will also be decryptable with our backup key, which is necessary for the “incremental” part to work. This is not an increased security risk, because if someone were hypothetically to gain access to our key, they would also have sufficient privileges to read your unencrypted files directly from our local servers. The goal of encryption in this case is only to ensure that a compromise restricted to rsync.net isn’t sufficient to compromsie your files.)

For those who prefer convenience and simplicity over maximum security and incremental backups, our system can also perform more traditional daily rsync backups instead. With this approach, a copy of your filesystem is simply mirrored to your backup account once per day.

It’s important to point out that to keep the number of “superuser” processes to a minimum, our backup process also runs with reduced privileges; it can only access files readable by the “me” or “web” groups. Files and directories that are not group accessible (those with 700 or 600 permissions) cannot be backed up. This typically includes the “private” directory unless you have changed the default permissions. We also exclude the site’s logs by default to save space, but this is optional.

rsync.net, as you might expect, has a flexible terms of service that sound a lot like something we might have written. Their respect for privacy is also evident, but it pretty much doesn’t matter because no one but us will know who a given numbered backup account belongs to, or what specific sites and MySQL process backups an account contains. This, combined with the use of one set of credentials to drop off the backups and another, independent set to pick them up is the reason we’re quasi-seriously calling this a “dead drop” service.

For the backup accounts we provide, Rsync.net storage billing is calculated daily in kilobytes and billed directly to the NearlyFreeSpeech.NET personal bandwidth account of your choice using our usual brand of pay-as-you-go calculation. The pricing for this service is 225,000 kilobyte-days per $0.01, which works out to be just under $1.40 per rsync.net-stored gigabyte per (30-day) month. Naturally there are no minimum commitments or filesystem quotas to deal with, and this price includes the cost for the daily backup transfers between their service and ours.

To sign up for this service, let us know what site(s) and/or MySQL process(es) you want set up with it via a secure support request. Don’t forget to provide your GPG public key if you want the backups to be encrypted!

I imagine one of the first questions we’ll get is why we didn’t use Amazon S3 as the backend for this service instead, given that it’s “so much cheaper.” There are several reasons for this:

1) We don’t feel Amazon S3 shares our views on issues like customer privacy as well as rsync.net does.

2) rsync.net has been very willing to work with us to support a solution that meets all of our objectives; Amazon S3 couldn’t care less.

3) While the storage fee at Amazon S3 is much less, they charge a fee every single time you read or write that data, which we calculated was not conducive to keeping that data up-to-date, and a backup that is not up-to-date is worthless.

Concurrent with this new feature, we have made a couple of changes to our local backups to run more frequently. We have also taken this opportunity to streamline our storage accounting in anticipation of future changes in that area, while also fixing a bug that was causing some sites with large numbers of directories or very small files to under-report their actual storage usage.

Update: By default, you will get an account at the San Diego rsync.net location, which is the fastest for us. If you prefer, you can now request their Zurich location instead when you contact us to set it up, at the same price. We plan to offer their geo-redundant option later at a (significantly) higher price at some point in the future, possibly as soon as next week.

This outage will not affect member services, only our own sites (www, blog, and members), FTP, and ssh will be affected during these brief downtimes.

We apologize for the short notice and hope this is not too disruptive to anyone’s site maintenance plans.

To mediate the contest, we’re using 99Designs.com. To meet their requirements, we’ve had to break the project up a little bit and submit just the design/layout (not the HTML/CSS coding), with a prize of $600. If the winning designer wants to produce the type of HTML/CSS we need, we’ve got an additional $400 for that. In addition to the cash payment, we’ll credit the winner here on our blog and in our public FAQ, which is where we send people when they ask us for referrals to web designers.

If you’re a NearlyFreeSpeech.NET member who likes to design websites, we urge you to get involved. There’s no charge for participating on 99Designs.com as a designer and it takes just a minute to get set up.

If you’re a NearlyFreeSpeech.NET member who may not like to design websites but who does have strong opinions, feel free to review the contest, entries, and comments, and post feedback about the submitted designs and your thoughts about priorities in the contest thread on our forum. We’ll review your feedback and pass it on to the entrants where appropriate, as well as including your thoughts and feelings in our decision making process.

One of the things I dig about 99Designs is how open everything is. Whether you’re participating or not, you can check it out and see every design submitted, every comment made, and every bit of feedback. I found 99Designs while exploring venues to recruit a PHP programmer. We’ve been kicking around the idea of a contest for awhile, but when I found this I said, “Why wouldn’t we do this? Why wouldn’t we do this right now?” The only drawback I’ve found is that the period seems a bit short, the contest is currently slated to end somewhere around Wednesday night. But I’m going to see if I can get them to extend it at least through the end of this week.

The contest is open to anyone, but I’d sure like to see a NearlyFreeSpeech.NET member win this. (NearlyFreeSpeech.NET members will not receive preference; you can only win it by being the best, which is exactly what my experience has consistently proven you all to be.)