Here’s the executive summary: FreeBSD 7 is, by and large, great for us and we will be aggressively deploying it throughout our network in the near future. ZFS is also great, but while it definitely has a future on our network someday, this is not that day.

FreeBSD 7 offers significant benefits for SMP scalability, which is pretty important to us since most MySQL processes and member sites run on eight-core servers. The performance benefits on MySQL are very real, particularly on CPU-intensive workloads (which tend to be common if your tables are indexed properly). The performance benefits for member sites are less pronounced for some pretty complicated reasons, but they’re still there and they’re still worth pursuing.

ZFS is a really neat technology. By and large, it works pretty well. It’s not a great match for a MySQL server, though, since it seems to want a lot of CPU and RAM at the same time MySQL wants lots of CPU and RAM, because it works hard when asked for data and MySQL asks for data when it’s working hard. Thus, we noticed a little bit of contention and we were concerned about what would happen in low memory conditions, such as if a bunch of MySQL processes happened to surge busily at once. ZFS seems best suited to sticking on super-tightly controlled file servers that have gigs of RAM free and don’t do anything else. Even so, we’re not 100% sold on its stability, so we’ll revisit it sometime in the future and do further testing before we start loading it up on mission-critical file servers.

All of our MySQL FreeBSD 7 ZFS beta volunteers have been migrated to production FreeBSD 7 (non-ZFS) MySQL servers. They’re just the vanguard, though; we will be aggressively upgrading almost all member MySQL processes over the next few days.

The actual process is a bit complicated, but here are the essentials: we’ve brought some new hardware online running the target environment (FreeBSD 7, MySQL 5.0.67). Call that “Server A.” We’ve moved all the MySQL processes from Server B (one of the older ones) onto server A. Then, we reinstall Server B. Then we move everyone on Server C to Server B and reinstall server C. And so on, down the line.

The upgrade requires no action on your part and entails MySQL downtime averaging less than five minutes; a significant percentage of our MySQL-using members will notice only one to two minutes of disruption, if they notice it at all. Even so, we’re limiting it to low-usage periods (certain times during weekends, late nights, and early mornings). Due to the number of processes being upgraded and our need to schedule upgrades based on specific server assignments, we will not be able to coordinate individual process upgrades.

We do still have some longtime members hanging around on MySQL 4.1 running on FreeBSD 4. Since MySQL 4.1 to MySQL 5.0 isn’t always a seamless migration, we’ll be contacting those affected to let them know their situation and try to coordinate their journey out of the stone age before we do anything there.

If you want to see what version of FreeBSD or MySQL is powering your MySQL process, you can use the SQL command “SHOW GLOBAL VARIABLES” and look at the “version” and “version_compile_os” variables, or access them in queries as “SELECT @@version” or “SELECT @@version_compile_os.” The former shows the MySQL server version (which is also displayed in phpMyAdmin and by the MySQL command line client), and the latter shows the OS version: “portbld-freebsd6.3″ for FreeBSD 6.3 and “portbld-freebsd7.0″ for FreeBSD 7.0.

We also have a significant RAM and CPU power upgrade planned for MySQL once this upgrade is complete. That will require a brief off-hours downtime for some MySQL processes, which we will schedule and announce at a future time.

Finally, though MySQL processes will never be able to flit around our network the way web requests do, we are taking baby steps in the direction of SAN-based MySQL storage in order to decrease the lock-tight correlation between MySQL processes and specific frontend hardware.

Hey, we have some heavy workloads…

However, as longtime members may know, when confronted with a choice between stability and performance, we tend to choose stability. In fact, that preference led us to skip FreeBSD 5 entirely while the FreeBSD team tirelessly worked out the changes needed to support modern SMP hardware. (I look back to the FreeBSD 4 days when we had to use single-core systems and I whimper.)

Another flagship technology in FreeBSD 7 is ZFS. The primary benefits of ZFS wouldn’t accrue directly to our members (although it might provide the underpinnings for some neat stuff), but rather it would make our lives a lot easier. The problem with that is that there’s some debate on whether or not ZFS is ready for prime-time usage. So far, it’s tested pretty well and we’ve done some fairly abusive things to it.

Even so, odds are that FreeBSD 7.0 will never see production in our web cluster; we will most likely give serious consideration to 7.1 or 7.2, depending on the results of our internal testing. That internal testing is (and has been) underway, and we’d like to offer the adventurous a chance to participate.

Putting FreeBSD 7.0 into our web cluster isn’t really a good option, because it would entail building FreeBSD 7 versions of a couple thousand software packages, CPAN modules, python eggs, and etc, and might cause problems for custom-built CGI binaries. Instead, we feel that heavy MySQL usage is most likely to show the benefits of improved SMP code in a controlled software environment.

Thus we have set up an experimental MySQL server running MySQL 5.0.51a on FreeBSD 7 with a ZFS filesystem. If you would like your MySQL process moved to this experimental server, please submit a secure support request indicating the name of your MySQL process. Migrating MySQL between servers is very fast, but it takes about 10 minutes for a moved process’s new IP address to propagate through our network.

The trade-offs should look like this:

• Your MySQL process will probably be a lot faster, especially if it’s busy.
• There might be an increased chance of MySQL downtime due to instability.

Naturally we won’t let stability issues get out of hand on that server. If ZFS presents a problem, we’ll blow it away and try FreeBSD 7 under UFS2, and if that’s also unstable, we’ll kill the test and try again at 7.1. We’d also move anyone back out of the test who encountered problems on request.

Give it some thought. If your MySQL process is not mission-critical and you’re interested in helping us plan the next generation of NearlyFreeSpeech.NET servers, please drop us a line.

If you’re currently running PHP 4 and you’re happy with it, don’t worry. We’re not planning to do anything drastic to you at this time, and we’ll continue to support PHP 4 as long as the developers do.

As part of the PHP 5 beta, we’ve also unified the versions of PHP 4 and PHP 5 that appear in the ssh environment, and we’ve updated the environment to detect which version your site uses and put it first in your default PATH variable. Concurrent with the update, the /usr/local/bin/php link has been switched from PHP 4 to PHP 5, but since the “php” you get when you type “php” will come straight from the version-specific directory appropriate for your site, that shouldn’t have much effect.

In case you haven’t been following the long-running PHP 5 Beta forum thread, you may not be aware that you can switch back and forth between server types on the fly, usually without downtime. (It’s remotely possible that someone else could trigger a second update right after you, which would cause your site to display “Unknown Site” for a minute or so after switching from PHP 4 to PHP 5 or vice-versa.) To switch PHP versions, all you have to do is click “Change Server Type” on the Site Information panel for your site in our member interface, select the radio button for the site type you want, click “Save Changes,” and wait around for a minute or so.

But, hmm, what’s this? There seem to be two PHP 5 server types listed on that page… “PHP 5 Fast” and “PHP 5 Flex (Beta).” What’s that all about?

Well, we just loved having PHP 5 in beta so much that we couldn’t let it go! OK, not really. Have a look at PHP 5 Flex phpinfo() output compared to PHP 5 Fast phpinfo() output. Spot any important differences?

PHP 5 Flex is a new implementation of PHP that runs inside our customized CGI environment. This means we can do fun stuff like turn off safe_mode and open_basedir. That’s right: you can now get non-safe_mode PHP at NearlyFreeSpeech.NET. Now, this has always been sort-of possible if you were willing to mess around with .htaccess, make all your PHP scripts executable and hack #!/usr/local/bin/php lines into them at the top, but when you do that, you’re running the command line version of PHP, so weird stuff sometimes happens. PHP 5 Flex, on the other hand, runs the specialized CGI version of PHP based on the same configuration we used for standard PHP 5, and it runs standard .php files unmodified. You don’t even need to make the PHP scripts executable; it just works. It also uses all the same /home-based paths you’re familiar with from the ssh command line and CGI, like /home/public.

So why would you ever not want that? Well, the answer lies in the name of the alternative, PHP 5 Fast. PHP 5 Fast is based on mod_php5, which runs significantly faster than CGI-based PHP: often three to five times faster, especially when handling lots of small requests. If you don’t need safe_mode off to get compatibility with some strange third-party application, there’s no need to take the speed hit associated with virtualizing every request. That’s why the new PHP default is PHP 5 Fast and why we recommend leaving it as-is unless you have a good reason to change it. PHP 5 Flex is also beta; there may be a few problems to kick around. If you try it and find them, let us know via the forums or support requests.

So that’s our big PHP 5 update. It’s been a long time coming and a lot of work; we hope you enjoy it!

If you are a current member and you have large, static content like music, videos, or file downloads, and you use 10+ GB of bandwidth per month, you are eligible for this beta. Submit a secure support request for more information. (You’re also eligible if you use less bandwidth, but since the service is highly optimized for bulk use, so is the pricing, meaning it may increase your costs rather than decrease them. Even so, you’d still enjoy the massive speed boost.)

Dynamic content (PHP, CGI, etc.) is not eligible for this new service. However, if you can segregate your static content into a subdirectory, one site can use both services at the same time!

We are pleased to announce that the NearlyFreeSpeech.NET API is about to enter beta. What is the NearlyFreespeech.NET API? In a nutshell, it’s a way to control your hosting services from a program or script.

The API platform we’ve created is fairly robust and very easy to use. (Here’s a shocker: it’s a web-based!) But since we’re just getting started with this, the functionality currently available is somewhat limited. However, the four most requested API functions are supported: adding and removing site aliases and email forwarding addresses.

We’ll add more functionality based on demand. Over time, we’ll also be migrating our own user interface to use the API, so someday it should be possible to use it to do pretty much anything you can do from our site, and plenty of things you can’t! (But don’t hold your breath waiting for a “increase account balance” API call. )

While we get this up to speed, beta space will be very limited, so if you want to try it out, drop us a Secure Support Request asking for your API key. You can access the API directly from the CGI or programming language of your choice, or use an example PHP class library that we’ve developed.

There’s also a documentation experiment attached to this. To document the API, we’ve created our first Wiki. We have a mountain of technical information about our service that we need to communicate to our members, and we know we need to find a better way to do that than our dauntingly-large FAQ. However, putting information out there in the form of a world-writable Wiki is a bit nerve-wracking, so we’re giving it a try with a special Wiki for the API. Check it out here. We’ll see how that goes.

Note: We are using a self-signed certificate for the api.nearlyfreespeech.net SSL site, because there are some license issues for us to explore with using a single SSL certificate on multiple servers. This means you’ll need to accept a certificate certified by an unknown authority (that’s us) to see the API site. A future version of the PHP library will include the NFSN CA certificate and use it to verify that you’re talking to the real API, but we don’t recommend importing it into your browser or anything.

As always, your feedback is welcomed and encouraged!