That’s all been prepped now, and what we’re going to do is automatically migrate everyone during the month of April. If you have affected sites, you can get a specific time for each site from our member interface, and the main sites page will star any site scheduled for an upgrade on your list of sites so you can see at a glance which sites are affected.

There are two drawbacks to this upgrade.

The first is that the upgrade involves minor downtime since the files need to be transferred from one file server to the other, which requires log files and any dynamic data files be closed and held inactive while they’re moved. The downtime will be roughly proportional to the size of the site, so this is a great time to peek in and check for any files you’re not using or log files you want to trim; it’ll make the upgrade go faster and you’ll save money on storage.

In addition, while most sites will be moved transparently, sites that use absolute paths in .htaccess files (or in PHP scripts, although we don’t advise that) will need to be updated to work properly after the move. If this is applicable at all (I personally moved all my sites without needing to change anything), it should just take a few minutes to update.

For those two reasons, we understand that you may want some control over upgrade schedule. To that end, you can log in and upgrade any of your eligible sites at any time between now and its scheduled date. You can also postpone a site’s upgrade if the scheduled time doesn’t work for you, within reason. You can’t postpone an upgrade past April 20, and postponements are for random intervals just so we don’t end up with an unmanageable number of sites to move on April 30th at 11pm.

The target file servers for this upgrade, f2 and f5, are a lot faster, have SSD-based L2 caches for busy files, superior network performance, and never require painfully long filesystem checks after an unclean shutdown, so we think people will be really happy with the change. Oh, and they have much newer and better redundant power supply setups!

We hope that the preparations we’ve made will make the transition as easy and painless as possible, and that everyone affected will agree that the end results are well worth having.

Those plans have now been finalized, and we will begin phasing them in this month.

Support Changes

We have broken up our human-provided support into three categories.

System Problem Reports

Any full member (sorry, not adjuncts) may submit a system problem report when they find something of ours malfunctioning that is affecting them. This might mean a site is down, or a MySQL process is crashed, or the member UI is giving them an error whenever they try to frotz the burin.

We will investigate the report as soon as we can (even outside our support hours if possible) and provide a brief response including a predefined finding and, at our option, a comment of up to one line. (This makes system problem reports not suitable for asking for other types of help.)

Assistance Requests

When a member needs us to do something that they are not able to do for themselves, either due to security limitations or a feature gap in our member interface, they can open an assistance request. These are no-cost support requests on specific topics that offer prepared guidance as far as what they encompass and what information you need to provide (if any). Examples include adding ssh keys, transferring stuff to someone else, or (whimper) canceling your membership.

We will take care of assistance requests during our regular support hours.

Secure Support Requests

These are the “catchall” for issues not covered by the other two avenues and not suitable for our member forums, and questions not covered by our FAQ or member wiki. This includes issues where you need us to investigate something on your behalf or provide basic advice about how our system works that isn’t covered elsewhere and isn’t suitable for our forum.

The following topics are specifically excluded from secure support issues because they are simply beyond the scope of the support we have the resources to provide:

• training of any sort
• web page design, uploading or consulting
• help using or configuring third-party software
• programming or debugging assistance
• re-explanation of things documented elsewhere on our site, other than very specific questions that have straightforward answers

Effective August 15th, 2009, you will need to have support points to open secure support issues. Support points will be available 10 for $5.00. They never expire, and can be “sold back” if you close your membership.

Each response we send you on a particular issue will carry a point cost from zero to ten points based on the time we spend and the complexity of the request, as determined by us. That cost will be deducted from your support points balance. If your support request requires multiple responses (e.g. multiple questions or back-and-forth exchanges), each response from us may carry a separate point cost. If you run out of support points, open tickets will be suspended until you add more, and new ones will not be able to be created.

The option to mark a secure support issue of “high” priority will be available for an extra up-front point cost: two support points during support hours and five support points outside support hours. While we frequently respond to “high” priority requests submitted outside business hours, we cannot guarantee any specific response time, only that we will look at them as soon as we can and, if necessary, move them to the front of the line when support opens.

Once the system has been operating for a few months, we will try to gather and make available some statistics about how much support requests cost to help members plan accordingly.

Service Pricing Changes

We are implementing five pricing changes:

The exact DNS charge will vary based on whether or not the domain is registered and/or hosted with us. See our public site for complete details; the detail page for each service is linked in the table above.

These changes will go into effect on or after September 1st, 2009 for affected services created after 2PM Eastern time today.

Temporary Grandfathering

Every site, domain, and MySQL process that existed at 2PM Eastern today has been exempted from these pricing changes until at least October 1st, 2009.

Beyond that, we’ve extended the grandfather period for each item by an amount based on when it was created. You will be able to see each “Exempt Until” date in the member interface later today.

The Road Ahead

We will be making many updates to the content of our public and member sites and our FAQ today to reflect these changes, so please bear with us while we go through that process. Once this post is up and our public site is updated, we will be doing a (rare) email notification to all of our members with funded accounts to inform them about the changes.

After that, we still have implementation and testing to do on these changes, so that’ll keep us busy for awhile.

Once that’s done, we do still plan to revise our single storage pricing fee into separate fees for storage and CPU/RAM usage to better apportion hardware costs to the sites that incur them (thereby sharply reducing the cost of hosting sites with large static content). We plan to do that in conjunction with the release of a couple of new features, hopefully within a few months.

Thank you!

Thanks for your patience with us while we worked out the changes that we feel will best befit our goal of continuing to promote legitimate freedom of expression by creating the best and most flexible hosting possible and offering it as affordably as possible to as many people as possible.

I’d like to (again) offer special thanks to the hundreds of people who offered comments, feedback, and suggestions in response to this proposal. You need only look at how different and how much better these changes are than our initial ideas to see what an impact you’ve had.

And, last but not least, thanks for being a member of NearlyFreeSpeech.NET!

Update

I thought I’d update this post to add some of the questions that seem to be getting asked over and over. Quite a bit of this was covered in the pricing discussion in our forums, but we recognize that not everybody wants to read 30+ pages of comments and feedback.

Q. Doesn’t this new pricing make your service very expensive?

No it does not. We ran the numbers, and over 50% of our current paid member accounts will pay less than $18.50 per year with our new pricing based on their current usage. That’s per year, not per month, and it includes domain registration fees, as well as MySQL, DNS, sites, bandwidth, email forwarding, and storage for everything they currently have set up with us.

We are 100% comfortable looking at those numbers and concluding from them that if 50% of our members can host, on average, multiple web sites including their supporting services and domain registration for about $1.50 a month, NearlyFreeSpeech.NET remains very affordable and a great value for small sites and light usage.

For people who work with our system to optimize their usage as much as they can, it’ll be even more affordable.

Q. Is it fair to charge a flat $0.01/day fee to tiny sites that used to pay a few cents per year?

Yes it is.

There are various ways to approach understanding this, but here’s the simplest way to explain it. “Fair” is defined by what happens if everybody does something. So, we ask this question: If every site were a small PHP/CGI-using site that got almost no traffic and got by on pennies per year, would we be able to stay in business from the revenue they generate? Quite simply, the answer is no.

That answer, all by itself, means that tiny sites are currently not paying their fair share. It means a baseline cost exists on a per-site basis regardless of the size of the site. (The details are quite a bit more complex and we’ll have to send you to the discussion thread for that.)

The followup question is: what is that baseline cost that needs to apply per-site to every single site, in order to make sure that we can continue in business if everybody ran tiny sites like that. It turns out that it’s $0.01/day. Every scheme that attempts to shift some or all of that expense to larger sites represents subsidizing small sites by charging large ones more. That is unfair.

Large sites already pay too much, as evidenced by the commenter below who remarked that he recently moved his site because it finally “got big” and implied that doing so was a foregone conclusion. Sadly, he’s not wrong. So, charging them more in order to preserve the “small site subsidy” is not an option.

We do understand that this is hard for many people to understand. We have spent seven years telling them that only machine costs matter, that it doesn’t cost anything to have people around to run the equipment for them 24×7, which is completely wrong, and we haven’t done anything to discourage “site sprawl” where people create sites at the drop of the hat because “they don’t cost anything.” These changes represent a huge shift, and we understand that some people will take time to adjust, and a few may not be willing or able to do so.

Q. Is it fair that all dynamic sites pay the same fee regardless of how processor intensive they are?

As far as the baseline fee which, as outlined above, we believe is fair for every site even if it does nothing at all, yes.

Beyond that, no. However, that isn’t a function of the per-site fee. Rather, the unfairness arises from tying sites’ resource-based billing solely to the amount of disk space used, rather than to some combination of disk space, CPU, and RAM. As outlined in the original blog post, this one, and many times in the discussion forums, we recognize and acknowledge the fundamental unfairness of that.

It is something we desperately need to move to correct. We will do so soon, but not now. The reasons for that have been covered elsewhere, so we will summarize them very simply by saying: we need this change to get to that change.

Q. If all this is true, how did you get this far?

At our current size, the founder of our company (that’s me) has been doing the work of 2-3 very expensive full time professionals without getting compensated for it. The effect of this has been to drastically hold down the cost of providing our service, that, even more than large sites, is where the existing subsidy of tiny sites comes from.

This introduces two problems. First, it doesn’t scale. As we continue to grow, the quantity of work increases but the quantity of founder does not. Second, it makes the operation of the company unacceptably dependent on one person; if anything bad ever happens to that person, the company and every member will be screwed because the money won’t be there to hire.

We’ve reached a point where we were faced with two choices: continue down that road and watch the quality of our service spiral into oblivion or take concrete steps to ensure that we have the people and resources to provide a strong, stable, sustainable organization that can provide the quality level we consider acceptable without being critically dependent on a single person who must never get sick, take a day off, or be killed by a falling bus. We chose the latter.

Q. How do I mark my site(s) as static so I won’t be charged the dynamic site fee?

Please see this entry in our FAQ.

We will provide additional reference information about static sites in the near future to help you get the most out of them.

Q. How do I minimize the impact of these changes on me?

There are several things you can do:

First and foremost, remove services you set up but aren’t using. This one is huge. So far, of the people who have written private feedback or cancelled, more people have commented that this change prompted them to clean up stuff they forgot about than have said they were put off by the pricing changes. One person said that after cleaning up stuff he wasn’t using, he’ll be paying less with the changes than he was before.

Second, make sure your domains are lined up with our registration, DNS, and hosting wherever possible. The “trifecta” discount for DNS is about 89%. If you’re using our service to provide DNS for random domains that are registered and hosted elsewhere, you’ll want to take a hard look at alternatives.

Third, switch to our static site type for any sites you have that don’t need scripting (PHP & CGI support). This includes the majority of sites built with web design tools.

Fourth, if you have a bunch of dynamic sites or multiple MySQL processes, explore what opportunities you may have to consolidate them. We will do a whole blog post sometime in the next few weeks discussing various options in this area. We also plan to offer features that will greatly enhance the power and flexibility of each site, a side effect of which will be making consolidation even easier, but of course we can only do that if doing so is financially viable (meaning we need these changes to be already in place).

Fifth, if you are using are email forwarding service, make sure you’re not doing so from inertia. It is a giant pain for us to maintain and eats a lot of resources, and it is priced accordingly. We consider it undoubtedly the worst value proposition of any service we offer as far as what you get for what you pay for. If you need it, we have it, but there are a lot of really good alternatives that are worth a look.

Q. Do you plan to give bulk discounts to people with tons of tiny sites?

We do not; this is behavior that incurs real costs for us. Such a plan would simply unfairly shift those costs onto people who use less services.

What we will do in the future is provide additional ways to do more with fewer sites, as outlined above.

Q. What are the moderation criteria for comments?

We have been able to approve almost all of the comments people have made on this posting, with a handful of exceptions.

Comments will not be approved if they:

• Ask questions that are off-topic or too tangential. (Please post them in our forums.)
• Repeat questions that have already been asked and answered.
• Threaten or insult us or other commenters
• Factually misrepresent the changes or their effects (such as claiming that we’ve instituted a fee to report system problems).

Most of that should be common sense, but we wanted to be very clear about it anyway.

One of the things we are running into with our pricing model is that the resource-based pricing we currently use doesn’t take everything into account, and doesn’t always do so accurately. That’s something we need to address.

For example, the price for storage is problematic, because it’s a machine resource we can measure, whereas CPU and RAM are (currently) not. It’s not that this charge is too high, as people sometimes like to use apples-to-oranges comparisons to claim, it’s that it’s improperly apportioned. People with large, static websites pay too much and people with small, CPU-intensive websites pay too little. This is a measurement problem more than anything, and it’s one we’ll address over time as we roll out the “arbitrary server process” technology that’s been under development for so long.

However, although that’s a good example, we have a more immediate problem that is not only keeping us from solving it but also gaining significance as we continue to grow. In addition to CPU and RAM, there is another hugely expensive resource that we are not recovering costs for: people. Since we are small organization that is very efficient and low-overhead, this comes in two forms: member support, which scales with the number of members we have, and system administration, which scales with the volume of services we provide.

Our service was designed with no “human” cost component because, at that time, I was the only person involved and I didn’t get paid. Support was intended to be limited to helping members when the system was broken, or with things that our system prevents them from doing themselves, and the service was priced accordingly. However, that was a long time ago. Now we have reached a point where over 50% (and approaching 75%) of our support requests do not fit the original design: they are from people who want to pay our prices but still get unlimited technical support and assistance. And we have to pay people to handle them. On average, each support request costs us $3.30, so that’s just not sustainable.

On the system administration front, our members (rightfully) expect us to provide a server environment that is carefully monitored, ruthlessly secured, relentlessly kept up to date, continually enhanced with new features, and where even the smallest performance and downtime problems are detected and responded to immediately by experts 365×24. But, despite that expectation, our members do not pay us to do so because here again, the prices were designed back in 2002 when I was the only person involved.

Therefore, as with support, the now-immense costs associated with system administration go unrecovered. Unlike support, the consequence is that we’re seriously understaffed in this area, and every NearlyFreeSpeech.NET member has in some way or another felt the repercussions: downtime, suboptimal performance, and endlessly delayed new features. As we continue to grow and the workload increases, the rate at which our list of unfinished to-dos expands simply accelerates. That, too, is unsustainable.

In order to properly staff NearlyFreeSpeech.NET to the level where we can provide the quality of service and support that our members expect, and get research and development of the new features everybody wants off of the 0-2-hours-a-week back burner, we need to start recovering an average of $2.50 more per member per month.

We’re going to do that, and we’re going to do it very, very soon. The only question is how we go about it. We want to be very careful not to repeat our storage pricing mistake: we want to apportion costs among our members as fairly and accurately as possible in proportion to their resource usage. That means imposing at least two new charge structures, one for support and one for system administration.

That’s because a whole lot of our members never use support at all. We strongly feel that they should not have to subsidize the other extreme: people who submit “high” priority support requests in the middle of the night demanding that we answer questions they could have just as easily found in our FAQ. So, what we’re leaning toward is a three-tier membership model:

- A “basic” membership, which carries no monthly charge but has no included support eligibility. Support requests can be submitted at normal or low priority for a $3.30 charge.

- A “standard” membership, which has a $1.00/month charge and includes one support request at normal or low priority per quarter. Additional support requests can be submitted at any priority for a $3.00 charge.

- A “premium” membership, which has a $2.50/month charge and includes one support request per month. Additional support requests can be submitted at any priority for a $2.50 charge.

In addition to support requests, our forum will remain free-to-post and we plan to implement a system to compensate the dedicated forum participants who we recognize are volunteering their time to make our service better by helping out their fellow members.

On the other hand, every single member needs us to have sufficient system administration resources to keep things running. The best plan we’ve come up with so far is to consider the “objects” that form a person’s services. As we define it, each of these is one object:

- one web site
- DNS for a single domain
- one MySQL process
- email forwarding for one domain

We’re considering applying a one-cent-per-day-per-object system administration surcharge to each account. The average member has 5 objects, so that would be an increase of about $1.50/month to them. That seems pretty reasonable in exchange for ensuring that we have the staff needed to keep an important website secure and available. But for people who need to squeeze out the maximum hosting-per-penny, it’ll still be possible to host a site in our domain (or with 3rd party DNS) that uses SQLite and limit the system administration surcharge to one penny per day.

Neither of these plans are finalized, but we’re moving rapidly in that direction. I’m posting this now for two reasons. First, because we’re committed to transparency in a way that nobody else can touch, and that demands we do so. Second, because we want to get as much constructive, helpful feedback as we can about how we can fairly distribute these costs over our member base.

Threats to cancel if we do (or don’t do) XYZ will be ignored. Simple economics dictates that any increase in prices will decrease demand; we already know that whatever we do will cause some people to stop using our service.

Some people may genuinely not be able to afford any increase; to those who are not able to rearrange their usage under whatever plan we come up with to reach a cost structure they can afford, we apologize that we are likewise not able to afford continuing to subsidize them. But we expect the loudest complaints to come from a small subset of members who voraciously consume our time and feel entitled to do so for free. This latter group is essentially strangling our business, so although converting such people to have sustainable, success-aligned goals (where they and we both gain) is our first choice, eliminating them as members if they are unwilling to cooperate is an inferior but acceptable alternative.

Likewise, exhortations not to change anything “because its fine the way it is” will also go unheeded, because it is not fine the way it is. We are not happy with the system quality we are able to deliver at current levels, so our options are to raise prices for the people incurring the costs or start zeroing in on members that we lose money on and terminating them, just like a lot of web hosts already do. And if that’s not enough to convince you, head over the the feature voting page and take a look at how far behind we are.

NearlyFreeSpeech.NET is not on the precipice of doom or anything; we balance our checkbook at the end of every month without going into the red, just like we’ve always done, and there is usually money left over for pizza. That, by us, is success. But we can see that if we don’t take corrective action, the quality of our service will inch downward until it reaches a threshold of “it’s cheap, but it sucks” that we’re not willing to approach, let alone cross. Knowing that we need to do something, the responsible approach is to do it as soon and as well as we can. So here we are.

Currently, we have to draw money out of our service and hardware budget to pay our human costs. With these changes, human costs will be paid for by the new charges instead. If things turn out the way we expect, we plan to use part of what that frees up from existing revenues to accelerate our service buildout (software and hardware) and we plan to return part in the form of reduced costs on some of our services. But first, we have to see things turn out the way we expect. So the changes outlined above aren’t intended to be the final word on the subject: things will get a bit more expensive, and then they should get a bit cheaper again.

As I said at the beginning, NearlyFreeSpeech.NET is not designed to turn a profit, its fee structure is designed to recover the costs of providing the service in the most fair and straightforward manner. That will remain true, even as we make the necessary adjustment to include the human cost in our fee structure. “As cheap as possible, but no cheaper.”

Thanks for your time, for being a member of NearlyFreeSpeech.NET, and, I hope, for your valuable feedback as we contemplate the best way to implement this change together.

We want the response to this to be a discussion, and blog comments don’t allow the easy quoting and attribution needed to facilitate that — I cringe every time I have to edit someone’s comment to add a response — so we are closing comments on the blog post and directing follow-ups to this thread in our discussion forum. That discussion is going to move fast, decisions will be made quickly, and changes will follow sooner rather than later. So if you want to be heard, now is the time.

Update

Based on the feedback we have received, we have made several significant changes to the plan. A summary is presented here for people who don’t want to read the whole discussion thread. This is getting close to being finalized, but is not yet set in stone, so please feel free to provide additional feedback. We would like to announce the final plan to all members via email sometime around the weekend, if possible.

Pricing Changes

Support Pricing

With this change, most support will be provided through the use of “support points,” which can be purchased in increments of 10 for $5. Unused support points do not expire and can be “sold back” when a membership is closed.

Submitting a support request will require you to have at least 1 support point. Each response you receive will be assigned a cost from 0 to 10 support points based on the time and complexity of the request, as determined by us, which will be deducted from your support points balance. If your support request requires multiple responses (e.g. back-and-forth exchanges), each response from us may carry a separate point cost. If you run out of support points, open tickets will be suspended until you add more, and new ones will not be able to be created.

Certain requests will default to zero points until they are implemented as features in the UI. These include:

- SSH key management
- Your first IP access control modification in a given week
- Chowning files from the “web” user to your site UID.
- asset transfers between members/accounts
- Canceling membership

We plan to implement “stub” features in the UI to open tickets for these types of requests without the need to obtain support points. If additional help is needed with such a request (such as troubleshooting a non-working ssh key), we may assign point costs as appropriate to cover our involvement.

The option to mark a request as “high” priority will be available. This will require 2 support points during our business day and 5 points at other times. This is a way to bump your request to the front of the line. While we frequently respond to “high” priority requests submitted outside business hours, we cannot guarantee any specific response time, only that they will be handled no later than first the following day.

To prevent people from having to pay a fee to report problems, we will implement a “problem report” option that will allow opening a ticket even if zero support points are available. To minimize misuse, our responses to such will be minimal, possibly stock answers, unless we need more information. In cases where misuse is a recurring problem, we reserve the right to block individual members from submitting problem reports. We intend to treat problem reports as high priority requests unless the rate of false positives proves problematic for us.

The estimated effective date for new support charges is August 1st – 15th.

Hosting Pricing Changes

Our new table of hosting service fees will be as follows:

- Bandwidth – $1/GB “or less” (existing sliding scale)
- Storage – $0.01/megabyte-month*
- Web Sites – $0.01/day/site
- First MySQL process – $0.02/day
- Additional MySQL process – $0.03/day
- InnoDB MySQL – $0.01/day/process
- DNS service – $0.01/day/zone
- RespectMyPrivacy – $0.01/day/domain
- Email Forwarding – $0.03/day/domain
- Domain Registration – $8.59/domain/year

Two discounts will be available when this change takes effect.

With respect to DNS, if the domain is registered with us or if www.(domain) is pointed at a site hosted here, then the DNS object pricing will be cut by a factor of 3 ($0.01/3-days). If both apply, then the DNS object pricing will be cut by a factor of 9 ($0.01/9-days). (This is a bit of a conscious effort on our part to reward people who are using our DNS and domain registration the way we intend — with our hosting.)

With respect to the per-site charge, sites using the currently-experimental “static content” site type will be exempt from the fee, as we will be handling those sites in a way that requires significantly less overall maintenance on our part.

*It is our intention to reduce storage pricing and add a “workload” billing factor to account for CPU/RAM resources used. It is our intention that this change will be revenue-neutral with the current storage charge but will more accurately apportion the machine costs of providing the service between individual sites. Specifics and timeframe are TBD.

The estimated effective date for new hosting charges is September 1st – October 1st.

Site Canonical Names

Site canonical names and types can now be set from the member interface by selecting the “Set Canonical Name” action from the site info panel. No more waiting for support requests!

For your information…

The site information panel for each site will now display a “For your information…” box if our system detects potential problems with your site, ranging from missing favicon.ico files to insecure “public” directory permissions.

Domain transfer codes

Domain transfer codes (needed for outbound domain transfers) can now be obtained from the member interface if the domain is unlocked. Please don’t use this feature, we really hate seeing domains transferred away!

Organization-owned accounts

Accounts can now be held in the name of an organization, such as a business, NGO, or club. This is designed to complement role membership, but should be useful for web developers hosting one or more business customers’ sites as well.

Alternate Emergency Contacts

The unintended consequence of having as draconian a privacy policy as we do and enforcing a strict 1:1 relationship between memberships and people is that we’ve had a couple of gut-wrenching cases arise where a company’s domain expired and their regular webmaster, who manages their account with us, was not available for whatever reason. In such situations, people tend to wind up screaming us about service interruptions we’re obligated not to talk to them about.

To help address that, you can now appoint an “alternate emergency contact” on a per-account basis. This is someone who can step in and take a limited subset of actions, like renewing a domain or depositing additional funds to keep a site running, if you’re unavailable. Finally, all our members managing their company’s or customers’ websites can go on vacation! (See our member FAQ for complete details.)

phpMyAdmin

phpMyAdmin has received a significant version upgrade. In addition, we have changed the way we interface with it so you can now use it to connect to MySQL processes other than your own. So if you’re managing MySQL for a friend, you no longer need to maintain a private copy of phpMyAdmin. (Naturally this feature will alert us if someone attempts to abuse it.) phpMyAdmin has been moved up to the main MySQL tab in our member interface since it is no longer process-specific. It’s now also possible to directly bookmark the phpMyAdmin login page if it’s something you access frequently.

Please note the “exit / log out” option in phpMyAdmin is very thorough; it will log you all the way out of our member interface! Click our logo in the upper left instead if you want to escape phpMyAdmin but remain logged in.

That’s all… for now! These are small things, but they represent the tip of the behind-the-scenes changes we needed to do to make certain longstanding feature requests able to work in our interface. Hopefully they will also improve the member experience a little bit while we get that finished up.

Our system still works with PHP 4, and we still have about 32.5% of our active hosted sites running on PHP 4. But its days are clearly numbered, so we’re taking appropriate, measured steps to curtail PHP 4 support on our network as well. We’ve removed most public references to PHP 4, and using our interface it’s no longer possible to switch a site to PHP 4, only away from it. With that done, we’re not making specific plans to go out and yank PHP 4 out from under our members whose sites are still using it before they are ready.

However, our plans are not the only factor at work. Sooner or later, a security problem or critical bug will emerge with PHP 4 that we won’t be able to fix ourselves or work around. Without developer support, we’ll be out of options. So the same day that happens, every PHP 4 site we host will be upgraded to the then-current version of PHP 5 whether they’re ready or not.

We have no way to predict if or when that will happen. Therefore, if your site is running PHP 4, you should make plans to upgrade your site sometime soon at a time when you’ll be available to handle any issues that arise. The upgrade is pretty straightforward, and problems are fairly rare, but it’s still better to block out a bit of time just in case.

To help people make this move, I want to address three points.

1. What version of PHP is my site running?

To find the version of PHP used by your site, go the site info panel for your site (click on the site name on the Sites tab), and look for the Server Type line in the Config Information box. If it says “Apache 1.3, PHP 4 Fast, CGI” then that site is running PHP 4. If it says PHP 5 Fast or PHP 5 Flex, then you don’t need to worry, because that site is already up-to-date.

2. How do I upgrade my site?

From the same page, select the Change Server Type action from the Actions box. Select the “Apache 1.3, PHP 5 Fast, CGI” radio button, and click the “Save Changes” button. Give it about 5 minutes, and your site should be up and running on PHP 5 with negligible disruption. The actual switchover should be nearly instant, it just sometimes takes a few minutes to process. When upgrading your site, do not accidentally choose the PHP 5 Flex option; it is a special-purpose option that will cause slower performance and much bigger compatibility issues in most cases.

3. What kind of problems could I have when upgrading?

Most people won’t have any problems at all. If you’re running a prewritten application developed by someone else (e.g. WordPress), then just make sure you are running an up-to-date version identified by the developers as working with PHP 5. That’s about it.

For people who program their own sites, we do have a few tips. When we upgraded our own PHP code from PHP 4 to PHP 5, the main thing that bit us was that “public” and “protected” are now reserved words in PHP 5. So, a couple of variable names ($public) had to be changed. There’s also an HTTP extension (pecl_http) we provide with PHP 5 with some common function names that sometimes collide with names others might have used in PHP 4 (http_request, http_get); if your site uses functions with these names, they’ll have to be renamed. We encountered similar issues with a SOAP library provided by our domain registrar for interacting with their system.

There is also some ugly, ugly code floating around that redefines $this inside objects in PHP 4. That should be fixed, because it was a bad idea in the first place. We encountered it in the PEAR Net_DNS module and a couple of other similar places.

The official PHP site provides a helpful list of backward-incompatible changes in PHP 5. In addition to the main content, the comments on that page are also quite helpful.

If people have other tips for migrating from PHP 4 to PHP 5, please feel free to submit them as comments.

But please, if you have a site still running on PHP 4, update it soon, at a time of your choosing, rather than waiting until the choice is taken away.

First, we have upgraded the ssh environment to more powerful hardware in order to allow for continued growth and to make sure scheduled tasks (still coming soon) will have enough resources to run without driving anybody’s site into the ground.

Second, we have updated Perl in the CGI/ssh environment from 5.8.8 to 5.8.9. Despite being a minor-version upgrade, this required the rebuild of nearly 1,000 CPAN modules and dozens of supporting libraries. Thus, we wanted to let people know to be on the lookout for resulting weirdness or incompatibilities. A few CPAN modules bought the farm on this upgrade, and we’ll list those below.

Two modules have been permanently removed. The first is Apache::Test, which can’t build because Apache isn’t actually present in the CGI environment. The second is Weather::Google. It seems the API used by Weather::Google is intended for desktop “widget” use and they don’t really want it used from other sites; they’ve blocked our entire network from using it. (This is speculation on our part, as there is little public information other than that this is an “unofficial” API for widgets to use, and so far we haven’t gotten a definitive response to our inquiries about it.)

Four other modules have been temporarily removed because they’re too badly broken to function in their current incarnation. They are:

• Catalyst::Engine::Test (fails its own tests)
• LWP::Parallel (appears to be badly out of date compared to LWP)
• Template::Alloy::XS (also appears out of date or obsolete)
• WWW::HostipInfo (fails its own tests, looks really broken)

We’ll keep all four of these modules on our list. If they start working in the future with new releases, they’ll reappear.

if you have any problems arising from either upgrade, please let us know via a Secure Support Request.

We did use that situation as an opportunity to make some big improvements to the quarantine feature that make dealing with bounce messages more palatable, including changing the “forward” option to just re-send the original message (where possible, since sometimes bounce messages from other servers don’t include the whole message) once the problem is fixed.

The quarantine is now also paginated, so you can see the whole thing if you have more than one page of quarantined messages. It also has an option to decode properly-formatted bounce messages, so you don’t have to stare at a list of 30 identical MAILER-DAEMON “Returned Mail” messages trying to figure out which is which. That option is resource-intensive, though, so it cuts the number of messages displayed from 50 to 10 to keep page load times down.

With these changes, the quarantine is starting to look and feel a little like a rudimentary webmail client. I want to remind everyone that, although it’s essentially now possible (if not pleasant) to read quarantined mail as if it were a mailbox, that’s not what it’s for. It is a temporary, volatile holding tank for messages that can’t or shouldn’t be forwarded and can’t or shouldn’t be returned. Our policy is to provide at least 10 megabytes per domain for quarantine. Currently, once a domain’s quarantine exceeds 12 megabytes, the system will expire out messages until it drops under 10. 10 megabytes can go faster than you think if something with a large attachment winds up in there, so don’t treat the quarantine like permanent storage.

The other glitch pertains to a handful of people who were participating in an email hosting trial we offered earlier this year using a reseller relationship that no longer exists. They were left in the lurch by this change, but we’ve contacted those people privately, and we’ll continue to do our best to help each of them make alternate arrangements.

Those two issues aside, I am very happy with the way this change managed, and super proud of our new system. Almost proud enough to make a foray of our own into email hosting. Almost. Maybe next year.

Since we felt we finally had the service we wanted to provide, we restarted the billing for email forwarding last night. Anyone using third-party DNS need not worry, we’ve pointed all our MX server names at the new email cluster.

We’ll be updating all of our documentation in the coming weeks to reflect the changes. In the future, we may offer the ability to search the quarantine; the new servers are so much faster than the old ones that such an option is now viable, where it wasn’t before.

But first we’ve got some other things to take care of, like the cron support we’re committed to releasing by the end of the year.

The older one is called Legacy Forwarding. It is a service we provide using our own servers, and adheres to our draconian email acceptance policy.

The newer one is simply called Email Forwarding. It is a service we provide in conjunction with a partnership with a third-party company. It is supposed to work a little differently; we call it a “hybrid” forwarding system because it combines certain aspects of a mailbox with certain aspects of email forwarding. Their servers are willing to accept virtually any email, but they then perform content-filtering checks and are supposed to divert suspect messages into a special “spam quarantine” accessible via our member interface.

I say “supposed to” because at some point in the recent past, without consulting with or notifying us, the third-party company unilaterally changed their system to discard all spam-suspect messages instead of placing them into the quarantine folder. At first we thought this was isolated to a few domains, but we have since discovered that the change was global. This means that for all of our members who have “Email Forwarding,” any messages you receive that are believed by their system to be spam, including false positives, are currently being silently discarded. This is absolutely not acceptable to us, but our efforts to redress this with them have not been successful.

Ultimately, our email forwarding members are paying us for email forwarding services, not a third party company, so it is our responsibility to make sure our members get the service they are paying for. To that end, we are taking the following steps:

• We will stop billing for all email forwarding domains until we feel things are working the way we intend.
• We are deploying a new hybrid email forwarding system, described below, which will be applied to all members’ email forwarding domains.
• Because we need to make changes, we will be temporarily disabling the email forwarding portions of our UI. We are not disabling email forwarding, just the UI that controls it.

The new email forwarding system will apply all of our existing email acceptance policy criteria, but it will do so a little differently. As with our legacy forwarding system, messages that meet all of our criteria will sail through the system with no disruption or delay, as will messages sent by domains with proper SPF configurations, even if they have other problems. Also as with the legacy system, messages from blacklisted servers will be immediately rejected.

For messages sent by servers with improper or no SPF and one or more other configuration problems, instead of bouncing them as the legacy forwarding system does, we will be trying a technique called greylisting, which tells the server sending the message to try again in a few hours. If the sending server does that, the message will be accepted.

We will also be adding a spam/virus checking component that will reject email containing viruses and quarantine egregious spam that somehow passes all of the above filters. (The current email forwarding scheme quarantines both spam and viruses.) Also new will be a feature that places post-forwarding bounce messages into the same quarantine, so if there are problems with delivering forwarded messages to you, you will be better able to detect and resolve them.

We feel that this system will make the best trade-offs between meeting our members’ need for reliable forwarding, keeping our forwarding mail servers off of global blacklists, and making life absolutely as miserable and unrewarding as we possibly can for spammers. It will also let us have one forwarding service for all members, and document it appropriately.

All of our email forwarding uses MX records in the nearlyfreespeech.net domain, so we retain full control over how they are handled. The above system is still in final development, but we believe it can be completed and deployed without any major disruption to anyone’s forwarded email. Due to the way Internet email works, even if we do wind up needing to take forwarding offline for a few hours in the middle of the night, messages will simply queue and redeliver when it comes back up.

We need to disable the member UI to work on it because we need the overall configuration to hold still long enough for us to update the UI code and move everyone over to the new system. It is our goal to have the entire transition completed by Saturday. Since we won’t be billing for email forwarding until this is resolved, we will be very highly motivated to make it happen as quickly and accurately as possible. If for any reason you do need to make an urgent email forwarding change that just can’t wait, or if you want to remove it altogether, just drop us a Secure Support Request and we’ll do our best to take care of you.

Since this is still an evolving situation, the above reflects what we want to do, and what we reasonably believe we can do. If and when circumstances change, we will update you accordingly. These changes will not affect the cost of email forwarding, and while they may ultimately put us in a more flexible position with respect to offering email hosting in the future, that is not a goal of this project and nothing is changing on that front at this time.

I apologize for the short notice and rushed character of this change. We spent a lot of effort (probably too much) trying to come to a less dramatic solution, and we were entirely unsuccessful.

The primary purpose of this maintenance window will be to add RAM and CPU power to some of our MySQL servers. Just under 40% of member MySQL processes will be affected, and we estimate the total time affected will be about 30 – 45 minutes.

You can tell if you are likely to be affected if your MySQL @@hostname variable is “m2″ or “m3.” However, while we don’t plan to move any MySQL processes between now and then, we do reserve the right to move them at any time for any reason if appropriate to maintain network stability and performance.

This upgrade will increase total RAM available to member MySQL processes by 60%, with a corresponding increase in CPU power. After the upgrade is complete, we will be able to finish the migration to FreeBSD 7 and the combined result will be better MySQL performance for all.

At the same time, we will be adding RAM and CPU cores to the hosting cluster, but that should not cause downtime since we can rotate the affected servers out of production prior to the upgrade and redistribute the load; there is plenty of spare capacity at that particular time.

As always, we appreciate your patience as we update and upgrade our network to make your hosting better, faster, and more reliable.

Here’s the executive summary: FreeBSD 7 is, by and large, great for us and we will be aggressively deploying it throughout our network in the near future. ZFS is also great, but while it definitely has a future on our network someday, this is not that day.

FreeBSD 7 offers significant benefits for SMP scalability, which is pretty important to us since most MySQL processes and member sites run on eight-core servers. The performance benefits on MySQL are very real, particularly on CPU-intensive workloads (which tend to be common if your tables are indexed properly). The performance benefits for member sites are less pronounced for some pretty complicated reasons, but they’re still there and they’re still worth pursuing.

ZFS is a really neat technology. By and large, it works pretty well. It’s not a great match for a MySQL server, though, since it seems to want a lot of CPU and RAM at the same time MySQL wants lots of CPU and RAM, because it works hard when asked for data and MySQL asks for data when it’s working hard. Thus, we noticed a little bit of contention and we were concerned about what would happen in low memory conditions, such as if a bunch of MySQL processes happened to surge busily at once. ZFS seems best suited to sticking on super-tightly controlled file servers that have gigs of RAM free and don’t do anything else. Even so, we’re not 100% sold on its stability, so we’ll revisit it sometime in the future and do further testing before we start loading it up on mission-critical file servers.

All of our MySQL FreeBSD 7 ZFS beta volunteers have been migrated to production FreeBSD 7 (non-ZFS) MySQL servers. They’re just the vanguard, though; we will be aggressively upgrading almost all member MySQL processes over the next few days.

The actual process is a bit complicated, but here are the essentials: we’ve brought some new hardware online running the target environment (FreeBSD 7, MySQL 5.0.67). Call that “Server A.” We’ve moved all the MySQL processes from Server B (one of the older ones) onto server A. Then, we reinstall Server B. Then we move everyone on Server C to Server B and reinstall server C. And so on, down the line.

The upgrade requires no action on your part and entails MySQL downtime averaging less than five minutes; a significant percentage of our MySQL-using members will notice only one to two minutes of disruption, if they notice it at all. Even so, we’re limiting it to low-usage periods (certain times during weekends, late nights, and early mornings). Due to the number of processes being upgraded and our need to schedule upgrades based on specific server assignments, we will not be able to coordinate individual process upgrades.

We do still have some longtime members hanging around on MySQL 4.1 running on FreeBSD 4. Since MySQL 4.1 to MySQL 5.0 isn’t always a seamless migration, we’ll be contacting those affected to let them know their situation and try to coordinate their journey out of the stone age before we do anything there.

If you want to see what version of FreeBSD or MySQL is powering your MySQL process, you can use the SQL command “SHOW GLOBAL VARIABLES” and look at the “version” and “version_compile_os” variables, or access them in queries as “SELECT @@version” or “SELECT @@version_compile_os.” The former shows the MySQL server version (which is also displayed in phpMyAdmin and by the MySQL command line client), and the latter shows the OS version: “portbld-freebsd6.3″ for FreeBSD 6.3 and “portbld-freebsd7.0″ for FreeBSD 7.0.

We also have a significant RAM and CPU power upgrade planned for MySQL once this upgrade is complete. That will require a brief off-hours downtime for some MySQL processes, which we will schedule and announce at a future time.

Finally, though MySQL processes will never be able to flit around our network the way web requests do, we are taking baby steps in the direction of SAN-based MySQL storage in order to decrease the lock-tight correlation between MySQL processes and specific frontend hardware.