• One server type to rule them all?
• A forum facelift.

The Kitchen Sink: One server type to rule them all?

For quite some time, we have offered the “Apache 2.4, PHP, CGI” for PHP users and the “Apache 2.4 Generic” type for people who want to run Node.JS or other custom web server processes. It’s possible to run PHP under Apache 2.4 Generic as FastCGI, but it often requires you to rethink the structure of your application. That can be tricky if you didn’t write the application. There are workarounds but… they kinda suck. We’ve heard from several people in that position that they hate being in that position and wish they could just have both. Now, they can. We’ve added a server type called “The Kitchen Sink” that includes support for Apache, native PHP support, CGI, and custom daemons and proxies. This is great for people who need to run apps that are part PHP and part not, as well as for people with PHP apps who want to jam something like Memcached or Redis in there with it.

“The Kitchen Sink” is available on an experimental basis right now; from the Site Information panel, it’s enabled by editing your site’s service type. It may… or may not… get a different name when it leaves experimental status.

A forum facelift

As some of our longtime members may know, our member forum was originally based on phpBB 2. phpBB 2, however, reached end-of-life in 2009, so we’ve long since ditched most of the innards in favor of more modern, up-to-date code designed for PHP 8, not PHP 3. What we haven’t ditched, until now, is the now-brutally-outdated early-2000s-era aesthetic. We’ve made some behind-the-scenes fixes over the last couple of weeks designed to make the forum easier to understand and use. We’ve also finally put some effort into the design. That update went out today.

This is nothing groundbreaking; we tend to avoid “groundbreaking” when it comes to usability. But it’s a solid update from the early 2000s to at least the mid-2010s. This also isn’t the most important thing in the world but we wanted to close out the year with something a little bit fun but still beneficial. It’s been an absolutely wild year!

We hope the functional and design changes will make the forum easier to browse and read and more pleasant to interact with. I’ll still post there as my usual curmudgeonly self, though; apparently, no upgrade can fix that.

More information about the forum updates is available in the forum.

That’s all for now! More updates to come as soon as we’re finished tamping down the bugs flushed out by these!

• We’re replacing our Intel Xeon servers with brand-new AMD Epyc servers.
• All our existing file storage will be migrated from SATA SSDs to NVMe PCIe 4.0 SSDs.
• Most of our content will be served from New York City rather than Phoenix after the upgrade.
• Various things may be intermittently weird or slow for the next couple of weeks as we shift them around, but we’re working hard to minimize and avoid disruptions to hosted services.

NearlyFreeSpeech goes Team Red

There’s no question that Intel has been good to us. Xeons are great processors. But these days, AMD Epyc… wow. The processors aren’t cheap, but the compute performance and I/O bandwidth are outstanding. 128 PCIe 4.0 lanes? Per CPU? Except for the speedup, this change should be transparent to most people. By and large, we’ve tried to protect people from building things too specific to exact CPU models by masking certain features, but there is probably some random instruction set supported on the old machines that isn’t present on the new ones. So if you’ve done something super-weird, you may have to recompile.

I don’t want to make any specific promises about performance. After all the speculative branch execution fixes, the security layers needed for our system to protect you properly, and other overhead, these things never quite reach their maximum potential. But, so far, they’re so fast!

Here’s the catch. Some ancient site plans bill based on storage space but not CPU usage. These plans have been gone for about ten years. They were an incredibly bad deal for people who wanted to store lots of data, but it cost basically nothing if your site was tiny and used lots of CPU. That wasn’t sustainable for us. We grandfathered those sites at the time because we’ve always paid a flat rate for a fixed amount of electricity whether we use it or not, and those sites have been running on the same hardware ever since (Intel Xeon X5680s!). Neither of those things will be true going forward, so it’s the end of the road for those plans. We plan to temporarily allocate a bare minimum amount of hardware to those sites for a few months and then let affected people know that they’ll be migrated to current plans around the end of the year.

If you want to check this now:

1. Go to the Site Information panel for your site.
2. Find the “Billing Information” box.
3. If there’s been a red-text message “($10.24/GB/Month – Legacy Billing!)” on the “Storage Class” line for the last ten years, you’re affected.

To change it, find the “Config Information” box and edit the Server Type. Pick the closest option. (If in doubt, “Apache 2.4, PHP, CGI.”)

Quoth the raven, “NVMe more!”

It’s something of a sore point that our file storage performance has always been a bit lackluster. That’s largely because of the tremendous overhead in ensuring your data is incredibly safe. Switching from SATA SSDs to NVMe will give a healthy boost in that area. The drives are much faster, and the electrical path between a site and its data will be shorter and faster. And it’ll give all those Epyc PCIe lanes something to do.

But there’s a little more to the story. To get adequate resiliency, sacrificing some performance is a necessary evil. It just flat-out takes longer to write to multiple SSDs in multiple physical servers and wait for confirmation than to YOLO your data into the write cache of a device plugged into the motherboard and hope for the best. We accept that. And we’ve always accepted that our less-than-stellar filesystem performance was the compromise we had to make to get the level of resiliency we wanted. However, we’ve always suspected we were giving up too much. It’s taken years, but we’ve finally confirmed that some weird firmware issues have created intermittent slowness above and beyond the necessary overhead.

So we expect our filesystem performance to be dramatically better after the upgrade. Don’t get me wrong; it won’t be a miracle. The fastest SAN in the world is still slower than the NVMe M.2 SSD on the average gaming PC (or cheap VPS). But one keeps multiple copies of your data live at all times and does streaming backups, and one doesn’t. And it should be a hell of a lot better than it has been.

Related to this, we’ve made some structural changes to site storage that will make moving them easier and faster. That has some other benefits we care a lot about that you probably don’t, like making storage accounting super fast. It should also make some other neat things possible. But we need to explore that a little more before we announce anything.

New York, New York!

Things have changed quite a bit since we started. As much as I love Phoenix, it’s not the Internet hub it was when I lived there in the 1990s. While some benefits remain, I no longer believe it’s the best place for our service. We see dumb stuff we can’t control, like Internet backbones routing traffic for the US east coast and Europe from Phoenix through Los Angeles because it’s cheaper. New York, on the other hand, is functionally the center of the Internet. (More specifically, the old Western Union building at 60 Hudson Street in Manhattan.)

It will surprise no one that Manhattan real estate is not exactly in our budget, but we got close. And, more importantly, we are parked directly on top of the fiber serving that building. It’d cost about ten times more to shave 0.1 milliseconds of our ping times.

This change will make life demonstrably better for most people visiting hosted sites; they’re in the eastern US and Europe. But we’re not leaving the west hanging out to dry. We can finally do what I always wanted: deploy our own CDN. After we’re finished, traffic for customer sites will be able to hit local servers in Phoenix, New York, and Boston. Those servers will transparently call back to the core for interactive stuff but can serve static content directly, much like our front-end servers do today. That’s already tested and working. You might be using it right now.

The new design is completely flexible. It doesn’t matter where your site is located; traffic enters our network at the closest point to the requestor, and then our system does the right thing to handle it with maximum efficiency.

It’s now technically possible for us to run your site’s PHP in New York, store your files in Boston, and have your MySQL database in Phoenix. But “could” doesn’t always mean “should.” We’re still constrained by the speed of light; a two-thousand-mile round trip on every database query would suck pretty hard. (But I’ve done it myself with the staging version of the member site. It works!) So everything’s going to New York for now.

Keeping it weird

This change means we have to move all your data across the country. Sometime in the next few weeks, each site and MySQL process will be briefly placed in maintenance and migrated across our network from Phoenix to New York. For most sites, this should take less than a minute. We’ll start with static sites because they don’t have any external dependencies. Then we’ll move each member’s stuff all at once so we don’t put your MySQL processes and site software into a long-distance relationship for more than a few minutes. Once we have a specific schedule, we’ll attempt to make some information and, hopefully, some control available via the member UI to help you further minimize disruption. But our goal is that most people won’t even notice.
There may be some other weirdness during this period, like slowness on the ssh server, and you may actually have to start paying attention to what ssh hostname to use. All that will be sorted out by the time we’re done.

Some longtime members may recall the 2007 move where it took us over a day to move our service a few miles across town. At the time, we wrote, “Should we ever need to move facilities in the future, no matter how long it takes or how much it costs, we will just build out the new facility in its entirety, move all the services between the two live facilities, and then burn down the old one for the insurance money.” Oh my god, it took a long time and cost so much money, but that’s exactly what’s happening. (Sans burning down the facility! We love our Phoenix facility and hope to continue to have equipment there as long as Arizona remains capable of sustaining human life.)

Final thoughts

These changes represent an enormous investment. Thus, much like everyone else these past couple of years, we will have to pass along a huge price increase.

No, just kidding.

Our prices will stay exactly the same, at least for now. (Except for domain registration, where constant pricing fuckery at the registries and registrar remain the status quo. Sadly, there’s nothing we can do about that. Yet.) In fact, they might go down. We bill based on how much CPU time you use, and it’s likely to take less time to do the same amount of work on the new hardware.

The last few years have been pretty weird. COVID aside, NearlyFreeSpeech.NET has been keeping pretty quiet. There’s a reason for that. I’m proud of what NearlyFreeSpeech.NET is. But there’s a gap between what is and what I think should be. There always has been. And that gap is probably bigger than you think.

So I spent some time… OK, nearly three years… more or less preserving the status quo while I did a very deep dive to learn some things I felt I needed to know. And then, I spent a year paying off tech debt, like getting our UI code cleaned up and onto PHP 8 and setting up this move. So four years went by awfully fast with little visible change, all in pursuit of a long-term plan. And in a few weeks, we’ll be finished. With the foundation.

“It’s a bold strategy, Cotton. Let’s see if it pays off for ’em!”

To make a long story short, we spent so long on the 2023Q2 quarterly software build that it was July, and we still had problems. We finally have a clean build that passes all of our hundreds of internal tests. But we also have the 2023Q3 quarterly build running just as smoothly. Since 2023Q2 won’t get any security updates through the FreeBSD ports team, having our non-production members test it doesn’t seem useful. And we’re sure not going to roll it out to production sites untested.

And so, we are skipping it. The default realm for production sites will be the (now very thoroughly tested) 2023Q1 realm. And the default realm for non-production sites will be the shiny new 2023Q3 realm. As always, we’ll backport security fixes as needed from 2023Q3 to 2023Q1.

No more PHP 7!

For those sites being upgraded from 2022Q4 to 2023Q1, it’s worth reiterating that PHP 7.4 was deprecated in 2021, and security support ended in November 2022. If your site still runs on PHP 7 eight months later, you’re in for a bad time. The PHP developers are ardent adherents of “move fast & break things,” and backward compatibility is the thing they break the most. Back in February, we posted information about this, including some advice for updating, in our forums.

Despite having no single point of failure from the hardware perspective, each site’s content is still backed by a single system image (necessary for coherency), so these updates may cause some temporary disruptions to affected sites. We will do our best to minimize that.

We do also plan to upgrade our core database servers. These are fully redundant, so we do not anticipate disruption, but the possibility does exist. We hope this upgrade will resolve an issue that mainly manifests as intermittent errors in our member interface early in the morning (UTC) on Sundays.

• TLS enhancements
• HTTP/2 support
• Automatic gzip compression
• Major Access Control List (ACL) improvements
• Shared IP blacklist support
• Websockets support
• Wildcard alias support

To encourage people to help us test out the new stuff, we’re exempting participating HTTP requests from bandwidth charges for the duration of the test. You can opt-in to the test for a particular site by selecting the “Use Free Beta Bandwidth” action on the Site Information panel for that site in our member interface. That page has all the fine print about the test, which mostly cover two central points:

• Reminding people that it is a test and things might not work.
• Clarifying that although there is no fixed limit to the amount of bandwidth a site can use under this test, there is a “floating” limit: don’t cause problems.

This test (and the free bandwidth) will run through at least March 15th, 2016.

Below, we’ll also discuss each new feature briefly.

TLS enhancements

The major enhancement to TLS (transport layer security, the technology that makes http:// URLs into https:// URLs) has to do with scalability. As people may know, we currently use Apache as a front-end TLS processor. As a test, we generated test keys and certificates for every site we host and loaded them all into a single Apache config just to see what would happen. The resulting server process took nine minutes to start and consumed over 32 GiB of RAM before answering its first request. That’s… not going to work. So we’ve written a great deal of custom code to solve that problem.

We’ve also always been worried that the overhead of TLS would require us to charge more for using it. One side-effect of this work is that we’ve reduced the fixed resources required to support TLS so much that we can now definitively say that that won’t be an issue.

The new system also improves the performance of TLS requests, and made a couple of other changes we were able to backport to the existing setup. First, we’ve eliminated TLS as a single point of failure. Second, due to our use of Apache as a TLS frontend, the last hop of an HTTPS request is handled as unsecured HTTP on our local LAN. Although the probability of anyone monitoring our local LAN without our knowledge is pretty small, in a post-Snowden world one has to acknowledge that taking reasonable precautions against improbable things isn’t as paranoid as it used to be. So last-hop HTTP traffic (all last-hop HTTP traffic, not just HTTPS) is now secured with IPSEC while it is on our LAN.

We’ll have more to say about TLS in the near future.

HTTP/2

RFC 2616 established HTTP/1.1 way back in 1999. It took many years for it to get properly adopted. Since then, there have been many attempts to improve it, like SPDY. In the end, RFC 7540 laid out HTTP/2 as the official successor, bringing many of the advantages of SPDY and similar protocols, and a lot of combined wisdom and experience to the new protocol.

Our beta service supports HTTP/2 right now.

In order to take advantage of it with a web browser, you need TLS configured. HTTP/2 can work over unencrypted connections, and although we do support that, no browser does. The default for the future of web browsing is intended to be encryption.

Automatic gzip compression

Contrary to popular belief, we’ve supported gzip encoding for a long time. The problem historically was that getting there has been a bit too tedious for most people. Delivering gzip-encoded static content requires maintaining two copies (regular and compressed) and twiddling around in .htaccess. Dynamic content is much easier; we’ve actually enabled gzip encoding for PHP by default since PHP 5.5. But still the word on the street is that we don’t have it, because when people think compression, they think mod_deflate.

We’ve never supported mod_deflate because it’s one of those solutions that is simultaneously easy and terrible. With mod_deflate, if someone requests a piece of static content and says they support gzip encoding, the server compresses the content and sends it to them. If another person requests the same content and says they support gzip encoding, the server compresses the same content again the same way, and sends it to them. Over and over, performing the same compression on the same input every time, wasting lots of resources and hurting the throughput of the server. (In testing, we found it was not unusual for requests handled this way to take longer than if no compression was used, even though the overall size is smaller.) Easy. And terrible.

Our beta service is capable of fully automatic gzip encoding of any compressible content. If someone requests a piece of static content and says they support gzip encoding, our system compresses the content and sends it to them. And then it stuffs it in a cache, so when the next person requests the same content with compression, it’s already ready to go.

Major Access Control List (ACL) improvements

ACLs (currently called IP access control in our UI) are how you decide who is or isn’t allowed on your site. People use them to block spammers and bandwidth leeches, or to limit access to their home network while a site is being developed.

First and foremost, the performance of ACLs has been dramatically improved with the new software. We greatly underestimated the degree to which some people would get carried away with ACLs. The site on our network with the largest ACL currently has over 4000 entries. That takes a lot of processing and really slows down access to that site. We could argue that such a large ACL is fundamentally unreasonable and that if using it has a performance impact, so be it. Or we could make the new system capable of processing an incoming request against that site’s ACL in 3 microseconds. We chose the latter.

At the same time, we’ve also dramatically expanded what can be included in an ACL. It’s now possible to filter inbound requests based not only on IP address (now including both IPv4 and IPv6) but also on protocol (http or https), request method (GET, POST, etc.), and URL prefix. So, as a purely hypothetical example that I’m sure won’t be of any practical interest, an ACL can now be used to block POST requests to a WordPress blog’s login script if they don’t originate from specific IP’s you know are OK without interfering with public access to the rest of the site.

Shared IP blacklists

We’ve also added the ability to filter incoming requests against a sort of giant shared ACL, a list of IPs flagged for bad behavior.

We haven’t turned this on yet, because we’d really like to include Project Honeypot’s http:bl in the list, but we’d need their cooperation to set that up, and they haven’t gotten back to us yet.

We can’t guarantee this will be effective, attacks tend to adapt and some botnets are huge, but we’re committed to finding new and better ways to keep our members’ sites safe.

Regardless of how the details shake out, this feature will be opt-in. At some point in the distant future, well after this test is over, if the shared list works really well and causes few problems, we may eventually make it the default for new sites. We’ll wait a long while on that, and then make the right decision at that time.

Websockets support

Websockets are a way to convert a web request into an efficient bidirectional pipe between a web browser and a server. They’re super handy for high-performance and interactive apps. They were very high on the list of things there was absolutely no way our infrastructure could ever support. Yesterday.

When things settle down, we’ll try to do a brief tutorial showing how to use them.

Wildcard aliases

Wildcard aliases refers to the ability to add an alias like *.example.org to your site and have all traffic for whatever name people enter (e.g. www.example.org, an.example.org, another.example.org, whatever.example.org, perfect.example.org, etc.) wind up on that site.

We’ve never supported wildcard aliases because they’re not super-common (in most cases, example.org/perfect is just as good as perfect.example.org) and because our existing system uses a hash table to speed up alias lookups; you can’t hash wildcards. The new system removes this limitation without sacrificing performance. We still don’t recommend using them unless you have a specific need, but there are a couple of use cases where there’s just no substitute. (One site which is perhaps not surprisingly no longer hosted here had 6000 aliases at the time it was deleted. That same site today could have gotten by with one wildcard alias.)

The “specific beats general” rule applies to wildcard aliases. If the site example has the alias www.example.org and the site wild-example has the alias *.example.org, requests for www.example.org will go to example not wild-example.

A caveat

Although these features now exist with the beta service, most of them aren’t reflected in the UI yet (where applicable). It seemed cruel to provide an interface to set up cool functionality that wasn’t actually available.

Now that it is, we’ll be rectifying that over the coming weeks as we refine and troubleshoot everything. In the meantime, if you want early access to one of the features listed here that requires custom configuration and you’re a subscription member, just drop us a line through our site and we’ll see what we can do.

Last words

This is at once very exciting and very daunting. The software being replaced is 15 years old and showing its age; the new features we’re bringing out are fantastic (and, in some cases, long overdue) and we couldn’t have done them with the old architecture. But on the other hand, the old software is a legitimate tough guy. It’s handled tens of billions of web requests. It built our business from nothing. We know exactly what it does under a dozen different types of DDOS attack. And here we are, replacing it.

There is absolutely, positively no way the new software is as bug-free or battle-tested as the old stuff. The latest bug logged against the existing software was a memory leak in 2009. The latest bug against the new software was fixed less than 24 hours ago. There will be problems. (Which we’ll fix.) Then there will be more problems. (Which we’ll fix.) It will inevitably crash at the worst possible time at least once. (Which we’ll fix.) And, there will no doubt be something obscure that works great on the current system but which doesn’t work on the new one that we won’t be able to fix. (But not to worry, we’ll be keeping the old one around for quite awhile.)

So this is a daunting move for us, but we’ve never made decisions based on fear and we’re not going to start now. So it’s time to push this technology out of the lab and onto the street so it can get started on its five hundred fights.

Please help us out and opt as many sites as you can into the beta, so we can test against the broadest possible cross-section of traffic and types of site content. Every little bit helps!

Thanks for your time, help, and support!

]]> https://blog.nearlyfreespeech.net/2016/02/19/unlimited_free_bandwidth_some_limitations_apply/feed/ 17 https://blog.nearlyfreespeech.net/2015/05/13/new-payment-features-methods-and-options/ https://blog.nearlyfreespeech.net/2015/05/13/new-payment-features-methods-and-options/#comments Wed, 13 May 2015 01:20:52 +0000 https://blog.nearlyfreespeech.net/?p=555 We’ve added a number of new payment features and options to our site designed to make things better for our members. This includes a new deposit form that allows arbitrary deposit amounts, the ability to choose either a specific payment or a specific deposit amount and let our system work out the fees, support for Dwolla and Bitcoin as payment methods, and the option to set up your site to accept contributions from the general public toward its hosting costs.


New payment features

Because of the admittedly complex nature of our “payment-fee+rebate=deposit” system and the number of factors that went into calculating the rebate, we have traditionally limited people to fixed deposit amounts. Thanks to a few simplifications in the rebate calculations (which have caused some deposits to become a few cents cheaper and the cost of others to go up a few cents) and some other behind-the-scenes changes, we’ve (finally!) been able to do away with that limitation. This is something that people have been asking for for a long time, and it’s frankly overdue, so we’re very happy that this feature has finally made it to production.

As a counterpart to that change, we’ve also placed the decision about how to calculate deposit fees into your hands. If you want to pay $20, have the $1.00 net fee deducted and receive $19.00 in your account, that’s the way it has always worked and it still does today. However, if you want to deposit $20.00 into your account, you now also have the option to click a button and let the system figure out that to do that, you should make a $21.03 payment. This will be very helpful for many use cases, like depositing just enough to register that domain name. (But don’t forget to account for privacy service!)

New payment methods

At long last, we have introduced support for Dwolla. Although limited to US customers, Dwolla is an interesting alternative to traditional payment processors. As they have a very favorable fee structure, we quite naturally hope they will be very popular with our members.

And, at some past point, I was caught in public stating that we not would revisit our decision not to accept bitcoin until after we had implemented arbitrary HTTP servers and we had implemented Dwolla support. Well, we now have both of those things. And thanks to the bitcoin experts at Bitpay, we now have bitcoin support as well! Bitpay not only made integration very straightforward for us, but they also assume most of the volatility risk by accepting bitcoins from you and paying us in US dollars. That combination removed most of our major objections to bitcoin acceptance and made it easy for us to change our long-held position on this.

Checks aren’t a new payment option, but they’re definitely improved. If you’ve ever sent one, you’re familiar with the onerous deposit form we make you fill out and include. Well, computers are supposed to automate routine tasks, so that form is now filled out automatically. All you have to do is click to print when you set up a mailed-in deposit and the form happens for you. Or, use the new streamlined process for online bill-pay. Either way, you’ll save some time and hassle.

Finally, on the credit card front, in addition to Visa, Mastercard, American Express and Discover, we now also accept JCB and Diner’s Club cards. Or, we assume we do. We’ve never actually seen either one to try it.

New payment options

Is your website something you do to make information available to the world? Does the world find it helpful, and often tell you that. Have you ever thought to yourself, “hey, if I’m doing all the work on this site to help/entertain others, why am I the one paying for it?”

Do you run a popular forum site with lots of users that would be more than happy to help pitch in to cover the costs, but maybe you don’t want 1000 people sending $2.00 checks to your house?

Or, are you a web developer who hosts sites with us for clients who want to pay their own bills, but for whom the responsibility and technical requirements of their own NearlyFreeSpeech.NET membership are not a good fit?

We now have the ability to let you allow the public (or a subset of the public) to make contributions to a site’s hosting costs through our service, without those contributors having to have a NearlyFreeSpeech.NET membership.

This is something we’ve always wanted to do, but we’ve only ever done it poorly. We’ve offered ad-hoc options for accepting donations via PayPal to a few sites, and we’ve had a form on our site to allow anyone to mail-in a check. That’s all done with. As of today, sites that apply to accept contributions (and that are accepted), can send their contributors to a special page on our public site that will let them use any of the payment options we support. Their contributions will go directly into your account to help fund your site.

The contribution process also has privacy built into its very core. The contributor identifies themselves to us, so we can process their payment, and you identify yourself to us so we can provide you hosting. But we do not identify them to you, nor you to them. During the process, they have the option to send you a brief message which they can, if they choose, use to identify themselves to you. Each contribution will also be assigned a unique ID code provided to both parties. The name of the site receiving the contribution, the ID code, the amount of the contribution, and the contributor’s message (if any) will be the only information shared between the contributor and the site operator.

Accepting payments from the general public is fraught with risk, and we need to make sure this feature isn’t abused or used as an eCommerce substitute, so there is naturally a solid page of fine print associated with accepting contributions for your site, and it will always be at our discretion whether or not to accept contributions for a particular site. If you’re interested in finding out more about this capability, start here.

In addition to the contribution feature, we’ve also made it easier than ever to transfer funds between NearlyFreeSpeech.NET memberships. If you know someone’s account number, you can transfer funds from your account to theirs instantly by using the “Transfer Funds Between Accounts” action on the Accounts tab in our member interface. Just like always, you can also use this option to transfer funds between your own accounts; it’s just easier and prettier now.

Perhaps this stuff is not as exciting as adding features or upgrading our equipment, and in some cases it required a ton of work to rebuild what we already had — work no one will ever see — in order to make a better foundation for the new stuff. But it really was a ton of work to build and test all of this, so we’re thrilled that it’s ready and really hope our members will find the new functionality helpful and get good use out of it!

]]> https://blog.nearlyfreespeech.net/2015/05/13/new-payment-features-methods-and-options/feed/ 13 https://blog.nearlyfreespeech.net/2015/03/03/upcoming-updates-upgrades-and-maintenance/ https://blog.nearlyfreespeech.net/2015/03/03/upcoming-updates-upgrades-and-maintenance/#comments Tue, 03 Mar 2015 23:45:24 +0000 https://blog.nearlyfreespeech.net/?p=545 We have accumulated some housekeeping tasks that we’ll be taking care of over the next couple of months. They’re all necessary things to make sure our service keeps running at its best, and though we work hard to prevent these types of things from impacting services, occasionally they do intrude. As a result, we want to let everyone know what we’re up to and what the effects will be.


Retiring file server f2

We still have quite a few sites using the file server designated as “f2.” This is the oldest file server still in service, and although it has been a great performer for many years, it is reaching the end of its useful life. It is also one of two remaining file servers (and the only one that holds member site files) that has a single point of failure. Our newer file servers use different technology; they are faster (100% SSD), have no single points of failure, allow hardware maintenance while they are running, and allow us to make major changes (like adding capacity or rebalancing files) behind the scenes without you having to change the configuration of your site.

So, we are quite anxious to get rid of f2. We’ve been offering voluntary upgrades for some time now, but it’s time to move things along. We’ve set an upgrade date and time for every site on f2 in April. If you have a site on this file server, you can see your upgrade time in our member interface and, if it doesn’t suit you, upgrade at any earlier time or postpone it closer to the end of April.

Please note, the file server f2 is distinct from and has no relation to site storage nodes that contain the text fs2. If your site’s file storage tag contains fs2, you are not affected by this.

Migrating a site does entail placing it into maintenance mode briefly, for a period proportional to the size of the site. Beyond that it usually has no ill effects. Some sites do have complications, especially if they have hardcoded paths in their .htaccess files. After our system migrates your site, it will attempt to scan the site for affected files and send you an email listing them if it finds any. This isn’t 100% foolproof, but we previously did it for a lot more sites under considerably greater pressure with the f5 server, and problems were relatively few and far between.

Discontinuing PHP Flex

As part of our continued (slow) migration away from Apache 2.2, we will be discontinuing PHP Flex. PHP Flex refers to running PHP as a CGI script, which is a terrible way to do things. In the bad old days, it was useful in some cases for compatibility with PHP applications that didn’t work with safe_mode, if you didn’t mind the horrible performance. But, even in the bad old days, it mostly ended up being used not because it was necessary, but because it was easier than dealing with safe_mode.

These days, PHP safe_mode is long gone, so there’s no real reason to have PHP Flex anymore. Our new PHP types are highly compatible with (and much faster than) PHP Flex, and most people have already happily upgraded. However, there are still some stragglers out there and, as time goes by, they are starting to have problems. Those problems often completely go away simply by switching to a currently-supported version of PHP. Thus, we feel it’s time to phase out PHP Flex. In the month of April, we will auto-migrate PHP Flex sites (which mostly run PHP 5.3 and in some cases 5.2) to PHP 5.5.

MySQL software upgrades

We are currently working on both long-term and short-term upgrades for MySQL. In the short term, we need to perform a series of OS and MySQL server updates on existing MySQL processes to keep them up-to-date and secure. This will require either one or two brief downtimes for each MySQL node, typically about 5-10 minutes. We will be performing these updates throughout the month of March, and we will announce them on our network status feed (viewable on our site and Twitter).

In the long term, MariaDB 5.3 is getting a bit long in the tooth, so we are working to jump straight to MariaDB 10 and all its great new functionality, as well as offering better scalability and configuration flexibility. This is likely to be somewhat more resource intensive, and hence more expensive, so it will be optional for people who are perfectly happy with the way things are. (If you like your MySQL plan, you can keep it!) More on this as it gets closer to release.

Physical maintenance

We also need to do some maintenance on the power feeds to one of our server shelves. Ordinarily that isn’t an issue that affects our members, but in this case it’s being converted between 120V and 208V. Hypothetically that can be done while the equipment is running, but doing so entails a nonzero risk of death by electrocution and after careful consideration we’ve decided that none of the current field techs are expendable at this time. Also, it could burn down the datacenter. So, we’re going to go ahead and do it by the book, which means shutting it off.

That’s a few dozen CPU cores and hundreds of gigs of RAM we need to take offline for a little while. In a real disaster, our infrastructure could survive, but there would be a period of degraded service while things balance out on the remaining hardware. That period would be significantly longer and affect significantly more people than the actual maintenance. So, we feel our best course of action is just to shut it off for the few minutes it will take to rewire the power feeds. The service impact should be low, but will probably not be zero.

We want to complete the MySQL maintenance listed above first, so we are likely to do this toward the end of March. We will post updates on our network status feed with more precise timing as we get closer.

Realm upgrade reminder

We have finally finished rolling sites off of the dreaded “legacy” realms (freebsd6, freebsd72, and 2011Q4). Every site is now on a “color” realm. This means that, for people who have selected late realm upgrades for their site in our UI and who are currently running on the red realm, they will receive an automatic upgrade to violet in April, after quarterly realm rotation has occurred. Compatibility between the two is excellent and we anticipate very few problems.

That’s all for now. All in all, the upgrades and maintenance shouldn’t affect too many people, but we regret and apologize in advance for any problems they do cause. These steps are part of a process designed to eliminate some very old stuff that causes stuff like this to be intrusive. In other words, the goal is to do this maintenance is in large part so that the next time we do it, you’ll be even less likely to notice.

Thanks for reading!

]]> https://blog.nearlyfreespeech.net/2015/03/03/upcoming-updates-upgrades-and-maintenance/feed/ 6 https://blog.nearlyfreespeech.net/2014/12/02/domain-registration-updates/ https://blog.nearlyfreespeech.net/2014/12/02/domain-registration-updates/#comments Tue, 02 Dec 2014 22:50:54 +0000 https://blog.nearlyfreespeech.net/?p=514 We have a few small updates to announce with respect to domain registration. There will be some small price changes, both up and down. We are also adding support for registering some of the new competitive gTLD’s. Finally, for those who choose our RespectMyPrivacy service, it will now be automatically prepaid (including the 10% prepayment discount) during all registrations and renewals.


Domain registration pricing changes

In the past, our system has always limited us to charging the exact same amount for a domain registration, regardless of TLD. But actually TLD prices vary widely, so that not only meant that we weren’t always able to give the best price, but also, a lot of higher-priced TLDs were simply not feasible to add. That limitation has been removed.

The main effect is that the prices for some TLD’s (biz, info, and org) will be going up a little, but prices for others (com, net, and name) will be going down a little. Here’s a quick table:

TLD	Before	After
com	$9.49	$9.34
net	$9.49	$9.34
org	$9.49	$9.89
biz	$9.49	$10.29
info	$9.49	$9.79
name	$9.49	$8.99

Across the entire portfolio of our members’ registered domain names, this is a net decrease in cost, largely because com is so much more popular than everything else. And those price decreases will be live today. However, we don’t want to spring price increases on people without notice, so those won’t take effect until January 1st. If you’ve got any of those, get your renewals in now!

This change will also let us track various “promotions” that some gTLDs run fairly often on new registrations. (Mainly biz, info, and org.) We’re not a huge fan of these “deals;” they never apply to transfers, renewals, or multiple-year registrations so it seems like they only exist to lure people in with artificially low first-year prices. But, still, if we can go get a lower price for our members, why wouldn’t we? For example, through the end of the year, new 1-year registrations in biz can be had for $3.49 and new 1-year registrations in org are $4.99. If that’s something you were going to do anyway, congratulations, we found you a few extra bucks.

New gTLDs

We’re also adding support for three additional gTLDs: .click, .club, and .guru. This is sort of an experiment. Of the dozens of competitive gTLDs that have been introduced in the past year or so, we picked these three for two key reasons:

A) They were easy to implement.
B) According to registration statistics, they are relatively popular.

In principle, we’re huge fans of gTLD diversification. The .com TLD is kind of a lost cause. Not only is it getting pretty crowded, it is run by Verisign, and ICANN gave them a sweetheart contract that auto-renews into perpetuity and allows significant price increases at regular intervals that require no technical or business justification. Things are not going to improve over time in .com. GTLD diversification is really the only way to compete with that, by breaking the “.com = the Internet” default mindset. And they should be able to compete vigorously on price and quality.

In practice, that hasn’t worked out yet. Most of the new gTLDs have been backed by speculators who seem to be in a race to see how ridiculous they can be. True story: there is a .luxury TLD that is almost $500/year. With a few exceptions (.click seems pretty low-cost at $6.59), they’ve all got some idea of why they’re worth vastly more than regular domains. We think that with a few exceptions, they’re probably wrong about that. But it’ll take some time, possibly a couple of years, for the operators of most of those gTLD’s to figure out that they’re not one of the exceptions. Once that happens, we anticipate the cost level of competitive gTLD registrations will fall sharply; there’s no reason a general-purpose “domains for the rest of us” gTLD couldn’t operate profitably charging $5/year or less. But we’ve been wrong before, and the stakes to enter the gTLD game are a cool $500,000, so don’t hold your breath.

Still, we’re sticking our toes in the water to try to help things along. If this works out, we’ll rapidly add a lot more gTLD’s. Pretty much anything that’s not too much hassle and doesn’t make that weird “huhuhuh” laugh come out when we look at the price. Feel free to campaign for your favorite in the comments!

To avoid raising any hopes, I want to make clear that we still don’t have any immediate plans to support in-house registration of ccTLDs (e.g. .uk and .de). They present both legal and technical problems that simply don’t exist with the new gTLDS. We fiddle with .us from time to time, since the hassles there are only technical, but we were not able to add it for this update.

Expanding prepaid privacy

For most of our services, pay-per-day makes a lot of sense: they can be added and removed at any time. But pay-per-day privacy service isn’t a good fit for pay-per-year domains, and that occasionally leads to problems.

Sometimes, people register a domain with privacy and then forget about the privacy. Likewise, a lot of people are really confused because privacy service doesn’t go away when the domain expires. Instead, it hangs around for about 75 more days, until the deletion date, since that’s how long a domain’s information remains publicly visible. And, on top of those two things, privacy is the only service intentionally allowed to overdraft member accounts.

The interaction between these factors can (and, unfortunately, sometimes does) lead to what we could euphemistically call a bad member experience, and it’s one of a few specific things about our service that contribute to the creation of angry ex-members.

To combat that, we have long offered the option to prepay RespectMyPrivacy service on domains in exchange for a 10% discount. It generally doesn’t benefit us for people to pay in advance, which is why most services don’t offer that type of discount, but in this case, giving up that 10% is worth it to prevent the chance of overdrawn accounts and the occasional nasty comments about our character that sometimes result. But anecdotal reports indicate that relatively few people know about that option.

As a result, we’re automatically going to apply privacy prepayment, with the discount, during all future registrations and renewals of domains that use our privacy service. We will soon expand that to prepaying privacy on auto-renewals and transfers as well. If you have a domain that isn’t currently prepaid, nothing will change until you renew it. (But the option to log in and manually prepay is still there if you want to save the money.)

That’s similar to what other providers do. Less similar to what they do, privacy service will remain fully pro-rata refundable if you remove it from a domain or if you transfer the domain away prior to its deletion. So this is mostly a price cut and a convenience increase at the cost of a slight loss of flexibility that has limited utility and a nasty habit of blowing up in our faces.

]]> https://blog.nearlyfreespeech.net/2014/12/02/domain-registration-updates/feed/ 14 https://blog.nearlyfreespeech.net/2014/09/24/more-power-more-control-more-insight-less-cost/ https://blog.nearlyfreespeech.net/2014/09/24/more-power-more-control-more-insight-less-cost/#comments Wed, 24 Sep 2014 19:51:35 +0000 https://blog.nearlyfreespeech.net/?p=452 We’ve made some very big changes to the fundamental nature of our service. We now support persistent processes and delegating requests for your site to those processes using HTTP, SCGI, or FastCGI. We’ve added the ability to graph up to two weeks of your site’s resource usage. And we’ve slashed resource charges by almost 70%.


More power & more control.

Ever since we started, one of the inviolable limitations of our service is that we don’t let you run your own persistent processes. No more! Using the technology underpinning our NFGI PHP implementation, we’ve added support for configuring and controlling persistent processes from our member interface. These processes run as part of your web site, and when you’re not around, our system takes care of them. For example, it keeps an eye on them and can restart them if they exit.

There are two main uses for this feature.

First, although we support a huge variety of languages for web development, unless your favorite language is spelled PHP then you were stuck running it as a CGI script. For any kind of modern web framework, that causes some substantial performance problems. Ruby on Rails, Django, Catalyst, Node.JS, Network.Wai? Sure, technically you can run them here, but it’s not what you’d call a good idea.

Take that long-standing well-known fact about our service, and pitch it straight into the trash. We’ve built a new server type that can delegate incoming requests to the web application server of your choice running as a persistent process. This currently supports HTTP, FastCGI, or SCGI. (Although it’s not clear that anyone actually uses SCGI.) You can even mix-and-match technologies on different paths of the same site. Or with HTTP, you have the option to take advantage of our Apache-based infrastructure or to bypass it entirely for maximum speed and control. (Either way our edge network will still have your back, accelerating static content and protecting you from a variety of web ne’er-do-wells, so you can focus on your app.)

Second, there’s no requirement that your persistent process be a web app. Although nothing but web apps will be accessible from outside our network, from inside our network, you have a lot more flexibility. Want a private memcached instance to turbocharge your site? You got it. Redis? Check. PostgreSQL? Now possible.

This feature is going to go through a very brief open beta period for a couple of weeks; we want a bit of a slow start because trying to anticipate all the creative things our members will do is simply impossible, so we want to keep a close eye on the first handful of setups to see if anything needs tweaking.

During the beta, persistent processes will only be available on the proxy site type. If that’s not what you need, but you still want to try this out, you can always set up a process on a proxy site and talk to it from another site hosted here. Or you can just set up PHP-FPM.

To get in on this now, just log in to our member interface and submit an assistance request. We’ll convert the site(s) of your choice and you can take it from there; These features are fully implemented and integrated into the UI. Just start by setting the server type of your site to “Custom” or “Apache 2.4 Generic” and the Proxy and Daemon management options will appear on the Site Information Panel.

As this comes out of beta, watch our blog for a series of tutorial articles demonstrating how to use this feature in conjunction with various technologies.

Although it might seem like it at first glance, this does not turn our service into a VPS. We (still) have no interest in going down that road. This feature is immensely powerful, but it doesn’t include root access and can’t make anything but web services available over the Internet. If you want to run an email server, voice chat server, or game server, you’ll still need a VPS for that.

This is designed as an alternative for people running web sites who might have felt forced to move to a VPS or dedicated server in the past to get more flexibility (at the cost of all the 24×7 administrative headaches that come with it). Like all of our services, it’s pay-as-you-go and fully managed, which means that we’re the ones keeping all that infrastructure up-to-date, secure, tuned, and running. You just do what you do. We’ll take care of the rest.

More insight.

One of our most common requests that has always had a technical barrier is getting you better visibility into the resource usage of your site. We’ve now been able to add an interactive Javascript chart that shows up to two weeks of RAM & CPU usage for most currently-supported server types (all the Apache 2.4 + PHP site types and the new proxy-based server type).

As part of this change, we’ve introduced a new internal tracking system that can take resource measurements on a per-site basis for a bazillion tiny sites without collapsing under its own weight. Naturally we’ve applied that directly to our resource-based billing. So if the random sampling element of our resource billing made you uncomfortable, just make sure your site is using one of the above types, and your usage will be calculated using the new, deterministic method.

In the future, we hope to expand on this functionality to gather and report more statistics about your site that will help you explore and optimize your site’s resource usage.

Less cost.

Our resource-based billing has been a huge success. Between the economies of scale associated with the popularity of this model and the slow decline in the cost of hardware, we’ve been able to greatly reduce what we charge for resources. The cost of a resource accounting unit (RAU) has dropped from 3.25 RAUs = $0.01 to 10.59 RAUs = $0.01. That’s almost a 70% drop. You don’t need to do anything to take advantage of this; it’s automatically applied to all resource-billed sites. This change means that over 2/3rds of resource-billed dynamic sites (and well over 99% of static sites) now pay less than $0.01/day for their resource usage.

These are all great changes. They will make more power available to more people for less money. But these changes are all steppingstones. Each one can be improved, extended, enhanced. And that’s not all. We’ll have more good stuff for you in the coming months. But now, we get to my favorite part, where we hand this stuff to our members and see the amazing things they’ll do with it.

Updated 2014-11-16: The daemon and proxy features went out of beta a few weeks ago with almost no fanfare. We’ve updated this entry to reflect the non-beta setup process; also be sure to check out the first in our how-to series about these features, which covers Django.

]]> https://blog.nearlyfreespeech.net/2014/09/24/more-power-more-control-more-insight-less-cost/feed/ 24 https://blog.nearlyfreespeech.net/2014/08/16/improving-support-and-how-we-talk-about-it/ https://blog.nearlyfreespeech.net/2014/08/16/improving-support-and-how-we-talk-about-it/#comments Sat, 16 Aug 2014 03:45:59 +0000 https://blog.nearlyfreespeech.net/?p=435 Looking back, it’s hard to believe we’ve been using our new subscription-based support system for almost eight months. Wow, time flies. Overall, we’ve been very happy with this change. We really enjoy getting to spend time working with our members and helping them be successful with our service. The response from the people who’ve used it has also been almost uniformly very positive. So, right now, we feel like we finally got it right. Which isn’t to say that we can’t do even better.

It’s hard even to describe how fantastic it is for support to (finally) be self-financing like the rest of our service. It’s good for everybody, even people who don’t ever use it, because support no longer takes time and money away from other areas. To give just one example, it’s enabled us to put more sysadmin time into managing software upgrades, so we have more realms, more software packages, and upgrades make it to you quicker. Case in point: the PHP developers released PHP 5.3.29, a final post-EOL release of PHP 5.3 (which we salute them for doing) yesterday. It went live on our system several hours ago. That used to take us two weeks.

A rose by another name…

We have found that there’s still room for improvement, though, particularly in terms of how we talk about support with people who don’t understand the subscription model. The best way we’ve found to explain how it works is this: there is a choice inherent in our system. If you want individual support, like other hosting companies, we offer that for a small monthly fee, like other hosting companies, and the service we provide is a very good value. If you (like most of our members) are comfortable supporting yourself or using our community-based support options, you can get a substantial discount by foregoing the support.

Problem is, that isn’t how we were explaining it. So we’re revamping how we talk about subscription-based support quite a bit, though we’re making very few actual changes to how it works. We’re starting with the name. The “support subscription” is now called the “subscription membership.” If you don’t have one, you have a “baseline membership.”

Rather than having support as an optional add-on to your membership, we’ve decided to present the “support or not” choice as two types of memberships: baseline and subscription. The baseline membership is exactly what everyone is used to: a membership where the only costs are based on the hosting resources used, and support is available through the forum, FAQ, and wiki. The subscription membership has a $5.00 setup fee the first time, costs up to $5.00 per month, and includes individual support through email and our site.

So, basically, exactly the same as before except we’re presenting it as an either-or choice between two options that include different things rather than one plan with an optional add-on. This seems to be a lot more familiar to a lot of people, as it’s commonly not just by hosting companies, but also all sorts of software products and services. It also lets us present the two choices in an easy-to-understand table.

But don’t worry, we’re still us and it’ll still be a cold day in hell before we start offering “Gold,” “Silver,” and “Bronze” packages and filling up that table with lots of “UNLIMITED,” exclamation points!!!, and misleading monthly costs based on paying for thirty years in advance. We just want the people coming to us from those companies to better understand what they are (or aren’t) getting from us.

We’ve also changed how we describe the term of subscription membership. With support subscriptions, there was a minimum term of five months. With subscription memberships, there is a one-time fee for switching back to baseline membership that decreases over time, reaching zero after five months. These are almost identical. The only difference is that with a support subscription, a person who wanted to get rid of it had to make a note on their calendar to log in after the five months and switch it off. Subscription memberships can be switched off at any time. Importantly, subscription members who pay the switch-back fee still get the benefits of subscription corresponding to what they’re paying. So both plans end up costing the same and providing the same benefits for the same length of time. Just a big change in description, and a minor improvement in convenience.

The only material change we’ve made affects only people who have a baseline membership, switch to a subscription membership, switch back to a baseline membership, and then switch to a subscription membership again. The cost to set up the first subscription will be the same as before, $5.00. The cost to set up the second subscription, however, will range from $0.00 to $25.00 based on how long they were unsubscribed. The purpose of that is very simple and we’re not going to try to pretend it’s anything other than what it is: a way to ensure that unless that person waits a very long time between subscriptions, they were better off staying subscribed. Support subscriptions were never intended to be per-incident support, and the same is true of subscription memberships.

(We remain uninterested in providing per-incident support; we’ve learned our lesson there. The subscription plan is definitely a case where we did the right thing after exhausting all the other options.)

Making life better for baseline members

Aside from support, there are a few things that come up where people need us to do stuff. This includes things like recovering a domain that’s gone into redemption, restoring files that have been deleted, or having us generate an SSL certificate for you. After we introduced subscription-based support, people who weren’t subscribers had a rough time getting access to that stuff, and the workarounds we came up with were, charitably, suboptimal. (“Open an unpaid support ticket and then post the issue number on the forum so we can find it.” Ugh.)

To address that, we’ve expanded our Assistance Request feature. These new assistance requests aren’t free, but they outline the costs right in the description and are available to all members. There is still an advantage for subscription members, though. Since they’re already funding our ability to have people there to interact with them through their subscription fee, they get almost all of these at lower or no cost.

This will make things a lot easier for baseline members who run into those weird edge cases where they need something specific from us, but for which a support subscription is not appropriate. If we find more things that fit that bill, we’ll add them.

Cleaning up our support tab

The other major issue we’ve found was the support tab of our UI. There’s a word that is often applied to well-designed interfaces: discoverability. It refers to how easy it is to figure out how the interface works just by browsing around. The old support tab lit a poop-filled bag on discoverability’s front porch and ran off. We have a bunch of different support options to choose from and there’s almost no overlap so it’s easy to get it wrong. And effectively all of them were at least two clicks from the support tab, often hidden behind labels or link text that, sure, it’s relevant. (If you squint a little.) This worked fine for the (admittedly large) percentage of our membership that was born with genetic knowledge of how hosting works. But for people who were already having a rough time and were looking for help, it tended to kick them while they were down. Which is not cool. So we’ve cleaned that up.

The new support tab does three basic things:

• It lists your open and recent issues (if any).
• It briefly explains the difference between baseline and subscription membership and tells you which one you are and how to change.
• It enumerates and directly links to all the support options available to you and, if you don’t know which one to use, gives you a specific recommendation about where to start based on what type of membership you have.

We’ve separated our system status out to its own page, which shows both open system issues and anything from our Twitter feed that’s marked as a network status update. (#status) We’ve also added notification about open system issues to the main member page so you’ll see them right when you log in.

It’s our hope that the changes to our site will make support easier to find and obtain, and that the changes to how we talk about support will make the cost vs. support tradeoff easier to understand, and help each member feel like they’re choosing the right option for them.

]]> https://blog.nearlyfreespeech.net/2014/08/16/improving-support-and-how-we-talk-about-it/feed/ 8