We’ve been hard at work behind the scenes developing the next-generation of our core hosting technology, and we’re ready to move it to public testing. It has some exciting new features:
- TLS enhancements
- HTTP/2 support
- Automatic gzip compression
- Major Access Control List (ACL) improvements
- Shared IP blacklist support
- Websockets support
- Wildcard alias support
To encourage people to help us test out the new stuff, we’re exempting participating HTTP requests from bandwidth charges for the duration of the test. You can opt-in to the test for a particular site by selecting the “Use Free Beta Bandwidth” action on the Site Information panel for that site in our member interface. That page has all the fine print about the test, which mostly cover two central points:
- Reminding people that it is a test and things might not work.
- Clarifying that although there is no fixed limit to the amount of bandwidth a site can use under this test, there is a “floating” limit: don’t cause problems.
This test (and the free bandwidth) will run through at least March 15th, 2016.
Below, we’ll also discuss each new feature briefly.
The major enhancement to TLS (transport layer security, the technology that makes http:// URLs into https:// URLs) has to do with scalability. As people may know, we currently use Apache as a front-end TLS processor. As a test, we generated test keys and certificates for every site we host and loaded them all into a single Apache config just to see what would happen. The resulting server process took nine minutes to start and consumed over 32 GiB of RAM before answering its first request. That’s… not going to work. So we’ve written a great deal of custom code to solve that problem.
We’ve also always been worried that the overhead of TLS would require us to charge more for using it. One side-effect of this work is that we’ve reduced the fixed resources required to support TLS so much that we can now definitively say that that won’t be an issue.
The new system also improves the performance of TLS requests, and made a couple of other changes we were able to backport to the existing setup. First, we’ve eliminated TLS as a single point of failure. Second, due to our use of Apache as a TLS frontend, the last hop of an HTTPS request is handled as unsecured HTTP on our local LAN. Although the probability of anyone monitoring our local LAN without our knowledge is pretty small, in a post-Snowden world one has to acknowledge that taking reasonable precautions against improbable things isn’t as paranoid as it used to be. So last-hop HTTP traffic (all last-hop HTTP traffic, not just HTTPS) is now secured with IPSEC while it is on our LAN.
We’ll have more to say about TLS in the near future.
RFC 2616 established HTTP/1.1 way back in 1999. It took many years for it to get properly adopted. Since then, there have been many attempts to improve it, like SPDY. In the end, RFC 7540 laid out HTTP/2 as the official successor, bringing many of the advantages of SPDY and similar protocols, and a lot of combined wisdom and experience to the new protocol.
Our beta service supports HTTP/2 right now.
In order to take advantage of it with a web browser, you need TLS configured. HTTP/2 can work over unencrypted connections, and although we do support that, no browser does. The default for the future of web browsing is intended to be encryption.
Automatic gzip compression
Contrary to popular belief, we’ve supported gzip encoding for a long time. The problem historically was that getting there has been a bit too tedious for most people. Delivering gzip-encoded static content requires maintaining two copies (regular and compressed) and twiddling around in .htaccess. Dynamic content is much easier; we’ve actually enabled gzip encoding for PHP by default since PHP 5.5. But still the word on the street is that we don’t have it, because when people think compression, they think mod_deflate.
We’ve never supported mod_deflate because it’s one of those solutions that is simultaneously easy and terrible. With mod_deflate, if someone requests a piece of static content and says they support gzip encoding, the server compresses the content and sends it to them. If another person requests the same content and says they support gzip encoding, the server compresses the same content again the same way, and sends it to them. Over and over, performing the same compression on the same input every time, wasting lots of resources and hurting the throughput of the server. (In testing, we found it was not unusual for requests handled this way to take longer than if no compression was used, even though the overall size is smaller.) Easy. And terrible.
Our beta service is capable of fully automatic gzip encoding of any compressible content. If someone requests a piece of static content and says they support gzip encoding, our system compresses the content and sends it to them. And then it stuffs it in a cache, so when the next person requests the same content with compression, it’s already ready to go.
Major Access Control List (ACL) improvements
ACLs (currently called IP access control in our UI) are how you decide who is or isn’t allowed on your site. People use them to block spammers and bandwidth leeches, or to limit access to their home network while a site is being developed.
First and foremost, the performance of ACLs has been dramatically improved with the new software. We greatly underestimated the degree to which some people would get carried away with ACLs. The site on our network with the largest ACL currently has over 4000 entries. That takes a lot of processing and really slows down access to that site. We could argue that such a large ACL is fundamentally unreasonable and that if using it has a performance impact, so be it. Or we could make the new system capable of processing an incoming request against that site’s ACL in 3 microseconds. We chose the latter.
At the same time, we’ve also dramatically expanded what can be included in an ACL. It’s now possible to filter inbound requests based not only on IP address (now including both IPv4 and IPv6) but also on protocol (http or https), request method (GET, POST, etc.), and URL prefix. So, as a purely hypothetical example that I’m sure won’t be of any practical interest, an ACL can now be used to block POST requests to a WordPress blog’s login script if they don’t originate from specific IP’s you know are OK without interfering with public access to the rest of the site.
Shared IP blacklists
We’ve also added the ability to filter incoming requests against a sort of giant shared ACL, a list of IPs flagged for bad behavior.
We haven’t turned this on yet, because we’d really like to include Project Honeypot’s http:bl in the list, but we’d need their cooperation to set that up, and they haven’t gotten back to us yet.
We can’t guarantee this will be effective, attacks tend to adapt and some botnets are huge, but we’re committed to finding new and better ways to keep our members’ sites safe.
Regardless of how the details shake out, this feature will be opt-in. At some point in the distant future, well after this test is over, if the shared list works really well and causes few problems, we may eventually make it the default for new sites. We’ll wait a long while on that, and then make the right decision at that time.
Websockets are a way to convert a web request into an efficient bidirectional pipe between a web browser and a server. They’re super handy for high-performance and interactive apps. They were very high on the list of things there was absolutely no way our infrastructure could ever support. Yesterday.
When things settle down, we’ll try to do a brief tutorial showing how to use them.
Wildcard aliases refers to the ability to add an alias like *.example.org to your site and have all traffic for whatever name people enter (e.g. www.example.org, an.example.org, another.example.org, whatever.example.org, perfect.example.org, etc.) wind up on that site.
We’ve never supported wildcard aliases because they’re not super-common (in most cases, example.org/perfect is just as good as perfect.example.org) and because our existing system uses a hash table to speed up alias lookups; you can’t hash wildcards. The new system removes this limitation without sacrificing performance. We still don’t recommend using them unless you have a specific need, but there are a couple of use cases where there’s just no substitute. (One site which is perhaps not surprisingly no longer hosted here had 6000 aliases at the time it was deleted. That same site today could have gotten by with one wildcard alias.)
The “specific beats general” rule applies to wildcard aliases. If the site example has the alias www.example.org and the site wild-example has the alias *.example.org, requests for www.example.org will go to example not wild-example.
Although these features now exist with the beta service, most of them aren’t reflected in the UI yet (where applicable). It seemed cruel to provide an interface to set up cool functionality that wasn’t actually available. 🙂
Now that it is, we’ll be rectifying that over the coming weeks as we refine and troubleshoot everything. In the meantime, if you want early access to one of the features listed here that requires custom configuration and you’re a subscription member, just drop us a line through our site and we’ll see what we can do.
This is at once very exciting and very daunting. The software being replaced is 15 years old and showing its age; the new features we’re bringing out are fantastic (and, in some cases, long overdue) and we couldn’t have done them with the old architecture. But on the other hand, the old software is a legitimate tough guy. It’s handled tens of billions of web requests. It built our business from nothing. We know exactly what it does under a dozen different types of DDOS attack. And here we are, replacing it.
There is absolutely, positively no way the new software is as bug-free or battle-tested as the old stuff. The latest bug logged against the existing software was a memory leak in 2009. The latest bug against the new software was fixed less than 24 hours ago. There will be problems. (Which we’ll fix.) Then there will be more problems. (Which we’ll fix.) It will inevitably crash at the worst possible time at least once. (Which we’ll fix.) And, there will no doubt be something obscure that works great on the current system but which doesn’t work on the new one that we won’t be able to fix. (But not to worry, we’ll be keeping the old one around for quite awhile.)
So this is a daunting move for us, but we’ve never made decisions based on fear and we’re not going to start now. So it’s time to push this technology out of the lab and onto the street so it can get started on its five hundred fights.
Please help us out and opt as many sites as you can into the beta, so we can test against the broadest possible cross-section of traffic and types of site content. Every little bit helps!
Thanks for your time, help, and support!
Sorry, the comment form is closed at this time.