A small billing error corrected

A few days ago, we noticed a tweet from one of our members mentioning that he had been charged twice in one day for his MySQL process, but that he didn’t plan to look into it. We, however, did. What we eventually uncovered was that for three of our services: MySQL, RespectMyPrivacy, and email forwarding, our system could hit a very rare case where it would “stutter” and double-bill somebody’s daily charge.

Of course, the first thing we did was turn off the billing on those services until the problem could be identified and fixed. So, pretty much everybody using MySQL, email forwarding, or RespectMyPrivacy received a day or two free this week.

In the case of MySQL, each charge “advanced the clock” by one day, so if a person got double-billed one day, they wouldn’t be billed at all the next; they were “early charged” rather than “overcharged.” Email and privacy work a little differently, however, and so people in a few cases did get double-billed for an extra penny. Fortunately, we do keep detailed billing records for active accounts, and were able to find all the cases where this had ever happened and over the past couple of days we have credited the accounts funding all affected email and privacy services. It turned out to be easier to credit both the original charge and the duplicate, rather than just the duplicate, so we did that.

A small minority of our members’ accounts were affected, and the amounts involved were fairly small (the average credit issued, which includes both the original charge and the duplicate, about 3 cents per incident). All told, we were off by around $56 over five years.

About $12 of that was related to accounts that no longer exist. As we do not have the ability to identify who those people were (nor would there be any viable way to refund a few cents to them even if we could), that amount will be included in our monthly donation to the EFF.

It may be tempting to blow this off due to the small amount of money involved. The original Twitter poster who mentioned it didn’t seem too concerned. But we did not blow it off. Being able to bill accurately is the cornerstone of our service. We take it incredibly seriously and no discrepancy like this, however small, will be tolerated.

We’re sorry this happened. To make it right, we’ve fixed the cause, credited the difference, and because we believe in transparency, we’re letting everyone know about it. And we appreciate the Twitter poster for bringing it to our attention.

Scheduled maintenance November 22 and December 15

We are scheduling two maintenance windows in the next month to move some equipment:

Date: November 22nd, 2010
Window: 9am to noon UTC (4am to 7am US Eastern, 1am to 4am US Pacific)
Affecting: MySQL nodes m2, m3, and m21

Date: December 15th, 2010
Window: 8am to 1pm UTC (3am to 8am US Eastern, midnight to 5am US Pacific)
Affecting: File servers f2 and f5

Each server should be offline for about one hour, not the whole window. This will cause some downtime. While the MySQL nodes are offline, those MySQL processes hosted on them will be unreachable. While the file servers are offline, sites hosted by those file servers will show an official maintenance page.
Continue reading Scheduled maintenance November 22 and December 15…

Removing deprecated IP block

Many years ago, we were assigned the IP address block 64.238.220.0/23 by one of our upstream network providers. We officially deprecated the use of that block way back in 2008, and we will be returning it on December 1st, 2010, so it will not work after that point.
Continue reading Removing deprecated IP block…

Brief Network Maintenance July 20-22

This is just a quick announcement about some upcoming network maintenance.

Due to our load-balancing capabilities, most of this will be done with no disruption to our services. There are a few exceptions, though. We’ll be doing maintenance over the next few days in the early morning hours (between 1am and 5am US Eastern time — 5am and 9am UTC), the following services will be briefly disrupted (should be about 5-10 minutes each):
Continue reading Brief Network Maintenance July 20-22…

Domain registration price increases

Verisign has pricing increases going into effect on July 1st. As a result of these pricing changes, our registrar partner has informed us of corresponding increases which we must, in turn, pass along.

Effective July 1st, the new price for domain registrations, renewals, and transfers will increase from $8.59/year to $8.99/year.

Get your domain renewals in now to avoid the increase.

These changes have nothing to do with us, are not under our control, and are of no benefit to us. They’re bad for us, and they’re bad for you. Unfortunately, the same (auto-renewing into perpetuity) contracts with ICANN that grant Verisign a monopoly over .COM, .NET, and .NAME domains (among others) enshrine Verisign’s right to unilaterally raise prices by 7% every year, and they never miss a chance to do so. Their 30% profit margin in 2009 will almost certainly increase after they finish unloading everything not domain-related onto Symantec. It’s not bad work if you can get it. (And live with yourself afterward.)

Pools: Arbitrary HTTP Servers, Resource Reservation and Scalability

We are pleased to announce that we are beginning the beta of our new “pools” service. Pools are a way to reserve memory and CPU power for one or more web sites. This approach makes it possible to discard many of the limitations traditionally associated with our service.
Continue reading Pools: Arbitrary HTTP Servers, Resource Reservation and Scalability…

A RespectMyPrivacy discount, a few UI upgrades, and Twitter?

We’ve released a minor update to our member UI with a few new features, one of which is of particular note: a 10% discount on RespectMyPrivacy service is now available in exchange for prepayment.
Continue reading A RespectMyPrivacy discount, a few UI upgrades, and Twitter?…

File server “f1” replacement

Our venerable old file server “f1” had some problems last month that left us with some doubt as to the viability of its redundant power supplies over the long term. Since then, we’ve been planning and preparing to migrate all the sites it handles to other, newer file servers.

That’s all been prepped now, and what we’re going to do is automatically migrate everyone during the month of April. If you have affected sites, you can get a specific time for each site from our member interface, and the main sites page will star any site scheduled for an upgrade on your list of sites so you can see at a glance which sites are affected.
Continue reading File server “f1” replacement…

Scheduled Downtime for Friday, November 20

We have some facilities maintenance scheduled for this Friday. As part of this maintenance, we will need to physically move a handful of critical file and database servers between racks in our Phoenix datacenter. Since that equipment forms the heart of our hosting service, we’ll need to shut almost everything down briefly, just long enough to move it.

The maintenance window will be from 10am to 4pm MST (5pm to 11pm UTC) on Friday, November 20th, 2009.
Continue reading Scheduled Downtime for Friday, November 20…

Service & pricing changes finalized

Our recent announcement that we were preparing to make pricing changes provoked quite a bit of discussion that resulted in significant improvements to our plans. (Please see both links if you want more information about the rationale and justification for these changes; both have been discussed in exhaustive detail.)

Those plans have now been finalized, and we will begin phasing them in this month.
Continue reading Service & pricing changes finalized…