Update 04/17/14 2:47pm Eastern
Alright guys, I messed up – I never ran the other test I was working on, and I apologize for dropping the ball there. Sounds like the issue was with an interconnect, and it also sounds like Netflix is setting up deals with ISP’s to avoid this type of problem.
Update 02/13/14 9:25am Eastern
Just a quick update for you all – I’m still working out a testing solution that standardizes as many variables as possible. Just for fun (and for the sake of actionable data) I’ve been running the original test periodically both on AWS and Akamai content. Neither has reached anything approaching the speeds my connection is capable of, however, in most cases the speeds have been usable (more like 1 – 2mb/s instead of 60kb/s). It does seem like something is consistently interfering with the data rate, though it’s still impossible to say with certainty whether it’s throttling, a dodgy backbone or node, or something all together different. I’ll update with more info when I can.
Update 02/11/14 9:24pm Eastern
As some of you have pointed out, it’s hard to say anything definitive about whether Fios is throttling without traceroute information. Additionally, it’s come to my attention that the file I was testing on today is hosted on Akamai, not AWS. Turns out I had some outdated info – sorry about that. It does, hwoever, look like Akamai is subject to the same slowdown based on what I found. I’m going to look into testing Akami further, and I’ll be running more tests on AWS in the coming days.
Articles and tweets have been popping up over the past couple of days accusing Verizon of throttling content hosted by AWS, Amazon’s cloud-based content storage solution. Today, I decided to spend a couple of hours running tests on my Fios internet connection to see if I could find signs of throttling, and the results were shocking.
First, let’s get the basic details out of the way. I’m in southern NJ, and while my Fios plan only includes 75mbps / 25mbps speeds, speed tests (and real world performance) generally look more like this for me.
Up until about a week ago I had no reason to complain about the performance. Then I started noticing videos hosted on AWS kept hitching, and stopping. I sort of just assumed Amazon was having trouble (wouldn’t be the first time) and I figured it would pass.
It never did pass, and then I started seeing articles about a guy in Texas who had some strong evidence to support the theory that Verizon was throttling his connection. So I decided to run some tests of my own to see if the same was happening here in New Jersey.
First thing’s first – is AWS up? Yes.
So I found a large (2.0 GB to be precise) file hosted on
AWS Akamai to download, and I downloaded that file earlier this morning. My connection was running fine at the time, but I was never able to get an AWSAkamai-hosted file to download faster than 60kps.
So I decided to run a second set of tests. First, I turned on my VPN, then I downloaded the same file from my previous test. A VPN encrypts all of your traffic so that your ISP (or anyone else for that matter) won’t be able to tell exactly what you’re doing on the internet. OK, before someone calls me out on it, it’s a little more complicated than all that. If you care for the details, read the wiki. When Verizon couldn’t “see” exactly where the bandwidth was going, the results were much better.
So I disconnected from the VPN, and immediately retried the download. Back down below 60kps again.
Once Verizon could “see” what I wanted to do with my bandwidth again, suddenly the speed plummeted. This behavior implies two things: first, that the problem isn’t on Amazon’s end. Second, given AWS-hosted content is the only content experiencing these slow speeds, it implies Amazon’s servers have been singled out for special treatment.
When the Washington Post approached Verizon about the topic of throttling, they gave the following statement:
“Many factors can affect the speed of a customer’s experience for a specific site, including that site’s servers, the way the traffic is routed over the Internet and other considerations. We are looking into this specific matter, but the company representative was mistaken. We’re going to redouble our representative education efforts on this topic.”
I can’t speak for all of Verizon’s Fios customers, but in southern NJ it seems fairly certain that Verizon is in fact slowing down traffic to and from AWS. My tests don’t explain why it’s happening though – it’s easy to assume that it was intentional but this could be an unintended byproduct of a traffic management strategy gone awry. Still, it’s hard to ignore the numbers.