Just How Hard Is Verizon Throttling AWS? Spoiler: AWS Is The Shoe

Update 04/17/14 2:47pm Eastern

Alright guys, I messed up – I never ran the other test I was working on, and I apologize for dropping the ball there. Sounds like the issue was with an interconnect, and it also sounds like Netflix is setting up deals with ISP’s to avoid this type of problem.

Update 02/13/14 9:25am Eastern

Just a quick update for you all – I’m still working out a testing solution that standardizes as many variables as possible. Just for fun (and for the sake of actionable data) I’ve been running the original test periodically both on AWS and Akamai content. Neither has reached anything approaching the speeds my connection is capable of, however, in most cases the speeds have been usable (more like 1 – 2mb/s instead of 60kb/s). It does seem like something is consistently interfering with the data rate, though it’s still impossible to say with certainty whether it’s throttling, a dodgy backbone or node, or something all together different. I’ll update with more info when I can.

Update 02/11/14 9:24pm Eastern

As some of you have pointed out, it’s hard to say anything definitive about whether Fios is throttling without traceroute information. Additionally, it’s come to my attention that the file I was testing on today is hosted on Akamai, not AWS. Turns out I had some outdated info – sorry about that. It does, hwoever, look like Akamai is subject to the same slowdown based on what I found. I’m going to look into testing Akami further, and I’ll be running more tests on AWS in the coming days.

Original Story

Articles and tweets have been popping up over the past couple of days accusing Verizon of throttling content hosted by AWS, Amazon’s cloud-based content storage solution. Today, I decided to spend a couple of hours running tests on my Fios internet connection to see if I could find signs of throttling, and the results were shocking.

First, let’s get the basic details out of the way. I’m in southern NJ, and while my Fios plan only includes 75mbps / 25mbps speeds, speed tests (and real world performance) generally look more like this for me.

Verizon speed test throttling AWSUp until about a week ago I had no reason to complain about the performance. Then I started noticing  videos hosted on AWS kept hitching, and stopping. I sort of just assumed Amazon was having trouble (wouldn’t be the first time) and I figured it would pass.

It never did pass, and then I started seeing articles about a guy in Texas who had some strong evidence to support the theory that Verizon was throttling his connection. So I decided to run some tests of my own to see if the same was happening here in New Jersey.

First thing’s first – is AWS up? Yes.

aws is up

So I found a large (2.0 GB to be precise) file hosted on AWS Akamai to download, and I downloaded that file earlier this morning. My connection was running fine at the time, but I was never able to get an AWSAkamai-hosted file to download faster than 60kps.

verizon fios slow speeds for AWS

So I decided to run a second set of tests. First, I turned on my VPN, then I downloaded the same file from my previous test. A VPN encrypts all of your traffic so that your ISP (or anyone else for that matter) won’t be able to tell exactly what you’re doing on the internet. OK, before someone calls me out on it, it’s a little more complicated than all that. If you care for the details, read the wiki. When Verizon couldn’t “see” exactly where the bandwidth was going, the results were much better.

fios with vpn

 

So I disconnected from the VPN, and immediately retried the download. Back down below 60kps again.

moar throttling

 

Once Verizon could “see” what I wanted to do with my bandwidth again, suddenly the speed plummeted. This behavior implies two things: first, that the problem isn’t on Amazon’s end. Second, given AWS-hosted content is the only content experiencing these slow speeds, it implies Amazon’s servers have been singled out for special treatment.

When the Washington Post approached Verizon about the topic of throttling, they gave the following statement:

“Many factors can affect the speed of a customer’s experience for a specific site, including that site’s servers, the way the traffic is routed over the Internet and other considerations. We are looking into this specific matter, but the company representative was mistaken. We’re going to redouble our representative education efforts on this topic.”

I can’t speak for all of Verizon’s Fios customers, but in southern NJ it seems fairly certain that Verizon is in fact slowing down traffic to and from AWS. My tests don’t explain why it’s happening though – it’s easy to assume that it was intentional but this could be an unintended byproduct of a traffic management strategy gone awry. Still, it’s hard to ignore the numbers.

 

  • mike

    I’ve experienced similar issues on my (different) provider, however simply turning on a VPN is not enough to state that they are throttling based on content/destination. It’s possible that your vpn connection exits your provider network at a different peering point, one that is less congested, and then proceeds to amazon. In this scenario your provider (or someone upstream) has not provisioned enough peering bandwidth resulting in service degradation when you are not connected to your vpn. It would be very hard to distinguish between this scenario and the one you have outlined based solely on your observations, unfortunately it would require your provider telling you what the problem is, which seems unlikely.

    • whatthegeek

      After talking it out with a couple of people on Reddit, I’m actually looking into setting up a test (assuming I can find someone to let me in their AWS server) whereby I can see if the route is changing. When I do a traceroute on their EC2 server, all the hops are the same, but I’d like to dig a little deeper here if I can.

      • Jack

        From what I recall EC2 has a free tier. Its not too hard to setup, as in half an hour, and you could do the testing for free.

  • overhere2000

    Try disabling encryption on your VPN and see if the effect remains during all three tests. Most VPNs allow disabled encryption.

  • A

    I have this exact same issue with AWS, Netflix, and YouTube on my FiOS connetion in NY. Keep it up Verizon, I’m headed back to the assholes at the cable company. Lesser of two evils.

  • Richard Buckingham

    Urgh, very difficult to prove this without knowing the routes taken. For me this likely means the bandwidth from AWS to your VPN provider is different than in the route from AWS direct to you. So if anything it’s likely AWS’s fault for not making sure their transit route to Verizon customers is not congested, they’re either being cheap and routing that traffic indirectly through a cheaper provider or it is direct but congested because there’s not enough bandwidth allocated (which they buy as transit from Verizon.)

  • spiffae

    It’s not just AWS. They seem to be throttling Astraweb usenet in the NYC metro area as well.

    Without VPN on 50/25 FiOS – 1.0 MB/s (~8mb/s)
    With VPN – 6.3 MB/s (50mb/s roughly)

    Awful.

  • alex

    That file is not hosted on AWS. It’s on Akamai. You’re not necessarily wrong that Verizon may be doing something weird, but I don’t think they are specifically targeting AWS.

    • whatthegeek

      Wait, I could have sworn Giant Bomb was on AWS? How do you know it’s on Akamai?

      • Brandon Schlinker

        Perform a traceroute — it lands at an Akamai CDN point of prescence. You can also do a simple DNS query to see this as well.

  • Anonymous

    There are at least two ways, even when ‘Net Neutrality’ was in effect, that a large ISP could limit the amount of bandwidth a provider like AWS or Netflix can consume on their networks. While truthfully telling the world that they are not ‘throttling’ said traffic. They both involve peering agreements.

    The first way, which I believe Xfinity was already caught doing, is to establish peering agreements with al the high bandwidth providers that you ‘like’ and force the rest of the internet through a small 1Gbit or 10Gbit default route.

    The second way, if you wanted to be a bit more targeted, would be to establish peering agreements with those sites you want to restrict, but only allocate a few Gbits to this link. Thereby firmly capping bandwidth from those sites without employing any traffic shaping.

    I suspect a traceroute would tell you if Verizon has a peering agreement either with AWS or Netflix or not.

  • eL.Gee

    FYI Akamai hosts 30% of the internet on their CDN (their stat not mine). I doubt that Verizon is blanket throttling connections to any Akmai servers otherwise it would have huge implications to both end-users and Akamai’s business (and thus other business’).

    Sites you may know with akamai: http://www.akamai.com/html/customers/customer_list.html it’s a non trivial list.

    • whatthegeek

      Thanks for the list – you’re right, there are some significant sites there. I’m gonna work out a better (or maybe it would be more accurate to say “real” :/ ) AWS test sometime later today, and I may put something together for Akamai later in the week

  • PatrickTulskie

    Downloading the same file, with Time Warner in Manhattan from my office, I’m getting 80kb/s. If I download it from home in the burbs on Verizon Fios, I get 2-3mb/s. I have a hunch that it’s more of a network congestion issue than a throttling issue.

  • http://matthewkrieger.tumblr.com/ Matthew Krieger

    Your post name totally cracks me up…

  • http://plus.ly/carlos Carlos Limardo

    Not just AWS & Netflix but I signed up to Hulu (14 days free) just to test this same thing and it’s happening with Hulu as well. Even Spotify when I play music but not Google Play Music. SMH