benchmarking the cloud

RSS Feed

Cloud Articles From Our Blog

02/14/2010 14:33 PST

Cloud Storage Showdown Part 1 - Cloud to Consumer

This is a follow up to our previous post Cloud Speed Test Results where we analyzed the results from our custom cloud speedtest as they pertained to cloud servers and content delivery networks. In this post, we'll use the same test results to analyze the bandwidth performance of 4 cloud storage services including Amazon's Simple Storage Service (S3) (all 3 regions), Rackspace's Cloud Files, Nirvanix's Storage Delivery Network (SDN), and Microsoft's Azure Blog Storage. To date, about 1350 unique users in 115 countries have run the speedtest. This speedtest tracks the amount of time required to download a 1MB test file from these 6 cloud storage services, and records the transfer rate for each test run. We use MaxMind's geoip database to track where the user is running the test from (country + state/province for US/Canada). The order of the tests is always random and we enforce a time limit such that users with dial up or slow connections are excluded. We also limit individual users to running the speedtest only once per day. 90% of the speedtests were run by users on residential high-speed Internet connections, thus this post is focused on Cloud to Consumer bandwidth performance. In the next post (Part 2), we'll focus on Cloud to Cloud or Intracloud storage bandwidth performance.

As you'll see in the results below, consumer cloud storage performance is highly dependent on the consumer's geographical location. Although cloud vendors may tell you it isn't important where their services are run out of, this is not entirely true. In the previous post we separated the results into very general geographical regions (US and non-US). Since this post is more focused on consumer cloud storage usage, and because there is a large difference in bandwidth performance based on geographical origin, we have broken these results into 8 geographical regions.

A few admin notes regarding the speedtest results:
  • The results are based on download performance. Consumers will generally not achieve this same throughput for uploads due to ISP uplink caps
  • Nirvanix's SDN is unique in that it is geographically optimized like a content delivery network (users connect to the fastest/closest of 4 storage nodes). We used a 4-node SDN (US West, US East, Germany and Japan) in conducting the Nirvanix tests
  • Rackspace Cloud Files is not directly browser accessible and thus not compatible with our speedtest. The results shown below are based on performance for Rackspace Cloud Servers which to our knowledge are run out of the same data centers. Also, based on other bandwidth tests we have conducted, there is a very close correlation between Rackspace Cloud Files/Cloud Servers bandwidth performance
  • Results for Azure storage are based on Azure Platform performance. These results should be most closely correlated with the South Central US storage region. Now that Azure is out of CTP, we will be including the other regions in future tests: US North Central, North Europe and Southeast Asia
Global Results
The global results (all tests performed) show a fairly close spread between the 6 storage services with the exception of S3 US West which seems to be more geographically sensitive.
US Results
In the US, tests were run from all 50 states. S3 US East, Azure Storage and Rackspace Cloud Files were all pretty close in the top 3 spots. Surprisingly, Amazon's S3 region in Europe performed significantly better than the new S3 US West region in the US.
US West Results
US West (AK, AZ, CA, HI, ID, NM, NV, OR, UT, WA) is where Amazon's new S3 US West region really shined out performing other US services by 20-40%.
US Central Results
In US Central (CO, IA, IL, IN, KS, LA, MI, MN, MO, MS, ND, NE, OH, OK, SD, TX, WI, WY), Rackspace's Cloud Files service was the clear top performer followed by Azure (7% slower) and S3 US East (13% slower).
US East Results
In US East (AL, AR, CT, DC, DE, FL, GA, KY, MA, MD, ME, NC, NH, NJ, NY, PA, RI, SC, TN, VA, VT, WV), Amazon's S3 US East region was the top performer. Surprisingly, S3 EU West was not close behind. Azure followed at a distant third (15% slower).
Non US Results
Outside of the US is where Nirvanix's storage outperformed other services. Azure also performed well followed by S3 US East.
Europe Results
In Europe, Nirvanix's SDN did extremely well, out performing second place S3 EU West by 20%. Azure performed comparably with S3 EU West. Rackspace Cloud Files was a bit further behind. S3 US West did not perform well in Europe.
Asia Results
In Asia Azure and Nirvanix were about equal in performance. Rackspace Cloud Files and S3 US West followed at about 15% slower. Amazon will be deploying a new AWS Asia region later this year.
Summary
Bandwidth is but one factor for consumers to consider when choosing a cloud storage service. Most of these storage services are not natively mountable (i.e. you can't browse them using your operating system's file browser), so third party tools like JungleDisk (S3 or Rackspace Cloud Files only), CloudBerry (S3 or Nirvanix) or Windows Explorer Virtual Network Drive (S3, Azure, Nirvanix) are necessary in order to use them. The decision on which storage service to use really depends on a variety of factors including geography, bandwidth, third party tools available and bandwidth/storage costs.