benchmarking the cloud

RSS Feed

Cloud Articles From Our Blog

42 months ago 20:54 PDT

Introducing Web Services for Cloud Performance Metrics

Over the past year we've amassed a large repository of cloud benchmarks and metrics. Today, we are making most of that data available via web services. This data includes the following:

  • Available Public Clouds: What public clouds are around and which cloud services they offer including:
    • Cloud Servers/IaaS: e.g. EC2, GoGrid
    • Cloud Storage: e.g. S3, Google Storage
    • Content Delivery Networks/CDNs: e.g. Akamai, MaxCDN, Edgecast
    • Cloud Platforms: e.g. Google AppEngine, Microsoft Azure, Heroku
    • Cloud Databases: e.g. SimpleDB, SQL Azure
    • Cloud Messaging: e.g. Amazon SQS, Azure Message Queue
  • Cloud Servers: What instance sizes, server configurations and pricing are offered by public clouds. For example, Amazon's EC2 comes in 10 different instances sizes ranging from micro to 4xlarge. Our cloud servers pricing data includes typical hourly, daily, monthly pricing as well as complex pricing models such as spot pricing (dynamically updated) and reserve pricing where applicable
  • Cloud Benchmark Catalog: This includes names, descriptions and links to the benchmarks we run. Our benchmarks cover both system and network performance metrics
  • Cloud Benchmark Results: Access to our repository of 6.5 million benchmarks including advanced filtering, aggregation and comparisons. We are continually conducting benchmarks so this data is constantly being updated

We are releasing this data in hopes of improving transparency and making the comparison of cloud services easier. There are many ways that this data might be used. In this post, we'll go through a few examples to get you started and let you take it from there.

Our web services API provides both RESTful (HTTP query request and JSON or XML response) and SOAP interfaces. The API documentation and SOAP WSDLs are published here: http://api.cloudharmony.com/api

The sections below are separated into individual examples. This is not intended to be comprehensive documentation for the web services, but rather a starting point and a reference for using them. More comprehensive technical documentation is provided for each web service on our website.

Example 1: Lookup Available Clouds

In this first example we'll use the getClouds web service to lookup all available public clouds. The table at the top of the web service documentation describes the data structure that is used by this web service call.

Request URI: JSON Response

Request URI: XML Response

Note on pagination

Due to the large amount of data that can be returned, our web services utilize results pagination (similar to to getting multiple pages of results for a web search). The maximum number of results request to this web service will return is 10. You may set a limit lower than 10 using the ws-limit request parameter, but not greater than 10. The example request URIs above return only the first 10 results (as determined by the limit response value). At the time of this writing there were 37 total records (as determined by the count response value). To return the remaining 27 results, utilize the following URIs:

Request URI: Results 11-20

Request URI: Results 21-30

Request URI: Results 31+

Note on SOAP

In this post we'll only be showing use of the RESTful API interface. A SOAP interface is also provided. The base API documentation page includes links to WSDLs you may use to import and utilize the SOAP interface (some IDEs let you import WSDLs). The parameters and response structure for SOAP requests are very similar, but not identical to the REST interface (XML names may differ slightly from HTTP request and response names). The WSDL for the getClouds web service is available here: http://api.cloudharmony.com/getClouds/wsdl

Example 2: Search for cloud with servers, storage and CDN services

In this Example you'll use the same getClouds web service again, but we'll add a few constraints so only clouds with server, storage and CDN services are returned. Unless otherwise stated, from this point forward the same rules with regards to pagination apply (see Example #1 for more details). Additionally, only JSON URIs will be shown (to use XML responses, simply add the parameter ws-format=xml to the URI).

Request URI

Note on constraints

In the example URI above, we used ws-constraint parameters to filter the results. Constraints can be applied to specific attributes defined by the data structure for a given web service. The data structure is documented in a table at the top of the web services documentation page. In this example, we used 3 such attributes: hasServers, hasStorage, and hasContentDelivery. Because these are boolean type attributes, we assigned constraint values 1, signifying that only clouds where those attributes are TRUE should be returned.

The API also supports more complex constraint parameters. This example utilizes the most simple form of constraints by testing for equality and joining the 3 constraints with an AND connective. Constraints can also be used to check that an attribute is less or greater than a desired value, and use of an OR connective if multiple constraints are specified. We'll go into these types of constraints in a proceeding example.

Response

In this Example, only 5 public clouds are returned instead of the 37 returned in the previous example, signifying that only 5 public clouds offer all cloud servers, storage and CDN services.

Example 3: Retrieve a specific public cloud

Each data structure has an attribute that is the unique identifier. This attribute is called the primary key. The getClouds web service can be used to retrieve a specific public cloud if you know the primary key for the cloud. In this example, we'll use this feature to retrieve the AWS (Amazon Web Services) public cloud.

Request URI

Response

The response is almost identical to the previous 2 requests with the exception that the base response value is not an array. When a web service is invoked for a specific cloud using the primary key as we've done here, the response will always be a single data structure value. This is in contrast to the previous 2 requests that returned multiple clouds using an array as the base data structure.

Example 4: Retrieve Cloud Server Service for AWS

In this Example we'll use the getCloudServerServices web service to return the cloud server/IaaS service for AWS (Amazon Web Services) - EC2. We know AWS has such a service because the boolean hasServers attribute was true in the previous examples. The API documentation shows that the CH_CloudServer data structure contains an attribute named cloud that references the cloud that the service belongs to. In order for this web service to return only the cloud server service belonging to AWS, we'll just need to add a single ws-constraint for this attribute.

Request URI

Response

Even though only a single result for EC2 was returned, the base response data structure is an array. This will always be the case when invoking the API for a data structure without a primary key (as we did in Example 4 above), because when this is the case, there is always the possibility that multiple results could be returned.

Example 5: Find all cloud services that support Windows

Suppose you are looking to deploy a Windows server in the cloud. Because the CH_CloudServer data structure has an attribute operatingSystemsSupported that defines which operating systems are supported by that service, we can use it in conjunction with a ws-constraint request parameter to filter the results accordingly. In our previous use of constraints, we used the default equality operator. In this example, we'll need to change the operator to a substring search. This is because the operatingSystemsSupported attribute is an array which may contain multiple values representing all of the operating systems supported by the service (i.e. Linux, Windows, etc.). By using the substring operator, the request will search for services where the operatingSystemsSupported attribute contains Windows. The substring operator is the numeric value 32 (operators are numeric to support multiple operators using bitmasks). The operators supported and their corresponding values are shown on the API documentation.

Request URI

Response

At the time of this writing, this request returned 14 services that support the Windows operating system.

Example 6: Find other cloud services

In addition to the getCloudServerServices web service discussed in Examples 4-5, the following additional web services are provided: getCloudDatabaseServices, getCloudMessagingServices, getCloudPlatformServices,getCloudStorageServices and getCDNs. The usage for each of these is identical. In the example below we'll use them for various lookups.

We are still in the process of populating vendor profiles for different cloud services. Currently only basic information is provided by the web services. In the future, the data structures for these services will be expanded to include many additional details such as pricing, SLAs, features, technical details, etc.

Lookup all CDNs

Lookup storage services for AWS

Lookup database services for Azure

Lookup all cloud platforms (i.e. Google AppEngine, Microsoft Azure)

Lookup all cloud messaging services (i.e. AWS SQS)

Example 7: Get the full benchmark catalog

Up to this point we've been using web services to lookup which clouds and cloud services are available. The remaining Examples will involve retrieving benchmarking related data. To get started, we'll first need to determine which benchmarks are available using the getBenchmarks web service. This web service provides access to information about the benchmarks we conduct. Unlike the previous web services, getBenchmarks does not support the ws-constraint parameters to filter results. This is always the case when the top section of the API documentation page does not show a data structure table. Instead of ws-constraint filters, getBenchmarks supports 4 request parameters (these are shown on the right column of the API documentation table):

  • aggregateOnly: set to TRUE if only aggregate benchmarks should be returned (see not on aggregate benchmarks below)
  • nonAggregateOnly: set to TRUE if only non-aggregate benchmarks should be returned (see not on aggregate benchmarks below)
  • category: return only benchmarks in this category. Multiple categories may be specified separated by pipe characters (see Example 8 below)
  • serverOnly: set to TRUE if only cloud server benchmarks should be returned. These are benchmarks we run on cloud servers only

Benchmarks are assigned to 1 or more categories. The getBenchmarkCategories web service may be used to obtain all of the available benchmark categories (see Example 8 below).

In this example, we'll retrieve all benchmarks (or at least the first 10 due to pagination).

Request URI: Results 1-10

Request URI: Results 11-20

Response

The response from this web service is an array of benchmarks each containing the follow values:

  • benchmarkId: the identifier of the benchmark
  • title: the benchmark title
  • subtitle: the benchmark subtitle
  • categories: the categories for this benchmark (an array)
  • description: the benchmark description
  • url: URL to this benchmark's website (if available)
  • lowerIsBetter TRUE if a lower score is better for this benchmark
  • aggregate: TRUE if this benchmark is an aggregate of multiple benchmarks (see not on aggregate benchmarks below)
  • benchmarks: if this is an aggregate benchmark, this value will provide details about which individual benchmarks are included in it and their corresponding weights. This return value is an array of values each containing the following keys:
    • benchmarkId: the id of the individual benchmark
    • weight: the weight assigned to this benchmark
    • alternates: if this benchmark has alternate benchmarks (benchmarks used if this benchmark is not available), this will be an array representing the IDs of those alternate benchmarks
  • baseline: if this is an aggregate benchmark, this return value will be provided defining the baseline definition for it. The baseline definition determines how the aggregate benchmark metric is calculated. This response value is an array containing one or more server/value pairs. The aggregate score is based on determining how the benchmarked server performs relative to the baseline servers. If the server performs better, the metric will be higher than the baseline. If the server performed worse, the metric will be lower. This response is a hash of key/value pairs where the key is the serverId and the value is the score that is assigned if the benchmarked server performed exactly the same as that server. For more information on baselines and aggregate metric calculation is available on the What is an ECU? CPU Benchmarking in the Cloud post on our blog

Note on aggregate benchmarks

Aggregate benchmarks are a special type of benchmark that aren't benchmarks themselves, but rather a compilation of multiple benchmark result metrics. This compilation is used in conjunction with a baseline configuration to produce a more comprehensive benchmark metric related to some facet of performance. A more detailed description of aggregate benchmarks and baselines is discussed on our blog. CCU is one such aggregate benchmark discussed here.

Example 8: Get benchmark categories

Every benchmark is assigned to one or more categories. The getBenchmarkCategories web service returns a list of all possible benchmark category name. This web service is very simple. It does not use any parameters or pagination.

Request URI

Example 9: Get only server benchmarks in category System: CPU

In this example we'll use the same getBenchmarks web service to retrieve only server benchmarks in the category System: CPU (we discovered this category previously using the getBenchmarkCategories web service). To accomplish this, we'll use the serverOnly and category request parameters.

Request URI

Example 10: What server benchmarks have been run

Before attempting to analyze benchmark results, it may be helpful to first determine what benchmark results data is available including which clouds and server configurations have been benchmarked. Generally, we conducted cloud server benchmarking 3-4 times each year. Every benchmark test run has a unique testId. The typical format of a testId is MMYY-[SEQ]. For example, the test 0410-1 was conducted in April 2010. Do determine what tests have been run within clouds the getServerCloudsBenchmarked web service may be used. This web service uses the following parameters:

  • serviceId: the ID of a service or cloud to return test information for. Multiple IDs may be specified each separated by a pipe character
  • start: if specified, only services that have been benchmarked on or after this date will be returned
  • stop: if specified, only services that have been benchmarked on or before this data will be returned

The return value is an array of cloud server services and the corresponding testing information for those services including testIds and testing dates.

Request URI

Response

The response from this web service is an array of services and information about the benchmark tests that have been conducted within those services.

  • id: the id of this service
  • name: the name of the service
  • testIds: the IDs of tests performed (array)
  • numTests: the number of tests that have been conducted for this service (same as number of elements in testIds)
  • lastTestId: the ID of the last test that was run for this service
  • lastTestDate: the date of the last test that was run for this service
  • url: the URL to the service's website

Example 11: What server benchmarks have been run in the GoGrid and Amazon clouds after June 2010

In this example, we'll use the same getServerCloudsBenchmarked web service to determine when testing has occurred only in the AWS and GoGrid clouds on or after June 2010. To do so, we'll use the serviceId and start parameters to filter the results. The serviceId parameter can be either the ID of the specific server service or the ID of a cloud.

Note on dates and times

Whe specifying dates or dates and times, most standard formats are supported such as 6/1/2010 or 2010-06-01 or June 1 2010. Date data types are returned by web services as a text value unless the ws-js-dates parameter is set to TRUE in which case it will be returned using a javascript Date object (only applicable to JSON responses).

Request URI

Example 12: Get all Geekbench benchmark results for Rackspace Cloud and GoGrid

When it comes down to retrieving cloud server benchmark metrics we'll use the getServerBenchmarkResults web service. This web service requires 2 parameters:

  • benchmarkId: the identifier(s) of the benchmarks that should be returned (REQUIRED). Multiple IDs may be specified separated by pipe characters
  • serviceId: the identifier(s) of the cloud server service to return the benchmarks for (REQUIRED). Multiple IDs may be specified separated by pipe characters. Alternatively, this parameter may be left out if the serverId parameter below is specified

Additionally, the following parameters may optionally be provided:

  • serverId: the identifier(s) of the server to return benchmark metrics for. Multiple IDs may be specified each separated by a pipe character
  • dataCenter: the identifier(s) of a specific service data center to return benchmarks for (if the cloud server service operates out of multiple data centers). Multiple data centers may be specified separated by pipe characters. For example, AWS EC2 operates out of 4 regions - US West, US East, EU West and APAC currently. These regions are located in California, Virginia, Ireland and Singapore respectively. To return only results for the US West data center, this parameter should be set to CA, US. To return metrics for both US West and EU West data centers, this parameter would be CA, US|IE (IE is the ISO 3166 code for Ireland)
  • testId: the identifier of a specific test for the benchmarks that should be returned. Multiple IDs may be specified separated by pipe characters
  • lastBenchmarksOnly: set to TRUE if only the latest benchmark test should be included in the results. This guarantees that only a single set of results will be returned
  • combineMultiple: If multiple benchmark metrics are included in the results, this parameter defines how those values should be returned as a single value. Valid options are:
    • average: use an average of all values (default)
    • lowest: return the worst value (may be the lowest or highest value depending on whether higher or lower scores are better for the benchmark)
    • highest: return the best value
    • earliest: return the value from the earliest test
    • latest: return the value from the latest test

As you can see, requests using this web service can be quite complex if desired. In this example, we'll keep it simple by using only the benchmarkId and serviceId parameters. The Geekbench benchmark produces a metric that rates CPU and memory performance

Request URI

Response

The response is an array of benchmark result metrics each consisting of the following values:

  • serverId: the ID of the server this metric pertains to
  • serviceId: the ID of the service serverId pertains to
  • benchmarkId: the ID of the benchmark this metric pertains to
  • value: the benchmark metric. If this result consists of multiple benchmark values, value will be an average of all results unless the combineMultiple request parameter specifies otherwise
  • testDate: the date of the test (if result is from a single benchmark test)
  • testId: the ID of the test (if result is from a single benchmark test)
  • resultsUrl: some benchmark result artifacts are accessible online. When this is the case, this will be the URL to those artifacts (if result is from a single benchmark test)
  • values: the values of the tests (if result is from multiple benchmark tests)
  • testDates: the dates of the tests (if result is from multiple benchmark tests)
  • testIds: the ID of the tests (if result is from multiple benchmark tests)
  • resultsUrls: the URLs to the test results (if result is from multiple benchmark tests)
  • numTests: the number of tests used to calculate the value

Example 13: Get the latest CCU benchmarks for the EC2 APAC region

In this example, we'll use the dataCenter and lastBenchmarksOnly parameters to return all of the CCU benchmark results for Amazon EC2's APAC region (this region is located in Singapore - hence the dataCenter parameter is set to the ISO 3166 country code SG). Unlike the previous example where multiple test results were returned, in this example because lastBenchmarkOnly is TRUE, the web service will only return a single benchmark value (the values, testDates, testIds and resultsUrls values will not be included in the response). CCU is an aggregate benchmark consisting of many underlying CPU performance related benchmarks as discussed here.

Request URI

Example 14: Get available cloud server configurations for the Amazon EC2 APAC region

Before proceeding any further with getServerBenchmarkResults examples, we'll demonstrate how to find out what server configurations are available for a given cloud service. This is useful because the getServerBenchmarkResults supports a serverId parameter that can be used to filter benchmark results using a specific server identifier. For example, you may want to compare benchmark results between EC2 m2.4xlarge and GoGrid 16GB cloud servers only.

The getCloudServerConfigurations web service allows you to lookup cloud server configurations. This web service uses a data structure containing various details about cloud servers including CPU, memory, and storage specifications; pricing and more (review the API documentation for full details). Because this web service is based on a data structure, we'll be able to use ws-constraint parameters to filter the results. In this example, we'll use 2 constraints (cloud and dataCenter) to filter the results so that only Amazon EC2 APAC region servers are returned.

Request URI

Example 15: Compare IOP benchmark results between Rackspace Cloud and GoGrid 4GB cloud servers

Now that we've been able to obtain the identifiers of cloud servers using the getCloudServerConfigurations web service, we can go back to the getServerBenchmarkResults web service and compare cloud servers using those IDs and the serverId parameter. In this example, we'll compare storage IO performance between 4GB Rackspace Cloud and GoGrid cloud servers (gg-4gb and rs-4gb) using the aggregate IOP benchmark. IOP is an an aggregate storage IO benchmark based on 7 IO related benchmarks as documented here. This is benchmark is NOT the same as IOPS. To invoke retrieve the IOP benchmark results for only Rackspace and GoGrid 4GB cloud servers, we'll set the serverId parameter to gg-4gb|rs-4gb (multiple IDs can be specified each separated by a pipe character).

Request URI

Example 16: Lookup all cloud servers in the US with at least 2GB memory and costing $0.10/hr or less

In this example, we'll use the getCloudServerConfigurations to lookup US-based cloud services offering cloud servers with at least 2GB memory and costing $0.10/hr or less. This will involve use of 4 filtering constraints: dataCenter, memory, priceHourly and priceCurrency. In order to apply these constraints, we'll first need to determine what operators should be used.

The dataCenter attribute value is either [state/province], [country] (US or Canada only) OR [country]. Thus, we'll want the dataCenter attribute to "end with" "US". According to the API documentation, the "ends with" operator is 16.

The memory attribute is a numeric value representing the # of gigabytes included with a cloud server. We'll want this attribute to be equal to or greater than 2. The operator for "equal to" is 1. The operator for "greater than" is 2. Thus, an "equal to or greater than" operator is 1+2=3 (bitmask addition).

The priceHourly attribute is also numeric representing the price of the server per hour. We'll want this attribute to be equal to or less than 0.10. The operator for "equal to" is 1, and the operator for "less than" is 4. Thus, an "equal to or less than" operator is 1+4=5.

The priceCurrency attribute is a string representing the currency code for pricing defined in the server configuration (USD = US dollar). Thus we want this attribute to be equal to "USD". Equality is the default operator, so we do not need to provide an operator value for this constraint.

Request URI

Response

At the time of this writing, only gigenet cloud offers a cloud server with these specifications.

Example 17: Determine average uplink throughput from GoGrid US West to Amazon S3 US West, Zetta and Google Storage

In addition to system benchmarks, we also continually collect networking benchmark metrics. These include both throughput and latency metrics within clouds, between clouds, and from clouds to consumer (i.e. residential Internet connections such as DSL and cable to various cloud services).

Suppose you are evaluating cloud services and decide to use GoGrids' cloud servers. Your business and customers are in California, so you opt to use GoGrid's US West data center. For added protection against a large scale failure, you decide to use an external storage service for backups (instead of GoGrid's own storage service). You've narrowed your storage choices down to either Amazon S3, Zetta or Google's Storage for Developers. You'd like to know which of these storage services will provide the fastest uplink throughput from your cloud servers at GoGrid in order to ensure that backups can be uploaded as quickly as possible. The getNetworkBenchmarkResults web service provides access to this sort of data. This web service uses the following parameters:

Request Parameters

  • serviceId: the ID of the service to get network benchmark metrics for (REQUIRED). Multiple IDs can be specified each separated by a pipe character. Adding multiple IDs each ID specified will essentially double execution time for this web service, so use this feature sparingly
  • dataCenter: if the serviceId specified is operated out of multiple data centers, this parameter must also be provided defining the location of the data center to return results for. Multiple data centers may be specified each separated by a pipe character. Adding multiple data centers will essentially double execution time so use this feature sparingly
  • testId: we conducted multiple network performance tests. This parameter should be the name of the network test that results should be returned for. The following network tests are currently used:
    • intercloud: results originate from our intracloud/intercloud network performance tests. These tests are run throughout the day at varying times to test throughput and latency between and within cloud services (DEFAULT)
    • speedtest: get results from our browser-based cloud speedtest. We allow Internet users to run this test for free. We also pay about 1000 users each month to run this test using Amazon's Mechanical Turk. When the speedtest is run, we capture the user's location (city, state, country), ISP and connection speed (netspeed) using MaxMind's GeoIP databases. Users select a test file between 1-5 MB to test download throughput, or a test file between 0.5-2.5MB to test upload throughput. User's may also test latency
    • speedtest-web: this is the same as speedtest, except that instead of downloading a single large file, many small files are downloaded to simulate an actual web page load. Users select a small (10 files), medium (19 files) or large (51 files) website to test
  • endpoint_*: the endpoint parameters are used to define a service, region, location (city, state or country), ISP or netspeed (or some combination of those) for which network benchmark metrics should be returned. At least 1 endpoint parameter must be specified. The endpoint parameters used must correspond with the testId specified. The following endpoint parameters are allowed:
    • endpoint_cloudId: return results for all services in this cloud. This parameter applies only to the intercloud testId and cannot be used in conjunction with any other endpoint parameters except for endpoint_dataCenter. Multiple IDs may be specified each separated by a pipe character. Results will be grouped by service and data center (multiple results possible for each service and data center)
    • endpoint_serviceId: return results for a specific cloud service. This parameter applies only to the intercloud testId and cannot be used in conjunction with any other endpoint parameters except for endpoint_dataCenter. Multiple IDs may be specified each separated by a pipe character. Results will be grouped by service and data center (multiple results possible for each data center)
    • endpoint_dataCenter: if the endpoint_cloudId or endpoint_serviceId parameters are specified and services are operated out of multiple data centers, this parameter may be used to limit the results to a specific data center location. Multiple data center locations may be specified each separated by a pipe character
    • endpoint_region: a specific region identifier to return results for. This parameter may be used for any test type (speedtest, speedtest-web or intercloud) but may not be used in conjunction with any other endpoint parameters except for endpoint_isp and endpoint_netspeed. Region identifiers and configurations are available using the getRegions web service. Only a single region may be specified, and results are grouped by region (single result for each invocation unless endpoint_isp or endpoint_netspeed are also specified)
    • endpoint_city: a specific city to return benchmark results for. If used, endpoint_country MUST also be specified. This parameter applies only to speedtest or speedtest-web tests. This parameter is not case sensitive. Multiple cities may be specified each separated by a pipe character. Set this parameter to the wildcard character * to return results for all cities for the endpoint_state (optional) and endpoint_country specified. Results will be grouped by city (multiple results possible for each city specified)
    • endpoint_state: a specific state or province to return benchmark results for. If used, endpoint_country will be automatically determined if not specified (US for US states, CA for Canadian provinces). This parameter applies only to speedtest or speedtest-web tests. This parameter can only be used for US and Canada test results because the GeoIP database only supports state/provinces in those countries. It should be the 2 character code for the state or province and is not case sensitive (i.e. NY, CA or QC). Multiple states/provinces may be specified each separated by a pipe character. Set this parameter to the wildcard character * to return results for all states/provinces for the endpoint_country specified. Results will be grouped by state (multiple results possible for each state). If used in conjunction with the endpoint_city parameter, results will be grouped by city
    • endpoint_country: a specific country to return benchmark results for. This parameter applies only to speedtest or speedtest-web tests only. May be used in conjunction with the endpoint_city and endpoint_state parameters. It should be the 2 character ISO 3166 code for the country and is not case sensitive (i.e. US, CA or FR). Multiple countries may be specified each separated by a pipe character. Set this parameter to the wildcard character * to return results for all countries. Results will be grouped by country (multiple results possible for each country) unless endpoint_city or endpoint_state are also specified in which case results will be grouped according to those parameters
    • endpoint_isp: the name of a specific ISP to return benchmark results for. This parameter may be used alone or in conjunction with the endpoint_region, endpoint_city, endpoint_state or endpoint_country parameters. This parameter applies to speedtest or speedtest-web tests only. This parameter is not case sensitive and can also be a substring match to the ISP name (e.g. Verizon will return multiple results for Verizon Business,Verizon Internet Services and Verizon Australia PTY Limited). Multiple ISPs may be specified each separated by a pipe character. Set this parameter to the wildcard character * to return results for all ISPs. The getSpeedtestIsps web service may be used to obtain the names of ISPs for which results are available. If this parameter is specified, results will be grouped by ISP in addition to existing grouping. For example, if the endpoint_city parameter was also specified, the results will be grouped by ISP and then city. This parameter may NOT be used in conjunction with the endpoint_netspeed parameter
    • endpoint_netspeed: a specific connection type to filter results on. This parameter should be one of the following:
      • cabledsl
      • corporate
      • dialup
      • unknown

      A majority of our speedtest results are of type cabledsl. More information on how netspeed is determined is available here. Multiple netspeeds may be specified each separated by a pipe character. Set this parameter to the wildcard character * to return results for all connection speeds. If this parameter is specified, results will be grouped by netspeed in addition to existing grouping. For example, if the endpoint_city parameter was also specified, the results will be grouped by netspeed and then city. This parameter may not be used in conjunction with endpoint_isp
  • metric: the network benchmark metric to return. One of the following values:
    • downlink: the average downlink throughput measured in megabits per second (Mb/s) (DEFAULT)
    • uplink: the average uplink throughput measured in megabits per second (Mb/s)
    • latency: the average latency measured in milliseconds (ms)
  • start: only consider results from tests that occurred on or after this date
  • stop: only consider results from tests that occurred on or before this date
  • minNumTests: the minimum # of tests for a result to be included. The larger the number of tests in a result, the more reliable and accurate that metric will be
  • order: the ordering method, one of the following: asc: order results in ascending order; or desc: order results in descending order. The default ordering is descending for throughput and ascending or latency benchmark results

Constructing the Request

As you can see, the getNetworkBenchmarkResults web service support a complex array of parameters. For the purposes of this example, we'll only be using a few parameters:

  • serviceId: The serviceId we'll use is GoGrid:Servers which is the ID for the GoGrid server service
  • dataCenter: GoGrid currently operates out of 2 data centers, us-west and us-east. The us-west data center is located in California, so the dataCenter parameter we'll use is CA, US
  • testId: We are looking for results from the intercloud test. This is the default value for this parameter, so we do not need to include it in the request
  • endpoint_serviceId: We want to get throughput metrics for AWS S3, Zetta and Google Storage. This parameter supports multiple service IDs each separated by a pipe character. Thus, this parameter will be AWS:S3|Zetta:Storage|Google:Storage
  • endpoint_dataCenter: AWS, Zetta and Google all run storage services out of California. AWS also offers storage in Virginia, Ireland and Singapore. Since we do not want to include those data centers in the results, we'll set this parameter to CA, US
  • metric: Since we'll be doing a lot of uploading to the storage service, our primary are of concern is uplink throughput, so we'll set this parameter value to uplink (the default is downlink)

Request URI

Response

The API documentation states that the results will be an array of hashes each with the following possible values:

  • value: the average downlink (Mb/s), uplink (Mb/s) or latency (ms) for this network benchmark result. This value is based on the metric parameter specified (default is downlink)
  • originServiceId: The ID of the service this result originates from. Returned only when multiple serviceId parameters were specified
  • serviceId: The ID of the endpoint service this result pertains to. Returned if the endpoint_cloudId or endpoint_serviceId parameters were used
  • originDataCenter: The location of the data center this result originates from. Returned only if multiple dataCenter parameters were specified
  • dataCenter: The location of the endpoint data center this result pertains to. Returned if the endpoint_cloudId or endpoint_serviceId parameters were used
  • region: the geographical region this result pertains to. Returned if the endpoint_region, endpoint_city, endpoint_state or endpoint_country parameters were specified. For more information, see the API documentation for the getRegions web service
  • city: The name of the city this result pertains to. Returned if the endpoint_city parameter was specified. More information on how this data is obtained is available here
  • state: The 2 character identifier of the state or province this result pertains to. Only available for US or Canada based results. Returned if the endpoint_state parameter was specified. More information on how this data is obtained is available here
  • country: The 2 character ISO 3166 identifier of the country this result pertains to. Returned if the endpoint_country parameter was specified. More information on how this data is obtained is available here
  • isp: The name of the ISP this result pertains to. Returned if the endpoint_isp parameter was specified. More information on how this data is obtained is available here
  • netspeed: The connection speed used by the tester. Returned if the endpoint_netspeed parameter was specified. More information on how this data is obtained is available here
  • numTests: The number of tests that were averaged to produce this result
  • earliestTest: The date/time of the earliest test included in this result
  • latestTest: The date/time of the latest test included in this result

Because we are testing throughput from one cloud service to another, the results only include value, serviceId, dataCenter, numTests, earliestTest and latestTest. In the proceeding examples we'll see when the other response values are used. The results for this example at the time of this writing are:

  • AWS S3 US West: 161.22 Mb/s uplink (out of 212 tests)
  • Zetta: 63.1 Mb/s uplink (out of 213 tests)
  • Google Storage: 31.07 Mb/s uplink (out of 215 tests)

These results signify that AWS S3 US West region storage will generally provide the fastest uplink throughput from GoGrid US West cloud servers and may be the best service to use for backups (subject to other decision making criteria like price and support).

Example 18: Which CDN has the lowest latency in Europe

In the previous example we obtained network performance results based on our intercloud network testing. These tests are run periodically throughout the day to test throughput and latency between and within cloud services and Internet data centers. We also host a browser-based cloud speedtest to track throughput and latency between cloud services and primarily consumer-based high-speed Internet connections such as DSL and Cable. Users of the cloud speedtest select one or more cloud services to test, a test file size (1-5MB for download tests or 0.5-2.5MB for upload tests) and test to perform (uplink, downlink or latency). The speedtest then uploads/downloads the test file to/from the select cloud services and displays the latency and/or throughput results. We use MaxMind's GeoIP databases to track where the user is (city, state, country), the name of their ISP, and their connection speed using their IP address. This is a generally reliable method for obtaining this data with accuracy of about 99.8%. In addition to allowing Internet users to run this test for free, we also pay about 1000 users per month to run the test using Amazon's Mechanical Turk. All of these results are stored in our database and accessible through the getNetworkBenchmarkResults web service.

In this example, we want to find the CDN (Content Delivery Network) with the lowest throughput in Europe. We used the getRegionsweb service to discover that the region code for Europe is eu. CloudHarmony currently collects network benchmark metrics for about a dozen different CDNs. However, in this example, we've narrowed our CDN choices down to four: AWS CloudFront, MaxCDN, Edgecast or Akamai (resold by VPS.net). The request will be fairly simple, using the serviceId, testId, endpoint_region and metric parameters. The serviceId parameter supports multiple IDs each separated by a pipe character, so we will use that to specify the IDs of each of these 4 CDNs.

Request URI

Response

At the time of this writing, the results from these benchmarks were:

  • AWS CloudFront: 51.15ms (out of 191 tests)
  • MaxCDN: 56.65ms (out of 190 tests)
  • Edgecast: 51.38ms (out of 182 tests)
  • Akamai: 34.22ms (out of 195 tests)

So in this example, the clear winner was Akamai by a margin of about 35%. However, latency is not bad for any of these CDNs.

Example 19: What is the average downlink throughput for CDNs in California

In this example, we'll use the endpoint_isp and endpoint_state parameters to view performance of the Internap and AWS CloudFront CDNs in California, grouped by ISP. The endpoint_isp parameter can either be the name (or partial name) of an ISP such as Verizon, or a wildcard character * to indicate that all ISPs should be returned in the results. In this example, we'll use the wildcard option so the results are grouped by ISP. We will also use the minNumTests parameter so that only results with at least 5 tests completed are returned. The order=asc parameter is also used signifying that the slowest ISPs will show first in the results.

Request URI

Response

The response includes the average downlink throughput value, name of the isp, and the region identifier (us_west_pacific for all results in this example). Because more than 10 results are returned, we'll have to use the ws-offset=10 parameter to view the second page of results, ws-offset=20 for the third page and so on.

Example 20: Which CDN provides the best throughput in the APAC region

In this example, we'll determine which out of a handful of CDNs provides the best overall downlink throughput in the APAC region. We used the getRegions web service to discover that the region code for APAC is asia_apac. In this example, we'll evaluate Akamai, Edgecast, CloudFront, Microsoft Azure CDN and Limelight (resold by Rackspace Cloud).

Request URI

Response

At the time of this writing, the results from these benchmarks were:

  • Akamai: 2.69 Mb/s (out of 489 tests)
  • Edgecast: 2.51 Mb/s (out of 483 tests)
  • AWS CloudFront: 2.61 Mb/s (out of 493 tests)
  • Azure CDN: 3.29 Mb/s (out of 496 tests)
  • Limelight: 2.6 Mb/s (out of 483 tests)

So in this example it appears that Microsoft's Azure CDN service provides almost 20% better downlink throughput in APAC countries with almost 500 tests recorded.

Example 21: Which cloud server vendor has the best throughput in New York City

In this example, we'll use the endpoint_city parameter to determine which of a handful of cloud service provides has best downlink throughput in New York City. We will evaluate the following cloud server providers: AWS EC2 (US East region), GoGrid (US East region), Storm on Demand, Speedyrails (Quebec, CA), VoxCLOUD (New York) and Rackspace Cloud Servers (Texas data center). Because we are dealing with multiple services and multiple data centers, the serviceId and dataCenter parameters need to corresponding with the IDs of all 4 services and and data center locations. The web service will ignore data centers that are not valid for a given service (i.e. only Speedyrails has a data center in Quebec and only Voxel has a data center in New York).

Request URI

Response

At the time of this writing, the results from these benchmarks were:

  • AWS EC2 US East: 10.91 Mb/s (out of 45 tests)
  • GoGrid US East: 6.16 Mb/s (out of 12 tests)
  • Storm on Demand: 7.33 Mb/s (out of 39 tests)
  • Speedyrails: 10.39 Mb/s (out of 20 tests)
  • VoxCLOUD New York: 8.88 Mb/s (out of 36 tests)
  • Rackspace Cloud Servers - Chicago: 5.72 Mb/s (out of 65 tests)

So, with a limited number of test results (less than 100 results should not be considered to be reliable), AWS EC2 US East, Speedyrails and VoxCLOUD New York appear to provide the fastest downlink throughput to New York City (primarily consumer) Internet connections.

Conclusion

For now, we are offering free access to these web services for up to 10 requests per rolling 24-hour period. After 10 requests, you will receive a 503: Service Unavailable http response. This is a beta service and usage and terms are subject to change. If you would like an increased quota or professional support, please contact us. We'd of course also appreciate feedback and bug reports (send to info [at] cloudharmony.com).

10/03/2010 11:46 PDT

Cloudscaling & KT: Private cloud validation using benchmarking

A few months ago we were contacted by Cloudscaling CEO Randy Bias regarding our work in benchmarking of public IaaS clouds (see previous blog posts). His team was working on a large private cloud deployment for KT, Korea's largest landline and second largest mobile carrier, and was interested in using similar techniques to validate that private cloud. This validation would include not only raw benchmark results, but also comparisons of how the private cloud stacked up against existing public clouds such as EC2 and GoGrid. This data would be useful not only for Cloudscaling to validate their own work, but also as a reference for their client KT. We agreed to the project and benchmarking was conducted over a 4 day period last August. Our deliverables included raw benchmark data and an executive report highlighting the results. In this post we will provide the results of these benchmarks.

Multi-Tenancy and Load Simulation
Benchmarking of public IaaS clouds involves a certain amount of ambiguity due to the scheduling and allocation of resources in multi-tenant virtualized environments. One of the fundamental jobs of a hypervisors such as VMware and Xen is to allocate shared resources in a fair and consistent manner. In order to maximize performance and utilization, they are designed to allocate resources such as CPU and Disk IO using a combination of fixed and burstable methods. For example, when a VM requests CPU resources, the hypervisor will generally provide more resources when neighboring VMs are idle versus when they are also requesting CPU resources. In very busy environments, this often results in variable and inconsistent VM performance.

Because the KT cloud benchmarking was conducted pre-launch, there was no other load in the environment besides our benchmarking. To offset this, we ran the benchmarks twice. In the first run, the benchmarks were run individually to provide maximum performance. In the second run, we attempted to simulate a loaded environment by filling the cloud to about 70% capacity with VMs instructed to perform a random sample of load simulating benchmarks (using mostly non-synthetic benchmarks like tpcc, blogbench and pgbench). The benchmarks for the second run were conducted concurrently with the load simulation. The tables and graphs below provide the unloaded benchmark results. Differences between those and the loaded results are noted above the results.

Organization of Results
The results below are separated into 2 general VM types, a large (16 & 32GB) VM and small (2GB) VM. Comparative data is also shown from public clouds including
BlueLock, GoGrid, Amazon EC2, Terremark vCloud Express and Rackspace Cloud. We conducted similar benchmarking in these public clouds earlier this year. The results provided are based on 5 aggregate performance metrics we created and discussed in previous blog posts including:
Note on CPU Stats
The servers tested in all of these benchmarks run within virtualized environments. The cores shown in the benchmark tables below are the # of cores or vCPUs exposed to the virtual server by the hypervisor. This is often not the same as the # of physical cores available on the host system.

Benchmark Results

CPU Performance
CPU benchmark results during the loaded versus unloaded benchmark runs were roughly the equivalent.

Large Server

Cloud Server CPU Memory CCUs
BlueLock 16gb-8cpu Xeon X5550 2.67 GHz [8 cores] 16 GB 29.2
KT 32GB/6x2GHz Xeon L5640 2.27 GHz [6 cores] 32 GB 28.66
GoGrid 8gb Xeon E5450 2.99 GHz [6 cores] 8 GB 27.36
Amazon EC2 m2.2xlarge Xeon X5550 2.67 GHz [4 cores] 34.2 GB 25.81
KT 16GB/3x2GHz Xeon L5640 2.27 GHz [3 cores] 16 GB 15.27
Terremark 16gb-8vpu AMD 8389 2.91 GHz [8 cores] 16 GB 9.81
Amazon EC2 m2.xlarge Xeon X5550 2.67 GHz [2 cores] 17.1 GB 9.1
Rackspace Cloud 16gb AMD 2374 2.20 GHz [4 cores] 16 GB 5.1

CPU Performance
Small Server

Cloud Server CPU Memory CCUs
BlueLock 2gb Xeon X5550 2.67 GHz [1 core] 2 GB 6.37
KT 2GB/1x2GHz Xeon Xeon L5640 2.27 GHz [1 core] 2 GB 5.98
Terremark 2gb AMD 8389 2.91 GHz [1 core] 2 GB 5.57
Rackspace Cloud 2gb AMD 2374 2.20 GHz [4 cores] 2 GB 5.08
GoGrid 2gb Xeon E5520 2.27 GHz [2 cores] 2 GB 5.02
Amazon EC2 c1.medium Xeon E5410 2.33 GHz [2 cores] 1.7 GB 3.49


Disk IO Performance
Disk IO performance was 20-30% slower than shown below during the loaded benchmark run. The KT cloud uses external SAN storage for VM instance storage. This, combined with the fact that the load simulation benchmarks were fairly disk IO intensive (probably more so than an actual production environment), are likely the reason this. Despite this, disk IO performance was very good. It should be noted that GoGrid and Rackspace Cloud do not utilize external VM instance storage.

Large Server

Cloud Server CPU Memory IOP
KT 16GB/3x2GHz Xeon L5640 2.27 GHz [3 cores] 16 GB 127.05
KT 32GB/6x2GHz Xeon L5640 2.27 GHz [6 cores] 32 GB 125.31
GoGrid 8gb Xeon E5450 2.99 GHz [6 cores] 8 GB 122.62
Terremark 16gb-8vpu AMD 8389 2.91 GHz [8 cores] 16 GB 112.59
Rackspace 16gb AMD 2374 HE 2.20 GHz [4 cores] 16 GB 100.15
Amazon EC2 m2.2xlarge Xeon X5550 2.67 GHz [4 cores] 34.2 96.22
Amazon EC2 m2.xlarge Xeon X5550 2.67 GHz [2 cores] 17.1 87.87

Disk IO Performance
Small Server

Cloud Server CPU Memory IOP
GoGrid 2gb Xeon E5520 2.27 GHz [2 cores] 2 GB 143.35
KT 2gb Xeon L5640 2.27 GHz [1 core] 2 GB 133.08
Terremark 2gb AMD 8389 2.91 GHz [1 core] 2 GB 96.9
Rackspace 2gb AMD 2374 HE 2.20 GHz [4 cores] 2 GB 62.46
BlueLock 2gb Xeon X5550 2.67 GHz [1 core] 2 GB 49
Amazon EC2 c1.medium Xeon E5410 2.33 GHz [2 cores] 1.7 39.69


Programming Language Performance
Benchmark performance in this category was about 10-15% slower during the loaded benchmark run that what is shown below.

Large Server

Cloud Server CPU Memory Score
KT 32GB/6x2GHz Xeon L5640 2.27 GHz [6 cores] 32 GB 123.43
GoGrid 8gb Xeon E5450 2.99 GHz [6 cores] 8 GB 122.22
Amazon EC2 m2.2xlarge Xeon X5550 2.67 GHz [4 cores] 34.2 GB 115.45
BlueLock 16gb-8cpu Xeon X5550 2.67 GHz [8 cores] 16 GB 115.41
KT 16GB/3x2GHz Xeon L5640 2.27 GHz [3 cores] 16 GB 108.45
Terremark 16gb-8vpu AMD 8389 2.91 GHz [8 cores] 16 GB 106.9
Amazon EC2 m2.xlarge Xeon X5550 2.67 GHz [2 cores] 17.1 GB 102.27
Rackspace 16gb AMD 2374 2.20 GHz [4 cores] 16 GB 78.66

Programming Language Performance
Small Server

Cloud Server CPU Memory Score
BlueLock 2gb Xeon X5550 2.67 GHz [1 core] 2 GB 101.31
KT 2GB/1x2GHz Xeon L5640 2.27 GHz [1 core] 2 GB 95.72
Terremark 2gb AMD 8389 2.91 GHz [1 core] 2 GB 94.82
GoGrid 2gb Xeon E5520 2.27 GHz [2 cores] 2 GB 80.82
Rackspace 2gb AMD 2374 2.20 GHz [4 cores] 2 GB 73.71


Memory IO Performance
There was no notable different in memory IO benchmark performance between the loaded and unloaded runs.

Large Server

Cloud Server CPU Memory MIOP
BlueLock 16gb-8cpu Xeon X5550 2.67 GHz [8 cores] 16 GB 117.88
KT 32gb Xeon L5640 2.27 GHz [6 cores] 32 GB 114.48
Amazon EC2 m2.2xlarge Intel Xeon X5550 2.67 GHz [4 processors, 4 cores] 34.2 GB 113.04
KT 16gb Xeon L5640 2.27 GHz [3 cores] 16 GB 108.55
Amazon EC2 m2.xlarge Xeon X5550 2.67 GHz [2 cores] 17.1 GB 102.18
GoGrid 8gb Xeon E5450 2.99 GHz [6 cores] 8 GB 88.25
Rackspace 16gb AMD 2374 2.20 GHz [4 cores] 16 GB 70.09
Terremark 16gb-8vpu AMD 8389 2.91 GHz [8 cores] 16 GB 64.74


Memory IO Performance
Small Server

Cloud Server CPU Memory MIOP
BlueLock 2gb Xeon X5550 2.67 GHz [1 core] 2 GB 103.73
KT 2gb Xeon L5640 2.27 GHz [1 core] 2 GB 99.29
GoGrid 2gb Xeon E5520 2.27 GHz [2 cores] 2 GB 83.74
Terremark 2gb AMD 8389 2.91 GHz [1 core] 2 GB 66.06
Rackspace 2gb AMD 2374 2.20 GHz [4 cores] 2 GB 63.04


Encoding & Encryption Performance
Benchmark performance in this category was about 5-10% slower during the loaded benchmark run that what is shown below.

Large Server

Cloud Server CPU Memory Score
GoGrid 8gb Xeon E5450 2.99 GHz [6 cores] 8 GB 146.51
KT 16gb Xeon L5640 2.27 GHz [3 cores] 16 GB 139.25
KT 32gb Xeon L5640 2.27 GHz [6 cores] 32 GB 139.02
Amazon EC2 m2.2xlarge Xeon X5550 2.67 GHz [4 cores] 34.2 GB 136.32
Amazon EC2 m2.xlarge Xeon X5550 2.67 GHz [2 cores] 17.1 GB 135.81
BlueLock 16gb-8cpu Xeon X5550 2.67 GHz [8 cores] 16 GB 130.11
Rackspace 16gb AMD 2374 2.20 GHz [4 cores] 16 GB 111.2
Terremark 16gb-8vpu AMD 8389 2.91 GHz [8 cores] 16 GB 95.25


Encoding & Encryption Performance
Small Server

Cloud Server CPU Memory Score
KT 2gb Xeon L5640 2.27 GHz [1 core] 2 GB 137.21
Terremark 2gb AMD 8389 2.91 GHz [1 core] 2 GB 131.27
BlueLock 2gb Xeon X5550 2.67 GHz [1 core] 2 GB 119.57
Rackspace 2gb AMD 2374 2.20 GHz [4 cores] 2 GB 108.98
GoGrid 2gb Xeon E5520 2.27 GHz [2 cores] 2 GB 103.78
Amazon EC2 c1.medium Xeon E5410 2.33 GHz [2 cores] 1.7 GB 101.56

Conclusion
Overall the KT cloud performed very well relative to other public IaaS clouds. In particular, disk IO performance was exceptional considering Cloudscaling's use of external storage. By using external storage versus local storage, the KT cloud offers higher fault tolerance because VMs can be quickly or even automatically migrated to another host should the host they are running on fail. This feature is often referred to as high availability. Use of Intel Westmere L5640 processors also helped to provide very good CPU and memory IO performance. VM sizing also showed good linear performance increase from smaller to larger sized instances.

09/08/2010 22:04 PDT

Benchmarking of EC2's new Cluster Compute Instance Type

Two months ago Amazon Web Services released a new "Cluster Compute" EC2 instance type, the cc.4xlarge. This new instance type is targeted for High-Performance Computing (HPC) such as computationally intensive scientific applications. The major differences between this and other EC2 instance types are:
  • Dual quad core "Nehalem" X5570 2.93 processors: compared with X5550 2.67 processors for the next largest m2 instance types. Amazon states this CPU configuration provides 33.5 ECU (EC2 Compute Units) compared with 26 ECUs for their m2.4xlarge instance type (previously the largest instance type)
  • Hardware-Assisted Virtualization (HVM): compared with paravirtualization used by other instance types
  • Multi-node 10 Gbps clustering capabilities: instances can be deployed to separate "Placement Groups" wherein each such group has non-blocking, low latency 10 Gbps network connectivity
Previously, we published 5 blog posts on cloud server performance which did not include this new EC2 instance type (cc.4xlarge) including:
The purpose of this post is to highlight the new EC2 Cluster Compute instance type in the context of these benchmarks and how it performs relative to the other EC2 instance types and servers in other IaaS clouds. For specifics on how the benchmarks are conducted and scores calculated, review the previous blog posts linked above. The benchmarks were performed on an individual cc.4xlarge instance and measure performance of a single instance only. The most beneficial feature of this new instance type is the clustering capabilities via 10 Gbps non-blocking network, which is not highlighted in this post.

The new cluster compute instance type is currently only available in Amazon's US-East region. The benchmark results tables below show only EC2 instances from that same region. NOTE: Although the EC2 documentation states that the cluster compute instance is assigned 2 quad core processors (8 cores total), the processors' hyper-threading capabilities resulted in benchmarks reporting 16 total cores.

CPU Performance
As described in the original post, we calculated CPU performance using a metric we created called the CCU. This metric is based on Amazon's ECU. Amazon states that the new cluster compute instance type should provide 33.5 ECUs. This is fairly close to our calculated 36.85 ECUs. Overall, CPU performance was exceptionally good, exceeding the performance of 134 cloud servers in 28 IaaS clouds from the previous post with the exception of the Storm on Demand 48GB X5650 Westmere cloud server which scored 42.87.

ID CPU Mem Price Geekbench Unixbench CCUs
cc.4xlarge Xeon X5570 2.93 [16 cores] 23 $1.6/hr 12306 1044.3 36.85
m2.4xlarge Xeon X5550 2.67 [8 cores] 68.4 $2/hr 5877 1511 30.72
m2.2xlarge Xeon X5550 2.67 [4 cores] 34.2 $1/hr 5163 1332 25.81
c1.xlarge Xeon E5410 2.33 7 $0.68/hr 5118 780 10.66
m2.xlarge Xeon X5550 2.67 17.1 $0.5/hr 3952 935.8 6.5
m1.xlarge Xeon E5430 2.66 15 $0.68/hr 4256 938.6 5.15
m1.large Xeon E5430 2.66 7.5 $0.34/hr 3092 663.4 4.17
c1.medium Xeon E5410 2.33 1.7 $0.17/hr 2680 758.4 3.49
m1.small Opteron 2218 2.60 1.7 $0.085/hr 1726 179.7 0.9

Disk IO Performance
Disk IO performance was likewise very good. The score of 104.42 signifies that it performed better than the baseline system for this benchmark, a "bare-metal" server running 4 x 15K RPM SAS drives configured with hardware Raid 1+0. For more information, review the previous post. Cloud server storage comes in both local and external storage flavors. External storage provides generally higher reliability (High Availability is only possible with external storage), while local storage provides generally better performance. With one exception (a Terremark vCloud Express Cloud Server), disk IO performance for the new cluster compute instance type was better than any other "external storage" cloud servers including any of EC2 existing instance types.

IDCPUMem
Price (USD)IOP
cc.4xlargeXeon X5570 2.93 [16 cores]23
$1.6/hr104.42
m2.2xlarge
m2.4xlarge
Xeon X550 2.67 [4 cores]
Xeon X5550 2.67 [8 cores]
34.2
68.4

$1/hr
$2/hr
96.22
87.56
m2.xlargeXeon X5550 2.67 [2 cores]17.1
$0.5/hr86.37
c1.xlargeXeon E5410 2.33 [8 cores]7
$0.68/hr74.29
m1.xlargeXeon E5430 2.66 [4 cores]15
$0.68/hr57.34
m1.largeXeon E5430 2.66 [2 cores]7.5
$0.34/hr54.29
c1.mediumXeon E5410 2.33 [2 cores]1.7
$0.17/hr35.76
m1.smallOpteron 2218 HE 2.60 [1 core]1.7
$0.085/hr25.34



Interpreted Programming Language Performance (Java, Python, Ruby, PHP)
In this benchmark category the new cluster compute instance type really shined. It performed significantly better than any of the other 134 cloud servers benchmarked in the previous post. The previous top performers were the Ec2 m2.4xlarge instance with a score of about 139, followed by the Storm on Demand 48GB Westmere cloud server with a score of 124.

ID CPU Memory PHP Python Ruby SPECjvm Score
cc.4xlarge Xeon X5570 2.93 [16 cores] 23 55383 3430 2.84 212.97 159.47
m2.4xlarge Xeon X5550 2.67 [8 cores] 68.4 50328 3725 4.12 197.42 138.58
m2.2xlarge Xeon X5550 2.67 [4 cores] 34.2 50253 3737 4.08 108.09 115.45
m2.xlarge Xeon X5550 2.67 [2 cores] 17.1 50774 3743 3.56 58.73 106.49
c1.xlarge Xeon E5410 2.33 [8 cores] 7 44460 4586 5.46 131.12 105.76
m1.xlarge Xeon E5430 2.66 [4 cores] 15 38737 5279 6.45 68.09 79.62
m1.large Xeon E5430 2.66 [2 cores] 7.5 38625 5324 6.58 38.21 71.29


Memory IO Performance
Memory IO performance was also exceptional for the new cluster compute instance. Its score of 130.6 was the second highest of the 134 cloud servers included in the previous post. Only the 48GB Storm on Demand Westmere server was higher with a score of 136.4.

IDCPUMemoryRedisMIOP
cc.4xlargeXeon X5570 2.93 [16 cores]2358909.51130.6
m2.4xlargeXeon X5550 2.67 [8 cores]68.444330.61117.88
m2.2xlargeXeon X5550 2.67 [4 cores]34.253150.28113.04
m2.xlargeXeon X5550 2.67 [2 cores]17.142735.76104.94
c1.mediumXeon E5410 2.33 [2 cores]1.715766.0968.98
m1.xlargeXeon E5430 2.66 [4 cores]1527314.7663.67
m1.largeXeon E5430 2.66 [2 cores]7.528656.1457.05




Encoding & Encryption Performance
In this benchmark category, the new cluster compute instance type scored second highest again out of the 134 cloud servers benchmarked in the previous post.

ID CPU Memory Encode
cc.4xlarge Xeon X5570 2.93 [16 cores] 23 148.38
m2.2xlarge Xeon X5550 2.67 [4 cores] 34.2 136.32
m2.4xlarge Xeon X5550 2.67 [8 cores] 68.4 136.12
m2.xlarge Xeon X5550 2.67 [2 cores] 17.1 136.09
c1.xlarge Xeon E5410 2.33 [8 cores] 7 119.77
m1.xlarge Xeon E5430 2.66 [4 cores] 15 103.33
m1.large Xeon E5430 2.66 [2 cores] 7.5 103.06
c1.medium Xeon E5410 2.33 [2 cores] 1.7 100.86
m1.small Opteron 2218 HE 2.60 [1 core] 1.7 43.25

Conclusion
The new EC2 cluster compute instance type is an excellent performing cloud server. Performance exceeded that of most of the "bare metal" cloud servers we benchmarked previously. Combined with 10 Gbps non-blocking clustering capabilities, and on-demand deployment & hourly billing, this new instance type provides exceptional value and capabilities for HPC applications.

06/28/2010 06:11 PDT

Cloud Server Benchmarking Part 5: Encoding & Encryption

This is the fifth post in our series on cloud server performance benchmarking. In this post, we'll look at encoding and encryption performance using a compilation of 7 different benchmarks.

Benchmark Setup
All benchmarked cloud servers were configured almost identically using CentOS 64-bit (or 32-bit in the case of EC2 m1.small, c1.medium, Gandi, and IBM cloud servers).

Benchmark Methodology
Individual benchmark scores are calculated using the Phoronix Test Suite. Phoronix runs each test 3 times or until the standard deviation between each execution is less than 3.5% to improve statistical accuracy.

We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is the NewServers Jumbo server configured with dual Intel E5504 quad core 2.00 GHz processors and 48GB DDR3 ECC ram. We chose NewServers because they are the only IaaS cloud that does not utilize a virtualization layer that could adversely affect the benchmark results. All NewServers servers run on top of physical hardware. We assigned the baseline server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 7 benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Benchmarks
The following benchmarks were used to calculate the aggregate encoding performance (Encode) score displayed in the results tables below.
  • Monkey Audio Encoding [weight=100]: This test times how long it takes to encode a sample WAV file to APE format.
  • WAV To FLAC [weight=100]: This test times how long it takes to encode a sample WAV file to FLAC format.
  • WAV To MP3 [weight=100]: LAME is an MP3 encoder licensed under the LGPL. This test measures the time required to encode a WAV file to MP3 format.
  • WAV To Ogg [weight=100]: This test times how long it takes to encode a sample WAV file to Ogg format.
  • WAV To WavPack [weight=100]: This test times how long it takes to encode a sample WAV file to WavPack format.
  • FFmpeg AVI to NTSC VCD [weight=100]: This test uses FFmpeg for testing the systems audio/video encoding performance.
  • GnuPG [weight=100]: This test times how long it takes to encrypt a 2GB file using GnuPG of a file.
Results
The results are divided into tables separated by provider. If the provider has more than one data center location, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), and the aggregate baseline relative score (as described above) linked to the complete Phoronix results. On the high end, there wasn't as much performance variation between providers in this post when compared to the previous 4 performance posts. Additionally, the results show that these benchmarks appear to be most influenced by CPU model and clock speed versus number of CPUs/cores. The top performers in this post are NewServers, GoGrid and Bluelock.


Amazon Web Services (AWS) [US East]
ID CPU Memory Encode
us-east.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.32
us-east.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 136.12
us-east.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 135.81
us-east.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.77
us-east.m1.xlarge Xeon E5430 2.66 [4 processors, 4 cores] 15 103.33
us-east.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 103.06
us-east.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 101.56
us-east.m1.small Opteron 2218 HE 2.60 [1 processor, 1 core] 1.7 43.25





Amazon Web Services (AWS) [US West]
ID CPU Memory Encode
us-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.29
us-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 136.27
us-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 135.86
us-west.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.77
us-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 103.9
us-west.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 103.28
us-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 100.89
us-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 45.2





Amazon Web Services (AWS) [EU West]
ID CPU Memory Encode
eu-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 136.4
eu-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.21
eu-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 136.12
eu-west.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.69
eu-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 103.83
eu-west.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 103.79
eu-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 101.28
eu-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 44.92





Amazon Web Services (AWS) [APAC]
ID CPU Memory Encode
apac.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.12
apac.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 135.83
apac.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 135.49
apac.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.54
apac.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 103.8
apac.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 103.68
apac.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 101.32
apac.m1.small Xeon E5430 2.67 [1 processor, 1 core] 1.7 45.45





Rackspace Cloud [Dallas]
ID CPU Memory Encode
16gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 111.2
2gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 108.98
1gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 108.25
4gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 106.02





Rackspace Cloud [Chicago]
ID CPU Memory Encode
16gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 112.44
8gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 8 112.44
4gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 111.58
1gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 108.5
2gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 107.61





Storm on Demand [MI, US]
ID CPU Memory Encode
48gb Xeon X5650 2.67 [1 processor, 12 cores] 45.9 135.59
i5-750-4gb Core i5 750 2.67 [1 processor, 4 cores] 4 130.83
4gb Core i5 750 2.67 [1 processor, 2 cores] 3.5 130
i5-750-2gb Core i5 750 2.67 [1 processor, 4 cores] 2 128.51
2gb Core 2 Quad Q9400 2.66 [1 processor, 1 core] 1.7 124.27
8gb Xeon X3440 2.53 [1 processor, 4 cores] 7 123.64
x3440-8gb Xeon X3440 2.53 [1 processor, 8 cores] 8 123.4
16gb Opteron 2378 2.40 [2 processors, 4 cores] 15.2 121.05
32gb Opteron 2378 2.40 [4 processors, 8 cores] 30.4 120.76
e5506x2-4gb Xeon E5506 2.13 [1 processor, 8 cores] 8 108.32
e5506x2-4gb Xeon E5506 2.13 [1 processor, 8 cores] 8 108.31
e5506x2-8gb Xeon E5506 2.13 [1 processor, 8 cores] 8 108.1
e5506x2-8gb Xeon E5506 2.13 [1 processor, 8 cores] 8 107.75
amd2350x2-32gb Opteron 2350 2.00 [4 processors, 8 cores] 32 95.23





GoGrid [US West]
ID CPU Memory Encode
8gb Xeon E5450 2.99 [1 processor, 6 cores] 8 146.51
4gb Xeon E5520 2.27 [1 processor, 4 cores] 4 108.36
1gb Xeon E5520 2.27 [1 processor, 1 core] 1 104.3
2gb Xeon E5520 2.27 [1 processor, 2 cores] 2 103.78





Voxel [New York]
ID CPU Memory Encode
4gb-ny Xeon L5520 2.26 [2 processors, 2 cores] 4 119.95
8gb-ny Xeon L5520 2.26 [4 processors, 4 cores] 8 118.91
14gb-ny Xeon L5520 2.26 [7 processors, 7 cores] 14 118.79
2gb-ny Xeon L5520 2.26 [1 processor, 1 core] 2 106.67





Voxel [Amsterdam]
ID CPU Memory Encode
8gb-nl Xeon L5520 2.26 [4 processors, 4 cores] 8 122.62
14gb-nl Xeon L5520 2.26 [7 processors, 7 cores] 14 122.6
4gb-nl Xeon L5520 2.26 [2 processors, 2 cores] 4 122.58
2gb-nl Xeon L5520 2.26 [1 processor, 1 core] 2 114.22





Voxel [Singapore]
ID CPU Memory Encode
14gb-sg Xeon L5520 2.26 [7 processors, 7 cores] 14 115.9
8gb-sg Xeon L5520 2.26 [4 processors, 4 cores] 8 115.83
4gb-sg Xeon L5520 2.26 [2 processors, 2 cores] 4 115.8
2gb-sg Xeon L5520 2.26 [1 processor, 1 core] 2 112.29





NewServers [Miami]
ID CPU Memory Encode
fast Xeon E5450 2.99 [1 processor, 4 cores] 4 152.43
large Xeon E5405 2.00 [1 processor, 4 cores] 4 101.06
jumbo Xeon E5504 2.00 [2 processors, 8 cores] 48 100
med Xeon 3.20 [1 processor, 2 cores] 2 94.28
small Xeon 2.80 [1 processor, 1 core] 1 81.55





Linode VPS Hosting [Atlanta]
ID CPU Memory Encode
5760-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 5.54 115.71
14400-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 14.06 115.66
8640-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 8.45 108.25
1080-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 1.05 108.05
2880-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 2.81 106.65





SoftLayer [Dallas]
ID CPU Memory Encode
1gb-dallas Xeon X3460 2.80 [1 processor, 1 core] 1 126.76
4gb-dallas Xeon E5520 2.27 [4 processors, 4 cores] 4 111.51
8gb-dallas Xeon E5520 2.27 [8 processors, 8 cores] 8 111.28
2gb-dallas Xeon X3460 2.80 [2 processors, 2 cores] 2 96.74





SoftLayer [WDC]
ID CPU Memory Encode
4gb-wdc Xeon X3460 2.80 [4 processors, 4 cores] 4 137.71
1gb-wdc Xeon X3460 2.80 [1 processor, 1 core] 1 131.49
2gb-wdc Xeon X3460 2.80 [2 processors, 2 cores] 2 129.06
8gb-wdc Xeon E5520 2.27 [8 processors, 8 cores] 8 116.04





SoftLayer [Seattle]
ID CPU Memory Encode
4gb-seattle Xeon X3460 2.80 [4 processors, 4 cores] 4 131.51
1gb-seattle Xeon X3460 2.80 [1 processor, 1 core] 1 122.06
2gb-seattle Xeon X3460 2.80 [2 processors, 2 cores] 2 117.75
8gb-seattle Xeon E5520 2.27 [8 processors, 8 cores] 8 111.14





Terremark vCloud Express [Miami]
ID CPU Memory Encode
1gb Opteron 8389 2.91 [1 processor, 1 core] 1 138.13
2gb Opteron 8389 2.91 [1 processor, 1 core] 2 131.27
4gb-2vpu Opteron 8389 2.91 [1 processor, 2 cores] 4 128.26
8gb-4vpu Opteron 8389 2.91 [1 processor, 4 cores] 8 105.19
16gb-8vpu Opteron 8389 2.91 [1 processor, 8 cores] 16 95.25





VPS.NET [Atlanta]
ID CPU Memory Encode
18node-atl Xeon E5520 2.27 [7 processors, 7 cores] 6.6 106.3
12node-atl Xeon E5520 2.27 [5 processors, 5 cores] 4.4 102.46
6node-atl Xeon E5520 2.27 [3 processors, 3 cores] 2.2 94.66
3node-atl Xeon E5520 2.27 [2 processors, 2 cores] 1.1 89.67





OpSource Cloud [VA, US]
ID CPU Memory Encode
1gb Xeon X7460 2.66 [1 processor, 1 core] 1 72.27
4gb-2cpu Xeon X7460 2.66 [1 processor, 2 cores] 4 70.06
2gb Xeon X7460 2.66 [1 processor, 1 core] 2 68.77
16gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 16 63.81
8gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 8 62.54
32gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 32 61.01





Speedyrails [Quebec]
ID CPU Memory Encode
4gb Xeon E5520 2.27 [2 processors, 8 cores] 4 113.13
1gb Xeon E5520 2.27 [2 processors, 8 cores] 1 111.37
2gb Xeon E5520 2.27 [2 processors, 8 cores] 2 106.45





Zerigo [CO, US]
ID CPU Memory Encode
4gb Opteron 2374 HE 2.20 [2 processors, 4 cores] 4 110.53
2gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 109.61
1gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 108.97





ReliaCloud [MN, US]
ID CPU Memory Encode
4gb Xeon E5504 2.00 [1 processor, 4 cores] 4 38.95
8gb 8 38.81
2gb Xeon E5504 4.40 [1 processor, 2 cores] 2 35.2
1gb Xeon E5504 2.00 [1 processor, 1 core] 1 24.15





Gandi.net [France]
ID CPU Memory Encode
6gb Xeon L5640 2.27GHz 2.26 [1 processor, 6 cores] 6 125.01
4gb Opteron 8346 HE 1.79 [1 processor, 4 cores] 4 72.22
1gb Opteron 8346 HE 1.79 [1 processor, 1 core] 1 70.61
2gb Opteron 8346 HE 1.79 [1 processor, 2 cores] 2 67.37





CloudSigma [Switzerland]
ID CPU Memory Encode
8gb-8gh Six-Core Opteron 2427 2.21 [1 processor, 4 cores] 8 111.01
16gb-16gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 16 110.75
4gb-4gh Six-Core Opteron 2427 2.21 [1 processor, 2 cores] 4 110.19
32gb-20gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 32 109.59
1gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 1 95.24
2gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 2 91.15





IBM Development & Test Cloud [NY, US]
ID CPU Memory Encode
dev-large Xeon X5570 2.93 [1 processor, 8 cores] 3.5 126.89
dev-med Xeon X5570 2.93 [1 processor, 4 cores] 1.75 98.55





BlueLock [IN, US]
ID CPU Memory Encode
8gb-4cpu Xeon X5550 2.67 [1 processor, 4 cores] 8 142.31
4gb-2cpu Xeon X5550 2.67 [1 processor, 2 cores] 4 133.29
16gb-8cpu Xeon X5550 2.67 [1 processor, 8 cores] 16 130.11
2gb Xeon X5550 2.67 [1 processor, 1 core] 2 119.57
1gb Xeon X5550 2.67 [1 processor, 1 core] 1 113.3





Cloud Central [Australia]
ID CPU Memory Encode
huge Opteron 2374 HE 2.20 [4 processors, 4 cores] 16 112.23
reg Opteron 2374 HE 2.20 [4 processors, 4 cores] 4 112.1
large Opteron 2374 HE 2.20 [4 processors, 4 cores] 8 111.82
med Opteron 2374 HE 2.20 [4 processors, 4 cores] 2 103.44
small Opteron 2374 HE 2.20 [4 processors, 4 cores] 1 100.05





RimuHosting [Dallas]
ID CPU Memory Encode
tx-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 101.27





RimuHosting [Auckland]
ID CPU Memory Encode
nz-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 105.98





ElasticHosts [Peer1 UK]
ID CPU Memory Encode
8gb-20gh Xeon E5420 2.50 [1 processor, 6 cores] 8 124.54
1gb-2gh Xeon E5420 2.50 [1 processor, 1 core] 1 118.62
4gb-8gh Xeon E5420 2.50 [1 processor, 4 cores] 4 118
2gb-4gh Xeon E5420 2.50 [1 processor, 2 cores] 2 113.55





Flexiscale [UK]
ID CPU Memory Encode
4gb Dual-Core Opteron 8220 2.80 [1 processor, 2 cores] 4 126.58
8gb Opteron 8218 2.62 [1 processor, 4 cores] 8 115.72
2gb Opteron 8218 2.60 [1 processor, 1 core] 2 109.76
1gb Opteron 8218 2.60 [1 processor, 1 core] 1 105.06


06/21/2010 09:01 PDT

Cloud Server Benchmarking Part 4: Memory IO

This is the fourth post in our series on cloud server performance benchmarking. In this post, we'll examine memory IO performance using a compilation of 7 different benchmarks. This is the last of the performance posts to focus on synthetic benchmarks. Memory IO performance is particularly important for applications that read and write from memory heavily like caching systems and memory-based data stores such as memcache and redis.

Benchmark Setup
All benchmarked cloud servers were configured almost identically using CentOS 64-bit (or 32-bit in the case of EC2 m1.small, c1.medium, Gandi, and IBM cloud servers).

Benchmark Methodology
We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is the NewServers Jumbo server configured with dual Intel E5504 quad core 2.00 GHz processors and 48GB DDR3 ECC ram. We chose NewServers because they are the only IaaS cloud that does not utilize a virtualization layer that could adversely affect the benchmark results. All NewServers servers run on top of physical hardware. We assigned the baseline server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 7 benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Benchmarks
The following benchmarks were used to calculate the aggregate memory IO performance (MIOP) score displayed in the results tables below. Geekbench and Unixbench measure both CPU and memory IO performance and are included in the aggregate MIOP metric with a weight of 25 points each. To view the raw Geekbench and Unixbench scores for each of the servers in this post see the post What is an ECU? CPU Benchmarking in the Cloud.
  • CacheBench [weight=100]: This is a performance test of CacheBench, which is part of LLCbench. CacheBench is designed to test the memory and cache bandwidth performance. 50 points are assigned to each of the benchmarks read and write.
  • Geekbench [weight=25]: Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand, Geekbench takes the guesswork out of producing robust and reliable benchmark results.
  • hdparm cached reads [weight=50]: Determines the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.
  • RAMspeed [weight=100]: This benchmark tests the system memory (RAM) performance. 33 points are assigned to each of the benchmarks add, copy and scale.
  • Redis Benchmark [weight=50]: Redis is an in-memory key-value store. It includes the redis-benchmark utility that simulates SETs/GETs/INCRs/LPUSHs/LPOPs/PINGs/LRANGEs done by N clients at the same time sending M total queries (it is similar to the Apache's ab utility). Our benchmark is performed with 50 simultaneous clients performing 100000 requests (./redis-benchmark -n 100000). The result is the average requests per second for all benchmark actions (gets, sets, etc.). For specific results, download the benchmark output source.
  • Stream [weight=100]: This benchmark tests the system memory (RAM) performance. 25 points are assigned to each of the benchmarks add, copy, scale and triad.
  • Unixbench [weight=25]: UnixBench provides a basic indicator of the performance of a Unix-like system. Multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system.
Results
The results are divided into tables separated by provider. If the provider has more than one data center location, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), Redis Benchmark score (linked to the complete Redis benchmark results), and the aggregate baseline relative score (as described above). The top performers in this benchmark category were EC2, Storm on Demand (6 servers with scores 116 and above), BlueLock, and IBM Developer Cloud.


Amazon Web Services (AWS) [US East]
ID CPU Memory Redis MIOP
us-east.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 44330.61 117.88
us-east.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 53150.28 113.04
us-east.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 43767.98 102.18
us-east.m1.xlarge Xeon E5430 2.66 [4 processors, 4 cores] 15 27314.76 63.67
us-east.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 28656.14 57.05




Amazon Web Services (AWS) [US West]
ID CPU Memory Redis MIOP
us-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 42603.58 118.12
us-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 48163.06 112.11
us-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 52766.05 106.11
us-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 23420.06 70.83
us-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 31635.65 65.49
us-west.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 28608.91 64.93
us-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 9214.6 31.81




Amazon Web Services (AWS) [EU West]
ID CPU Memory Redis MIOP
eu-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 44084.09 117.27
eu-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 48563.97 111.74
eu-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 46370.88 104.07
eu-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 37373.2 73.48
eu-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 28149.67 65.94
eu-west.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 27864.93 63.22
eu-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 9038.17 33.52




Amazon Web Services (AWS) [APAC]
ID CPU Memory Redis MIOP
apac.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 15859.76 97.48
apac.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 15587.35 95.94
apac.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 31992.3 92.74
apac.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 16492.91 68.46
apac.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 23171.72 66.47
apac.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 21667.04 59.67
apac.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 18181.59 59.47
apac.m1.small Xeon E5430 2.67 [1 processor, 1 core] 1.7 7465.1 31.67




Rackspace Cloud [Dallas]
ID CPU Memory Redis MIOP
16gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 23844.89 70.09
4gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 16928.56 63.44
2gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 19167 63.04
1gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 15945.68 62.83




Rackspace Cloud [Chicago]
ID CPU Memory Redis MIOP
16gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 25400.7 70.77
8gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 8 25528.68 68.65
2gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 24860.47 68.14
1gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 25719.31 67.84
4gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 25248.93 66.34




Storm on Demand [MI, US]
ID CPU Memory Redis MIOP
48gb Xeon X5650 2.67 [1 processor, 12 cores] 45.9 62269.22 136.4
i5-750-4gb-2disks Core i5 750 2.67 [1 processor, 4 cores] 4 79473.07 124.49
i5-750-2gb-1disk Core i5 750 2.67 [1 processor, 4 cores] 2 81301.01 122.88
x3440-8gb-2disks Xeon X3440 2.53 [1 processor, 8 cores] 8 74553.53 121.15
8gb Xeon X3440 2.53 [1 processor, 4 cores] 7 74183.16 119.81
4gb Core i5 750 2.67 [1 processor, 2 cores] 3.5 76701.91 116.99
e5506x2-4gb-2disks Xeon E5506 2.13 [1 processor, 8 cores] 8 49970.66 83.18
e5506x2-4gb-1disk Xeon E5506 2.13 [1 processor, 8 cores] 8 49760.06 82.8
e5506x2-8gb-4disks Xeon E5506 2.13 [1 processor, 8 cores] 8 49775.14 76.5
32gb Opteron 2378 2.40 [4 processors, 8 cores] 30.4 37672.21 75.82
e5506x2-8gb-4disks Xeon E5506 2.13 [1 processor, 8 cores] 8 49178.7 75.36
2gb Core 2 Quad Q9400 2.66 [1 processor, 1 core] 1.7 22571.9 65
amd2350x2-32gb Opteron 2350 2.00 [4 processors, 8 cores] 32 29966.25 59.15
16gb Opteron 2350 2.00 [2 processors, 4 cores] 15.2 29179.3 52.32




GoGrid [CA, US]
ID CPU Memory Redis MIOP
4gb Xeon E5520 2.27 [1 processor, 4 cores] 4 46176.73 96.12
8gb Xeon E5450 2.99 [1 processor, 6 cores] 8 32590.17 89.74
2gb Xeon E5520 2.27 [1 processor, 2 cores] 2 39756.91 83.74
1gb Xeon E5520 2.27 [1 processor, 1 core] 1 23346.48 79.7




Voxel [New York]
ID CPU Memory Redis MIOP
14gb-ny Xeon L5520 2.26 [7 processors, 7 cores] 14 30566.6 100.02
8gb-ny Xeon L5520 2.26 [4 processors, 4 cores] 8 29376.08 89.78
4gb-ny Xeon L5520 2.26 [2 processors, 2 cores] 4 23520.43 88.62
2gb-ny Xeon L5520 2.26 [1 processor, 1 core] 2 14434.5 79.8




Voxel [Amsterdam]
ID CPU Memory Redis MIOP
14gb-nl Xeon L5520 2.26 [7 processors, 7 cores] 14 27423.78 105.25
8gb-nl Xeon L5520 2.26 [4 processors, 4 cores] 8 30788.54 101.71
4gb-nl Xeon L5520 2.26 [2 processors, 2 cores] 4 27979.96 96.63
2gb-nl Xeon L5520 2.26 [1 processor, 1 core] 2 14691.72 86




Voxel [Singapore]
ID CPU Memory Redis MIOP
14gb-sg Xeon L5520 2.26 [7 processors, 7 cores] 14 28547.88 87.16
8gb-sg Xeon L5520 2.26 [4 processors, 4 cores] 8 27832.58 84.95
4gb-sg Xeon L5520 2.26 [2 processors, 2 cores] 4 24189.04 77.39
2gb-sg Xeon L5520 2.26 [1 processor, 1 core] 2 15237.89 75.99




NewServers [FL, US]
ID CPU Memory Redis MIOP
jumbo Xeon E5504 2.00 [2 processors, 8 cores] 48 39567.78 100
large Xeon E5405 2.00 [1 processor, 4 cores] 4 34997.3 72.25
med Xeon 3.20 [1 processor, 2 cores] 2 22225.77 43.78
small Xeon 2.80 [1 processor, 1 core] 1 11758.45 31.8




Linode VPS Hosting [Atlanta]
ID CPU Memory Redis MIOP
14400-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 14.06 29486.24 99.9
5760-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 5.54 13240.74 96.33
2880-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 2.81 20404.32 93.26
1080-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 1.05 22810.13 87.48
8640-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 8.45 12350.71 86.39




SoftLayer [Dallas]
ID CPU Memory Redis MIOP
4gb-dallas Xeon E5520 2.27 [4 processors, 4 cores] 4 26091.52 98.27
8gb-dallas Xeon E5520 2.27 [8 processors, 8 cores] 8 25067.69 96.93
2gb-dallas Xeon X3460 2.80 [2 processors, 2 cores] 2 22067.26 95.74
1gb-dallas Xeon X3460 2.80 [1 processor, 1 core] 1 14838.57 94.58



SoftLayer [Seattle]
ID CPU Memory Redis MIOP
8gb-seattle Xeon E5520 2.27 [8 processors, 8 cores] 8 25503.99 105.11
4gb-seattle Xeon X3460 2.80 [4 processors, 4 cores] 4 37422.67 103.86
2gb-seattle Xeon X3460 2.80 [2 processors, 2 cores] 2 34444.9 99.62
1gb-seattle Xeon X3460 2.80 [1 processor, 1 core] 1 14836.2 93.01




Terremark vCloud Express [FL, US]
ID CPU Memory Redis MIOP
1gb Opteron 8389 2.91 [1 processor, 1 core] 1 37476.72 66.78
2gb Opteron 8389 2.91 [1 processor, 1 core] 2 37367.9 66.06
16gb-8vpu Opteron 8389 2.91 [1 processor, 8 cores] 16 36601.03 64.74
4gb-2vpu Opteron 8389 2.91 [1 processor, 2 cores] 4 42788.76 64.03
8gb-4vpu Opteron 8389 2.91 [1 processor, 4 cores] 8 41525.25 61.08




VPS.NET [Atlanta]
ID CPU Memory Redis MIOP
12node-atl Xeon E5520 2.27 [5 processors, 5 cores] 4.4 28676.91 98.17
18node-atl Xeon E5520 2.27 [7 processors, 7 cores] 6.6 17864.39 88.83
3node-atl Xeon E5520 2.27 [2 processors, 2 cores] 1.1 25017.57 81
6node-atl Xeon E5520 2.27 [3 processors, 3 cores] 2.2 23600.54 73.24




OpSource Cloud [VA, US]
ID CPU Memory Redis MIOP
1gb Xeon X7460 2.66 [1 processor, 1 core] 1 13552.35 38.13
2gb Xeon X7460 2.66 [1 processor, 1 core] 2 12775.31 36.56
4gb-2cpu Xeon X7460 2.66 [1 processor, 2 cores] 4 13772.58 36.52
8gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 8 13538.06 34.62
16gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 16 13131.61 33.88
32gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 32 12260.02 32.99




Speedyrails [Quebec]
ID CPU Memory Redis MIOP
1gb Xeon E5520 2.27 [2 processors, 8 cores] 1 20705.27 92.89
4gb Xeon E5520 2.27 [2 processors, 8 cores] 4 21118.42 87.52
2gb Xeon E5520 2.27 [2 processors, 8 cores] 2 19086.71 83.5




Zerigo [CO, US]
ID CPU Memory Redis MIOP
4gb Opteron 2374 HE 2.20 [2 processors, 4 cores] 4 16428.46 63.81




ReliaCloud [MN, US]
ID CPU Memory Redis MIOP
4gb Xeon E5504 2.00 [1 processor, 4 cores] 4 6499.32 34.04
2gb Xeon E5504 4.40 [1 processor, 2 cores] 2 5559.85 32.21
1gb Xeon E5504 2.00 [1 processor, 1 core] 1 5300.22 22.42




Gandi.net [France]
ID CPU Memory Redis MIOP
4gb Opteron 8346 HE 1.79 [1 processor, 4 cores] 4 33143.29 46.59
6gb Opteron 8346 HE 1.79 [1 processor, 6 cores] 6 10647.31 43.35
2gb Opteron 8346 HE 1.79 [1 processor, 2 cores] 2 10713.03 36.56
1gb Opteron 8346 HE 1.79 [1 processor, 1 core] 1 11255.61 33.6




CloudSigma [Switzerland]
ID CPU Memory Redis MIOP
16gb-16gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 16 37227.13 72.03
32gb-20gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 32 36628.6 67.26
4gb-4gh Six-Core Opteron 2427 2.21 [1 processor, 2 cores] 4 37030.85 60.68
2gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 2 21457.67 56.7
1gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 1 21682.06 55.96




IBM Development & Test Cloud [NY, US]
ID CPU Memory Redis MIOP
dev-large Xeon X5570 2.93 [1 processor, 8 cores] 3.5 62280.44 124.37
dev-med Xeon X5570 2.93 [1 processor, 4 cores] 1.75 46424.68 95.5




BlueLock [IN, US]
ID CPU Memory Redis MIOP
16gb-8cpu Xeon X5550 2.67 [1 processor, 8 cores] 16 57578.66 117.88
8gb-4cpu Xeon X5550 2.67 [1 processor, 4 cores] 8 48148.66 115.64
4gb-2cpu Xeon X5550 2.67 [1 processor, 2 cores] 4 57390.96 107.8
2gb Xeon X5550 2.67 [1 processor, 1 core] 2 29641.05 103.73
1gb Xeon X5550 2.67 [1 processor, 1 core] 1 24710.52 91.9




Cloud Central [AU]
ID CPU Memory Redis MIOP
large Opteron 2374 HE 2.20 [4 processors, 4 cores] 8 18585.76 60.91
reg Opteron 2374 HE 2.20 [4 processors, 4 cores] 4 19328.25 59.37
huge Opteron 2374 HE 2.20 [4 processors, 4 cores] 16 13726.38 57.93
med Opteron 2374 HE 2.20 [4 processors, 4 cores] 2 18325.34 57.23
small Opteron 2374 HE 2.20 [4 processors, 4 cores] 1 17260.76 56.05




RimuHosting [TX, US]
ID CPU Memory Redis MIOP
tx-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 24771.2 71.62




RimuHosting [NZ]
ID CPU Memory Redis MIOP
nz-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 22446.26 71.22




ElasticHosts [UK]
ID CPU Memory Redis MIOP
4gb-8gh Xeon E5420 2.50 [1 processor, 4 cores] 4 38376.45 74.91
1gb-2gh Xeon E5420 2.50 [1 processor, 1 core] 1 17456.46 67.03
2gb-4gh Xeon E5420 2.50 [1 processor, 2 cores] 2 24852.55 66.4




Flexiscale [UK]
ID CPU Memory Redis MIOP
8gb Opteron 8218 2.62 [1 processor, 4 cores] 8 20853.61 44.9
4gb Dual-Core Opteron 8220 2.80 [1 processor, 2 cores] 4 22004.85 44.51
2gb Opteron 8218 2.60 [1 processor, 1 core] 2 17149.07 42.86
1gb Opteron 8218 2.60 [1 processor, 1 core] 1 16410.28 37.14

06/12/2010 12:33 PDT

Cloud Server Benchmarking Part 3: Java, Ruby, Python & PHP

This is the third post in our series on cloud server performance benchmarking. The previous blog posts, What is an ECU? CPU Benchmarking in the Cloud and Disk IO Benchmarking in the Cloud focused on more synthetic (i.e. raw performance numbers with no real-world application) CPU and Disk IO performance. In this post, we'll look at performance using 4 common interpreted (full or byte-code) programming languages: Java, Ruby, Python and PHP.

This post is by no means intended to provide exhaustive and exact metrics for cloud server performance with these languages. The purpose of this and the other benchmarking posts is to provide a quick reference and starting point for further research and evaluation of different cloud providers. There is much variation in performance, features, reliability, support, pricing and other factors of different cloud providers, yet if you were to evaluate those providers based solely on marketing literature, you'd be hard pressed to distinguish one from another. Our goal is to reduce this ambiguity by providing objective, quantifiable measurements for comparing cloud providers.

Benchmark Setup
All benchmarked cloud servers were configured almost identically in terms of OS and software:
  • Operating System: CentOS 64-bit (except for IBM Developer Cloud using RHEL 5 32-bit)
  • File System: ext3
  • Sun JDK 1.6
  • Ruby 1.8.5
  • Python 2.4.3
  • PHP 5.1.6
Benchmark Methodology
We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is Storm on Demand's bare-metal instance on E5506 2.13 hardware (dual processors - 8 cores total), with 4 x 15K RPM SAS drives (Raid 10) and 8GB ram. This is a fairly high-end server with many cores and very fast IO. We assigned this server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 4 language benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Example Score Calculation:
Baseline aggregate score = SPECjvm2008: 100; ruby-benchmark-suite: 100; PyBench: 100; PHPBench 100
Total baseline score = 400

Server X aggregate score = SPECjvm2008: 80/100; ruby-benchmark-suite: 90/100; PyBench: 70/100; PHPBench 95/100
Total baseline score = 335
Server X Score = (335/400) * 100 = 83.75

Benchmarks
The following benchmarks were used in this post:
  • SPECjvm (Java Virtual Machine Benchmark) [Higher score is better]: A benchmark suite for measuring the performance of a Java Runtime Environment (JRE), containing several real life applications and benchmarks focusing on core java functionality. The suite focuses on the performance of the JRE executing a single application; it reflects the performance of the hardware processor and memory subsystem, but has low dependence on file I/O and includes no network I/O across machines. The SPECjvm2008 workload mimics a variety of common general purpose application computations. These characteristics reflect the intent that this benchmark will be applicable to measuring basic Java performance on a wide variety of both client and server systems.
  • Ruby [Lower score is better]: A suite for measuring the performance of Ruby implementations, including micro-benchmarks that focus on core Ruby functionality, as well as macro-benchmarks that represent a variety of real, common workloads. The project is aimed at providing a useful suite for comparing the performance of the available Ruby implementations and, as a positive side effect, to give VM implementers an additional tool to measure and identity performance related issues. The score for this benchmark is the average of the median times to execute 13 of the macro benchmarks (all macro benchmarks minus bm_hilbert_matrix.rb). Specific times for these benchmarks individually are available in the benchmark source file.
  • Python [Lower score is better]: A collection of tests that provides a standardized way to measure the performance of Python implementations. The score for this benchmark is the average of all benchmarks performed by pybench.
  • PHP [Higher score is better]: A benchmark suite for PHP. It performs a large number of simple tests in order to benchmark various aspects of the PHP interpreter. PHPBench can be used to compare hardware, operating systems, PHP versions, PHP accelerators and caches, compiler options, etc.
Results
The results are divided into tables separated by provider. If the provider has more than one data center locations, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), PHPBench score, PyBench score, ruby-benchmark-suite score, SPECjvm2008 score, and the aggregate baseline relative score (as described above). Each of the 4 individual scores are linked to the complete results for those specific tests (a single small text file for PHPBench, PyBench, ruby-benchmark-suite and a 3-4MB zip file for SPECjvm2008).

Note: Some smaller server configurations (e.g. EC2 m1.small and c1.medium) are not included in these results due to their inability to run one more more of the individual benchmarks.


Amazon Web Services (AWS) [US East]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50328 3725 4.12 197.42 138.58
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50253 3737 4.08 108.09 115.45
c1.xlarge Xeon E5410 2.33 8 cores 7 44460 4586 5.46 131.12 105.76
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50396 3756 4.1 58.15 102.27
m1.xlarge Xeon E5430 2.66 4 cores 15 38737 5279 6.45 68.09 79.62
m1.large Xeon E5430 2.66 2 cores 7.5 38625 5324 6.58 38.21 71.29



Amazon Web Services (AWS) [US West]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50553 3723 4.09 197.83 139.02
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50062 3797 4.1 108.53 114.82
c1.xlarge Xeon E5410 2.33 8 cores 7 44733 4566 5.48 130.26 105.75
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50832 3724 4.12 58.59 102.81
m1.xlarge Xeon E5430 2.67 4 cores 15 39001 5323 6.36 67.98 79.79
m1.large Xeon E5430 2.66 2 cores 7.5 38716 5382 6.46 38.17 71.38



Amazon Web Services (AWS) [EU West]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50775 3739 4.1 200.09 139.55
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50207 3723 4.1 107.32 115.22
c1.xlarge Xeon E5410 2.33 8 cores 7 44656 4563 5.51 131.52 105.94
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50275 3732 4.11 58.92 102.55
m1.xlarge Xeon E5430 2.67 4 cores 15 38387 5259 6.37 69.06 79.94
m1.large Xeon E5430 2.67 2 cores 7.5 38327 5299 6.39 39.15 71.91



Amazon Web Services (AWS) [APAC]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50336 3743 4.42 131.72 119.05
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50095 3762 4.42 89.86 108.51
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50042 3799 4.43 50.63 97.92
c1.xlarge Xeon E5410 2.33 8 cores 7 44992 4608 5.57 63.93 88.13
m1.xlarge Xeon E5430 2.67 4 cores 15 38793 5314 6.56 54.15 75.62
m1.large Xeon E5430 2.67 2 cores 7.5 38214 5299 6.49 34.1 70.28



Rackspace Cloud [Dallas]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 2374 HE 2.20 4 cores 4 39210 5465 5.74 63.07 79.76
16gb Opteron 2374 HE 2.20 4 cores 16 37914 5517 5.75 62.77 78.66
1gb Opteron 2374 HE 2.20 4 cores 1 38679 5396 5.81 56.22 77.73
2gb Opteron 2374 HE 2.20 4 cores 2 38868 5454 5.87 41.89 73.71



Rackspace Cloud [Chicago]
ID CPU Memory PHP Python Ruby SPECjvm Score
16gb-il Opteron 2374 HE 2.20 4 cores 16 38729 5397 5.7 63.86 80.08
4gb-il Opteron 2374 HE 2.20 4 cores 4 39065 5424 5.8 62.14 79.42
8gb-il Opteron 2374 HE 2.20 4 cores 8 38958 5455 5.66 63.88 78.31
1gb-il Opteron 2374 HE 2.20 4 cores 1 38443 5434 5.68 56.88 77.99



Storm on Demand [MI, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
48gb Xeon X5650 2.67 12 cores 45.9 50602 3810 6.17 174.81 124.05
i5-750-4gb-2disks Intel i5 750 2.67 4 cores 4 45560 4326 3.17 89.03 110.35
i5-750-2gb-1disk Intel i5 750 2.67 4 cores 2 45620 4352 3.16 82.16 108.53
x3440-8gb-2disks Xeon X3440 2.53 8 cores 8 43428 4599 3.38 97.18 107.55
8gb Xeon X3440 2.53 4 cores 7 43861 4565 3.35 86.55 105.51
e5506x2-4gb-1disk Xeon E5506 2.13 8 cores 8 40577 4774 3.93 108.06 103.52
e5506x2-4gb-2disks Xeon E5506 2.13 8 cores 8 40136 4766 3.94 105.9 102.67
32gb Opteron 2378 2.40 8 cores 30.4 40954 5104 3.8 104.01 101.96
4gb Core i5 750 2.67 2 cores 3.5 45983 4331 3.17 49.71 100.35
e5506x2-8gb-4disks Xeon E5506 2.13 8 cores 8 40731 4765 4.02 96.18 100
e5506x2-8gb-4disks Xeon E5506 2.13 8 cores 8 40306 4780 4.01 95.13 99.45
2gb Core 2 Quad Q9400 2.66 1 core 1.7 49905 4142 3.59 23.91 93.6
amd2350x2-32gb-4disks Opteron 2350 2.00 8 cores 32 32182 6271 4.96 86.73 81.55
16gb Opteron 2350 2.00 4 cores 15.2 32240 6187 4.87 53.1 73.48



GoGrid [CA, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5450 2.99 6 cores 8 55970 3616 3.53 101.76 121.46
4gb Xeon E5520 2.27 4 cores 4 40513 4806 3.8 74.34 95.42
2gb Xeon E5520 2.27 2 cores 2 38206 5090 4.09 36.13 80.82
1gb Xeon E5520 2.27 1 core 1 38215 4945 4.02 20.56 77.89



Voxel [New York]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-ny Xeon L5520 2.26 7 cores 14 44463 4214 4.98 119.82 106.89
8gb-ny Xeon L5520 2.26 4 cores 8 44043 4237 4.91 81.23 96.73
4gb-ny Xeon L5520 2.26 2 cores 4 40921 4446 4.95 45.71 84.1
2gb-ny Xeon L5520 2.26 1 core 2 44022 4210 4.94 24.47 82.02



Voxel [Amsterdam]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-nl Xeon L5520 2.26 7 cores 14 45069 4221 4.85 124.22 108.9
8gb-nl Xeon L5520 2.26 4 cores 8 45043 4219 4.9 82.02 97.71
4gb-nl Xeon L5520 2.26 2 cores 4 44880 4217 4.83 45.52 88.44
2gb-nl Xeon L5520 2.26 1 core 2 44436 4229 4.86 24.57 82.51



Voxel [Singapore]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-sg Xeon L5520 2.26 7 cores 14 42488 4468 5.14 111.08 101.17
8gb-sg Xeon L5520 2.26 4 cores 8 42355 4473 5.15 75.75 91.84
4gb-sg Xeon L5520 2.26 2 cores 4 42433 4480 5.15 42.15 83.11
2gb-sg Xeon L5520 2.26 1 core 2 42153 4482 5.16 23.08 77.93



NewServers [FL, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
fast Xeon E5450 2.99 4 cores 4 57795 3600 3.24 81.3 120.71
jumbo Xeon E5504 2.00 8 cores 48 37593 5087 4.2 110.03 99.02
large Xeon E5405 2.00 4 cores 4 38161 5462 4.59 59.75 82.66
med Xeon 3.20 2 cores 2 29381 7383 5.89 25.74 57.92
small Xeon 2.80 1 core 1 25223 8183 6.9 12.37 47.82



Linode VPS Hosting [Atlanta]
ID CPU Memory PHP Python Ruby SPECjvm Score
5760 Xeon L5520 2.27 4 cores 5.54 42841 4418 4.67 78.51 95.19
14400 Xeon L5520 2.27 4 cores 14.06 42644 4410 4.65 78.45 95.19
2880 Xeon L5520 2.27 4 cores 2.81 39050 4503 5.02 66.78 87.8
1080 Xeon L5520 2.27 4 cores 1.05 40442 4579 5.12 59.57 85.95
8640 Xeon L5520 2.27 4 cores 8.45 32904 5632 5.28 67.85 78.02



SoftLayer [Dallas]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42414 4439 5.5 123.82 103.33
4gb Xeon E5520 2.27 4 cores 4 42471 4459 5.51 78.74 91.49
2gb Xeon X3460 2.80 2 cores 2 47816 4130 5.53 49.59 89.26
1gb Xeon X3460 2.80 1 core 1 47634 4118 4.61 27.86 87.21



SoftLayer [WDC]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42981 4441 5.48 127.55 104.7
4gb Xeon X3460 2.80 4 cores 4 47425 4101 4.62 91.07 103.58
2gb Xeon X3460 2.80 2 cores 2 47478 4108 4.4 48.95 93.7
1gb Xeon X3460 2.80 1 core 1 47614 4113 4.47 27.61 87.85



SoftLayer [Seattle]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42642 4451 5.54 125.75 103.76
2gb Xeon X3460 2.80 2 cores 2 48095 4138 4.63 48.7 92.67
1gb Xeon X3460 2.80 1 core 1 47472 4113 4.61 26.49 86.79



Terremark vCloud Express [FL, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
16gb-8vpu Opteron 8389 2.91 8 cores 16 48625 4623 3.99 100.42 106.9
1gb Opteron 8389 2.91 1 core 1 48997 4254 3.18 22 95.4
2gb Opteron 8389 2.91 1 core 2 48651 4321 3.17 21.87 94.82
4gb-2vpu Opteron 8389 2.91 2 cores 4 39078 5019 3.72 38.78 84.82
8gb-4vpu Opteron 8389 2.91 4 cores 8 32959 5388 4.19 70.56 84.67



VPS.NET [Atlanta]
ID CPU Memory PHP Python Ruby SPECjvm Score
18node Xeon E5520 2.27 7 cores 6.6 39447 4762 5.85 96.35 91.45
12node Xeon E5520 2.27 [5 processors, 5 cores 4.4 42843 5035 5.39 77.12 88.65
6node Xeon E5520 2.27 [3 processors, 3 cores 2.2 33213 5730 6.8 45.81 67.86
3node Xeon E5520 2.27 2 cores 1.1 34188 5452 6.54 35.72 67.49



OpSource Cloud [VA, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
2gb Xeon X7460 2.66 1 core 2 28083 7721 7.93 11.13 48.23
8gb-4cpu Xeon X7460 2.66 4 cores 8 24639 8758 9.08 9.03 42.14
32gb-4cpu Xeon X7460 2.66 4 cores 32 23604 8790 8.87 9.04 41.72
16gb-4cpu Xeon X7460 2.66 4 cores 16 24636 8738 9.43 8.77 41.69



Speedyrails [Quebec]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Xeon E5520 2.27 8 cores 4 41819 4735 5.26 100.74 96.12
1gb Xeon E5520 2.27 8 cores 1 41541 4975 5.28 56.54 83.18



Zerigo [CO, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 2374 HE 2.20 4 cores 4 39030 5400 5.58 55.54 78.46



ReliaCloud [MN, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Xeon E5504 2.00 4 cores 4 13287 13209 10.7 3.85 27.56
8gb 8 14159 15567 10.79 3.69 26.62
2gb Xeon E5504 4.40 2 cores 2 12131 13819 11.36 5.36 26.31



CloudSigma [Switzerland]
ID CPU Memory PHP Python Ruby SPECjvm Score
32gb-20gh Six-Core Opteron 2427 2.21 8 cores 32 37912 5646 4.27 97.2 93.17
16gb-16gh Six-Core Opteron 2427 2.21 8 cores 16 37599 5654 4.3 98.23 93.05
8gb-8gh Six-Core Opteron 2427 2.21 4 cores 8 37473 5619 4.2 59.99 83.72
4gb-4gh Six-Core Opteron 2427 2.21 2 cores 4 37409 5582 4.26 32.54 76.35
1gb-2gh Six-Core Opteron 2427 2.21 1 core 1 37326 5609 4.27 17.2 72.16
2gb-2gh Six-Core Opteron 2427 2.21 1 core 2 36561 5789 4.25 17.82 71.3



IBM Development & Test Cloud [NY, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
large Xeon X5570 2.93 8 cores 3.5 45260 4711 3.6 140.6 117.53



BlueLock [IN, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb-4cpu Xeon X5550 2.67 4 cores 8 54399 3761 3 94.08 123.02
16gb-8cpu Xeon X5550 2.67 8 cores 16 48314 3859 3.36 96.07 115.41
4gb-2cpu Xeon X5550 2.67 2 cores 4 45071 3894 3.21 51.9 103.05
2gb Xeon X5550 2.67 1 core 2 50021 3739 3.16 26.72 101.31
1gb Xeon X5550 2.67 1 core 1 49499 3745 3.06 16.01 99.2



Cloud Central [Australia]
ID CPU Memory PHP Python Ruby SPECjvm Score
reg Opteron 2374 HE 2.20 4 cores 4 38725 5469 5.55 60.59 79.41
large Opteron 2374 HE 2.20 4 cores 8 38921 5439 5.64 60.58 79.36
huge Opteron 2374 HE 2.20 4 cores 16 37989 5573 6.43 61.86 76.4
small Opteron 2374 HE 2.20 4 cores 1 37356 5597 6.18 49.6 73.36



RimuHosting [TX, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
tx-2gb Xeon E5506 2.13 2 cores 2 39096 4883 4.89 39.28 79.16



RimuHosting [New Zealand]
ID CPU Memory PHP Python Ruby SPECjvm Score
nz-2gb Xeon E5506 2.13 2 cores 2 39529 4854 5.08 39.22 78.78



ElasticHosts [UK]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb-20gh Xeon E5420 2.50 6 cores 8 47168 4431 4.1 92.82 104.47
2gb-4gh Xeon E5420 2.50 2 cores 2 43723 4508 4.11 34.92 86.79
1gb-2gh Xeon E5420 2.50 1 core 1 44943 4470 4.1 20.94 84.19



Flexiscale [UK]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 8220 2.80 2 cores 4 36319 6033 4.03 34.1 69.57
8gb Opteron 8218 2.62 4 cores 8 33878 6570 4.35 53.46 67.87
2gb Opteron 8218 2.60 1 core 2 35141 6505 4.16 17.19 63.09
1gb Opteron 8218 2.60 1 core 1 34497 6590 4.39 16.16 63.05




Summary
We think it is important for these and other cloud performance benchmarks to be made available publicly. With all the hype surrounding cloud computing today, new cloud users may not understand what exactly they are getting in terms of performance and scalability. As these benchmarks show, not all clouds are created equal, and there can be very pronounced performance variations from one provider to another. One of the greatest benefits of the cloud is simplicity and a perceived ability to infinitely scale. However, it should be understood that with that simplicity and abstraction, the cloud is still powered by physical hardware, CPUs, disks and memory, and subject to the same performance limitations that same hardware exhibits in a non-cloud environment.

06/11/2010 15:28 PDT

Google Storage, a CDN/Storage Hybrid?

We received our Google Storage for Developers account today, and being the curious types, immediately ran some network tests. We maintain a network of about 40 cloud servers around the world which we use to monitor cloud servers, test cloud-to-cloud network throughput, and our cloud speedtest. We used a handful of these to test Google Storage uplink and downlink throughput and latency. We were very surprised by the low latency and consistently fast downlink throughput to most locations.

Most storage services like Amazon's S3 and Azure Blob Storage are physically hosted from a single geographical location. S3 for example, is divided into 4 regions, US West, US East, EU West and APAC. If you store a file to an S3 US West bucket, it is not replicated to the other regions, and can only be downloaded from that regions' servers. The result is much slower network performance from locations with poor connectivity to that region. Hence, you need to chose your S3 region wisely based on geographical and/or network proximity. Azure's Blob storage uses a similar approach. Users are able to add CDN on top of those services (CloudFront and Azure CDN), but the CDN does not provide the same access control and consistency features of the storage service.

In contrast, Google's new Storage for Developers service appears to store uploaded files to a globally distributed network of servers. When a file is requested, Google uses some DNS magic to direct a user to a server that will provide the fastest access to that file. This is very similar to the approach that Content Delivery Networks like Akamai and Edgecast work, wherein files are distributed to multiple globally placed PoPs (point-of-presence).

Our simple test consisted of requesting a test 10MB file from 17 different servers located in the US, EU and APAC. The test file was set to public and we used wget to test downlink and Google's gsutil to test uplink throughput (wget was faster than gsutil for downloads). In doing so, we found the same test URL resolved to 11 different Google servers with an average downlink of about 40 Mb/s! This hybrid model of CDN-like performance with enterprise storage features like durability, consistency and access control represents an exciting leap forward for cloud storage!

Google Storage Network Performance Tests
Service Location Resolved To Latency Upload (Mb/s) Download (Mb/s)
Gigenet IL, US 209.85.225.132 15.05 15.13 68.4
CloudSigma Switzerland 209.85.227.132 24.25 1.75 21.5
Linode (London) UK 209.85.229.132 7.13 5.72 41.6
Bluelock IN, US 64.233.169.132 23.1 4.42 56.8
EC2 (EU West) Ireland 66.102.9.132 2.19 6.58 29.5
CloudCentral Australia 66.102.11.132 5.96 4.84 8.2
Rimu Hosting New Zealand 66.102.11.132 27.05 5.53 9.64
Gandi Cloud VPS France 66.102.13.132 19.9 2.81 31.1
Zerigo CO, US 72.14.203.132 57.05 5.63 24.04
Voxel (Amsterdam) The Netherlands 72.14.204.132 83.2 3.86 40.6
EC2 (US East) VA, US 72.14.204.132 2.72 10.14 39.4
EC2 (US West) VA, US 72.14.204.132 27.7 8.77 9.5
Speedyrails Canada 72.14.204.132 24.5 8.28 33.1
VPS.net (UK) UK 74.125.79.132 11.6 3.48 32.1
Rackspace (Chicago) IL, US 74.125.95.132 13.05 4.23 70.3
GoGrid CA, US 74.125.127.132 24.2 6.32 39.9
Zerigo CO, US 74.125.155.132 55.3 10.51 42.08

06/05/2010 19:22 PDT

Disk IO Benchmarking in the Cloud

This is the second post in our series on cloud server performance benchmarking. The previous blog post, What is an ECU? CPU Benchmarking in the Cloud, focused strictly on CPU performance. In this post, we'll look at disk IO performance. Choosing a cloud provider should be based on a many factors including performance, price, support, reliability/uptime, scalability, network performance and features. Our intent is to provide a good reference for those looking to use cloud services by providing objective analysis in regards to all of these factors.

Benchmark Setup
All benchmarked cloud servers were configured almost identically in terms of OS and software, CentOS 5.4 64-bit (or 32-bit in the case of EC2 m1.small and c1.medium and IBM's Development Cloud where 64-bit is not supported). File systems were formatted ext3.

Benchmark Methodology
In the previous post we used Amazon's EC2 ECU (Elastic Compute Unit) as a baseline for comparison of CPU performance between providers. In terms of disk performance, there really isn't a common term synonymous to the ECU. However, most readers will be at least be somewhat familiar with hardware disk IO performance factors such drive types (SAS, SATA), spindle speeds (10K, 15K) and RAID levels. So, we chose to use a "bare-metal" cloud server as the baseline for disk IO performance in this post. Our experience has been that most providers will not disclose much technical detail about their underlying storage systems. However, in the case of Storm on Demand's new Bare Metal Cloud Servers they fully disclose most technical details about their servers (one of the selling points of this service). With this service, you are assigned a dedicated server. Your OS still runs on a hypervisor (Storm uses Xen), but the underlying hardware is not shared with any other virtual servers, and you thereby have the full resources of that server available (no CPU limits, disk or memory sharing).

The server model we chose as the performance comparison baseline is the dual processor Intel E5506 2.13 GHz (8 cores total) with 4 x 15K RPM SAS drives configured in hardware managed Raid 1+0. SAS 15K is one of the fastest storage interfaces with throughput up to 6 Gbp/s, and Raid 1+0 can improve performance using striping.

To use this server as the baseline we assigned it an aggregate IO Performance score (IOP) of exactly 100 points. Other server disk IO benchmark results were then compared to baseline results and assigned a relative score, where 100 is equal in performance, less than 100 worse, and greater than 100 better. For example, a server with a score of 50 scored 50% lower than the baseline server, while a server with a score of 125, scored 25% higher.

To compute the IOP, the results from each of the benchmarks is first calculated for the baseline server. Each benchmark has a weight for the overall aggregate score. The baseline server benchmark scores are the 100% mark for each of these weights (i.e. the baseline server receives the full weight of each benchmark for its IOP). Once the weights are calculated, they are then summed to create an aggregate score. This score is then compared with the aggregate score of the baseline server and use to produce the IOP.

Example IOP Calculation:
Baseline aggregate score = blogbench-read: 200/200; bonnie++: 100/100; dbench: 30/30; fio: 30/30; hdparm: 100/100; iozone: 200/200; tiobench: 30/30; Total aggregate score = 690
Baseline IOP = (690/690) * 100 = 100
Server X aggregate score = blogbench-read: 180/200; bonnie++: 80/100; dbench: 25/30; fio: 20/30; hdparm: 90/100; iozone: 175/200; tiobench: 28/30; Total aggregate score = 598
Server X IOP = (598/690) * 100 = 86.67

Benchmarks
We used a combination 7 disk IO performance benchmarks to create the IOP score. The following is a description of the benchmarks and corresponding weights use:
  • Blogbench [weight=200]: BlogBench is designed to replicate the load of a real-world busy file server by stressing the file-system with multiple threads of random reads, writes, and rewrites. The behavior is mimicked of that of a blog by creating blogs with content and pictures, modifying blog posts, adding comments to these blogs, and then reading the content of the blogs. All of these blogs generated are created locally with fake content and pictures.
  • Bonnie++ [weight=100]: Bonnie++ is based on the Bonnie hard drive benchmark by Tim Bray. This program is used by ReiserFS developers, but can be useful for anyone who wants to know how fast their hard drive or file system is.
  • Dbench (128 clients) [weight=30]: Dbench is a benchmark designed by the Samba project as a free alternative to netbench, but dbench contains only file-system calls for testing the disk performance.
  • Flexible IO Tester (fio) [weight=30]: fio is an I/O tool meant to be used both for benchmark and stress/hardware verification. It has support for 13 different types of I/O engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more), I/O priorities (for newer Linux kernels), rate I/O, forked or threaded jobs, and much more.
  • hdparm buffered disk reads [weight=100]: Determines the speed of reading through the buffer cache to the disk without any prior caching of data. This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead.
  • IOzone (4GB reads & writes) [weight=200]: The IOzone benchmark tests the hard disk drive/file-system performance.
  • Threaded I/O Tester (64 MB random write; write; read) [weight=30]: Tiotester (Threaded I/O Tester) benchmarks the hard disk drive / file-system performance.
We credit the Phoronix Test Suite for making it easier to run the benchmarks. The tests above come from the Disk test suite (except for bonnie++). If you'd like to compare your own server to the baseline server in this post, you can install Phoronix and use the comparison feature when running your own IO benchmarks. The full baseline server results are available here (including the ID for comparison tests). The baseline server bonnie++ results are available here.

Results
The following results are divided into sections based on provider. If the provider has more than one data center, multiple tables are displayed one for each. Each table shows the server identifier, CPU architecture, memory (GB), storage description (if known) or size, the server price, and the IOP score.

Amazon EC2 servers can use either ephemeral (local host) or external Elastic Block Storage (EBS) storage. Regarding EBS, Amazon states "The latency and throughput of Amazon EBS volumes is designed to be significantly better than the Amazon EC2 instance stores in nearly all cases.". EBS is a block level off-instance storage, meaning it exists independent of the underlying host. Striping of multiple EBS volumes can be used to improve performance. We used a single EBS volume only (no striping or other performance enhancements).

Because EBS is off-instance, it has the advantage of added durability because if the host system fails the instance can be quickly restarted on another host (this is not automatic). EBS volumes are automatically replicated to prevent data loss due to failure of any single hardware component. EBS is billed at $0.10/GB/month + $0.10 per 1 million I/O requests. The prices below do not include EBS usage charges.

Amazon states that larger instances will see faster and more consistent throughput performance. We found this to be generally true, until we got into the m2 instances where larger instance performance increase was not apparent.

Following suit with the CPU benchmarks, we saw a notable difference in performance in the APAC region compared with instances of the same size in other regions.

We've noted the standard EC2 hourly pricing in the tables. EC2 also offers reserve pricing (pay a 1 or 3 year setup in exchange for significantly discounted hourly rates) and spot/bid pricing.

Amazon Web Services (AWS) [US East]
ID CPU Memory Storage Price IOP
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 2.4/hr 96.88
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.5/hr 88.9
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.4/hr 87.56
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.68/hr 70.88
m1.xlarge Xeon E5430 2.66 4 cores 15 EBS 0.68/hr 57.03
m1.large Xeon E5430 2.66 2 cores 7.5 EBS 0.34/hr 53.01
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.17/hr 34.82
m1.small Opteron 2218 2.60 1 core 1.7 EBS 0.085/hr 22.2


Amazon Web Services (AWS) [US West]
ID CPU Memory Storage Price IOP
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.57/hr 104.36
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 1.34/hr 97.87
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.68/hr 89.56
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.76/hr 59.53
m1.xlarge Xeon E5430 2.67 4 cores 15 EBS 0.76/hr 58.45
m1.large Xeon E5430 2.66 2 cores 7.5 EBS 0.38/hr 49.11
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.19/hr 38.65
m1.small Xeon E5430 2.66 1 core 1.7 EBS 0.095/hr 26.47


Amazon Web Services (AWS) [EU West]
ID CPU Memory Storage Price IOP
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 1.34/hr 92.66
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.57/hr 91.82
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.68/hr 81.91
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.76/hr 71.31
m1.large Xeon E5430 2.67 2 cores 7.5 EBS 0.38/hr 54.81
m1.xlarge Xeon E5430 2.67 4 cores 15 EBS 0.76/hr 54.48
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.19/hr 35.98
m1.small Xeon E5430 2.66 1 core 1.7 EBS 0.095/hr 22.28


Amazon Web Services (AWS) [APAC]
ID CPU Memory Storage Price IOP
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 1.34/hr 81.59
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.68/hr 80.81
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.57/hr 76.44
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.76/hr 68.68
m1.large Xeon E5430 2.67 2 cores 7.5 EBS 0.38/hr 66.12
m1.xlarge Xeon E5430 2.67 4 cores 15 EBS 0.76/hr 57.47
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.19/hr 35.43
m1.small Xeon E5430 2.67 1 core 1.7 EBS 0.095/hr 27.89



Rackspace Cloud
Rackspace Cloud uses local storage for cloud servers. This provides generally good performance on nodes of all sizes. However, the tradeoff is that it does not provide durability should the host system fail. Rackspace does offers scheduled hot backup/imaging capabilities for added durability.

Rackspace Cloud [Dallas]
ID CPU Memory Storage Price IOP
rs-16gb Opteron 2374 2.20 4 cores 16 620 GB 0.96/hr 86.75
rs-4gb Opteron 2374 2.20 4 cores 4 160 GB 0.24/hr 63.1
rs-2gb Opteron 2374 2.20 4 cores 2 80 GB 0.12/hr 57.91
rs-1gb Opteron 2374 2.20 4 cores 1 40 GB 0.06/hr 57.36


Rackspace Cloud [Chicago]
ID CPU Memory Storage Price IOP
rs-8gb-il Opteron 2374 2.20 4 cores 8 320 GB 0.48/hr 77.95
rs-16gb-il Opteron 2374 2.20 4 cores 16 620 GB 0.96/hr 74.43
rs-4gb-il Opteron 2374 2.20 4 cores 4 160 GB 0.24/hr 72.12
rs-2gb-il Opteron 2374 2.20 4 cores 2 80 GB 0.12/hr 56.33
rs-1gb-il Opteron 2374 2.20 4 cores 1 40 GB 0.06/hr 45.84


Storm on Demand
Storm offers a diverse range of cloud servers to chose from including traditional cloud servers and their new "bare metal" dedicated cloud servers. The row highlighted in green is the baseline server for the IOP metric in this post. The 48GB cloud server was the top performer at almost 70% faster than the baseline server! We aren't sure how they do this, but we do know that the Intel
Westmere CPU it runs on is very high performing (it was also the top performer in the CPU benchmarks post), and most likely the storage components are very high end as well. To our knowledge, all Storm's cloud servers run off of local host storage. They offer automated daily backups and imaging for added durability.

Storm Cloud [MI, US]
ID CPU Memory Storage Price IOP
storm-48gb Xeon X5650 2.67 12 cores 45.9 1500 GB 1.37/hr 168.48
e5506x2-8gb Xeon E5506 2.13 8 cores 8 4 SAS Raid 10 261GB 0.48/hr 100
storm-8gb Xeon X3440 2.53 4 cores 7 600 GB 0.27/hr 96.94
storm-32gb Opteron 2378 2.40 8 cores 30.4 1500 GB 0.69/hr 90.99
e5506x2-8gb Xeon E5506 2.13 8 cores 8 4 SATA Raid 10 917GB 0.391/hr 90.3
x3440-8gb Xeon X3440 2.53 8 cores 8 2 SATA Raid 1 919GB 0.274/hr 78.62
amd2350x2-32gb Opteron 2350 2.00 8 cores 32 4 SATA Raid 10 1853GB 0.713/hr 77.61
storm-16gb Opteron 2350 2.00 4 cores 15.2 750 GB 0.34/hr 72.49
i5-750-4gb Intel i5 750 2.67 4 cores 4 2 SATA Raid 1 220GB 0.206/hr 69.47
storm-4gb Intel i5 750 2.67 2 cores 3.5 300 GB 0.14/hr 60.86
i5-750-2gb Intel i5 750 2.67 4 cores 2 1 200GB SATA No Raid 0.171/hr 49.76
e5506x2-4gb Xeon E5506 2.13 8 cores 8 1 453GB SATA No Raid 0.274/hr 48.37
e5506x2-4gb Xeon E5506 2.13 8 cores 8 2 SATA Raid 1 453GB 0.322/hr 46.8
storm-2gb Intel Q9400 2.66 1 core 1.7 150 GB 0.07/hr 34.95


GoGrid
GoGrid's disk IO performance was excellent across instances of all sizes. Even the low end 1GB cloud server scored almost 10% faster than the baseline. The 4GB cloud server was the #2 performer in this post at over 60% better performance than the baseline.
GoGrid's cloud servers run off of local host storage. They offer prepaid plans starting at $199/mo which discounts usage to $0.08/GB/hr (versus non-prepaid $0.19/GB/hr).

GoGrid [CA, US]
ID CPU Memory Storage Price IOP
gg-4gb Xeon E5520 2.27 4 cores 4 240 GB 0.76/hr 161.14
gg-2gb Xeon E5520 2.27 2 cores 2 120 GB 0.38/hr 133.61
gg-8gb Xeon E5450 2.99 6 cores 8 480 GB 1.52/hr 130.68
gg-1gb Xeon E5520 2.27 1 core 1 60 GB 0.19/hr 108.19


Voxel
Voxel's cloud server storage resides on external SANs in each data center. Disk IO performance was roughly 50% of the baseline across the board, about average for providers using external instance storage. The exact technical details of Voxel's SAN storage are not available.

Voxel [New York]
ID CPU Memory Storage Price IOP
vx-4gb-ny Xeon L5520 2.26 2 cores 4 40 GB 0.211/hr 49.65
vx-2gb-ny Xeon L5520 2.26 1 core 2 20 GB 0.106/hr 41.9

Voxel [Amsterdam]
ID CPU Memory Storage Price IOP
vx-2gb-nl Xeon L5520 2.26 1 core 2 20 GB 0.106/hr 40.61

Voxel [Singapore]
ID CPU Memory Storage Price IOP
vx-4gb-sg Xeon L5520 2.26 2 cores 4 40 GB 0.211/hr 51.52
vx-2gb-sg Xeon L5520 2.26 1 core 2 20 GB 0.106/hr 43.49

NewServers
NewServers offers a true bare metal cloud. When you create a server instance it actually deploys onto physical hardware including local disk storage (no hypervisor overhead). Like the baseline server, the ns-fast instance below uses SAS storage. According to NewServers, the SAS drives used in this instance are Seagate 10K RPM. The slower spindle speed and Raid 1 configuration may have attributed to the reduced performance of that server compared with the baseline.

NewServers [FL, US]
ID CPU Memory Storage Price IOP
ns-jumbo Xeon E5504 2.00 8 cores 48 2 x 500GB SATA RAID 1 0.6/hr 84.15
ns-fast Xeon E5450 2.99 4 cores 4 2 x 300GB SAS RAID 1 0.53/hr 51.97
ns-large Xeon E5405 2.00 4 cores 4 1 x 250GB SATA 0.25/hr 41.17
ns-med Opteron 3.20 2 cores 2 2 x 73GB SCSI Raid 1 0.17/hr 29.42
ns-small Opteron 2.80 1 core 1 1 x 36GB SCSI 0.11/hr 22.79


Linode
Linode markets itself as a VPS provider. However, because they are a very common and popular provider, and support some cloud-like features, we included them in our benchmarks. Linode uses Raid 1 local storage. The higher end servers generally performed better. The ln-14400 performed slightly better than the baseline server.

Linode VPS Hosting [Atlanta]
ID CPU Memory Storage Price IOP
ln-14400 Xeon L5520 2.27 4 cores 14.06 640 GB 26.67/day 106.06
ln-5760 Xeon L5520 2.27 4 cores 5.54 256 GB 10.67/day 96.21
ln-8640 Xeon L5520 2.27 4 cores 8.45 383 GB 16/day 95.38
ln-2880 Xeon L5520 2.27 4 cores 2.81 128 GB 5.33/day 47.17
ln-1080 Xeon L5520 2.27 4 cores 1.05 48 GB 2/day 34.47


SoftLayer
SoftLayer's cloud server storage resides on external SANs. Disk IO performance ranged from OK to painfully slow and showed a general increase in performance on larger sized instances. We believe storage is based on a GigE iSCCI SAN.

SoftLayer [Dallas]
ID CPU Memory Storage Price IOP
sl-2gb-dallas Xeon X3460 2.80 2 cores 2 100 GB 0.25/hr 41.07
sl-4gb-dallas Xeon E5520 2.27 4 cores 4 100 GB 0.35/hr 31.22
sl-1gb-dallas Xeon X3460 2.80 1 core 1 100 GB 0.15/hr 22.6


SoftLayer [WDC]
ID CPU Memory Storage Price IOP
sl-4gb-wdc Xeon X3460 2.80 4 cores 4 100 GB 0.35/hr 51.07
sl-2gb-wdc Xeon X3460 2.80 2 cores 2 100 GB 0.25/hr 41.53
sl-1gb-wdc Xeon X3460 2.80 1 core 1 100 GB 0.15/hr 19.33


SoftLayer [Seattle]
ID CPU Memory Storage Price IOP
sl-4gb-seattle Xeon X3460 2.80 4 cores 4 100 GB 0.35/hr 49.49
sl-2gb-seattle Xeon X3460 2.80 2 cores 2 100 GB 0.25/hr 39.74
sl-1gb-seattle Xeon X3460 2.80 1 core 1 100 GB 0.15/hr 24.89


Terremark
Terremark is a VMWare vCloud provider. Terremark was the best performing provider in this post using external SAN storage and providing automatic failover capabilities. Use of the SAN and VMWare provide a high availability feature wherein servers are automatically migrated to another host should the existing host fail.

Terremark vCloud Express [FL, US]
ID CPU Memory Storage Price IOP
tm-8gb-4vpu Opteron 8389 2.91 4 cores 8 80 GB 0.64/hr 117.5
tm-16gb-8vpu Opteron 8389 2.91 8 cores 16 160 GB 1.672/hr 101.85
tm-4gb-2vpu Opteron 8389 2.91 2 cores 4 40 GB 0.305/hr 95.86
tm-2gb Opteron 8389 2.91 1 core 2 20 GB 0.137/hr 88.99
tm-1gb Opteron 8389 2.91 1 core 1 10 GB 0.074/hr 74.68


OpSource Cloud
OpSource is another VMWare-based provider. C
loud server storage resides on an external SAN. Performance was generally slower than other SAN/VMWare providers in this post. Performance was roughly flat across instances of all sizes.

OpSource Cloud [VA, US]
ID CPU Memory Storage Price IOP
os-2gb Xeon X7460 2.66 1 core 2 20 GB 0.296/hr 42.98
os-16gb-4cpu Xeon X7460 2.66 4 cores 16 160 GB 0.808/hr 40.84
os-32gb-4cpu Xeon X7460 2.66 4 cores 32 320 GB 1.256/hr 40.73
os-8gb-4cpu Xeon X7460 2.66 4 cores 8 80 GB 0.584/hr 39.75
os-4gb-2cpu Xeon X7460 2.66 2 cores 4 40 GB 0.392/hr 38.24
os-1gb Xeon X7460 2.66 1 core 1 10 GB 0.268/hr 37.34


Speedyrails
Speedyrails is a VPS provider based out of Quebec Canada. Cloud server storage is local to the host.

Speedyrails [QC, CA]
ID CPU Memory Storage Price IOP
sr-4gb Xeon E5520 2.27 8 cores 4 160 GB 8/day 66.72
sr-2gb Xeon E5520 2.27 8 cores 2 80 GB 4.27/day 57.99
sr-1gb Xeon E5520 2.27 8 cores 1 40 GB 2.27/day 33.43


Zerigo
Zerigo is a VPS and Cloud Server vendor based out of Denver, CO.
Cloud server storage is local to the host.

Zerigo [CO, US]
ID CPU Memory Storage Price IOP
zr-4gb Opteron 2374 2.20 4 cores 4 160 GB 0.24/hr 52.64
zr-2gb Opteron 2374 2.20 4 cores 2 80 GB 0.12/hr 42.2
zr-1gb Opteron 2374 2.20 4 cores 1 40 GB 0.06/hr 29.12


ReliaCloud
ReliaCloud uses an external SAN for cloud server instance storage. Average performance was slower than other external storage providers.

ReliaCloud Cloud Services [MN, US]
ID CPU Memory Storage Price IOP
rc-8gb 8 800 GB 0.64/hr 29.82
rc-2gb Xeon E5504 4.40 2 cores 2 200 GB 0.16/hr 27.41
rc-1gb Xeon E5504 2.00 1 core 1 100 GB 0.08/hr 23.83
rc-4gb Xeon E5504 2.00 4 cores 4 400 GB 0.32/hr 23.32


IBM Development & Test Cloud
IBM Developer Cloud also uses SAN storage. Disk IO performance was about average compared with other external storage providers.

IBM Development & Test Cloud [NY, US]
ID CPU Memory Storage Price IOP
ibm-dev-large Xeon X5570 2.93 8 cores 3.5 30 GB 56.53
ibm-dev-med Xeon X5570 2.93 4 cores 1.75 20 GB 51.82


BlueLock
BlueLock is a VMWare vCloud provider. Server storage resides on a SAN.
Combined SAN and VMWare provide a high availability feature wherein servers are automatically migrated to another host should the existing host fail. Performance was very good for external storage providers and generally showed a linear increase in performance on larger sized instances.

BlueLock [IN, US]
ID CPU Memory Storage Price IOP
bl-8gb-4cpu Xeon X5550 2.67 4 cores 8 160 GB 0.661/hr 71.68
bl-16gb-8cpu Xeon X5550 2.67 8 cores 16 320 GB 1.729/hr 62.49
bl-4gb-2cpu Xeon X5550 2.67 2 cores 4 80 GB 0.308/hr 54
bl-2gb Xeon X5550 2.67 1 core 2 40 GB 0.134/hr 41.35
bl-1gb Xeon X5550 2.67 1 core 1 20 GB 0.068/hr 25.75


CloudCentral
Cloud Central is a new cloud server provider in Australia. Cloud Central uses SAN storage. Performance was very good compared with other SAN providers.

Cloud Central [AU]
ID CPU Memory Storage Price (AUD) IOP
cc-large Opteron 2374 2.20 4 cores 8 512 GB 0.96/hr 77.11
cc-huge Opteron 2374 2.20 4 cores 16 1024 GB 1.92/hr 73.77
cc-reg Opteron 2374 2.20 4 cores 4 256 GB 0.48/hr 68.53
cc-med Opteron 2374 2.20 4 cores 2 128 GB 0.24/hr 62.42
cc-small Opteron 2374 2.20 4 cores 1 64 GB 0.12/hr 44.27


RimuHosting is a VPS provider based out of New Zealand. Cloud server storage is local to the host.

RimuHosting [NZ]
ID CPU Memory Storage Price (NZD) IOP
rh-nz-2gb Xeon E5506 2.13 2 cores 2 16 GB 210.88/mo 38.47


ElasticHosts is a UK cloud provider. We are unaware if storage is local or external.

ElasticHosts [UK]
ID CPU Memory Storage Price (GBP) IOP
eh-8gb-20gh Xeon E5420 2.50 6 cores 8 80 GB 0.654/hr 57.91
eh-4gb-8gh Xeon E5420 2.50 4 cores 4 40 GB 0.326/hr 33.06
eh-2gb-4gh Xeon E5420 2.50 2 cores 2 20 GB 0.164/hr 31.25
eh-1gb-2gh Xeon E5420 2.50 1 core 1 10 GB 0.082/hr 23.1


Flexiscale
Flexiscale is a UK cloud provider that has been around for a few years. They were recently acquired and renamed to Flexiant. They have recently released version 2.0 of their cloud server platform. They use external SAN storage.

8/17/2010 Update: Since originally conducting these tests in May 2010, Flexiscale has updated their external storage system. The top 8gb instance in the table below represents benchmark results conducted after than update. These results show a dramatic increase in IO performance (about 35%).


Flexiscale [UK]
ID CPU Memory Storage IOP
8gb AMD Opteron 8356 2.29 GHz [1 processor, 4 cores] 8 320 GB 84.93
8gb AMD Opteron 8218 2.62 GHz [1 processor, 4 cores] 8 320 GB 62.71
4gb AMD Opteron 8220 2.80 GHz [1 processor, 2 cores] 8 160 GB 41.77
2gb AMD Opteron 8218 2.60 GHz [1 processor, 1 core] 8 80 GB 36.56
1gb AMD Opteron 8218 2.60 GHz [1 processor, 1 core] 8 40 GB 19.46



Summary
This is our first attempt at comparing cloud provider disk IO performance. We acknowledge that it is not perfect and hope to make improvements over time. Please comment on this post if you have any suggestions on how we might improve our methods. We intend to continually run these benchmarks (every couple of months) to incorporate new providers, improve the quality and quantity of our data available, and check for provider upgrades and improvements.

Providers were about evenly split between those using external SAN storage versus those using local host storage for cloud server instance storage. Generally the local storage providers performed better in the benchmarks. However, most of the external SAN providers are able to incorporate better durability including implicit backups and automatic host server failover.

05/29/2010 16:06 PDT

What is an ECU? CPU Benchmarking in the Cloud

NOTE: This post has been updated after it's original writing. The original CPU performance metrics did not accurately depict performance on multi-core servers. The updated post utilizes an improved method of calculating CPU performance that applies more weight to multi-core aware benchmarks (see benchmarks description below for more info).

Over the past couple of months we've spent some time benchmarking about 150 different cloud server configurations with 20 different vendors. This included all 8 AWS EC2 instances types (m1.small - m2.4xlarge) in all 4 regions (32 servers total for EC2). The benchmark suite we ran includes about 100 different benchmarks from synthetic benchmarks measuring raw CPU performance such as Unixbench and Geekbench to higher level application benchmarks such as mysql-bench, pgbench, tpcc-mysql and blog bench. This post will be the first in a series highlighting the results of these benchmarks. In it, we'll focus purely on raw CPU performance. Future posts will focus on other aspects of performance such as disk IO and application specific performance metrics.

We believe choosing a cloud provider should be based on a variety of factors including performance, price, support, reliability/uptime, scalability, network performance and features. We've previously written a few posts regarding network performance are continue to compile network performance and uptime statistics for most of the major cloud providers. With all of hype surrounding the cloud, our goal, is to provide objective information and analysis to enable educated decisions pertaining the adoption of, and migration to cloud services.

Benchmark Setup
All benchmarked cloud servers were configured almost identically in terms of OS and software, CentOS 5.4 64-bit (or 32-bit in the case of EC2 m1.small and c1.medium and IBM's Development Cloud where 64-bit is not supported).

Benchmark Methodology
Most IaaS/server clouds are based on hypervisor/virtualization technology and running in multi-tenant environments (multiple virtual servers running on a single physical host). Different hypervisors support different methods of CPU allocation/sharing including fixed/weighted, burstable, and others. Because of this, it is difficult to compare CPU performance in different clouds. Vendors often use different terminology to define cloud server CPUs including ECU (EC2), VPU (vCloud), GHz (KVM), CPUs, Cores, and more. Many provide an approximation of how that terminology relates to physical resources (e.g. 1 ECU = 1.0-1.2 GHz 2007 Xeon), but this is generally not sufficient for an objective comparison of providers.

Amazon's EC2 in addition to being one of the oldest and most mature cloud server platforms, also provides clearly defined CPU tiers across its 8 different instance sizes. These are defined in terms of ECUs (EC2 Compute Unit) where 1 ECU is the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. Their instance sizes includes the following:

Small/m1.small (32-bit) = 1 ECU
Large/m1.large = 4 ECUs
High-CPU Medium/c1.medium (32-bit) = 5 ECUs
High-Memory Extra Large/m2.xlarge = 6.5 ECUs
Extra Large/m1.xlarge = 8 ECUs
High-Memory Double Extra Large/m2.2xlarge = 13 ECUs
High-CPU Extra Large/c1.xlarge = 20 ECUs
High-Memory Quadruple Extra Large/m2.4xlarge = 26 ECUs

With a few exceptions, most of our CPU benchmarks showed clear upward scaling (although not always proportional) from m1.small (1 ECU) up to m2.4xlarge (26 ECUs). Because of these factors, we feel that the ECU metric provides a good, standard, understandable metric for comparing cloud servers not only within EC2, but also within other IaaS clouds as well. However, although it is based on the ECU, there will be some subtle differences (as described below), so we will refer this new metric as a CCU (CloudHarmony Compute Unit)

To calculate the CCU metric we selected 19 CPU benchmarks that showed clear upward scaling on smaller to larger sized EC2 instances. We use the average of the highest scores in all 4 EC2 regions (generally from the m2.4xlarge 26 ECU instance) to produce a 100% baseline for each of these 19 benchmarks. Each instance is then assigned a relative score as a ratio of that instance's score to the highest average score (<= 100). The relative scores for all 19 benchmarks are then aggregated to produce a CPU comparison score (CCS) for each instance. In calculating the CCS, some benchmarks are weighted higher than others. For example, the Geekbench and Unixbench results are weighted 200 points for the baseline, while opstone and john-the-ripper are weighted 33 points each (the remaining benchmarks are all weighted 100). We then use these results to create a CCU evaluation table where the left column in the table is the # of CCUs, and the right column is the average CCS corresponding with that CCU value. This table was then populated with 5 rows, one for EC2 instance sizes m1.small, m1.large, m2.xlarge, m2.2xlarge and m2.4xlarge, using an average of the CCS for those instances in all 4 regions. Once the comparison table was populated, we use the same algorithm to compute CCS values for every cloud server benchmarked. To translate from CCS to CCU, we determine the closest matching row(s) to the CCS for a given cloud server using the left column, and then compute an CCU value using the right column (if the CCS falls between 2 rows, a proportional average CCU value is calculated).

Example CCU Evaluation Table
ECUs CCS EC2 Instance
26 1574 m2.4xlarge
13 1405 m2.2xlarge
6.5 1225 m2.xlarge
4 873 m1.large
1 312 m1.small

Example CCU Calculation
Eval_CCS = 1524
Enter table from top and find first row where Eval_CCS >= Column_2 (Row 2)
Since Eval_CCS resides between rows 1 and 2, find the proportional midpoint between ECUs:
CCU = 13 + ((1524 - 1405)/(1574 - 1405))*(26 - 13)
CCU = 13 + 9.15
CCU = 22.15

EC2 Discrepancies
During EC2 benchmarking we observed the CPU architecture reported for each instance type:

Small (m1.small) - US East Region Only: AMD Opteron 2218 2.6 GHz
Small, Large, Extra Large (m1.small, m1.large, m1.xlarge): Xeon E5430 "Harpertown" 2.66 GHz
High-CPU Medium and Extra Large (c1.medium, c1.xlarge): Xeon 5410 "Harpertown" 2.33 GHz
High-Memory Extra Large, 2 Extra Large, 4 Extra Large (m2.xlarge, m2.2xlarge, m2.4xlarge): Xeon X5550 "Nehalem" 2.66 GHz

We also noted the following EC2 ECU discrepancies with most of the 19 CPU performance benchmarks performed:
1. The m1.large (4 ECU) always outperformed the High-CPU c1.medium (5 ECU) instance. This might be attributed to the m1.large being 64-bit vs 32-bit for the c1.medium
2. Even the lowest High-Memory instance (m2.xlarge - 6.5 ECU) out performs the larger m1.large (8 ECU) and c1.xlarge (20 ECU) instances in many cases. This is most likely due to the newer and faster "Nehalem" CPUs used by the High-Memory instances
3. The performance increase between m2.4xlarge (13 ECU) and m2.2xlarge (26 ECU) was minimal (15-20% higher based on CCU)

Because of these discrepancies, we used only the m1.small, m1.large, m2.xlarge, m2.2xlarge and m2.4xlarge instance sizes to create the CCU comparison table used to calculate CCUs for other cloud servers.

Benchmarks
The following are the 19 benchmarks we use to compute the CCU comparison metrics (benchmarks prefixed with ** are multi-core aware):


**c-ray [weight=100]: This is a test of C-Ray, a simple raytracer designed to test the floating-point CPU performance. This test is multi-threaded (16 threads per core), will shoot 8 rays per pixel for anti-aliasing, and will generate a 1600 x 1200 image.

**crafty [weight=100]: Crafty is a popular open-source chess engine that can be used to benchmark your CPU speed and is part of SPEC2000 benchmark. The benchmark itself is very basic. It analyzes pre-determined chess games positions and calculates the number of "nodes" (moves) per second till certain "depth" is reached and displays the total NPS as well as the average NPS.

dcraw [weight=100]: This test times how long it takes to convert several high-resolution RAW NEF image files to PPM image format using dcraw.

espeak [weight=100]: This test times how long it takes the eSpeak speech synthesizer to read Project Gutenbergs The Outline of Science and output to a WAV file.

**geekbench [weight=200]: Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand, Geekbench takes the guesswork out of producing robust and reliable benchmark results.

**graphics-magick [weight=100]: This is a test of GraphicsMagick with its OpenMP implementation that performs various imaging tests to stress the systems CPU.

**hmmer [weight=100]: This test searches through the Pfam database of profile hidden markov models. The search finds the domain structure of Drosophila Sevenless protein.

john-the-ripper-blowfish [weight=33]: This is a benchmark of John The Ripper, which is a password cracker.

john-the-ripper-des [weight=33]: This is a benchmark of John The Ripper, which is a password cracker.

john-the-ripper-md5 [weight=33]: This is a benchmark of John The Ripper, which is a password cracker.

mafft [weight=100]: This test performs an alignment of 100 pyruvate decarboxylase sequences.

nero2d [weight=100]: This is a test of Nero2D, which is a two-dimensional TM/TE solver for Open FMM. Open FMM is a free collection of electromagnetic software for scattering at very large objects. This test profile times how long it takes to solve one of the included 2D examples.

**openssl [weight=100]: This test measures the RSA 4096-bit performance of OpenSSL.

opstone-svd [weight=33]: CPU Singular Value Decomposition test.

opstone-svsp [weight=33]: CPU Sparse-Vector Scalar Product test.

opstone-vsp [weight=33]: CPU Vector Scalar test

sudokut [weight=100]: This is a test of Sudokut, which is a Sudoku puzzle solver written in Tcl. This test measures how long it takes to solve 100 Sudoku puzzles.

tscp [weight=100]: CPU performance benchmark based on TSCP (Tom Kerrigan's Simple Chess Program).

**unixbench [weight=200]: UnixBench provides a basic indicator of the performance of a Unix-like system. Multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system. The parallel results are used when multiple CPUs exist for a cloud server.

We credit the Phoronix Test Suite for making it easier to run many of these benchmarks.

Results
The following results are broken down by cloud server vendor. If the vendor utilizes multiple data centers, multiple tables are displayed one for each data center. A total of 140 different cloud server configurations are included in this post. Each table shows our server identifier, the CPU architecture our benchmark server was placed on, the amount of memory for the server, raw Geekbench results linked to the full results page, raw Unixbench results linked to the full results page, and finally, the CCU score for that server instance.

EC2 is one of the oldest, most widely used, and mature cloud server platforms. EC2 currently supports 4 regions (US East, US West, EU West and APAC) and 10 availability zones. Each region consists of 2 or more availability zones each of which is basically a different physical data center in close proximity to the other availability zone in that region. EC2 uses the EC2 Compute Unit (ECU) term to describe CPU resources for each instance size where one ECU provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Although the CCU metric is based on EC2's ECU, the comparison table used to compute CCUs is based on only 5 instances sizes (m1.small, m1.large, m2.xlarge, m2.2xlarge and m2.4xlarge) and an average scores from all 4 regions. Because of this, the EC2 instance CCU will not be precisely equal to it's ECU allocation.

EC2 offers multiple pricing options including straight hourly, reserve (upfront reserve fee in exchange for lower hourly), and spot (bid pricing). The pricing shown in these table is for straight hourly pricing.


Amazon Web Services (AWS) [US East]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.4/hr 5877 1511 27.25
m2.2xlarge [13 ECUs] Xeon X5550 34.2 2.4/hr 5163 1332 14.89
linux.c1.xlarge [20 ECUs] Xeon E5410 7 0.68/hr 5118 780 8.78
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.5/hr 4049 932.1 7.05
m1.xlarge [8 ECUs] Xeon E5430 15 0.68/hr 4256 938.6 5.15
m1.large [4 ECUs] Xeon E5430 7.5 0.34/hr 3092 663.4 4.08
c1.medium [5 ECUs] Xeon E5410 1.7 0.17/hr 2635 776.2 3.43
m1.small [1 ECU] Opteron 2218 1.7 0.085/hr 1726 179.7 0.92




Amazon Web Services (AWS) [US West]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.68/hr 6109 1520 27.45
m2.2xlarge [13 ECUs] Xeon X5550 34.2 1.34/hr 5475 1329.2 15.9
c1.xlarge [20 ECUs] Xeon E5410 7 0.76/hr 4693 785 8.21
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.57/hr 3883 945.4 6.85
m1.xlarge [8 ECUs] Xeon E5430 15 0.76/hr 4185 916.8 5.14
m1.large [4 ECUs] Xeon E5430 7.5 0.38/hr 3026 643.4 3.95
c1.medium [5 ECUs] Xeon E5410 1.7 0.19/hr 2962 715.5 3.45
m1.small [1 ECU] Xeon E5430 1.7 0.095/hr 1312 277.2 1.04


Amazon Web Services (AWS) [EU West]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.68/hr 6188 1489.7 27.45
m2.2xlarge [13 ECUs] Xeon X5550 34.2 1.34/hr 5368 1337.6 15.19
c1.xlarge [20 ECUs] Xeon E5410 7 0.76/hr 4926 787 8.55
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.57/hr 3836 945.9 6.79
m1.xlarge [8 ECUs] Xeon E5430 15 0.76/hr 4147 934.7 5.14
m1.large [4 ECUs] Xeon E5430 7.5 0.38/hr 3160 644 4.12
c1.medium [5 ECUs] Xeon E5410 1.7 0.19/hr 2710 730.5 3.43
m1.small [1 ECU] Xeon E5430 1.7 0.095/hr 1288 277.3 1.02



Amazon Web Services (AWS) [APAC]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.68/hr 3472 1465.8 12.46
m2.2xlarge [13 ECUs] Xeon X5550 34.2 1.34/hr 3357 1228.4 9.62
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.57/hr 3469 821.2 6.27
c1.xlarge [20 ECUs] Xeon E5410 7 0.76/hr 3042 906.1 5.66
m1.xlarge [8 ECUs] Xeon E5430 15 0.76/hr 3271 867.4 4.65
m1.large [4 ECUs] Xeon E5430 7.5 0.38/hr 2594 598.9 3.92
c1.medium [5 ECUs] Xeon E5410 1.7 0.19/hr 2796 415.8 3.23
m1.small [1 ECU] Xeon E5430 1.7 0.095/hr 1522 183.5 1.02


Rackspace Cloud
Rackspace Cloud servers showed a very flat CPU performance variation between server instance sizes. All instances we benchmarked in both data centers utilized homogenous Opteron 2374 "Shanghai" 2.2 GHz hardware. Rackspace states that their CPU provides a minimum allocation based on instance size with bursting allowed for all instance sizes.

We tested servers in both their Dallas as well as the newer Chicago data centers. However, Rackspace does not allow users to chose which data center to deploy servers to. Their use of multiple data centers appears to deal more with capacity issues rather than to offer user choice. When you create an account, that account is assigned to a specific data center, and from that point forward you will only have the option to deploy to that assigned data center.

Rackspace Cloud [Dallas]
ID CPU Memory Price Geekbench Unixbench CCUs
rs-16gb Opteron 2374 16 0.96/hr 3222 933.2 4.95
rs-2gb Opteron 2374 2 0.12/hr 3330 935.4 4.94
rs-1gb Opteron 2374 1 0.06/hr 3462 934.2 4.93
rs-4gb Opteron 2374 4 0.24/hr 3211 935.1 4.9


Rackspace Cloud [Chicago]
ID CPU Memory Price Geekbench Unixbench CCUs
rs-16gb-il Opteron 2374 16 0.96/hr 4473 1019.2 5.41
rs-8gb-il Opteron 2374 8 0.48/hr 4291 1035.5 5.39
rs-1gb-il Opteron 2374 1 0.06/hr 4339 1042.9 5.38
rs-4gb-il Opteron 2374 4 0.24/hr 4368 1013.8 5.29
rs-2gb-il Opteron 2374 2 0.12/hr 4360 963.4 5.17


Storm on Demand was launched a few months ago by Liquid Web. They provide 2 types of cloud servers. The first is traditional 2-48GB cloud server running on multi-tenant hosts. The second, called "Bare Metal", allows you to select specific dedicated hardware (CPU, SATA or SAS disks, memory) to deploy your server on. Bare Metal servers are still virtualized, but do not share the underlying hardware with any other server instances.

Of all the IaaS vendors we reviewed, Storm offers by far the most diverse heterogenous infrastructure. This approach pays off big in terms of performance with 10 servers that scored 20+ CCUs. The only mismatched hardware we discovered was the cloud 16GB server running on Opteron 2350 hardware which performed poorly compared with the small 2, 4 and 8 GB servers. However, Storm has informed us that their 16GB Opteron 2350 hardware is being upgraded to Opteron 2378 which should improve performance on future benchmarks.

Storm's 48GB cloud server was the top performer out of all of our benchmarked servers with 42.5 CCUs and a Geekbench score of 13020! This is most likely due to the very new and extremely fast Xeon X5650 "Westmere" hardware it runs on. The Intel i5 CPUs also performed very well with our CPU benchmarks and provide an excellent performance to price ratio (26.5 CCUs for $0.171/hr)!

Storm Cloud [MI, US]
ID CPU Memory Price Geekbench Unixbench CCUs
Cloud: 48gb Xeon X5650 45.9 1.37/hr 13020 4448.5 42.87
Bare Metal: x3440-8gb Xeon X3440 8 0.274/hr 7597 2760.1 27.39
Bare Metal: e5506x2-4gb Xeon E5506 8 0.274/hr 7742 2906.7 27.24
Bare Metal: e5506x2-4gb Xeon E5506 8 0.322/hr 7704 2880.1 27.13
Bare Metal: e5506x2-8gb Xeon E5506 8 0.391/hr 7753 2837.5 26.72
Bare Metal: i5-750-2gb Core i5 750 2 0.171/hr 6417 2464.8 26.6
Bare Metal: i5-750-4gb Core i5 750 4 0.206/hr 6413 2450.1 26.47
Bare Metal: e5506x2-8gb Xeon E5506 8 0.48/hr 7686 2811.5 26.51
Cloud: 32gb Opteron 2378 30.4 0.69/hr 8326 2373.3 26.4
Cloud: 8gb Xeon X3440 7 0.27/hr 6162 2284.7 21.41
Cloud: 4gb Core i5 750 3.5 0.14/hr 4555 1465.1 9.33
Bare Metal: amd2350x2-32gb Opteron 2350 32 0.713/hr 6531 1803.6 7.31
Cloud: 2gb Core 2 Q9400 1.7 0.07/hr 3062 629.7 5.01
Cloud: 16gb Opteron 2350 15.2 0.34/hr 4034 1214 4.73


GoGrid's servers showed a decent linear performance increase with larger sized instances. The largest 8GB instance was deployed on an E5450 2.99 GHz host (compared with the E5520 2.27 GHz hosts for the other instances) which performed significantly better than the smaller sized instances.


GoGrid [CA, US]
ID CPU Memory Price Geekbench Unixbench CCUs
gg-8gb Xeon E5450 8 1.52/hr 8105 507.5 23.2
gg-4gb Xeon E5520 4 0.76/hr 5373 1866.9 9.28
gg-2gb Xeon E5520 2 0.38/hr 3549 993.3 4.87
gg-1gb Xeon E5520 1 0.19/hr 2860 941.2 4.42



Voxel maintains 3 cloud data centers in the US (New York), EU (Amsterdam), and Asia (Singapore). All of our cloud servers were deployed on homogenous harware: Xeon L5520 "Nehalem" 2.26 GHz. They appear to use more of a fixed CPU allocation because there was a notable increase in performance on larger instance sizes.


Voxel [NY, US]
ID CPU Memory Price Geekbench Unixbench CCUs
vx-14gb-ny Xeon L5520 14 0.727/hr 5176 1023.5 10.15
vx-8gb-ny Xeon L5520 8 0.421/hr 4022 846.4 6.12
vx-4gb-ny Xeon L5520 4 0.211/hr 3487 680.7 5.33
vx-2gb-ny Xeon L5520 2 0.106/hr 2876 483.4 4.67



Voxel [NL]
ID CPU Memory Price Geekbench Unixbench CCUs
vx-14gb-nl Xeon L5520 14 0.727/hr 6169 1178 16.2
vx-8gb-nl Xeon L5520 8 0.421/hr 4699 954.7 7.53
vx-4gb-nl Xeon L5520 4 0.211/hr 3416 645.8 5.48
vx-2gb-nl Xeon L5520 2 0.106/hr 3093 494 4.85



Voxel [SG]
ID CPU Memory Price Geekbench Unixbench CCUs
vx-14gb-sg Xeon L5520 14 0.727/hr 5419 1107.8 11.17
vx-8gb-sg Xeon L5520 8 0.421/hr 4428 909.8 6.24
vx-4gb-sg Xeon L5520 4 0.211/hr 3168 611.3 5.04
vx-2gb-sg Xeon L5520 2 0.106/hr 2779 463.7 4.4

NewServers is fairly unique in that their "bare metal" cloud servers actually run on physical hosts. There is no hypervisor layer between the server and the underlying hardware. When you deploy a server, the OS image is written directly to the physical disk(s). Their "Fast" server performed very well and is one of the better values at $0.53/hr for 26.41 CCUs.


NewServers [FL, US]
ID CPU Memory Price Geekbench Unixbench CCUs
ns-fast Xeon E5450 4 0.53/hr 6271 2193.5 27.21
ns-jumbo Xeon E5504 48 0.6/hr 5713 2809.7 19.15
ns-large Xeon E5405 4 0.25/hr 4505 1849.7 6.12
ns-med Xeon 3.20 2 0.17/hr 2672 708.5 3.43
ns-small Xeon 2.80 1 0.11/hr 1669 491.3 2.47



While Linode doesn't market itself as a "cloud", we included it in the benchmarks because they are a good and very popular service and provide many of the features common to the cloud including auto-provisioning, disk imaging. All instance sizes we benchmarked deployed on homogenous Xeon L5520 2.26 GHz hardware. The servers show a very flat CPU performance variation between instance sizes. We only benchmarked servers in their Atlanta data center. Linode also maintains data centers in Fremont CA, Dallas TX, Newark NJ, and London UK.


Linode VPS Hosting [Atlanta]
ID CPU Memory Price Geekbench Unixbench CCUs
ln-5760-atlanta Xeon L5520 5.54 10.67/day 4217 1160.5 6.4
ln-14400-atlanta Xeon L5520 14.06 26.67/day 3865 1144.1 6.3
ln-1080-atlanta Xeon L5520 1.05 2/day 3623 957.2 5.48
ln-2880-atlanta Xeon L5520 2.81 5.33/day 3625 979.4 5.38
ln-8640-atlanta Xeon L5520 8.45 16/day 3780 971 5.33



All our benchmark servers deployed on to either Xeon X3460 2.8 GHz or E5520 2.27 GHz hardware. SoftLayer is another provider where CPU performance was very flat. Disk I/O was also painfully slow, but that is a topic for another post.


SoftLayer [Dallas]
ID CPU Memory Price Geekbench Unixbench CCUs
sl-8gb-dallas Xeon E5520 8 0.5/hr 5862 1514.1 14.35
sl-1gb-dallas Xeon X3460 1 0.15/hr 3745 502.7 6.1
sl-4gb-dallas Xeon E5520 4 0.35/hr 4482 941.8 5.52
sl-2gb-dallas Xeon X3460 2 0.25/hr 4399 573.9 5.11



SoftLayer [WDC]
ID CPU Memory Price Geekbench Unixbench CCUs
sl-4gb-wdc Xeon X3460 4 0.35/hr 5599 1250.9 16.88
sl-8gb-wdc Xeon E5520 8 0.5/hr 5885 1527.7 16.84
sl-2gb-wdc Xeon X3460 2 0.25/hr 4265 871.5 7.62
sl-1gb-wdc Xeon X3460 1 0.15/hr 3710 507.7 5.81


SoftLayer [Seattle]
ID CPU Memory Price Geekbench Unixbench CCUs
sl-8gb-seattle Xeon E5520 8 0.5/hr 6040 1520.8 14.07
sl-4gb-seattle Xeon X3460 4 0.35/hr 5455 1324.4 12.3
sl-2gb-seattle Xeon X3460 2 0.25/hr 4429 822.8 6.49
sl-1gb-seattle Xeon X3460 1 0.15/hr 3716 505.8 5.52



Terremark vCloud Express
Terremark is one of the first VMWare vCloud providers. Deployment of servers in vCloud allows the user to select both desired memory and VPUs (the vCloud term for CPUs). All of our benchmark servers deployed on homogenous Opteron 8389 2.91 GHz hardware. CPU performance varied from benchmark to benchmark. Unixbench's parallel benchmark did show a notable increase in performance from 1 to 8 VPUs. However, other benchmarks did not show much increase leading to a flat CCU metric across all instance sizes and VPU combinations we benchmarked.


Terremark vCloud Express [FL, US]
ID CPU Memory Price Geekbench Unixbench CCUs
tm-16gb-8vpu Opteron 8389 16 1.672/hr 4304 2371.5 7.69
tm-8gb-4vpu Opteron 8389 8 0.64/hr 5999 1667 6.41
tm-1gb Opteron 8389 1 0.074/hr 2863 1279.3 5.44
tm-2gb Opteron 8389 2 0.137/hr 2779 1272.7 5.38
tm-4gb-2vpu Opteron 8389 4 0.305/hr 3793 1336.6 5.34



OpSource is another VMWare based cloud. OpSource allows you to configure cloud severs with 1-4 "CPUs". CPU performance was very flat between instances of varying sizes even showing a decrease in performance from smaller to larger sized instances. All servers deployed to identical Xeon X7460 2.66 GHz hardware.


OpSource Cloud [VA, US]
ID CPU Memory Price Geekbench Unixbench CCUs
os-1gb Xeon X7460 1 0.268/hr 2100 361.5 2.56
os-2gb Xeon X7460 2 0.296/hr 1894 338.1 2.43
os-4gb-2cpu Xeon X7460 4 0.392/hr 1923 241 2.24
os-8gb-4cpu Xeon X7460 8 0.584/hr 1830 149.3 1.87
os-16gb-4cpu Xeon X7460 16 0.808/hr 2004 129 1.87
os-32gb-4cpu Xeon X7460 32 1.256/hr 1908 130.8 1.82



Speedyrails is a VPS provider based out of Quebec Canada. All benchmark servers deployed on homogenous hardware: Xeon E5520 2.27 GHz.


Speedyrails [QC, CA]
ID CPU Memory Price Geekbench Unixbench CCUs
sr-1gb Xeon E5520 1 2.27/day 4413 1260.8 9.06
sr-4gb Xeon E5520 4 8/day 4359 1261.1 8.49
sr-2gb Xeon E5520 2 4.27/day 4347 1066.7 6.74



Zerigo is a VPS and Cloud Server vendor based out of Denver, CO. All benchmark servers deployed on identical hardware, Opteron 2374 2.20 GHz.


Zerigo [CO, US]
ID CPU Memory Price Geekbench Unixbench CCUs
zr-4gb Opteron 2374 4 0.24/hr 3829 805.5 4.83
zr-2gb Opteron 2374 2 0.12/hr 3274 813.9 4.73



While there was a notable increase in performance with larger sized instances, the overall CPU performance was not great. Hardware was also homogenous between different instance sizes.


ReliaCloud Cloud Services [MN, US]
ID CPU Memory Price Geekbench Unixbench CCUs
rc-4gb Xeon E5504 4 0.32/hr 1637 226.8 1.11
rc-2gb Xeon E5504 2 0.16/hr 1653 267.9 1.07
rc-1gb Xeon E5504 1 0.08/hr 1392 243.2 0.65



IBM's Development & Test Cloud is a free cloud service intended for only development and testing. Only 3 32-bit instance sizes are supported: small, medium and large. Instances are time limited to about 1 week with the option to extend. The large instance performed very well overall.


IBM Development & Test Cloud [NY, US]
ID CPU Memory Price (USD) Geekbench Unixbench CCUs
ibm-dev-large Xeon X5570 3.5 10256 3059.5 27.9
ibm-dev-med Xeon X5570 1.75 5536 1526.6 5.44



BlueLock is another VMWare vCloud provider. As with the other VMWare providers, they appear to use a homogenous hardware environment for all instance sizes. However, unlike most other homogenous platforms, BlueLock's instances showed a notable increase in performance on larger sized (more CPUs) instances. Strangely, the 4CPU/8GB instance outperformed the 8CPU/16GB instance.


BlueLock [IN, US]
ID CPU Memory Price Geekbench Unixbench CCUs
bl-8gb-4cpu Xeon X5550 8 0.661/hr 6311 2159.1 26.26
bl-16gb-8cpu Xeon X5550 16 1.729/hr 6665 2525.2 25.55
bl-4gb-2cpu Xeon X5550 4 0.308/hr 4575 1351 9.06
bl-1gb Xeon X5550 1 0.068/hr 3690 1153.3 6.3
bl-2gb Xeon X5550 2 0.134/hr 3727 1223.4 6.14



Cloud Central is a new cloud server provider based out of Australia. All benchmark instances deployed to homogenous AMD Opteron 2.20 GHz hosts. Prices shown are Australian Dollar.

Cloud Central [AU]
ID CPU Memory Price Geekbench Unixbench CCUs
cc-large Opteron 2374 8 0.96/hr 4224 767.3 4.97
cc-reg Opteron 2374 4 0.48/hr 3894 787.2 4.96
cc-huge Opteron 2374 16 1.92/hr 3674 785.2 4.87
cc-med Opteron 2374 2 0.24/hr 3913 755.1 4.69
cc-small Opteron 2374 1 0.12/hr 3727 756.6 4.63

RimuHosting is a VPS provider based out of New Zealand. They maintain data centers in Australia, New Zealand, London, and Texas. We benchmarked 2GB instances in their Auckland NZ and Dallas TX data centers.


RimuHosting [TX, US]
ID CPU Memory Price Geekbench Unixbench CCUs
rh-tx-2gb Xeon E5506 2 104.57/mo 3080 735.6 4.56



RimuHosting [NZ]
ID CPU Memory Price (NZD) Geekbench Unixbench CCUs
rh-nz-2gb Xeon E5506 2 210.88/mo 3122 662.3 4.49

ElasticHosts is a UK based cloud provider. They currently maintain 2 data centers in the UK and a new data center in Dallas, TX. We only benchmarked their London Peer1 data center. ElasticHosts runs on the Linux KVM hypervisor. This hypervisor is unique in that it allows you to selected a specific MHz metric when deploying cloud servers. We benchmarked 2 GHz - 20 GHz cloud servers. Although the hardware environment appears to be homogenous, the benchmarks showed a clear increase in performance on larger sized instances.


ElasticHosts [UK]
ID CPU Memory Price (USD) Geekbench Unixbench CCUs
eh-8gb-20gh Xeon E5420 8 0.654/hr 6544 881.7 9.98
eh-4gb-8gh Xeon E5420 4 0.326/hr 3919 814.3 5.54
eh-2gb-4gh Xeon E5420 2 0.164/hr 3409 631 4.75
eh-1gb-2gh Xeon E5420 1 0.082/hr 2545 539.8 4.3



Flexiscale is a UK based cloud server provider that has been around for a few years. They were recently acquired and renamed to Flexiant. They are currently in beta release of their Flexiscale 2.0 cloud server platform. These results were from testing 2.0 platform servers. Flexiant adopted a point-based subscription model for purchasing cloud servers. See their website for more details.


Flexiscale [UK]
ID CPU Memory Price (GBP) Geekbench Unixbench CCUs
fx-8gb Opteron 8218 8 4629 636.5 3.81
fx-4gb Opteron 8220 4 3407 541.6 3.66
fx-2gb Opteron 8218 2 2394 510 3.36
fx-1gb Opteron 8218 1 2354 481.9 2.95



Summary
This is our first attempt at defining a standard CPU performance metric for comparing servers in multiple clouds. We acknowledge that it is not perfect and hope to make improvements over time. Please comment on this post if you have any suggestions on how we might improve our methods. We intend to continually run these benchmarks (every couple of months) to improve the quality and quantity of our data available as well as to check for upgrades and improvements made by the providers.

One take-away point we observed is that heterogenous hardware environments (where host hardware is configured with faster CPUs for larger sized instances) appears to be more conducive to true cloud CPU scaling. Of the 20 server clouds we benchmarked, only 4 appear to be providing such an environment: EC2, Storm on Demand, GoGrid and NewServers.

03/14/2010 00:02 PDT

Connecting Clouds - Summary of Our Intercloud Network Testing

Over the past few months we've been conducting some intercloud network testing. To get started, we first setup testing nodes in 28 different infrastructure/server clouds (a.k.a IaaS). Twice daily at random times (hourly for latency), these nodes initiate throughput and latency network tests with each of the other 27 nodes in the test group. These tests consist of uploading and downloading a 5 MB test file and recording the throughput, and pinging to determine latency. The purpose is to determine which clouds are best connected to each other. In the results tables below, we've compiled averages from all of these tests ordered by downlink throughput and displayed the top 10 best connected clouds to each of the 28 we tested.

As reliable as cloud services generally are, they do occasionally fail. We believe a good cloud adoption strategy involves use of more than one cloud service to avoid risks of extended downtime (i.e. natural disaster, power failures, etc.). Such a strategy could be as simple as maintaining backups in a separate cloud for cold restores, to complex load balanced cross cloud clusters of application servers and data repositories. In both cases, good throughput and low latency between clouds will allow data to be transmitted quickly and efficiently. If latency is high or throughput low, you may run into data consistency issues and performance bottlenecks.

A few admin notes regarding our testing:
  1. With a few exceptions, most cloud services limit Internet uplinks to 100 Mb/s or less
  2. A 5MB test file is not sufficiently large to determine accurate throughput capacity between clouds. The downlink and uplink values provided here are for comparison purposes only and not meant to represent actual throughput capacity (which could be even higher for larger data transfers)
  3. We've found that routing is not always symmetrical between uplink and downlink tests. This is usually the reason for large discrepancies between uplink and downlink throughput
  4. The results are broken down by cloud. The table shows the 10 best connected clouds for each. The downlink, uplink and latency values are measured by the cloud shown in the left column of the table
These tests are ongoing. Over the next few months, we plan to make this data available in real-time via web services and on our website to include filtering criteria. If you have any suggestions on how we might improve the accuracy of these tests, please feel free to comment.

The Results:

EC2 - US East
EC2 - US West
EC2 - EU West
ElasticHosts
London, UK
Flexiscale
London, UK
GoGrid
San Francisco, CA
New York, NY
Linode
Linode - Atlanta, GA
Linode - Dallas, TX
Linode - Fremont, CA
Linode - London, UKLinode - Newark, NJ
NewServers
Dallas, TX
ReliaCloud
Minnesota, US
Rimu Hosting
Rimu - Auckland, NZ
Rimu - Dallas, TX
SoftLayer
SoftLayer - Dallas, TX
SoftLayer - Seattle, WA
SoftLayer - Washington DC
Quebec, Canada
Storm Cloud Servers
Voxel - New York, NY
Voxel - Amsterdam, NL
Voxel - Singapore
Zerigo
Denver, CO



03/02/2010 13:00 PST

Cloud Server Performance Benchmarking

We are in the process of developing a benchmark suite to run in the cloud and use as a basis for comparing different IaaS (cloud server) vendors. There are lots of uses for cloud servers be it web, application or database servers; scientific computing; video encoding; etc. In establishing our benchmark suite we'd like to be as comprehensive as possible in order to provide decent coverage of most computational needs. Our current list of benchmarks includes the following:

All benchmarks will be run on similar CentOS 64-bit server instances. Are there any benchmarks you'd like to see that are not in the list? We'd appreciate any comments, suggestions or feedback.

02/27/2010 23:56 PST

Cloud Storage Showdown Part 2 - What is the best storage for your cloud server?

In the previous post we discussed cloud storage for consumers. Probably a more common use case for cloud storage is to enable backups and/or extended storage from cloud servers and platforms. Over the past month, we have used our network of 25 global servers running in various public clouds to measure bandwidth throughput and latency to and from various cloud storage services including Microsoft's Azure Blob Storage, Amazon's Simple Storage Service (S3), SoftLayer's CloudLayer Storage, Nirvanix Storage Delivery Network, Rackspace Cloud Files, and Box.net. We conducted these bandwidth tests by reading and writing a 3MB test file (10MB for storage services in the same cloud such as EC2 to S3) to/from each storage services at random times twice daily. The results are separated by cloud server provider. The table displays all of the storage services tested within that cloud ordered by fastest downlink. We also tested storage in Microsoft's Azure platform.

The purpose of this test wasn't to measure the maximum throughput capacity between server and storage service, but rather to provide a comparison between different storage services. The 3MB test file is not sufficiently large for maximum capacity to be determined (for same cloud storage services the throughput will be more accurate because of the larger 10MB file size). Actual throughput for larger files will most likely be higher than the throughput calculations displayed here.

Bandwidth and storage pricing are other factors to consider when selecting a storage service. For example, Amazon does not charge for bandwidth to/from its S3 storage service and EC2 instances running in the same region. However, there are good reasons to use an external storage service for backups. If, for example, you use EC2 and store your backups using S3 in the same region, and that region happens to go down entirely for an extended period (an unlikely scenario of course), you will be without any means of recovering your data until the region is brought back online. For added fault tolerance, you may decide to keep backups in a separate Amazon S3 region (and pay the bandwidth costs) or even in a separate cloud like Microsoft's Azure.

Tests were performed using a small Azure instance.

South Central US (TX)
North Central US (IL)
Southeast Asia (Singapore)

Tests were performed using an m1.small instance in all regions.

US East Region
EU West Region
US West Region

All instances are Linode 360s

Newark, NJ
Atlanta, GA
Dallas, TX
Rackspace Cloud Files is run out of Dallas as well. Linode also provides GigE uplinks with all servers. This explains the very high downlink throughput result
London, UK
Fremont, CA

Our VoxCloud servers are the smallest 2GB model

New York
Singapore
Most likely Microsoft Azure and Voxel run out of data centers in very close proximity
Amsterdam, NL

GoGrid (CA, US)
We run a 512MB instance with GoGrid. Box.net tends to perform very well against US west coast servers.

Our rackspace node is also run on a 512MB instance. Throughput to and from Rackspace's own Cloud Files storage service was very good.



ReliaCloud (MN, US)



Dallas, TX
Auckland, NZ

02/14/2010 14:33 PST

Cloud Storage Showdown Part 1 - Cloud to Consumer

This is a follow up to our previous post Cloud Speed Test Results where we analyzed the results from our custom cloud speedtest as they pertained to cloud servers and content delivery networks. In this post, we'll use the same test results to analyze the bandwidth performance of 4 cloud storage services including Amazon's Simple Storage Service (S3) (all 3 regions), Rackspace's Cloud Files, Nirvanix's Storage Delivery Network (SDN), and Microsoft's Azure Blog Storage. To date, about 1350 unique users in 115 countries have run the speedtest. This speedtest tracks the amount of time required to download a 1MB test file from these 6 cloud storage services, and records the transfer rate for each test run. We use MaxMind's geoip database to track where the user is running the test from (country + state/province for US/Canada). The order of the tests is always random and we enforce a time limit such that users with dial up or slow connections are excluded. We also limit individual users to running the speedtest only once per day. 90% of the speedtests were run by users on residential high-speed Internet connections, thus this post is focused on Cloud to Consumer bandwidth performance. In the next post (Part 2), we'll focus on Cloud to Cloud or Intracloud storage bandwidth performance.

As you'll see in the results below, consumer cloud storage performance is highly dependent on the consumer's geographical location. Although cloud vendors may tell you it isn't important where their services are run out of, this is not entirely true. In the previous post we separated the results into very general geographical regions (US and non-US). Since this post is more focused on consumer cloud storage usage, and because there is a large difference in bandwidth performance based on geographical origin, we have broken these results into 8 geographical regions.

A few admin notes regarding the speedtest results:
  • The results are based on download performance. Consumers will generally not achieve this same throughput for uploads due to ISP uplink caps
  • Nirvanix's SDN is unique in that it is geographically optimized like a content delivery network (users connect to the fastest/closest of 4 storage nodes). We used a 4-node SDN (US West, US East, Germany and Japan) in conducting the Nirvanix tests
  • Rackspace Cloud Files is not directly browser accessible and thus not compatible with our speedtest. The results shown below are based on performance for Rackspace Cloud Servers which to our knowledge are run out of the same data centers. Also, based on other bandwidth tests we have conducted, there is a very close correlation between Rackspace Cloud Files/Cloud Servers bandwidth performance
  • Results for Azure storage are based on Azure Platform performance. These results should be most closely correlated with the South Central US storage region. Now that Azure is out of CTP, we will be including the other regions in future tests: US North Central, North Europe and Southeast Asia
Global Results
The global results (all tests performed) show a fairly close spread between the 6 storage services with the exception of S3 US West which seems to be more geographically sensitive.
US Results
In the US, tests were run from all 50 states. S3 US East, Azure Storage and Rackspace Cloud Files were all pretty close in the top 3 spots. Surprisingly, Amazon's S3 region in Europe performed significantly better than the new S3 US West region in the US.
US West Results
US West (AK, AZ, CA, HI, ID, NM, NV, OR, UT, WA) is where Amazon's new S3 US West region really shined out performing other US services by 20-40%.
US Central Results
In US Central (CO, IA, IL, IN, KS, LA, MI, MN, MO, MS, ND, NE, OH, OK, SD, TX, WI, WY), Rackspace's Cloud Files service was the clear top performer followed by Azure (7% slower) and S3 US East (13% slower).
US East Results
In US East (AL, AR, CT, DC, DE, FL, GA, KY, MA, MD, ME, NC, NH, NJ, NY, PA, RI, SC, TN, VA, VT, WV), Amazon's S3 US East region was the top performer. Surprisingly, S3 EU West was not close behind. Azure followed at a distant third (15% slower).
Non US Results
Outside of the US is where Nirvanix's storage outperformed other services. Azure also performed well followed by S3 US East.
Europe Results
In Europe, Nirvanix's SDN did extremely well, out performing second place S3 EU West by 20%. Azure performed comparably with S3 EU West. Rackspace Cloud Files was a bit further behind. S3 US West did not perform well in Europe.
Asia Results
In Asia Azure and Nirvanix were about equal in performance. Rackspace Cloud Files and S3 US West followed at about 15% slower. Amazon will be deploying a new AWS Asia region later this year.
Summary
Bandwidth is but one factor for consumers to consider when choosing a cloud storage service. Most of these storage services are not natively mountable (i.e. you can't browse them using your operating system's file browser), so third party tools like JungleDisk (S3 or Rackspace Cloud Files only), CloudBerry (S3 or Nirvanix) or Windows Explorer Virtual Network Drive (S3, Azure, Nirvanix) are necessary in order to use them. The decision on which storage service to use really depends on a variety of factors including geography, bandwidth, third party tools available and bandwidth/storage costs.