benchmarking the cloud

RSS Feed

Cloud Articles From Our Blog

05/29/2010 16:06 PDT

What is an ECU? CPU Benchmarking in the Cloud

NOTE: This post has been updated after it's original writing. The original CPU performance metrics did not accurately depict performance on multi-core servers. The updated post utilizes an improved method of calculating CPU performance that applies more weight to multi-core aware benchmarks (see benchmarks description below for more info).

Over the past couple of months we've spent some time benchmarking about 150 different cloud server configurations with 20 different vendors. This included all 8 AWS EC2 instances types (m1.small - m2.4xlarge) in all 4 regions (32 servers total for EC2). The benchmark suite we ran includes about 100 different benchmarks from synthetic benchmarks measuring raw CPU performance such as Unixbench and Geekbench to higher level application benchmarks such as mysql-bench, pgbench, tpcc-mysql and blog bench. This post will be the first in a series highlighting the results of these benchmarks. In it, we'll focus purely on raw CPU performance. Future posts will focus on other aspects of performance such as disk IO and application specific performance metrics.

We believe choosing a cloud provider should be based on a variety of factors including performance, price, support, reliability/uptime, scalability, network performance and features. We've previously written a few posts regarding network performance are continue to compile network performance and uptime statistics for most of the major cloud providers. With all of hype surrounding the cloud, our goal, is to provide objective information and analysis to enable educated decisions pertaining the adoption of, and migration to cloud services.

Benchmark Setup
All benchmarked cloud servers were configured almost identically in terms of OS and software, CentOS 5.4 64-bit (or 32-bit in the case of EC2 m1.small and c1.medium and IBM's Development Cloud where 64-bit is not supported).

Benchmark Methodology
Most IaaS/server clouds are based on hypervisor/virtualization technology and running in multi-tenant environments (multiple virtual servers running on a single physical host). Different hypervisors support different methods of CPU allocation/sharing including fixed/weighted, burstable, and others. Because of this, it is difficult to compare CPU performance in different clouds. Vendors often use different terminology to define cloud server CPUs including ECU (EC2), VPU (vCloud), GHz (KVM), CPUs, Cores, and more. Many provide an approximation of how that terminology relates to physical resources (e.g. 1 ECU = 1.0-1.2 GHz 2007 Xeon), but this is generally not sufficient for an objective comparison of providers.

Amazon's EC2 in addition to being one of the oldest and most mature cloud server platforms, also provides clearly defined CPU tiers across its 8 different instance sizes. These are defined in terms of ECUs (EC2 Compute Unit) where 1 ECU is the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. Their instance sizes includes the following:

Small/m1.small (32-bit) = 1 ECU
Large/m1.large = 4 ECUs
High-CPU Medium/c1.medium (32-bit) = 5 ECUs
High-Memory Extra Large/m2.xlarge = 6.5 ECUs
Extra Large/m1.xlarge = 8 ECUs
High-Memory Double Extra Large/m2.2xlarge = 13 ECUs
High-CPU Extra Large/c1.xlarge = 20 ECUs
High-Memory Quadruple Extra Large/m2.4xlarge = 26 ECUs

With a few exceptions, most of our CPU benchmarks showed clear upward scaling (although not always proportional) from m1.small (1 ECU) up to m2.4xlarge (26 ECUs). Because of these factors, we feel that the ECU metric provides a good, standard, understandable metric for comparing cloud servers not only within EC2, but also within other IaaS clouds as well. However, although it is based on the ECU, there will be some subtle differences (as described below), so we will refer this new metric as a CCU (CloudHarmony Compute Unit)

To calculate the CCU metric we selected 19 CPU benchmarks that showed clear upward scaling on smaller to larger sized EC2 instances. We use the average of the highest scores in all 4 EC2 regions (generally from the m2.4xlarge 26 ECU instance) to produce a 100% baseline for each of these 19 benchmarks. Each instance is then assigned a relative score as a ratio of that instance's score to the highest average score (<= 100). The relative scores for all 19 benchmarks are then aggregated to produce a CPU comparison score (CCS) for each instance. In calculating the CCS, some benchmarks are weighted higher than others. For example, the Geekbench and Unixbench results are weighted 200 points for the baseline, while opstone and john-the-ripper are weighted 33 points each (the remaining benchmarks are all weighted 100). We then use these results to create a CCU evaluation table where the left column in the table is the # of CCUs, and the right column is the average CCS corresponding with that CCU value. This table was then populated with 5 rows, one for EC2 instance sizes m1.small, m1.large, m2.xlarge, m2.2xlarge and m2.4xlarge, using an average of the CCS for those instances in all 4 regions. Once the comparison table was populated, we use the same algorithm to compute CCS values for every cloud server benchmarked. To translate from CCS to CCU, we determine the closest matching row(s) to the CCS for a given cloud server using the left column, and then compute an CCU value using the right column (if the CCS falls between 2 rows, a proportional average CCU value is calculated).

Example CCU Evaluation Table
ECUs CCS EC2 Instance
26 1574 m2.4xlarge
13 1405 m2.2xlarge
6.5 1225 m2.xlarge
4 873 m1.large
1 312 m1.small

Example CCU Calculation
Eval_CCS = 1524
Enter table from top and find first row where Eval_CCS >= Column_2 (Row 2)
Since Eval_CCS resides between rows 1 and 2, find the proportional midpoint between ECUs:
CCU = 13 + ((1524 - 1405)/(1574 - 1405))*(26 - 13)
CCU = 13 + 9.15
CCU = 22.15

EC2 Discrepancies
During EC2 benchmarking we observed the CPU architecture reported for each instance type:

Small (m1.small) - US East Region Only: AMD Opteron 2218 2.6 GHz
Small, Large, Extra Large (m1.small, m1.large, m1.xlarge): Xeon E5430 "Harpertown" 2.66 GHz
High-CPU Medium and Extra Large (c1.medium, c1.xlarge): Xeon 5410 "Harpertown" 2.33 GHz
High-Memory Extra Large, 2 Extra Large, 4 Extra Large (m2.xlarge, m2.2xlarge, m2.4xlarge): Xeon X5550 "Nehalem" 2.66 GHz

We also noted the following EC2 ECU discrepancies with most of the 19 CPU performance benchmarks performed:
1. The m1.large (4 ECU) always outperformed the High-CPU c1.medium (5 ECU) instance. This might be attributed to the m1.large being 64-bit vs 32-bit for the c1.medium
2. Even the lowest High-Memory instance (m2.xlarge - 6.5 ECU) out performs the larger m1.large (8 ECU) and c1.xlarge (20 ECU) instances in many cases. This is most likely due to the newer and faster "Nehalem" CPUs used by the High-Memory instances
3. The performance increase between m2.4xlarge (13 ECU) and m2.2xlarge (26 ECU) was minimal (15-20% higher based on CCU)

Because of these discrepancies, we used only the m1.small, m1.large, m2.xlarge, m2.2xlarge and m2.4xlarge instance sizes to create the CCU comparison table used to calculate CCUs for other cloud servers.

The following are the 19 benchmarks we use to compute the CCU comparison metrics (benchmarks prefixed with ** are multi-core aware):

**c-ray [weight=100]: This is a test of C-Ray, a simple raytracer designed to test the floating-point CPU performance. This test is multi-threaded (16 threads per core), will shoot 8 rays per pixel for anti-aliasing, and will generate a 1600 x 1200 image.

**crafty [weight=100]: Crafty is a popular open-source chess engine that can be used to benchmark your CPU speed and is part of SPEC2000 benchmark. The benchmark itself is very basic. It analyzes pre-determined chess games positions and calculates the number of "nodes" (moves) per second till certain "depth" is reached and displays the total NPS as well as the average NPS.

dcraw [weight=100]: This test times how long it takes to convert several high-resolution RAW NEF image files to PPM image format using dcraw.

espeak [weight=100]: This test times how long it takes the eSpeak speech synthesizer to read Project Gutenbergs The Outline of Science and output to a WAV file.

**geekbench [weight=200]: Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand, Geekbench takes the guesswork out of producing robust and reliable benchmark results.

**graphics-magick [weight=100]: This is a test of GraphicsMagick with its OpenMP implementation that performs various imaging tests to stress the systems CPU.

**hmmer [weight=100]: This test searches through the Pfam database of profile hidden markov models. The search finds the domain structure of Drosophila Sevenless protein.

john-the-ripper-blowfish [weight=33]: This is a benchmark of John The Ripper, which is a password cracker.

john-the-ripper-des [weight=33]: This is a benchmark of John The Ripper, which is a password cracker.

john-the-ripper-md5 [weight=33]: This is a benchmark of John The Ripper, which is a password cracker.

mafft [weight=100]: This test performs an alignment of 100 pyruvate decarboxylase sequences.

nero2d [weight=100]: This is a test of Nero2D, which is a two-dimensional TM/TE solver for Open FMM. Open FMM is a free collection of electromagnetic software for scattering at very large objects. This test profile times how long it takes to solve one of the included 2D examples.

**openssl [weight=100]: This test measures the RSA 4096-bit performance of OpenSSL.

opstone-svd [weight=33]: CPU Singular Value Decomposition test.

opstone-svsp [weight=33]: CPU Sparse-Vector Scalar Product test.

opstone-vsp [weight=33]: CPU Vector Scalar test

sudokut [weight=100]: This is a test of Sudokut, which is a Sudoku puzzle solver written in Tcl. This test measures how long it takes to solve 100 Sudoku puzzles.

tscp [weight=100]: CPU performance benchmark based on TSCP (Tom Kerrigan's Simple Chess Program).

**unixbench [weight=200]: UnixBench provides a basic indicator of the performance of a Unix-like system. Multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system. The parallel results are used when multiple CPUs exist for a cloud server.

We credit the Phoronix Test Suite for making it easier to run many of these benchmarks.

The following results are broken down by cloud server vendor. If the vendor utilizes multiple data centers, multiple tables are displayed one for each data center. A total of 140 different cloud server configurations are included in this post. Each table shows our server identifier, the CPU architecture our benchmark server was placed on, the amount of memory for the server, raw Geekbench results linked to the full results page, raw Unixbench results linked to the full results page, and finally, the CCU score for that server instance.

EC2 is one of the oldest, most widely used, and mature cloud server platforms. EC2 currently supports 4 regions (US East, US West, EU West and APAC) and 10 availability zones. Each region consists of 2 or more availability zones each of which is basically a different physical data center in close proximity to the other availability zone in that region. EC2 uses the EC2 Compute Unit (ECU) term to describe CPU resources for each instance size where one ECU provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Although the CCU metric is based on EC2's ECU, the comparison table used to compute CCUs is based on only 5 instances sizes (m1.small, m1.large, m2.xlarge, m2.2xlarge and m2.4xlarge) and an average scores from all 4 regions. Because of this, the EC2 instance CCU will not be precisely equal to it's ECU allocation.

EC2 offers multiple pricing options including straight hourly, reserve (upfront reserve fee in exchange for lower hourly), and spot (bid pricing). The pricing shown in these table is for straight hourly pricing.

Amazon Web Services (AWS) [US East]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.4/hr 5877 1511 27.25
m2.2xlarge [13 ECUs] Xeon X5550 34.2 2.4/hr 5163 1332 14.89
linux.c1.xlarge [20 ECUs] Xeon E5410 7 0.68/hr 5118 780 8.78
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.5/hr 4049 932.1 7.05
m1.xlarge [8 ECUs] Xeon E5430 15 0.68/hr 4256 938.6 5.15
m1.large [4 ECUs] Xeon E5430 7.5 0.34/hr 3092 663.4 4.08
c1.medium [5 ECUs] Xeon E5410 1.7 0.17/hr 2635 776.2 3.43
m1.small [1 ECU] Opteron 2218 1.7 0.085/hr 1726 179.7 0.92

Amazon Web Services (AWS) [US West]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.68/hr 6109 1520 27.45
m2.2xlarge [13 ECUs] Xeon X5550 34.2 1.34/hr 5475 1329.2 15.9
c1.xlarge [20 ECUs] Xeon E5410 7 0.76/hr 4693 785 8.21
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.57/hr 3883 945.4 6.85
m1.xlarge [8 ECUs] Xeon E5430 15 0.76/hr 4185 916.8 5.14
m1.large [4 ECUs] Xeon E5430 7.5 0.38/hr 3026 643.4 3.95
c1.medium [5 ECUs] Xeon E5410 1.7 0.19/hr 2962 715.5 3.45
m1.small [1 ECU] Xeon E5430 1.7 0.095/hr 1312 277.2 1.04

Amazon Web Services (AWS) [EU West]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.68/hr 6188 1489.7 27.45
m2.2xlarge [13 ECUs] Xeon X5550 34.2 1.34/hr 5368 1337.6 15.19
c1.xlarge [20 ECUs] Xeon E5410 7 0.76/hr 4926 787 8.55
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.57/hr 3836 945.9 6.79
m1.xlarge [8 ECUs] Xeon E5430 15 0.76/hr 4147 934.7 5.14
m1.large [4 ECUs] Xeon E5430 7.5 0.38/hr 3160 644 4.12
c1.medium [5 ECUs] Xeon E5410 1.7 0.19/hr 2710 730.5 3.43
m1.small [1 ECU] Xeon E5430 1.7 0.095/hr 1288 277.3 1.02

Amazon Web Services (AWS) [APAC]
ID CPU Memory Price Geekbench Unixbench CCUs
m2.4xlarge [26 ECUs] Xeon X5550 68.4 2.68/hr 3472 1465.8 12.46
m2.2xlarge [13 ECUs] Xeon X5550 34.2 1.34/hr 3357 1228.4 9.62
m2.xlarge [6.5 ECUs] Xeon X5550 17.1 0.57/hr 3469 821.2 6.27
c1.xlarge [20 ECUs] Xeon E5410 7 0.76/hr 3042 906.1 5.66
m1.xlarge [8 ECUs] Xeon E5430 15 0.76/hr 3271 867.4 4.65
m1.large [4 ECUs] Xeon E5430 7.5 0.38/hr 2594 598.9 3.92
c1.medium [5 ECUs] Xeon E5410 1.7 0.19/hr 2796 415.8 3.23
m1.small [1 ECU] Xeon E5430 1.7 0.095/hr 1522 183.5 1.02

Rackspace Cloud
Rackspace Cloud servers showed a very flat CPU performance variation between server instance sizes. All instances we benchmarked in both data centers utilized homogenous Opteron 2374 "Shanghai" 2.2 GHz hardware. Rackspace states that their CPU provides a minimum allocation based on instance size with bursting allowed for all instance sizes.

We tested servers in both their Dallas as well as the newer Chicago data centers. However, Rackspace does not allow users to chose which data center to deploy servers to. Their use of multiple data centers appears to deal more with capacity issues rather than to offer user choice. When you create an account, that account is assigned to a specific data center, and from that point forward you will only have the option to deploy to that assigned data center.

Rackspace Cloud [Dallas]
ID CPU Memory Price Geekbench Unixbench CCUs
rs-16gb Opteron 2374 16 0.96/hr 3222 933.2 4.95
rs-2gb Opteron 2374 2 0.12/hr 3330 935.4 4.94
rs-1gb Opteron 2374 1 0.06/hr 3462 934.2 4.93
rs-4gb Opteron 2374 4 0.24/hr 3211 935.1 4.9

Rackspace Cloud [Chicago]
ID CPU Memory Price Geekbench Unixbench CCUs
rs-16gb-il Opteron 2374 16 0.96/hr 4473 1019.2 5.41
rs-8gb-il Opteron 2374 8 0.48/hr 4291 1035.5 5.39
rs-1gb-il Opteron 2374 1 0.06/hr 4339 1042.9 5.38
rs-4gb-il Opteron 2374 4 0.24/hr 4368 1013.8 5.29
rs-2gb-il Opteron 2374 2 0.12/hr 4360 963.4 5.17

Storm on Demand was launched a few months ago by Liquid Web. They provide 2 types of cloud servers. The first is traditional 2-48GB cloud server running on multi-tenant hosts. The second, called "Bare Metal", allows you to select specific dedicated hardware (CPU, SATA or SAS disks, memory) to deploy your server on. Bare Metal servers are still virtualized, but do not share the underlying hardware with any other server instances.

Of all the IaaS vendors we reviewed, Storm offers by far the most diverse heterogenous infrastructure. This approach pays off big in terms of performance with 10 servers that scored 20+ CCUs. The only mismatched hardware we discovered was the cloud 16GB server running on Opteron 2350 hardware which performed poorly compared with the small 2, 4 and 8 GB servers. However, Storm has informed us that their 16GB Opteron 2350 hardware is being upgraded to Opteron 2378 which should improve performance on future benchmarks.

Storm's 48GB cloud server was the top performer out of all of our benchmarked servers with 42.5 CCUs and a Geekbench score of 13020! This is most likely due to the very new and extremely fast Xeon X5650 "Westmere" hardware it runs on. The Intel i5 CPUs also performed very well with our CPU benchmarks and provide an excellent performance to price ratio (26.5 CCUs for $0.171/hr)!

Storm Cloud [MI, US]
ID CPU Memory Price Geekbench Unixbench CCUs
Cloud: 48gb Xeon X5650 45.9 1.37/hr 13020 4448.5 42.87
Bare Metal: x3440-8gb Xeon X3440 8 0.274/hr 7597 2760.1 27.39
Bare Metal: e5506x2-4gb Xeon E5506 8 0.274/hr 7742 2906.7 27.24
Bare Metal: e5506x2-4gb Xeon E5506 8 0.322/hr 7704 2880.1 27.13
Bare Metal: e5506x2-8gb Xeon E5506 8 0.391/hr 7753 2837.5 26.72
Bare Metal: i5-750-2gb Core i5 750 2 0.171/hr 6417 2464.8 26.6
Bare Metal: i5-750-4gb Core i5 750 4 0.206/hr 6413 2450.1 26.47
Bare Metal: e5506x2-8gb Xeon E5506 8 0.48/hr 7686 2811.5 26.51
Cloud: 32gb Opteron 2378 30.4 0.69/hr 8326 2373.3 26.4
Cloud: 8gb Xeon X3440 7 0.27/hr 6162 2284.7 21.41
Cloud: 4gb Core i5 750 3.5 0.14/hr 4555 1465.1 9.33
Bare Metal: amd2350x2-32gb Opteron 2350 32 0.713/hr 6531 1803.6 7.31
Cloud: 2gb Core 2 Q9400 1.7 0.07/hr 3062 629.7 5.01
Cloud: 16gb Opteron 2350 15.2 0.34/hr 4034 1214 4.73

GoGrid's servers showed a decent linear performance increase with larger sized instances. The largest 8GB instance was deployed on an E5450 2.99 GHz host (compared with the E5520 2.27 GHz hosts for the other instances) which performed significantly better than the smaller sized instances.

GoGrid [CA, US]
ID CPU Memory Price Geekbench Unixbench CCUs
gg-8gb Xeon E5450 8 1.52/hr 8105 507.5 23.2
gg-4gb Xeon E5520 4 0.76/hr 5373 1866.9 9.28
gg-2gb Xeon E5520 2 0.38/hr 3549 993.3 4.87
gg-1gb Xeon E5520 1 0.19/hr 2860 941.2 4.42

Voxel maintains 3 cloud data centers in the US (New York), EU (Amsterdam), and Asia (Singapore). All of our cloud servers were deployed on homogenous harware: Xeon L5520 "Nehalem" 2.26 GHz. They appear to use more of a fixed CPU allocation because there was a notable increase in performance on larger instance sizes.

Voxel [NY, US]
ID CPU Memory Price Geekbench Unixbench CCUs
vx-14gb-ny Xeon L5520 14 0.727/hr 5176 1023.5 10.15
vx-8gb-ny Xeon L5520 8 0.421/hr 4022 846.4 6.12
vx-4gb-ny Xeon L5520 4 0.211/hr 3487 680.7 5.33
vx-2gb-ny Xeon L5520 2 0.106/hr 2876 483.4 4.67

Voxel [NL]
ID CPU Memory Price Geekbench Unixbench CCUs
vx-14gb-nl Xeon L5520 14 0.727/hr 6169 1178 16.2
vx-8gb-nl Xeon L5520 8 0.421/hr 4699 954.7 7.53
vx-4gb-nl Xeon L5520 4 0.211/hr 3416 645.8 5.48
vx-2gb-nl Xeon L5520 2 0.106/hr 3093 494 4.85

Voxel [SG]
ID CPU Memory Price Geekbench Unixbench CCUs
vx-14gb-sg Xeon L5520 14 0.727/hr 5419 1107.8 11.17
vx-8gb-sg Xeon L5520 8 0.421/hr 4428 909.8 6.24
vx-4gb-sg Xeon L5520 4 0.211/hr 3168 611.3 5.04
vx-2gb-sg Xeon L5520 2 0.106/hr 2779 463.7 4.4

NewServers is fairly unique in that their "bare metal" cloud servers actually run on physical hosts. There is no hypervisor layer between the server and the underlying hardware. When you deploy a server, the OS image is written directly to the physical disk(s). Their "Fast" server performed very well and is one of the better values at $0.53/hr for 26.41 CCUs.

NewServers [FL, US]
ID CPU Memory Price Geekbench Unixbench CCUs
ns-fast Xeon E5450 4 0.53/hr 6271 2193.5 27.21
ns-jumbo Xeon E5504 48 0.6/hr 5713 2809.7 19.15
ns-large Xeon E5405 4 0.25/hr 4505 1849.7 6.12
ns-med Xeon 3.20 2 0.17/hr 2672 708.5 3.43
ns-small Xeon 2.80 1 0.11/hr 1669 491.3 2.47

While Linode doesn't market itself as a "cloud", we included it in the benchmarks because they are a good and very popular service and provide many of the features common to the cloud including auto-provisioning, disk imaging. All instance sizes we benchmarked deployed on homogenous Xeon L5520 2.26 GHz hardware. The servers show a very flat CPU performance variation between instance sizes. We only benchmarked servers in their Atlanta data center. Linode also maintains data centers in Fremont CA, Dallas TX, Newark NJ, and London UK.

Linode VPS Hosting [Atlanta]
ID CPU Memory Price Geekbench Unixbench CCUs
ln-5760-atlanta Xeon L5520 5.54 10.67/day 4217 1160.5 6.4
ln-14400-atlanta Xeon L5520 14.06 26.67/day 3865 1144.1 6.3
ln-1080-atlanta Xeon L5520 1.05 2/day 3623 957.2 5.48
ln-2880-atlanta Xeon L5520 2.81 5.33/day 3625 979.4 5.38
ln-8640-atlanta Xeon L5520 8.45 16/day 3780 971 5.33

All our benchmark servers deployed on to either Xeon X3460 2.8 GHz or E5520 2.27 GHz hardware. SoftLayer is another provider where CPU performance was very flat. Disk I/O was also painfully slow, but that is a topic for another post.

SoftLayer [Dallas]
ID CPU Memory Price Geekbench Unixbench CCUs
sl-8gb-dallas Xeon E5520 8 0.5/hr 5862 1514.1 14.35
sl-1gb-dallas Xeon X3460 1 0.15/hr 3745 502.7 6.1
sl-4gb-dallas Xeon E5520 4 0.35/hr 4482 941.8 5.52
sl-2gb-dallas Xeon X3460 2 0.25/hr 4399 573.9 5.11

SoftLayer [WDC]
ID CPU Memory Price Geekbench Unixbench CCUs
sl-4gb-wdc Xeon X3460 4 0.35/hr 5599 1250.9 16.88
sl-8gb-wdc Xeon E5520 8 0.5/hr 5885 1527.7 16.84
sl-2gb-wdc Xeon X3460 2 0.25/hr 4265 871.5 7.62
sl-1gb-wdc Xeon X3460 1 0.15/hr 3710 507.7 5.81

SoftLayer [Seattle]
ID CPU Memory Price Geekbench Unixbench CCUs
sl-8gb-seattle Xeon E5520 8 0.5/hr 6040 1520.8 14.07
sl-4gb-seattle Xeon X3460 4 0.35/hr 5455 1324.4 12.3
sl-2gb-seattle Xeon X3460 2 0.25/hr 4429 822.8 6.49
sl-1gb-seattle Xeon X3460 1 0.15/hr 3716 505.8 5.52

Terremark vCloud Express
Terremark is one of the first VMWare vCloud providers. Deployment of servers in vCloud allows the user to select both desired memory and VPUs (the vCloud term for CPUs). All of our benchmark servers deployed on homogenous Opteron 8389 2.91 GHz hardware. CPU performance varied from benchmark to benchmark. Unixbench's parallel benchmark did show a notable increase in performance from 1 to 8 VPUs. However, other benchmarks did not show much increase leading to a flat CCU metric across all instance sizes and VPU combinations we benchmarked.

Terremark vCloud Express [FL, US]
ID CPU Memory Price Geekbench Unixbench CCUs
tm-16gb-8vpu Opteron 8389 16 1.672/hr 4304 2371.5 7.69
tm-8gb-4vpu Opteron 8389 8 0.64/hr 5999 1667 6.41
tm-1gb Opteron 8389 1 0.074/hr 2863 1279.3 5.44
tm-2gb Opteron 8389 2 0.137/hr 2779 1272.7 5.38
tm-4gb-2vpu Opteron 8389 4 0.305/hr 3793 1336.6 5.34

OpSource is another VMWare based cloud. OpSource allows you to configure cloud severs with 1-4 "CPUs". CPU performance was very flat between instances of varying sizes even showing a decrease in performance from smaller to larger sized instances. All servers deployed to identical Xeon X7460 2.66 GHz hardware.

OpSource Cloud [VA, US]
ID CPU Memory Price Geekbench Unixbench CCUs
os-1gb Xeon X7460 1 0.268/hr 2100 361.5 2.56
os-2gb Xeon X7460 2 0.296/hr 1894 338.1 2.43
os-4gb-2cpu Xeon X7460 4 0.392/hr 1923 241 2.24
os-8gb-4cpu Xeon X7460 8 0.584/hr 1830 149.3 1.87
os-16gb-4cpu Xeon X7460 16 0.808/hr 2004 129 1.87
os-32gb-4cpu Xeon X7460 32 1.256/hr 1908 130.8 1.82

Speedyrails is a VPS provider based out of Quebec Canada. All benchmark servers deployed on homogenous hardware: Xeon E5520 2.27 GHz.

Speedyrails [QC, CA]
ID CPU Memory Price Geekbench Unixbench CCUs
sr-1gb Xeon E5520 1 2.27/day 4413 1260.8 9.06
sr-4gb Xeon E5520 4 8/day 4359 1261.1 8.49
sr-2gb Xeon E5520 2 4.27/day 4347 1066.7 6.74

Zerigo is a VPS and Cloud Server vendor based out of Denver, CO. All benchmark servers deployed on identical hardware, Opteron 2374 2.20 GHz.

Zerigo [CO, US]
ID CPU Memory Price Geekbench Unixbench CCUs
zr-4gb Opteron 2374 4 0.24/hr 3829 805.5 4.83
zr-2gb Opteron 2374 2 0.12/hr 3274 813.9 4.73

While there was a notable increase in performance with larger sized instances, the overall CPU performance was not great. Hardware was also homogenous between different instance sizes.

ReliaCloud Cloud Services [MN, US]
ID CPU Memory Price Geekbench Unixbench CCUs
rc-4gb Xeon E5504 4 0.32/hr 1637 226.8 1.11
rc-2gb Xeon E5504 2 0.16/hr 1653 267.9 1.07
rc-1gb Xeon E5504 1 0.08/hr 1392 243.2 0.65

IBM's Development & Test Cloud is a free cloud service intended for only development and testing. Only 3 32-bit instance sizes are supported: small, medium and large. Instances are time limited to about 1 week with the option to extend. The large instance performed very well overall.

IBM Development & Test Cloud [NY, US]
ID CPU Memory Price (USD) Geekbench Unixbench CCUs
ibm-dev-large Xeon X5570 3.5 10256 3059.5 27.9
ibm-dev-med Xeon X5570 1.75 5536 1526.6 5.44

BlueLock is another VMWare vCloud provider. As with the other VMWare providers, they appear to use a homogenous hardware environment for all instance sizes. However, unlike most other homogenous platforms, BlueLock's instances showed a notable increase in performance on larger sized (more CPUs) instances. Strangely, the 4CPU/8GB instance outperformed the 8CPU/16GB instance.

BlueLock [IN, US]
ID CPU Memory Price Geekbench Unixbench CCUs
bl-8gb-4cpu Xeon X5550 8 0.661/hr 6311 2159.1 26.26
bl-16gb-8cpu Xeon X5550 16 1.729/hr 6665 2525.2 25.55
bl-4gb-2cpu Xeon X5550 4 0.308/hr 4575 1351 9.06
bl-1gb Xeon X5550 1 0.068/hr 3690 1153.3 6.3
bl-2gb Xeon X5550 2 0.134/hr 3727 1223.4 6.14

Cloud Central is a new cloud server provider based out of Australia. All benchmark instances deployed to homogenous AMD Opteron 2.20 GHz hosts. Prices shown are Australian Dollar.

Cloud Central [AU]
ID CPU Memory Price Geekbench Unixbench CCUs
cc-large Opteron 2374 8 0.96/hr 4224 767.3 4.97
cc-reg Opteron 2374 4 0.48/hr 3894 787.2 4.96
cc-huge Opteron 2374 16 1.92/hr 3674 785.2 4.87
cc-med Opteron 2374 2 0.24/hr 3913 755.1 4.69
cc-small Opteron 2374 1 0.12/hr 3727 756.6 4.63

RimuHosting is a VPS provider based out of New Zealand. They maintain data centers in Australia, New Zealand, London, and Texas. We benchmarked 2GB instances in their Auckland NZ and Dallas TX data centers.

RimuHosting [TX, US]
ID CPU Memory Price Geekbench Unixbench CCUs
rh-tx-2gb Xeon E5506 2 104.57/mo 3080 735.6 4.56

RimuHosting [NZ]
ID CPU Memory Price (NZD) Geekbench Unixbench CCUs
rh-nz-2gb Xeon E5506 2 210.88/mo 3122 662.3 4.49

ElasticHosts is a UK based cloud provider. They currently maintain 2 data centers in the UK and a new data center in Dallas, TX. We only benchmarked their London Peer1 data center. ElasticHosts runs on the Linux KVM hypervisor. This hypervisor is unique in that it allows you to selected a specific MHz metric when deploying cloud servers. We benchmarked 2 GHz - 20 GHz cloud servers. Although the hardware environment appears to be homogenous, the benchmarks showed a clear increase in performance on larger sized instances.

ElasticHosts [UK]
ID CPU Memory Price (USD) Geekbench Unixbench CCUs
eh-8gb-20gh Xeon E5420 8 0.654/hr 6544 881.7 9.98
eh-4gb-8gh Xeon E5420 4 0.326/hr 3919 814.3 5.54
eh-2gb-4gh Xeon E5420 2 0.164/hr 3409 631 4.75
eh-1gb-2gh Xeon E5420 1 0.082/hr 2545 539.8 4.3

Flexiscale is a UK based cloud server provider that has been around for a few years. They were recently acquired and renamed to Flexiant. They are currently in beta release of their Flexiscale 2.0 cloud server platform. These results were from testing 2.0 platform servers. Flexiant adopted a point-based subscription model for purchasing cloud servers. See their website for more details.

Flexiscale [UK]
ID CPU Memory Price (GBP) Geekbench Unixbench CCUs
fx-8gb Opteron 8218 8 4629 636.5 3.81
fx-4gb Opteron 8220 4 3407 541.6 3.66
fx-2gb Opteron 8218 2 2394 510 3.36
fx-1gb Opteron 8218 1 2354 481.9 2.95

This is our first attempt at defining a standard CPU performance metric for comparing servers in multiple clouds. We acknowledge that it is not perfect and hope to make improvements over time. Please comment on this post if you have any suggestions on how we might improve our methods. We intend to continually run these benchmarks (every couple of months) to improve the quality and quantity of our data available as well as to check for upgrades and improvements made by the providers.

One take-away point we observed is that heterogenous hardware environments (where host hardware is configured with faster CPUs for larger sized instances) appears to be more conducive to true cloud CPU scaling. Of the 20 server clouds we benchmarked, only 4 appear to be providing such an environment: EC2, Storm on Demand, GoGrid and NewServers.