benchmarking the cloud

RSS Feed

Cloud Articles From Our Blog

30 months ago 07:17 PDT

Encoding Performance: Comparing Zencoder, Encoding.com, Sorenson & Panda

A few months ago we were approached by Zencoder to conduct a sponsored performance comparison of 4 encoding services including their own. The purpose was to validate their claims of faster encoding times using an independent, credible external source. This was a new frontier for us. Our primary focus has been performance analysis of infrastructure as a service (IaaS). However, we are curious about all things related to cloud and benchmarking and we felt this could be useful data to make available publicly, so we accepted.

Testing Methodology

This is a description of the methodology we used for conducting this performance analysis.

Source Media

Following discussions with Zencoder, we opted to test encoding performance using 4 distinct media types. We were tasked with finding samples for each media type, they were not provided by Zencoder. All source media was stored in AWS S3 using the US East region (the same AWS region each of the 4 encoding services are hosted from). The 4 media types we used for testing are:
  • HD Video: We chose an HD 1080P trailer for the movie Avatar. This file was 223.1 MB in size and 3 mins, 30 secs in duration.
  • SD Video: We chose a 480P video episode from a cartoon series. The file was 519.2 MB in size and about 23 mins in duration.
  • Mobile Video: We created a 568x320 video using an iPhone (source here). The file was 2.9 MB in size, 30 secs in duration.
  • MP3 Audio: We used an MP3 file we found on the web about Yoga (source here). The file was 42.2 MB in size, 58 mins 41 secs in duration.

Encode Settings

We used the same encode options across all of the services tested. The following is a summary of the encode options used for each corresponding media type:
Media Type Video Codec Video Bitrate Audio Codec Audio Bitrate Encode Passes
HD Video H.264 3000 Kb/s AAC 96 Kb/s 2
SD Video H.264 1000 Kb/s AAC 96 Kb/s 2
Mobile Video H.264 500 Kb/s AAC 96 Kb/s 2
MP3 Audio NA NA AAC 96 Kb/s 2

Test Scheduling

Testing was conducted during a span of 1 week. We built a test harness that integrated with the APIs of each of the 4 encoding services. The test harness invoked 2 test iterations daily with each service. Testing included both single request and 4 parallel requests. Each test iteration consisted of the following 8 test scenarios:
  • Single HD Video Request
  • Single SD Video Request
  • Single Mobile Video Request
  • Single MP3 Audio Request
  • 4 Parallel HD Video Requests
  • 4 Parallel SD Video Requests
  • 4 Parallel Mobile Video Requests
  • 4 Parallel MP3 Audio Requests
The order of the test scenarios was randomized, but the same tests were always requested at the same time for each service. Each test was run to completion on all services before the next test was invoked. The start times for 2 daily test iterations were separated by about 12 hours and incremented by 100 minutes each day. The end result was a distribution of test scenarios during many different times of the day. A total of 112 tests were performed during the 1 week test span, followed by an additional 24 test scenarios on encoding.com to test different combinations of service specific encoding options (described below).

Performance Metrics

During testing, we captured the following metrics:
  • Encode Time: The amount of time in seconds required to encode the media (excludes transfer time)
  • Transfer Time: The amount of time in seconds required to transfer the source media from AWS S3 into the service processing queue
  • Failures: When a service failed to complete a job

Test Results

The following is a summary of the results for each of the 8 test scenarios. Result tables have the following columns:
  • Avg Encode Time: The mean (average) encoding time in seconds for all jobs in this test scenario
  • Standard Deviation %: Standard deviation as percentage of the mean. A lower values indicates more consistency in performance
  • Median Encode Time: The median encoding time in seconds for all jobs in this test scenario
  • Avg Total Time: The mean (average) total job time in seconds. The total time is the sum of transfer (source files were hosted in AWS S3 US East), queue and encoding times, essentially the total turnaround time from the moment the job was submitted. Sorenson and Panda may cache source media files, thus reducing transfer time for future requests

A graph is displayed below each results table depicting sorted average encode and total times using a dual series horizontal bar chart. Also depicted on the graph is a line indicating the actual duration of the source media. Encode time bars that terminate to the left of this line signify faster than realtime encoding performance.

encoding.com offers a few different encoding job options that can affect encode performance. These options include the following:

  • instant: using the instant option, encoding.com will begin encoding before the source media has been fully downloaded. For larger source files, this can decrease encode times
  • twin turbo: this setting causes jobs to be delegated to faster servers in exchange for paying a $2/GB premium for encoding. encoding.com states that this option will deliver 6-8X faster encoding time over standard servers

Test Results: Single HD Video Request

The following are the results from a total of 14 HD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 141.08 6 137 155.08
encoding.com 2 (max+TT) 213.38 16 193 244.53
encoding.com 1 (max+TT+instant) 213.5 11 213.5 238
encoding.com 4 (plus+TT+instant) 226.75 11 235.5 258.75
encoding.com 3 (max) 618.75 12 643.5 653.25
sorenson 974.31 2 975 982.46
panda 1246.31 4 1247 1255.93

Failures - None

Test Results: Single SD Video Request

The following are the results from a total of 14 SD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 229.14 9 225.5 281.64
encoding.com 4 (plus+TT+instant) 453 1 453.5 718.75
encoding.com 2 (max+TT) 454.07 11 458.5 689.21
encoding.com 1 (max+TT+instant) 474 7 474 732.5
encoding.com 3 (max) 1137.75 12 1162 1429
panda 1649.36 7 1640.5 1673.93
sorenson 2087.23 8 2052 2101

Failures

  • Sorenson: 1

Test Results: Single Mobile Video Request

The following are the results from a total of 14 mobile video encode jobs submitted at various times of the day over a period of 1 week. NOTE: During our testing, encoding.com jobs would occasionally experience excessive queue times with the status "Waiting for encoder". This is the reason for the long green section representing the transfer/queue time delta on the graph below.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 19.38 12 20 30.23
panda 25.69 4 25 26.31
encoding.com 4 (plus+TT+instant) 35 7 34 122.75
encoding.com 3 (max) 40.25 9 40.5 51.25
sorenson 47 6 46 68.31
encoding.com 2 (max+TT) 76.92 139 31 88.23
encoding.com 1 (max+TT+instant) 97 67 97 112

Failures - None

Test Results: Single MP3 Audio Request

The following are the results from a total of 14 audio encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 120.31 11 115 131
sorenson 189.38 11 182 196.23
panda 218.58 3 217 222.16
encoding.com 1 (max+TT+instant) 221 0 221 255
encoding.com 4 (plus+TT+instant) 224.75 9 225 248.75
encoding.com 3 (max) 240.5 7 238 355.75
encoding.com 2 (max+TT) 328.36 52 254 360.45

Failures

  • encoding.com max+TT+instant: 1
  • encoding.com max+TT: 2

Test Results: 4 Parallel HD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) HD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 141.23 4 140 153.71
encoding.com 1 (max+TT+instant) 236.88 19 250 280.13
encoding.com 2 (max+TT) 292.21 46 259.5 321.96
encoding.com 4 (plus+TT+instant) 298.69 37 256.5 354.44
encoding.com 3 (max) 723.44 25 751.5 796.38
panda 1322.19 17 1249 1336.9
sorenson 1751.44 3 1745 1769.71

Failures - None

Test Results: 4 Parallel SD Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) SD video encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 240.48 17 224.5 327.98
encoding.com 2 (max+TT) 439.5 21 445 700.15
encoding.com 4 (plus+TT+instant) 461.56 14 459.5 757.87
encoding.com 1 (max+TT+instant) 484.88 10 459 770.01
encoding.com 3 (max) 1274.63 11 1274 1552.69
panda 1680.96 10 1648.5 1710.9
sorenson 3777.54 10 3835 3808.94

Failures - None

Test Results: 4 Parallel Mobile Video Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) mobile video encode jobs submitted at various times of the day over a period of 1 week. In this test scenario, we again experience some long queue times during one of the encoding.com test phases.
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 20.4 15 20 27.27
panda 26.65 7 26 27.21
encoding.com 1 (max+TT+instant) 34 8 34 49.75
sorenson 55.2 15 53 77.54
encoding.com 4 (plus+TT+instant) 68 60 44.5 106.63
encoding.com 2 (max+TT) 87.46 100 37 104.92
encoding.com 3 (max) 115.88 197 46 149.32

Failures

  • sorenson: 8

Test Results: 4 Parallel MP3 Audio Requests

The following are the results from a total of 56 (14 sets of 4 parallel requests) audio encode jobs submitted at various times of the day over a period of 1 week:
Service Avg Encode Time Standard Deviation % Median Encode Time Avg Total Time
zencoder 115.62 6 114 126.16
sorenson 187.24 5 184 204.57
panda 225.15 17 214 227.92
encoding.com 4 (plus+TT+instant) 234.38 17 228.5 266.63
encoding.com 2 (max+TT) 265.43 19 261 313.56
encoding.com 1 (max+TT+instant) 273.67 17 274 344
encoding.com 3 (max) 285.69 31 259 317.15

Failures

  • sorenson: 3
  • encoding.com max+TT+instant: 5
  • encoding.com max+TT: 5
  • encoding.com max: 3

Encoding Service Accounts

The following encoding services are included in this analysis: Zencoder, encoding.com, Sorenson Media, and Panda. Each service offers different account and pricing options. We setup an account with each service (including Zencoder) using their standard signup process. Because 4 parallel job requests were part of the testing, we opted for service plans that would limit the effects of queue time under such conditions. The pricing data below is for informational purposes only. The purpose of this post is not to compare service pricing, as there are simply too many variations between services to be able to do so.The following is a summary of the account options we selected and pricing with each service:
Service Plan Used Plan Cost Plan Usage Included Encoding Cost Total Test Costs
Zencoder Launch $40/mo 1000 Encoding Mins $0.08/min HD; $0.04/min SD; $0.01/min audio $91
encoding.com Max $299/mo 75GB Encoded Media NA $299
Sorenson Media Squeeze Managed Server $199/mo 1200 Encoding Mins $8 per 60 Mins extra $375 (incl addl encoding mins)
Panda 4 dedicated encoders $396 Unlimited encoding - $0.15/GB to upload encoded media NA $436 (incl bandwidth)

Disclaimer

This comparison was sponsored by Zencoder. In order to sustain our ability to provide useful, free & publicly accessible analysis, we frequently take on paid engagements. However, in doing so, we try to maintain our credibility and objectivity by using reliable tests, being transparent about our test methods, and attempting to represent the data in a fair way.

Summary

In order to maintain our objectivity and independence, we generally do not recommend one service over another. We prefer to simply present the data as it stands, and let readers draw their own conclusions.