Google Cloud Platform Blog: Measuring cloud performance just got easier and better

Google Cloud Platform Blog

Product updates, customer stories, and tips and tricks on Google Cloud Platform

Measuring cloud performance just got easier and better

Thursday, December 10, 2015

February 2015 EPFL EcoCloud Web SearchEPFL EcoCloud Web Serving

Fig. 1 PerfKit Benchmarker architecture

“Cloud computing has become the dominant computing platform for delivering scalable online services to a global user base all over the world. The constant emergence of new services, growing user bases and data deluge result in ever-increasing computational requirements for the service providers. Popular online services, such as web search, social networks and video streaming, are hosted by private or public cloud providers in large cloud-server systems, which comprise thousands of servers. Since its inception, CloudSuite has emerged as a popular suite of benchmarks, both in industry and among academics, to benchmark the performance of cloud servers. CloudSuite facilitates research in the field of servers specialized for cloud workloads (e.g., Scale-out Processors) and the development of real products in the server industry (e.g., Cavium Thunderx Processor). We believe that PerfKit Benchmarker (PKB) is a step towards the standardization of cloud benchmarking. In essence, we envision PKB as the “SPEC for cloud-server systems.” Naturally, our goals match with PKB's and the strong consortium put together by Google convinced us to team up. On the technical side, we are excited about the standard APIs that PKB provides, enabling the automated deployment of our benchmarks into the most well-known cloud-server providers. We believe that PKB has all the potential to be established as the de-facto standard for cloud benchmarking. Therefore, we expect it to grab the attention of cloud providers, academics and industry, while integrating more and the most recent online services.”
Carlos TorresRackspace “There are two main cases where I use PKB. One is to provide data for comparative analysis of hardware/software configurations to understand their performance characteristics, and the other is for measuring and tracking performance across software releases. PKB has brought me multiple benefits, but if I had to choose three, I'd say, speed, reproducibility and flexibility. Speed: Before PKB, configuring and executing a complex benchmark that made use of a multi-node distributed system, such as a 9-node Hadoop cluster with HDFS, took hours of tedious setup, scripting and validation. Maintenance of those scripts, and knowing the current best practices for deploying such systems was a nightmare. Once you executed a benchmark, gathering the data from the tests usually involved manually executing scripts to scrape, parse and copy the data from multiple machines. Now, with PKB, it is very easy to execute, not one, but even multiple of these benchmarks, against every major cloud, usually with just one command. I can rely on the community's expertise, and for the most part, trust the configurations provided with each of the benchmarks. Finally, PKB makes is really easy to consume the data gathered from the tests, since it produces JSON output of the results. Reproducibility: Just like in science, reproducibility is a very important aspect of performance engineering. To confirm that either a bottleneck exists, or that it has been fixed, it is necessary to reliably reproduce a workload. Previous to PKB, it was tedious to keep track of all the software versions and configuration settings needed to replicate a benchmark, which sometimes were not documented and hence forgotten. This made reproducibility hard, and error prone. By using the same PKB version, with a single command, I can easily recreate complex benchmarks, and know that I'm executing the same set of software since PKB explicitly tracks versions of the applications, benchmarks and tools used. Also by just sharing the command I used for a test, other users can recreate the same test in no time. Flexibility: One of the best features of PKB is the ability execute the same benchmarks across different cloud providers, machine types and compatible operating systems images. While PKB ships with great defaults for most benchmarks, it makes it very easy to execute the benchmarks using custom settings, using commands switches or configuration files that a benchmark might optionally accept. PKB doesn't just make executing benchmarks easy, but contributing new benchmarks is simple as well. It provides a set of libraries that benchmark writers can use to write, for the most part, OS-agnostic benchmark installations.“Marcin KarkochaMateusz BlaszkowskiIntel “We try to determine how specific cloud configuration options impact on workloads running inside the instances. Based on this, we want to create reference architectures for different clouds. We also run Perfkit benchmarks in order to compare and calculate capabilities of reference architectures from different providers. In our case, PKB is used in private cloud — because of that we have slightly different requirements and problems to solve. We do not focus on comparing public cloud offerings. Instead we try to find out what is the most efficient HW/SW configuration for a specific cloud. PKB as a framework gives us a possibility to create new plugins for providers and benchmarks. Thanks to this, we are able to easily build a custom benchmarking solution which meets most of our requirements.”Daniel SanchezMIT “We are using PKB to investigate new multicore systems. In particular, we are designing new hardware and software techniques that allow servers to provide low, predictable response latencies efficiently. PerfKit has made it much easier for us to simulate new hardware techniques on a broad array of cloud computing benchmarks. This is crucial for our work, because traditional benchmark suites are more focused on batch applications, which have quite different needs from cloud computing workloads.“PerfKit Benchmarker V1Open Source Benchmarking effort on GitHub