Google Cloud Platform Blog: Performance advantages of the new Google Cloud Storage Connector for Hadoop

Google Cloud Platform Blog

Product updates, customer stories, and tips and tricks on Google Cloud Platform

Performance advantages of the new Google Cloud Storage Connector for Hadoop

Wednesday, January 15, 2014

Our guest blog post today comes from Mike Wendt, R&D Associate Manager at Accenture Technology Labs, who recently published a study detailing the real world performance advantages of Hadoop on Google Compute Engine. His team utilized the recently launched Google Cloud Storage Connector for Hadoop and observed significant performance improvements over HDFS on local filesystemsAccenture Technology LabsGoogle Cloud Storage Connector for HadoophereOriginal experiment setup

Figure 1. Data-flow model using input and output copies to local disk HDFS

Modified experiment setup using Google Cloud Storage Connector for Hadoop

Figure 2. Data-flow model using Google Cloud Storage Connector for Hadoop

Result of experimentsRecommendation engine

Figure 3. Recommendation engine execution times

Sessionization

Figure 4. Sessionization execution times

Document clustering

Figure 5. Document clustering execution times

Conclusionrecent study

Labels: Big Data & Machine Learning , Storage & Databases

Free Trial

GCP Blogs

Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog

Popular Posts

Understanding Cloud Pricing
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads

Labels

Announcements 193
Big Data & Machine Learning 134
Compute 271
Containers & Kubernetes 92
CRE 27
Customers 107
Developer Tools & Insights 151
Events 38
Infrastructure 44
Management Tools 87
Networking 43
Open 1
Open Source 135
Partners 102
Pricing 28
Security & Identity 85
Solutions 24
Stackdriver 24
Storage & Databases 164
Weekly Roundups 20

Feed

Subscribe by email

Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.

Technical questions? Check us out on Stack Overflow.
Subscribe to our monthly newsletter.

Company-wide

Official Google Blog
Enterprise Blog
Student Blog

Products

Official Android Blog
Chrome Blog
Lat Long Blog

Developers

Ads Developer Blog
Android Developers Blog
Developers Blog

Google
Privacy
Terms