Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
Big data, the cloud way
Thursday, April 16, 2015
The promise of big data is faster and better insight into your business. Yet it often turns into an infrastructure project. Why? For example, you might be collecting a deluge of information and then correlating, enriching and attempting to extract real-time insights. Should you expect such feats, by their very nature, to involve a large amount of resource management and system administration? You shouldn’t. Not in the cloud. Not if you’re using big data the
cloud way
.
Big data the
cloud way
means being more productive when building applications, with faster and better insights, without having to worry about the underlying infrastructure. More specifically, it includes:
NoOps
: Your cloud provider should worry about deploying, managing and upgrading infrastructure to make it scalable and reliable. “NoOps” means the platform handles such tasks and optimizations for you, freeing you up to focus on understanding and exploiting the value in your data.
Cost effectiveness
: In addition to increased ease of use and agility, a “NoOps” solution provides clear cost benefits via the removal of operations work; but the cost benefits of big data the
cloud way
go even further
–
the platform auto-scales and optimizes your infrastructure consumption, and eliminates unused resources like idle clusters. You manage your costs by dialing up or down the number of queries and the latency of your processing based on your cost/benefit analysis. You should never have to re-architect your system to adjust your costs.
Safe and easy collaboration
: You can share datasets from files in
Google Cloud Storage
or tables in
Google BigQuery
with collaborators inside or outside of your organization without the need to make copies or grant database access. There’s one version of the data – which you control – and authorized users can access it (at no cost to you) without affecting the performance of your jobs.
Google has been blazing the big data trail for the rest of the industry
–
so when you use Google Cloud Platform, big data the
cloud way
also means:
Cutting-edge features:
Google Cloud Dataflow
provides reliable, event-time-based stream processing, available by default with no extra work. But making stream processing easy and reliable doesn’t mean removing the option of running in batch. The same pipeline can execute in batch mode, which you can use to lower costs or analyze historical data. Now, consistently processing streaming data at large scale doesn’t have to be a complex and brittle endeavor that’s reserved for the most critical scenarios.
Google Cloud Platform
delivers these characteristic by making data analysis quick, affordable and easy. Today, at the
Hadoop Summit
in Brussels, we announced that
our big data services
are taking a big step forward – allowing everyone to use big data the
cloud way
.
Google Cloud Dataflow now available in beta
Today, nothing stands between you and the satisfaction of seeing your processing logic, applied in your choice of streaming or batch mode, executed via a fully managed processing service. Just write a program, submit it, and Cloud Dataflow will do the rest. No clusters to manage – Cloud Dataflow will start the needed resources, autoscale them (within the bounds you choose), and terminate them as soon as the work is done. You can
get started right now
.
Google BigQuery has many new features and is now available in European zones
BigQuery, the quintessential cloud-native, API-driven service for SQL analytics, has new security and performance features. For example, the introduction of row-level permissions makes data sharing even easier and more flexible. With its ease of ingestion (we’ve raised the default ingestion limit to 100,000 rows per second per table), virtually unlimited storage, and fantastic query performance even for huge datasets, BigQuery is the ideal platform for storing, analyzing and sharing structured data. It also supports repeated records and querying inside JSON objects for loosely structured data. In addition, starting today, BigQuery now offers the option to store your data in Google Cloud Platform European
zones
. You can contact
Google
technical support today to use this option.
A comprehensive set of big data services
Google Cloud Pub/Sub
is designed to provide scalable, reliable and fast event delivery as a fully managed service. Along with BigQuery streaming ingestion and Cloud Dataflow stream processing, it completes the platform’s end-to-end support for low-latency data processing. Whether you’re processing customer actions, application logs or IoT events, Google Cloud Platform allows you to handle them in real time, the
cloud way
. Leave Google Cloud Platform in charge of all the scaling and administration tasks so you can focus on
what
needs to happen, not
how
.
Using big data the
cloud way
doesn’t mean that Hadoop, Spark, Flink and other open source tools originally created for on-premises can’t be used in the cloud. We’ve ensured that you can benefit from the richness of the open source big data ecosystem via native connectors to
Google Cloud Storage
and
BigQuery
along with an
automated Hadoop/Spark cluster deployment
.
Google BigQuery customer
zulily
joined us recently for a
big data webinar
to share their experience using big data the
cloud way
and how it helped them increase revenue and overall business visibility while decreasing their operating costs. If you’re interested in exploring these types of benefits for your own company, you can easily get started today by
running your first query on a public dataset
or uploading your own data.
Here’s a simplified illustration of how Google Cloud Platform data processing services relate to each other and support all stages of the data lifecycle:
Scuba equipment helps humans operate under water, but divers still fall hopelessly short of the efficiency and agility of marine creatures. When it comes to big data in the cloud, be a dolphin, not a scuba diver. Google Cloud Platform offers a set of powerful, scalable, easy to use and efficient big data services built for the cloud. Embrace big data, the
cloud way
, by taking advantage of them today.
Learn more about Google Cloud Platform’s
big data
solutions or
get started with Dataflow
and BigQuery today. We can’t wait to see what you achieve when you use
big data the cloud way
.
-Posted by William Vambenepe, Product Manager
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
Understanding Cloud Pricing
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow