Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
Take your logs data to new places with streaming export to Cloud Pub/Sub
Thursday, April 30, 2015
Earlier this year, we
announced
the beta of the
Google Cloud Logging
service, which included the capability to:
Stream logs in real-time to
Google BigQuery
, so you can
a
nalyze log data and get immediate insights.
Export logs to
Google Cloud Storage
(including
Google Cloud Storage Nearline
), so you can archive logs data for longer periods to meet backup and compliance requirements.
Today we’re expanding Cloud Logging capabilities with the beta of Cloud Logging Connector that allows you to stream logs to
Google Cloud Pub/Sub
. With this capability you can stream log data to your own endpoints and further expand how you can make big data useful. For example, you can now transform and enrich the data in
Google Cloud Dataflow
before sending it to BigQuery for analysis. Furthermore, this provides easy real-time access to all your logs data, so you can export it to your private cloud or any third party application.
Cloud Pub/Sub
Google Cloud Pub/Sub
is designed to deliver real-time and reliable messaging in one global, managed service that helps you create simpler, more reliable, and more flexible applications. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication between independently written applications. With Cloud Pub/Sub, you can push your log events to another Webhook, or pull them as they happen. For more information, check out our Google Cloud Pub/Sub
documentation
.
High-Level Pub/Sub Schema
Configuring Export to Cloud Pub/Sub
Configuring export of logs to Cloud Pub/Sub is easy and can be done from the
Logs Viewer
user interface. To get to the export configuration UI start in the
Google Developers Console
, go to Logs under Monitoring and then click Exports on the top menu. Currently this supports export configuration for
Google App Engine
and
Google Compute Engine
logs.
One Click Export Configuration in the Developers Console
Transforming Log Data in Dataflow
Google Cloud Dataflow
allows you to build, deploy, and run data processing pipelines at a global scale. It enables reliable execution for large-scale data processing scenarios such as ETL and analytics, and allows pipelines to execute in either streaming or batch mode. You choose.
You can use the Cloud Pub/Sub export mechanism to stream your log data to Cloud Dataflow and dynamically generate fields, combine different log tables for correlation, and parse and enrich the data for custom needs. Here are a few examples of what you can achieve with log data in Cloud Dataflow:
Sometimes it is useful to see the data only for the key applications for top customers. In Cloud Dataflow, you can group logs by Customer ID or Application ID, filter out specific logs, and then apply some aggregation of system level or application level metrics.
On the flip side, sometimes you want to enrich the log data to make it easier to analyze, for example by appending marketing campaign information to customer interaction logs, or other user profile info. Cloud Dataflow lets you do this on the fly.
In addition to preparing the data for further analysis, Cloud Dataflow also lets you perform analysis in real time. So you can look for anomalies, detect security intrusions, generate alerts, keep a real-time dashboard updated, etc.
Cloud Dataflow can stream the processed data to BigQuery, so you can analyze your enriched data. For more details, please see the Google Cloud Dataflow
documentation
.
Getting Started
If you’re a current
Google Cloud Platform
user, the capability to stream logs to Cloud Pub/Sub is available to you at no additional charge. Applicable charges for
using Cloud Pub/Sub and Cloud Dataflow will apply
. For more information, visit the the
Cloud Logging documentation page
and share your
feedback
.
-Posted by Deepak Tiwari, Product Manager
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
Understanding Cloud Pricing
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow