Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
Near Real-Time Log Streaming and Analysis with Google Cloud Platform & Logentries
Monday, June 22, 2015
At Google we spend a lot of time thinking about how we can make DevOps easy for Google Cloud Platform customers. Whether you are using Google App Engine, Google Compute Engine, or any other service, you want access to logs produced by your system and applications.
Google Cloud Platform delivers support for centralized logging via
Google Cloud Logging
which provides you with the ability to view, search, and analyze log data. Cloud Logging includes the capability for log archival in
Google Cloud Storage
and the ability to send logs to
Google BigQuery
. In addition, Cloud Logging also allows you to forward these logs to any custom endpoint including third party log management services for advanced and tailored log analytics via the near real-time streaming
Google Cloud Pub/Sub API
.
We are happy to announce a real-time integration of Logentries, a third party log analytics service,
with the Google Cloud Platform. Log Management and Analytics is a critical customer need and we are excited to offer Google customers a choice to easily send logs to a key provider like Logentries. This integration offers Google Cloud Platform customers an easily configurable choice for log management and advanced analytics that includes anomaly detection. Customers can now use Logentries for Google App Engine and services like Cloud Dataflow as well and makes it even easier to get started. At Google, we are committed to creating an open ecosystem with easy path of integration for partners, and Logentries provides a great example of a leading partner.
"Thanks to Google Cloud Logging export feature to Pub/Sub, it was easy to build a direct integration between Logentries and Google Cloud," explained Marc Concannon, VP of Product at Logentries. "The Pub/Sub API was well documented and the Google's commitment to developing an open collaboration made the integration smooth”.
Overview of the Google Cloud Pub/Sub API
Cloud Pub/Sub is a powerful messaging service responsible for routing data between applications at scale that delivers notifications within milliseconds, even when handling more than 1 million messages per second. In essence it is a near real time many-to-many, asynchronous messaging service that helps to create simple, reliable, and flexible applications by decoupling senders and receivers. It allows for secure and highly available communication between independently written applications.
Cloud Pub/Sub is thus an ideal service for transporting your logs and it allows you to either push your log events, or pull them as they happen.
Figure one: Google Cloud Pub/Sub Data Flow Schema
Logentries
, a near real-time log analytics service, is the first third party service to integrate with Google Cloud Pub/Sub near real-time log streaming, allowing users to configure alerts, perform anomaly detection as well as advanced analytics.
How To Configure Logentries with Google Cloud Logging
Streaming Google Cloud Platform logs to Logentries can be configured as follows:
Enable the Cloud Pub/Sub API
Add the Logentries Service Account to your project
Configure Export to Cloud Pub/Sub
Add a Log in Logentries
Step by Step instructions
are available to get configured quickly.
Near Real-Time Log Analytics
Logentries uses a unique pre-processing engine to perform advanced analysis on Google logs in near real-time such that data is pre-analysed, thus reducing the requirement for complex search queries on your logs to identify important system or user activity.
Figure two: Logentries Log Management and Analysis Flow
Logentries’ integration with Google Cloud Platform enables you to pinpoint issues quickly as well as look at long term trends across your log data.
Some of the most useful capabilities of Logentries for Google Cloud Platform customers include:
Live Tail with Event Tagging
: The Logentries pre-processing engine automatically tags important events such as exceptions, warnings, or errors allowing users to easily spot issues in a live view of your log data.
Figure three: Live Tail of Logs in Logentries
Near Real-Time Notifications and Inactivity Alerts
: Get notified about important events within seconds of them occurring. Notifications can be configured to be sent to email, or can be integrated with other third party APIs and tools (e.g. Slack, HipChat, PagerDuty…).
Use your Logs as Data
: Logs contain lots of very useful information beyond stack traces and error codes.
Field level log analytics
allows you to extract key metrics (e.g. server resource usage, or API response time) from your logs and roll these metrics up into interesting charts and graphs.
Figure four: Live Charts in Logentries
Google Cloud Logging supports a long list of
known log formats
via the google-fluentd collector - e.g. Apache, Chef, MongoDB, NginX and several others. Logentries also provides out-of-the-box intelligence (tags, alerts and dashboards) for these log formats via the Logentries
community packs
such that you do not need to spend time configuring rules or queries.
Get started with Logentries, now paired with Google Cloud Logging service, today.
We are excited by this collaboration between Google Cloud Platform and Logentries, and we welcome your feedback. You can find more on the
Logentries forum
as well as send us feedback at
cloud-logging-feedback@google.com
.
- Posted by Deepak Tiwari (Product Manager, Google Cloud Platform) and Trevor Parsons (Co-founder and Chief Scientist, Logentries)
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
Understanding Cloud Pricing
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Enter the Andromeda zone - Google Cloud Platform’s latest networking stack
New in Google Cloud Storage: auto-delete, regional buckets and faster uploads
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow