Google Cloud Platform Blog: March 2015

Benchmarking Web search latencies

Tuesday, March 31, 2015

announcedOLDISIMMultiscale Architecture and Systems Team (MAST)

PerfKit Benchmarkershared it

OLDISIMPerfKit Benchmarke

Manage Google Cloud Platform on the go with Cloud Console for Android

Monday, March 30, 2015

Google App EngineCloud Platform resourcesGoogle Cloud MonitoringGet it Quickly view app status Cloud Platform service status

Get alerts and manage incidentsGoogle Compute EngineCloud Monitoring

View App Engine and Compute Engine properties and make quick changes

Get started nowthis quick guide

Feedback wanted

Introducing the Scalable and Resilient Web Apps Solution

Friday, March 27, 2015

Google App EngineScalable and Resilient Web ApplicationsGoogle Compute Engine Load Balancer easily handling 1,000,000 requests per secondwatched the live demoCloud Deployment ManagerRedmine project management web appGoogle Cloud SQLGoogle Cloud Storage

ChefGoogle Compute EnginearticleGitHub project page

Deploy popular software packages using Cloud Launcher

Thursday, March 26, 2015

high performance VMscontainer-based servicesPaaSGoogle Cloud LauncherGoogle Cloud PlatformGoogle Cloud Monitoringsupported packages

Google Compute Enginecloud launchermailing list

Personalized medicine with Northrop Grumman and Google Cloud Platform

Tuesday, March 24, 2015

Northrop GrummanGoogle Cloud PlatformGoogle GenomicsGoogle BigQuery The Architecture

Massive Data Storage: A storage layer leverages Google Genomics to efficiently store and access genomic data on the petabyte scale and Northrop Grumman knowledge engines and framework to efficiently process and store electronic health records (EHR) data.

Annotation Layer: The annotation layer provides tools to extract clinical knowledge from structured and unstructured EHR data sources. It also includes a database containing aggregated phenotypic and disease associations from public sources. These enable improved functional annotation of the genomic data.

Analytics Layer: The analytics layer is built on top of Google BigQuery and Google Compute Engine to provide high-performance modeling and analytics tools. With these, we can demonstrate genomic risk modeling with analysis time scales of only several seconds.

Visualization & Collaboration Layer: The visualization and collaboration layer provides a framework for high-level analytics, visualization, and collaboration tools.

The system architecture for Northrop Grumman’s personalized health analytics platform. A layered approach is designed to provide an integrated research environment with greater access to storage infrastructure, improved information extraction and annotation tools, more powerful computational platforms and improved collaboration and visualization tools.

New Breakthroughs in Personalized MedicinePHC@ngc.com

Announcing Google Cloud Dataflow runner for Apache Flink

Monday, March 23, 2015

Google Cloud DataflowSDKApache FlinkCloud Dataflow runner for Flinknew

Apache Flink, on-premises or in the cloud, as announced today

Apache Spark, on-premises or in the cloud, thanks to the Dataflow runner for Spark, contributed by Cloudera

Google Cloud Dataflow, a fully managed service (currently in Alpha, apply here to join)

blog postdata Artisans

“google-cloud-dataflow” tag on StackOverflow

Scaling MySQL in the cloud with Vitess and Kubernetes

Friday, March 20, 2015

Add more replicas for better read throughput and data durability

Introduce sharding to scale your write throughput and let your data set grow beyond a single machine

Create separate replica pools for batch jobs and backups, to isolate them from live traffic

Clone the whole deployment into multiple datacenters worldwide for disaster recovery and lower latency

journey300 hours of new video uploads per minuteVitessopen-source projectKubernetesGoogle Container Enginecontainer cluster managerany platform that Kubernetes supports

Horizontal scaling – add capacity by launching additional nodes rather than making one huge node

Dynamic placement – let the cluster manager schedule Vitess containers wherever it wants

Declarative specification – describe your desired end state, and let the cluster manager create it

Self-healing components – recover automatically from machine failures

run Vitess on KuberneteswebsiteforumGitHubGoogle Cloud SQL

Leverage the power of log data to drive operational and business decisions through Google Cloud Logging

Thursday, March 19, 2015

Google Cloud LoggingGoogle Compute EngineGoogle App EngineGoogle Cloud Monitoring

Ingest and view the log data, so that you can see all your logs in one place

Search the log data in real-time, so that you can resolve operational issues

Analyze the log data in real-time, so that you can glean actionable insights

Archive logs data for longer periods, to meet backup and compliance requirements

At Wix we use BigQuery to analyze logs of Compute Engine auto-scaled deployments. We get a large volume of syslog data that we send to BigQuery to get insights on system health state and error rates. We generate time series data and integrate it with Google Cloud Monitoring to monitor system performance and business metrics. This provides us with essential insight for the running of our operations. - Dmitry Shestak, Engineer@Infrastructure team, Wix Ingest and view the log data

Compute Engine VM logs can be automatically collected for about two dozen log types through the Google packaged fluentd agent, with additional logs possible through custom configuration.

Compute Engine Activity logs record all system actions and API calls are enabled by default, with no agent installation required.

App Engine logs that include syslog, request logs and application logs are automatically enabled for all App Engine projects, including applications using Managed VM runtimes.

Google Developers Console

When viewing logs in the Logs Viewer, you can filter results using filter text or drop-downs

Search the log data in real-time

A filtered view of logs data using the Logs Viewer

Analyze the log data in real-timeGoogle BigQueryExportsdetailed documentation

Log data in the BigQuery tables

Monitoring Code Performance: There are situations when something unexpected happened or is indicative of an imminent problem e.g. “disk space low.” With Compute Engine log data in BigQuery, you can generate a time series and monitor logs with a particular severity. It’s simple, you just query metadata.severity = “WARNING” in the relevant tables. E.g.

SELECT Count (*) AS total, Date(metadata.timestamp) AS time FROM   (Table_date_range(TABLE ID,  Timestamp('2015-03-01'), Timestamp('2015-03-12')))  WHERE  metadata.severity = "warning"  GROUP  BY time ORDER  BY total;

Monitoring Request Latency: High latency leads to poor user experience and failed requests, which can lead to frustrated users and lost revenue. With App Engine log data in BigQuery, you can create time series of latency data by aggregating and charting the “protoPayload.latency” field. You can see unusual latencies in real-time and take steps to resolve the issue.

Archive logs data for longer period Google Cloud StorageGoogle Cloud DataflowHadoop Google Cloud Storage Nearlinemore affordable Getting StartedCloud Logging documentation pagefeedback

Deploy Puppet on Google Compute Engine with a click

Wednesday, March 18, 2015

Open Source PuppetGoogle Compute EnginePuppet masternode_gcegce_computePuppet ForgeOpen Source Puppet on Google Compute Enginedeploy a Puppet master todayprofessional servicespremium supporttraining

Exploring genetic variation with Google Genomics and Tute

Monday, March 16, 2015

Today we hear from Tute Genomics on how they're using cloud-based technology and big data tools to support the scientific community and help advance genomics research. Tute has made its 8.5 billion record genetic annotation database publicly available to users of Google Genomics through Google BigQuery. Google BigQuery1000 Genomes Project

Google Cloud Platformdiscussion group

Fans come on stage in Azealia Banks’ new interactive video, built on Google Cloud Platform

Friday, March 13, 2015

Today’s guest blog comes from Brett Renfer, Director of Experience Design, at COLLINS, a New York City-based brand consultancy that uses design thinking and convergence of design, music, film, and technology to create unforgettable brand experiences.first-of-its-kind interactive music videoGoogle Cloud PlatformGoogle App EngineGoogle Cloud Storagefull story

Fuel your hungry applications with our new, larger virtual machines

Thursday, March 12, 2015

cloud pricing should track to Moore's Law

IndustriromantikThis is the best efficiency in scalability I've ever seen when talking about 3D renderingDevelopers Consolenew instanceinstructions on creating new virtual machineszonesfeedback channels

Introducing Google Cloud Storage Nearline: (near)online data at an offline price

Wednesday, March 11, 2015

– Google Cloud Storage Nearline

Fast Performance: all the benefits of cold storage while making the data immediately available. Unlike its competitors, Nearline enables ~3 second response times for data retrieval and improves SLAs.

Low-cost: capacity pricing is extremely low at 1c per GB for data at rest.

Security: redundant storage at multiple physical locations protects data. OAuth and granular access controls form strong, configurable security.

Integrated: fully integrated with other Google Cloud Storage services, providing a consistent method of access across the entire Google Cloud Storage service line.

Simple: no need to adopt new programming models – data manipulation behavior remains the same across Google Cloud Storage services.

––

Veritas/Symantec: NetBackup is the market leader in enterprise backup software and will support Google Cloud Storage Nearline in version 7.7. NetBackup integration with Google Cloud Storage Nearline will allow enterprises to increase business agility and information availability. Customers can seamlessly manage the lifecycle of their backups and keep a central catalog and recovery point for all their protected information, regardless of where it's stored.

NetApp: SteelStore is an on-premises appliance that de-duplicates, encrypts, and compresses data before rapidly streaming it to Google Cloud Storage Nearline. SteelStore reduces data volumes by up to 30x and speeds data transport times by 400%—making onramp of your data to the cloud fast and easy. SteelStore currently supports Google Cloud Storage and will support Nearline in the second half of 2015.

Iron Mountain: We’re working with Iron Mountain to design an on-ramp to the cloud via an offline ingestion service that builds on Iron Mountain's security, logistics and data management capabilities. If you have massive amounts of data and limited network connectivity, you’ll be able to simply package up and send your disks to Iron Mountain, where they’ll be uploaded directly into Google Cloud Storage Nearline.

Geminare: Get Disaster Recovery as a Service (DRaaS) solutions running on Google Compute Engine and Google Cloud Storage Nearline. Geminare lets you use the cloud as your secondary data center location, enabling both DR modernization and, for the first time, cost-effective replication of your current data centers.

Comparing online data storage solutions against Nearline and traditional cold storage offerings

Google Cloud Storage Nearlinesign up for our webinarNearlinedocumentation

#FutureCoders: Inspiring the next generation of female coders

Sunday, March 8, 2015

coding is the new second languageHour of Codetoys to groom kiddie codersprogramming positions whose numbers are growing at a rate twice the national averagePython for kidsturtle-graphics typewriterpicture-based guide

Two of David's three daughters, ages 6 and 12 here, working on examples from the Pencil Code guide

local teenGirls Who Code club

David helping his daughter, now 13, who is also in Girls Who Code, on her latest coding project

Khan AcademyCodecademyCheck back throughout MarchGrace HopperAda LovelaceFrances AllenAnita Borg#FutureCoders

Using Google Cloud Pub/Sub to Connect Applications and Data Streams

Wednesday, March 4, 2015

Google Cloud Pub/SubGoogle Cloud PlatformAPI

Scale: offering all customers, by default, up to 10,000 topics and 10,000 messages per second

Global deployment: dedicated resources in every Google Cloud Platform region enhance availability without increasing latency

Performance: sub-second notification even when tested at over 1 million messages per second

Google Cloud MonitoringDiscover

Integrated messaging between components of an application – for example, processing an office transfer in an HR system: developers need to control the distribution of updates to the company directory, to security badging, to the moving company, to payroll, and many other services.

Robust data collection from smart devices – for example, mobile device endpoints: providing developers with the ability to integrate sensor data from the endpoints with real-time data analysis pipelines, automatically routing the data streams to an application.

APIs & authGoogle Developers ConsoleGoogle Cloud Monitoringdirectlymailing list

Google Compute Engine uses Live Migration technology to service infrastructure without application downtime

Tuesday, March 3, 2015

IntroductionGoogle Compute Enginetransparent maintenance The Benefits of Transparent Maintenance

Regular infrastructure maintenance and upgrades

Network and power grid maintenance in the data centers

Bricked memory, disk drives, and machines

Host OS and BIOS upgrades

Security-related updates, with the need to respond quickly

System configuration changes, including changing the size of the host root partition, for storage of the host image and packages

unexpected

Flapping network cards — Network cards were intermittently failing. We were able to repeatedly try the VM migrations and successfully migrate them. This even worked with partially-failing NICs.

Cascading battery/power supply issues — Some overheating batteries were overheating the neighboring machines. We were able to migrate the VMs away before bringing down the machines to swap out their batteries.

A buggy update was pushed to production — We halted the rollout, but not before it reached some of our production machines (it didn't manifest in our canary environment). The buggy software would’ve crashed VMs within a week. Instead, we migrated the VMs on the affected machines to other hosts that didn’t have the buggy software.

Unexpected host memory consumption — One of our backend components consumed more memory than we had allocated and threatened to OOM (out of memory) the VMs. We migrated some VMs away from the over-loaded machines and avoided the OOM failures while patching the backend system to ensure it could not overrun its allocation.

Transparent Maintenance in Action “We took a look at our log files and all the data in the database and we saw…nothing unusual. In other words, if Google hadn’t told us that our instances had been migrated, we would have never known. All our logs and data looked normal, and we saw no changes in the RightScale Cloud Management dashboard to any of our resources, including the zone, instance sizes, and IP addresses.” “Just tested @googlecloud live migration of a @MongoDB replica set - no impact. None of the nodes noticed the primary was moved!” How it works

imminent

During pre-migration brownout, the VM is still executing on the source, while most state is sent from the source to the target. For instance, we copy all the guest memory to the target, while tracking the pages that have been re-dirtied on the source. The time spent in pre-migration brownout is a function of the size of the guest memory and the rate at which pages are being dirtied.

During blackout, which is a very brief moment when the VM is not running anywhere, it is paused, and all the remaining state required to begin running the VM on the target is sent. We go into blackout when sending state during pre-migration brownout reaches a point of diminishing returns. We use an algorithm that balances numbers of bytes of memory being sent against the rate at which the guest VM is dirtying pages, amongst other things.

During post-migration brownout, the VM executes on the target. The source VM is present, and may be providing supporting functionality for the target. For instance, until the network fabric has caught up with the new location of the VM, the source VM provides forwarding services for packets to and from the target VM.

logs Conclusionimmortality

Troubleshoot projects running on Google Compute Engine with Google Cloud Debugger

Tuesday, March 3, 2015

Google Cloud Platform Livereleased the beta of Google Cloud DebuggerGoogle Compute Enginetry it for yourself

64-bit Debian Linux image

Java JDK version 7 or 8

Project source code stored either in Cloud Repository or in a connected GitHub or BitBucket repository

direct feedbackStack Overflow