Google Cloud Platform Blog: Big Data & Machine Learning

Google Home meets .NET containers using Dialogflow

dijous, 12 de juliol del 2018

By Mete Atamel, Developer AdvocateChris BaconGitHub OverviewGoogle Cloud .NET librariesDialogflow

DialogflowDialogflow consoleintents

webhook ASP.NET Core on App Engine (Flex)Google Kubernetes EngineApp EngineHomeControllerConversationControllerConversationControllerDialogflowAppDialogflowAppConversationConversationBaseHandlerIntentAttribute Searching for imageVisionSearchHandlerGoogle Custom Search APIs

Vision APIVisionDescribeHandlerVision APIVision API .NET libraryVisionLandmarksHandlerVisionSafeHandler BigQueryBigQuerypublic datasetsHacker News DataNOAA Global Weather DataBigQueryNoaaextremeHandler

Logging and monitoringStackdriverProgramStackdriver Debugger

ConclusionGitHub

Cloud TPU now offers preemptible pricing and global availability

dimarts, 19 de juny del 2018

By Brennan Saeta, TensorFlow Tech Lead for Cloud TPUsTranslatePhotosSearchAssistantGmailCloud TPUsQuickstart

One Cloud TPU (v2-8) can deliver up to 180 teraflops and includes 64 GB of high-bandwidth memory. The colorful cables link multiple TPU devices together over a custom 2-D mesh network to form Cloud TPU Pods. These accelerators are programmed via TensorFlow and are widely available today on Google Cloud Platform.

Benchmarking Cloud TPU performance-per-dollarDAWNBenchImageNet Training CostAmoebaNetResNet-50ImageNet Training Timesix times fasterhereCloud TPU ResNet-50 implementationYou can now train ResNet-50 on ImageNet from scratch for just $7.50.Preemptible VMssaving and restoring from checkpoints

Select Open-Source Reference Models	Normal training cost (TF 1.8)	Preemptible training cost (TF 1.8)
ResNet-50 (with optimizations from fast.ai): Image classification	~$25	~$7.50
ResNet-50 (original implementation): Image classification	~$59	~$18
AmoebaNet: Image classification (model architecture evolved from scratch on TPUs to maximize accuracy)	~$49	~$15
RetinaNet: Object detection	~$40	~$12
Transformer: Neural machine translation	~$41	~$13
ASR Transformer: Speech recognition (transcribe speech to text)	~$86	~$27

Start using Cloud TPUs todayecently-improved preemptible pricingCloud TPU Quickstartopen source reference modelsfree trialEffective machine learning with Cloud TPUsTraining Performance: A user’s guide to converge faster.

A datacenter technician scoots past two rows of Cloud TPUs and supporting equipment.

GCP arrives in the Nordics with a new region in Finland

dilluns, 18 de juny del 2018

By Kirill Tropin, Product Manager, Google CloudhereGoogle Cloud Platform (GCP) region

www.gcping.com Services

BigQuerypoints of presenceService Specific Terms Build sustainably high-tech cooling system

Hear from our customers

“The road to emission-free and sustainable shipping is a long and challenging one, but thanks to exciting innovation and strong partnerships, Rolls-Royce is well-prepared for the journey. For us being able to train machine learning models to deliver autonomous vessels in the most effective manner is key to success. We see the Google Cloud for Finland launch as a great advantage to speed up our delivery of the project.”
– Karno Tenovuo, Senior Vice President Ship Intelligence, Rolls-Royce

“Being the world's largest producer of renewable diesel refined from waste and residues, as well as being a technologically advanced refiner of high-quality oil products, requires us to take advantage of leading-edge technological possibilities. We have worked together with Google Cloud to accelerate our journey into the digital future. We share the same vision to leave a healthier planet for our children. Running services on an efficient and sustainably operated cloud is important for us. And even better that it is now also available physically in Finland.”
– Tommi Touvila, Chief Information Officer, Neste

“We believe that technology can enhance and improve the lives of billions of people around the world. To do this, we have joined forces with visionary industry leaders such as Google Cloud to provide a platform for our future innovation and growth. We’re seeing tremendous growth in the market for our operations, and it’s essential to select the right platform. The Google Cloud Platform cloud region in Finland stands for innovation.”
– Anssi Rönnemaa, Chief Finance and Commercial Officer, HMD Global

“Digital services are key growth drivers for our renewal of a 108-year old healthcare company. 27% of our revenue is driven by digital channels, where modern technology is essential. We are moving to a container-based architecture running on GCP at Hamina. Google has a unique position to provide services within Finland. We also highly appreciate the security and environmental values of Google’s cloud operations.”
– Kalle Alppi, Chief Information Officer, Mehiläinen Partners in the Nordics

"Public cloud services like those provided by Google Cloud help businesses of all sizes be more agile in meeting the changing needs of the digital era—from deploying the latest innovations in machine learning to cost savings in their infrastructure. Google Cloud Platform's new Finland region enables this business optimization and acceleration with the help of cloud-native partners like Nordcloud and we believe Nordic companies will appreciate the opportunity to deploy the value to their best benefit.”
– Jan Kritz, Chief Executive Officer, Nordcloud AccentureAdaptyAppsPeopleAteaAvalon SolutionsBergeCap10Cloud2CloudpointComputasCrayonDataCenterFinlandDNADevoteamDobermanDeloitteEnfoEvryGappsGreenbirdHuman IT CloudIIH NordicKnowITKoivu SolutionsLamiaNetlightNordcloudOnline PartnerOutfox IntelligencePilviaPrecis DigitalPwCQuality of ServiceQvikSkyeSofthouseSolitaSymfoni NextSopra SteriaTietoUnifossVincitWizkidsWebstep Getting started Finland region pageCloud On-Airlocations pageContact us

Introducing improved pricing for Preemptible GPUs

dilluns, 11 de juny del 2018

By Chris Kleban, Product Manageryou could attach GPUsPreemptible VMsGoogle Compute EngineGoogle Kubernetes Engine

GPU Hourly Pricing *
GPU	Standard (Prices vary by location)	Previous Preemptible (All Locations)	New Preemptible (All Locations)
V100	$2.48	$1.24	$0.74
P100	$1.46	$0.73	$0.43
K80	$0.45	$0.22	$0.135

* GPU prices listed as hourly rate, per GPU attached to a VM that are billed by the second. Prices listed are for US regions. Prices for other regions may be different. Additional Sustained Use Discounts of up to 30% apply to GPU non-preemptible usage only.

custom machine typesThe use-case for Preemptible GPUs “Preemptible GPUs have been instrumental in enabling our research group to process large video collections at scale using our Scanner open-source platform. The predictable low cost makes it feasible for a single grad student to repeatedly deploy hundreds of GPUs in ML-based analyses of 100,000 hours of TV news video. This price drop enables us to perform twice the amount of processing with the same budget."
- Kayvon Fatahalian, Assistant Professor, Stanford University
Machine Learning Training and Preemptible GPUs
Training ML workloads is a great fit for Preemptible VMs with GPUs. Kubernetes Engine and Compute Engine’s managed instance groups allow you to create dynamically scalable clusters of Preemptible VMs with GPUs for your large compute jobs. To help deal with Preemptible VM terminations, Tensorflow’s checkpointing feature can be used to save and restore work progress. An example and walk-through is provided here.

Getting Started
To get started with Preemptible GPUs in Google Compute Engine, simply append --preemptible to your instance create command in gcloud, specify scheduling.preemptible to true in the REST API or set Preemptibility to "On" in the Google Cloud Platform Console, and then attach a GPU as usual. You can use your regular GPU quota to launch Preemptible GPUs or, alternatively, you can request a special Preemptible GPUs quota that only applies to GPUs attached to Preemptible VMs. Check out our documentation to learn more. To learn how to use Preemptible GPUs with Google Kubernetes Engine, head over to our Kubernetes Engine GPU documentation.

For a certain class of workloads, Google Cloud GPUs provide exceptional compute performance. Now, with new low Preemptible GPU pricing, we invite you to see for yourself how easy it is to get the performance you need, at the low, predictable price that you want.

Dialogflow adds versioning and other new features to help enterprises build vibrant conversational experiences

dimarts, 22 de maig del 2018

By Dan Aharon, Product ManagerDialogflowVersions and Environments ^BETA

Creating versions and environments

Publishing a version

Managing versions within an environment

Action Console'sthis tutorialImproved conversation history tool for easier debugginghistory toolGoogle Stackdriver Loggingdiagnostic information

Training with negative examples for improved precisionnegative examples

Try Dialogflow today using a free creditquickstart$300 free credit

Scale big while staying small with serverless on GCP — the Guesswork.co story

dimarts, 1 de maig del 2018

By Mani Doraisamy, founder at Guesswork.co and Google Developer Expert[Editor’s note: Mani Doraisamy built two products—Guesswork.co and CommerceDNA—on top of Google Cloud Platform. In this blog post he shares insights into how his application architecture evolved to support the changing needs of his growing customer base while still staying cost-effective.]ZaloraGaleries Lafayette GroupDaraz

Stay small. As a small team of three developers, we didn’t want to add any additional personnel even if we needed to scale up for a huge volume of users.

Stay profitable. Our revenue is based on the performance of our recommendation engine. Instead of a recurring fee, customers pay us a commission on sales to their users that come from our recommendations. This business model made our application architecture and infrastructure costs a key factor in our ability to turn a profit.

Embrace constraints. In order to increase our development velocity and stay flexible, we decided to trade off control over our development stack and embrace constraints imposed by managed cloud services.

Choosing a database: The Three MusketeersGoogle Cloud Platform

Cloud Datastore for our transactional database, because it can support high number of writes. In our case, the user events are in the billions and are updated in real time into Cloud Datastore.

BigQuery to analyze user behaviour. For example, we understand from BigQuery that users coming from a fashion blog usually buy a specific type of formal shoes.

Vision API to analyze product images and categorize products. Since we work with e-commerce companies across different geographies, the product names and descriptions are in different languages, and categorizing products based on images is more efficient than text analysis. We use this data along with user behaviour data from BigQuery and Cloud Datastore to make product recommendations.

First take: the App Engine approachApp EngineCloud Dataflow

We compare the price we receive in the user event with the product master and determine if there is a difference.

If there is a difference, we propagate it to the analytical and machine learning databases to trigger an alert and reflect that change in the product recommendation.

Take two: the Cloud Functions approach

Use memcache to load the product master in memory and compare the price/stock for every user event. With this option, we had no guarantee that memcache would be able to hold so many products in memory. So, we might miss a price change and end up with inaccurate product prices.

Use Cloud Firestore to record user events and product data. Firestore has an option to trigger Cloud Functions whenever there’s a change in value of an entity. In our case, the price/stock change automatically triggers a cloud function that updates the analytical and machine learning databases.

With Firestore, we replaced both App Engine and Datastore. Firestore was able to accept user requests directly from a browser without the need for a front-end service like App Engine. It also scaled well like Datastore.

We used Cloud Functions not only as a way to trigger price/stock alerts, but as an orchestration tool to synchronize data between Firestore, BigQuery and our recommendation engine.

What we gained

We increased our sales commissions by improving recommendation accuracy

The best thing that happened in this new version was the ability to A/B test new algorithms. For example, we found that users who browse e-commerce sites with an Android phone are more likely to buy products that are on sale. So, we included user’s device as a feature in the recommendation algorithm and tested it with a small sample set. Since, Cloud Functions are loosely coupled (with Cloud Pub/Sub), we could implement a new algorithm and redirect users based on their device and geography. Once the algorithm produced good results, we rolled it out to all users without taking down the system. With this approach, we were able to continuously improve the accuracy of our recommendations, increasing revenue.

We reduced costs by optimizing our algorithm

As counter intuitive it may sound, we also found that paying more money for compute didn't improve accuracy. For example, we analyzed a month of a user’s events vs. the latest session’s events to predict what the user was likely to buy next. We found that the latest session was more accurate even though it had less data points. The simpler and more intuitive the algorithm, the better it performed. Since Cloud Functions are modular by design, we were able to refactor each module and reduce costs without losing accuracy.

We reduced our dependence on external IT teams and signed more customers

We work with large companies and depending on their IT team, it can take a long time to integrate our solution. Cloud Functions allowed us to implement configurable modules for each of our customers. For example, while working with French e-commerce companies, we had to translate the product details we receive in the user events into English. Since Cloud Functions supports Node.js, we enabled scriptable modules in JavaScript for each customer that allowed us to implement translation on our end, instead of waiting for the customer’s IT team. This reduced our go-live time from months to days, and we were able to sign up new customers who otherwise might not have been able to invest the necessary time and effort up-front.

headless ChromeCompute Engine Let a thousand flowers bloom

Registration for the Associate Cloud Engineer beta exam is now open

divendres, 27 d’abril del 2018

By Katie Richardson, Google Cloud Certified Program ManagerGoogle Cloud ConsoleAssociate Cloud Engineerhire or fill existing skills gapsCloud ArchitectData EngineerGoogle Cloud PlatformGoogle Cloud Certified

The beta exam is now open for registration. The testing period runs May 9-30, 2018

To earn this certification, you must successfully pass our Associate Cloud Engineer exam

Save 40% on the cost of certification by participating in this beta

The length of the exam is four hours

BigQuery arrives in the Tokyo region

dimarts, 17 d’abril del 2018

By Chad W. Jennings, BigQuery Product ManagerBigQuery

“We believe BigQuery will become a significant driver to transform the way our data analysts work. Mizuho Bank currently runs SQL queries in on-premise data warehouses, but we used to experience issues of long processing time due to a fixed limit on our data-processing resources. We have wanted to move the data to the cloud for some time. Now that BigQuery is available at Google Cloud’s Tokyo region, we were able to conduct a proof of concept (PoC) under the conditions that satisfy Mizuho’s security requirements. We used real-world data to design the PoC to reflect our day-to-day operations. With BigQuery, we no longer need to worry about limited data-processing resources. We can aggregate processing tasks, perform parallel processing on large queries, and engage multiprocessing in order to substantially reduce working hours of data analysts collaborating across multiple departments. This streamlining can potentially generate more time for our data analysts to work on designing queries and models, and then interpreting the results. BigQuery can not only achieve cost-savings over the existing system but also provides integrated services that are very easy to use. The PoC has been a great success as we were able to confirm that by using data processing and visualization features like Dataprep and Data Studio with BigQuery, we can conduct all these data analysis processes seamlessly in the cloud.”

—Yoshikazu Kurosu, Senior Manager of Business Development Department, Mizuho Bank, Ltd. robust security Google Cloud offers “We’ve been using BigQuery on our Enterprise Cloud, a service we provide for enterprises, to detect and analyze anomalies in our network and servers. In our proof of concept (PoC) using BigQuery in Google Cloud’s Tokyo region, we performed evaluations of large-scale log data (cumulative total of 27.8 billion records) streamed real-time from nine regions around the world. Data analytics infrastructure requires real-time handling of extensive log data generated by both equipment and software. Our infrastructure also depends on speed and power to perform ultra-high speed analysis and output of time-series data.

BigQuery achieves high-speed response even in emergency situations, offers an excellent cost-performance ratio, and enables usage and application of large-scale log data that exceeds the capabilities of traditional data warehouses. We will continue to strengthen our cooperation with GCP services, BigQuery included, to provide cloud solutions to support secure data usage on behalf of our customers.” —Masaaki Moribayashi, Senior Vice President, Cloud Services, NTT Communications Corporation visit our websiteread our documentation

Dialogflow Enterprise Edition is now generally available

dimarts, 17 d’abril del 2018

By Dan Aharon, Product Manager, Cloud AI announcedDialogflow Enterprise EditionCloud Text-to-SpeechCloud Speech-to-TextsupportDialogflowplatformsdevices

A total of 30 supported languages and locales, including Hindi, Thai and Danish

Batch operations that allow you to combine operations that would have required multiple API calls into a single API call, reducing code surface and shortening development time

Actions on Google client library to help you integrate the Google Assistant features into your Dialogflow agent

Session Flow visualization in the Analytics dashboard to help you improve the performance and usage of your agent

Fulfillment Library for Node.js to help simplify your fulfillment code

KLM Royal Dutch Airlines, Domino’s and TicketmasterDPD Innovative businesses across industries are adopting Dialogflow Enterprise EditionUbisoft

(click to enlarge)

Best Buy Canada

(click to enlarge)

Ticketmasterbuild a conversational experience for event discovery and ticketing

Try Dialogflow today using a free creditquickstart$300 free credit