"Batterii, a consumer insights and collaboration platform, has been using Cloud Datastore for seven years. Previously, our backups took 8-10 hours. With managed export and import, we can now complete those backups in an hour."
—Greg Fairbanks, Senior Software Engineer
gcloud beta compute ssl-policies create web-front-end-app-policy \ --profile MODERN --min-tls-version 1.1
gcloud beta compute target-https-proxies update my_https_lb \ --ssl-policy my_ssl_policy
gcloud
$ gcloud spanner instances create my-instance \ --config=nam3 \ --description=my-instance \ --nodes=3 $ gcloud spanner databases create my-database --instance=my-instance $ gcloud spanner databases ddl update my-database --instance=my-instance --ddl="$(cat <<EOF CREATE TABLE Vault ( Key STRING(MAX) NOT NULL, Value BYTES(MAX), ) PRIMARY KEY (Key); EOF )"
# config.hcl storage "spanner" { database = "projects/my-default-project/instances/my-instance/databases/my-database" }
$ export VAULT_ADDR=http://127.0.0.1:8200 $ sudo vault server -dev -config=config.hcl
$ vault write secret/my-secret foo=bar
When you do not have consistent listings, there is a possibility of missing files. You cannot rely on the consistency of the data being read as you develop your products. Even worse, inconsistent listings lead to unforeseen issues. For example, our processing tooling will succeed reading partial data and may potentially produce seemingly valid outputs. Problems like these have a tendency to quickly propagate throughout the dependency tree.
When that happens, in the best-case we notice the failure and recompute all datasets produced within the dependency tree. In the worst-case, the failure goes unnoticed and we create invalid reports and statistics. Considering the large amount of data pipelines we run, even with a low probability of that happening, a lack of list-consistency in cloud storage offerings was a major blocker for data-processing at Spotify.
We considered multiple workarounds, such as using a global consistency cache based on NFS, porting Netflix’s s3mper as well as persisting listings in a manifest file stored alongside the data. All of the considered solutions were suboptimal as they either introduced a single point of failure or required us to put significant resources into developing our own solution and adjusting our tooling. Strong list consistency in Cloud Storage means we can continue using our existing data-processing stack without modifications and without worrying that data may be corrupted. List consistency on Cloud Storage is an essential feature for data processing at Spotify. We use a Hadoop-based data processing stack built on top of the Hadoop Distributed File System, which means we rely on its filesystem-like guarantees. Consistency is critical to running our business, and its absence creates many challenges.
--create-disk flag
$gcloud container clusters create camilia-cluster --num-nodes=3
$kops create cluster --zones us-east-1a k8saws.usualwebsite.com
$kops update cluster k8saws.usualwebsite.com --yes
$kubectl run camilia-nginx --image=nginx --port 80
$kubectl expose deployment camilia-nginx --target-port=80 --type=NodePort
$kubectl scale deployment camilia-nginx --replicas=3
"As part of our IoT integration strategy, Google Cloud IoT Core has helped us focus our engineering efforts on building oil and gas applications by leveraging existing IoT services to enable fast, reliable and economical deployment. We have been able to build quick prototypes by connecting a large number of devices over MQTT and perform real-time monitoring using Cloud Dataflow and BigQuery."
— Chetan Desai, VP Digital Technology, Schlumberger Limited
"Using Google Cloud IoT Core, we have been able to completely redefine how we manage the deployment, activation and administration of sensors and devices. Previously, we needed to individually set up each sensor/device. Now we allocate manufactured batches of devices into IoT Core for site deployments and then, using a simple activation smartphone app, the onsite installation technician can activate the sensor or device in moments. Job done!"
— John Heard, Group CTO, Smart Parking Limited
"Blaze is able to rapidly build the technology platform our customers and cyclists require on Google Cloud by more securely connecting our products and fleets of bikes to Cloud IoT Core and then run demand forecasting using BigQuery and Machine Learning."
— Philip Ellis, Co-Founder & COO, Blaze
"Agosto, a Google Cloud Premier partner, performed business and technical reviews of MOBILITY ADO’s existing architecture, applications and core data workflows which had been in place for about 12 years. These systems were originally very robust, but over time, we faced challenges with innovating on the existing technology stack, as well as with the optimization of operational costs. Agosto created a proof-of-concept which showcased that a Cloud IoT Core-based architecture was a viable path to modernization and functional optimization of many of our existing, core components. MOBILITY ADO now has real time access to bus diagnostic data via Google Cloud data and analytics services and a clear path to future-proof our platform."
— Humberto Campos, IT Director, MOBILITY ADO
"When preparing petabytes of global satellite imagery to be calibrated, cleaned up, and "science-ready" for our machine learning models, we do a tremendous amount of image compression. By leveraging the additional compute resources available with 96 vCPU machine types, as well as Advanced Vector Extensions such as AVX-512 with Skylake, we have seen a 38% performance improvement in our compression and a 23% improvement in our imagery expansions. This really adds up when working with petabytes of satellite and aerial imagery."
- Tim Kelton, Co-Founder, Descartes Labs
"We made a decision to focus our deep learning research on the cloud for many reasons, but mostly to gain access to the latest machine learning infrastructure. Google Cloud TPUs are an example of innovative, rapidly evolving technology to support deep learning, and we found that moving TensorFlow workloads to TPUs has boosted our productivity by greatly reducing both the complexity of programming new models and the time required to train them. Using Cloud TPUs instead of clusters of other accelerators has allowed us to focus on building our models without being distracted by the need to manage the complexity of cluster communication patterns."
— Alfred Spector, Chief Technology Officer, Two Sigma
“Since working with Google Cloud TPUs, we’ve been extremely impressed with their speed—what could normally take days can now take hours. Deep learning is fast becoming the backbone of the software running self-driving cars. The results get better with more data, and there are major breakthroughs coming in algorithms every week. In this world, Cloud TPUs help us move quickly by incorporating the latest navigation-related data from our fleet of vehicles and the latest algorithmic advances from the research community.
— Anantha Kancherla, Head of Software, Self-Driving Level 5, Lyft
"GKE clusters are ideal for deep learning workloads, with out-of-the box GPU integration, autoscaling clusters for our spiky training workloads, and integrated container logging and monitoring."
— Luc Vincent, VP of Engineering at Lyft
gcloud beta container node-pools create my-gpu-node-pool --accelerator=type=nvidia-tesla-p100,count=1 --cluster=my-existing-cluster --num-nodes 2 --min-nodes 0 --max-nodes 5 --enable-autoscaling
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.