Google Cloud Platform Blog
Product updates, customer stories, and tips and tricks on Google Cloud Platform
Financial services firm processes 25 billion stock market events per hour with Google Cloud Bigtable
Thursday, March 31, 2016
FIS
, a global financial technology and services firm and frequent leader of the
FinTech Top 100 list
, recently ran a load test of their system on
Google Cloud Platform
to process, validate and link U.S. stock exchange market events. FIS used
Google Cloud Dataflow
and
Google Cloud Bigtable
to process 25 billion simulated market events in 50 minutes, generating some impressive statistics in the process.
Cloud Bigtable achieved read rates in excess of 34 million events per second and 22 million event writes per second using 3500 Cloud Bigtable server nodes and 300 n1-standard-32 VMs with Cloud Dataflow. Additionally, Cloud Bigtable provided sustained rates of over 22 million event reads per second and 16 million event writes per second for extended periods of time.
Moreover, Cloud Bigtable was also able to achieve significant I/O bandwidth rates during the load test: read bandwidth peaked at 34 GB/s while write bandwidth peaked at 18 GB/s. Cloud Bigtable sustained significant bandwidth for input and output for 30 minutes as well: 22 GB/s for reads and 13 GB/s for writes.
For FIS, these performance capabilities make it possible to process an entire day’s worth of U.S. equities and options data and make it available for analysis within four hours.
For the complete set of benchmark results, see
these slides
. You can see a more detailed description of the overall system architecture presented by Neil Palmer and Todd Ricker from
FIS
and Carter Page, engineering manager for
Google Cloud Bigtable
:
We look forward to working with other innovative companies like FIS to help them address data processing challenges with the performance, scalability and NoOps approach that Cloud Bigtable provides.
-
Posted by Misha Brukman, Product Manager for Google Cloud Bigtable
Google Cloud Datastore simplifies pricing, cuts cost dramatically for most use-cases
Thursday, March 31, 2016
Google Cloud Datastore
is a highly-scalable NoSQL database for web and mobile applications. Today we’re announcing much simpler pricing, and as a result, many users will see significant cost-savings for this database service.
Along with the simpler pricing model, there’ll be a more transparent method of calculating stored data in Cloud Datastore. The new pricing and storage calculations will go into effect on July 1st, 2016. For the majority of our customers, this will effectively result in a price reduction.
New pricing structure
We’ve listened to your feedback and will be simplifying
our pricing
. The new pricing will go into effect on July 1st, 2016, regardless of how you access Datastore. Not only is it simpler, but also the majority of our customers will see significant cost savings. This change removes the disincentive our current pricing imposes on using the powerful indexing features, freeing developers from over-optimizing index usage.
We’re simplifying pricing for entity writes, reads and deletes by moving from internal operation counting to a more direct entity counting model as follows:
Writes
: In the current pricing, writing a single entity translated into one or more write operations depending on the number and type of indexes. In the new pricing, writing a single entity only costs 1 write regardless of indexes and will now cost $0.18 per 100,000. This means writes are more affordable for people using multiple indexes. You can use as many indexes as your application needs without increases in write costs. Since on average the vast majority of Entity writes previously translated to more than 4 write operations per entity, this represents significant cost savings for developers.
Reads
: In the current pricing, some queries would charge a read operation per entity retrieved plus an extra read operation for the query. In the new pricing, you'll only be charged per entity retrieved. Small ops (projections and keys-only queries) will stay the same in only charging a single read for the entire query. The cost per Entity read stays the same as the old per operation cost of $0.06 per 100,000. This means that most developers will see reduced costs in reading entities.
Deletes
: In the current pricing model, deletes translated into 2 or more writes depending on the number and type of indexes. In the new pricing, you'll only be charged a delete operation per entity deleted. Deletes are charged at the rate of $0.02 per 100,000. This means deletes are now discounted by at least 66% and often by more.
Free Quota
: The free quota limit for Writes is now 20,000 requests per day since we no longer charge multiple write operations per entity written. Deletes now fall under their own free tier of 20,000 requests per day. Over all, this means more free requests per day for the majority of applications.
Network
:
Standard Network costs
will apply.
New storage usage calculations
To coincide with our pricing changes on July 1st, Cloud Datastore will also use a new method for calculating bytes stored. This method will be transparent to developers so you can accurately calculate storage costs directly from the property values and indexes of the Entity. This new method will also result in decreased storage costs for the majority of customers.
Our current method relies heavily on internal implementation details that can change, so we’re moving to a fixed system calculated directly from the user data submitted. As the new calculation method gets finalized, we’ll post the specific details so developers can use it to estimate storage costs.
Building what’s next
With simpler pricing for Cloud Datastore, you can spend less time micro-managing indexes and focus more on building what’s next.
Learn more about
Google Cloud Datastore
or check out our
getting started guide
.
-
Posted by Dan McGrath, Product Manager, Google Cloud Platform
Introducing online resizing of Google Cloud Persistent Disks without downtime
Thursday, March 31, 2016
Google Compute Engine provides
Persistent Disks
to use as the primary block storage for your virtual machine instances. Provisioning the appropriate size of block storage has been a challenge for many cloud and on-premise customers because it requires planning for future data growth and performance needs. When a virtual machine runs out of space, there was no easy way to scale the size of your block storage.
Today we're announcing general availability of
online resizing for Persistent Disks
. It’s as easy as a button click or a single API call. It doesn’t cause any downtime to
Google Compute Engine
instances and doesn’t require snapshotting. It applies to all Persistent Disks, including the recently announced
64 TB volumes
.
With the introduction of this feature, Persistent Disk capacity planning becomes much simpler. Persistent Disks can be provisioned based on immediate needs and increased in size later when you require more space or performance
1
. Instead of implementing a complex workflow that would take the system offline
—
such as snapshot the disk, restore the snapshot to a larger device then bring back online again
—
there's a single command that makes physical devices larger. The device immediately has higher IOPS and throughput limits. After you resize a disk that's already mounted on a VM instance, resize the file system. Usually it's as simple as running resize2fs on Linux or resizing partitions in Windows Disk Manager.
Internally we've been using online disk resizing with
Cloud SQL Second Generation
. It has enabled automatic growth of Persistent Disks used by
Google Cloud SQL
with no downtime.
We hope you enjoy the new feature!
-
Posted by Igor Belianski, Software Engineer, Google Compute Engine
1
Persistent Disk performance depends on the size of the volume and the type of disk you select. Larger volumes can achieve higher I/O levels than smaller volumes.
IAM best practice guides available now
Tuesday, March 29, 2016
Google Cloud Identity & Access Managemen
t (IAM) service gives you additional capabilities to secure access to your
Google Cloud Platform
resources. To assist you when designing your IAM strategy, we've created a set of best practice guides.
The best practices guides include:
Using IAM Securely
Designing Resource Hierarchies
Understanding Service Accounts
The “
Using IAM Securely
” guide will help you to implement IAM controls securely by providing a checklist of best practices for the most common areas of concern when using IAM. It categorizes best practices into four sections:
Least privilege - A set of checks that assist you in restricting your users or applications to not do more than they're supposed to.
Managing Service Accounts and Service Account keys - Provides pointers to help you manage both securely.
Auditing - This covers practices that include reminding you to use
Audit logs
and
cloud logging roles
Policy Management - Some checks to ensure that you're implementing and managing your policies appropriately.
Cloud Platform resources are organized hierarchically and IAM policies can propagate down the structure. You're able to set IAM policies at the following levels of the resource hierarchy:
Organization level
. The Organization resource represents your company. IAM roles granted at this level are inherited by all resources under the organization.
Project level
. Projects represent a trust boundary within your company. Services within the same project have a default level of trust. For example, App Engine instances can access Cloud storage buckets within the same project. IAM roles granted at the project level are inherited by resources within that project.
Resource level
. In addition to the existing
Google Cloud Storage
and
Google BigQuery
ACL systems, additional resources such as
Google Genomics
Datasets and
Google Cloud Pub/Sub
topics support resource-level roles so that you can grant certain users permission to a single resource.
The diagram below illustrates an example of a Cloud Platform resource hierarchy:
The “
Designing Resource Hierarchies
” guide provides examples of what this means in practice and has a handy checklist to double-check that you're following best practice.
A Service Account is a special type of Google account that belongs to your application or a virtual machine (VM), instead of to an individual end user. The “
Understanding Service Accounts
” guide provides answers to the most common questions, like:
What resources can the service account access?
What permissions does it need?
Where will the code assuming the identity of the service account be running: on Google Cloud Platform or on-premises?
This guide discusses what the implications are of making certain decisions so that you have enough information to use Service Accounts safely and efficiently.
We’ll be producing more IAM best practice guides and are keen to hear from customers using IAM or wanting to use IAM on what additional content would be helpful. We’re also keen to hear if there are curated roles we haven’t thought of. We want Cloud Platform to be the most secure and the easiest cloud to use so your feedback is important to us and helps us shape our approach. Please share your feedback with us at:
GCP-iam-feedback@google.com
-
Posted by Grace Mollison, Solutions Architect
Snapchat shares security best practices for running on GCP
Friday, March 25, 2016
Snapchat security engineer, Subhash Sankuratripati, took the stage at
GCP NEXT
in San Francisco this week, to share his company’s best practices for running securely at scale on
Google Cloud Platform
. And when we say at scale, we mean at scale!
Snapchat has over 100 million daily users and supports 8 billion videos, viewed daily. The company runs about 100 separate GCP
projects
, each requiring different permissions for who at the company can do what on which GCP resources.
Until recently, Snapchat engineers exclusively used viewer/editor roles and built their own stopgaps to manage resources on the platform. With the launch of
IAM Roles
in beta, Snapchat now uses this service to set fine grained permissions it needs to help secure its users’ data. Essentially, Snapchat operates on the principle of least privilege.
The company is working on using our new iam.setpolicy feature to create what it calls Access Control List leases or “ACL leases.” These leases temporarily grant access to resources only when someone needs them, then the policy tears them down when the lease is over, for example:
AccessControlService can iam.SetPolicy
When bob@ needs access, AccessControlService adds bob@ to policy
AccessControlService removes bob@ after 1 hour
Like the nature of Snapchat itself, the company wants to treat access to its cloud resources as ephemeral for maximum security. Snapchat has implemented this leasing model for certain resources and privileges and is striving towards a goal where developers are granted access to the resources when they need it and for however long they need it.
Snapchat’s using the new
Organizational Node
, which sits above projects and manages GCP resources. This prevents shadow projects from being created, giving the company more control over all projects and the permissions of members associated to those projects. Sankuratripati said he’s also doing data siloing based on role using IAM Roles and is testing the
IAM Service Account API
, which can be used by programs to authenticate to Google and make API calls.
The possibilities this opens up are endless, according to Sankuratripati. He said microservice to microservice authentication would mean an even larger reduction in what his engineers can manage directly, locking down access to resources even further. Snapchat's strategy is essentially to ensure its developers have enough freedom to get their job done, but not enough to get themselves into trouble.
Stay tuned for more resources coming soon on using IAM on Cloud Platform and as you check out these services, please share your feedback with us at
GCP-iam-feedback@google.com
.
-
Posted by Jo Maitland, Managing Editor, Google Cloud Platform
Google shares data center security and design best practices
Thursday, March 24, 2016
GCP NEXT 2016
— San Francisco — Attendees at
Google Cloud Platform
’s user conference this week got a chance to hear from two of the company’s leaders — Joe Kava, VP of data center operations and Niels Provos, distinguished engineer for security and privacy
—
on how the company designs, builds, operates and secures its data centers globally. They shared some of the secret sauce that makes Google's data centers so unique and what this means for GCP customers running inside them.
Security and data protection
Google’s focus on security and protection of data is a key design criteria. Our physical security features a layered security model, including safeguards like custom-designed electronic access cards, alarms, vehicle access barriers, perimeter fencing, metal detectors and biometrics. The data center floor features laser beam intrusion detection. Our data centers are monitored 24/7 by high-resolution interior and exterior cameras that can detect and track intruders. Access logs, activity records and camera footage are available in case an incident occurs.
Data centers are also routinely patrolled by experienced security guards who have undergone rigorous background checks and training (look closely and you can see a couple of them in this
360 degree data center tour
). As you get closer to the data center floor, security measures increase. Access to the data center floor is only possible via a security corridor which implements multi-factor access control using security badges and biometrics. Only approved employees with specific roles may enter. Less than one percent of Google employees will ever set foot in one of our data centers.
We employ a very strict end-to-end chain of custody for storage, tracking everything from cradle to grave, from the first time a HD goes into a machine until it’s verified clean/erased or destroyed. Information security and physical security go hand-in-hand. Data is most vulnerable to unauthorized access as it travels across the Internet or within networks. For this reason, securing data in transit is a high priority for Google. Data traveling between a customer’s device and Google is encrypted using HTTPS/TLS (Transport Layer Security). Google was the first major cloud provider to enable HTTPS/TLS by default.
We build our own hardware and monitoring systems
Google servers don’t include unnecessary components such as video cards, chipsets or peripheral connectors, which can introduce vulnerabilities. Our production servers run a custom-designed operating system (OS) based on a stripped-down and hardened version of Linux. Google’s servers and their OS are designed for the sole purpose of providing Google services. Server resources are dynamically allocated, allowing for flexibility in growth and the ability to adapt quickly and efficiently, adding or reallocating resources based on customer demand.
For these teams to be successful they must have advanced, real-time visibility into the status and functionality of our infrastructure. As you might know, Google is obsessed with data, which is a bit of an understatement. To aid our teams, we've built monitoring and controls systems for all functional areas, from the servers, storage and networking systems, to the electrical distribution, mechanical cooling systems and security systems. We're monitoring all aspects of performance and operations from “chip to chiller.”
Using machine learning to optimize data center operations
To help in this endeavor, we’re
using our machine learning
/ deep learning algorithms for data center operations. As you can imagine, our data centers are large and complex, with electrical, mechanical and controls systems all working together to deliver optimal performance. Because of the sheer number of interactions and possible settings for these systems, it's impossible for mere mortals to visualize how best to optimize the data center in real time. However, it's fairly trivial for computers to crunch through these possible scenarios and find the optimal settings.
Over the past couple years we've developed this algorithm and trained it with billions of data points from our sites all over the world. We now use this machine learning model to help visualize the data so the operations teams can set up the data center electrical and cooling plants for the optimal, most efficient performance on any given day considering up to 19 independent variables that affect performance. This helps the team identify discontinuities or efficiency inflection points that aren't intuitive.
Powered by renewable energy
On the energy side, we're committed to powering our infrastructure with renewable energy. We're the world's largest private investor in renewable energy. To date we've invested more than $2 billion in renewable energy Power Purchase Agreements. These PPA's are very important because (1) we're buying the entire output of wind and solar farms for long periods, typically 10-20 years, (2) these wind farms are on the same power grids as our data centers, and (3) wind farms and data centers sharing power grids gives the project developer the financial commitment they need to get the project built, so we know our investment is adding renewable power to the grid that wouldn’t otherwise have been added.
For cooling, we've redesigned our fundamental cooling technology on average about every 12-18 months. Along the way, we've developed and pioneered innovations in water-based cooling systems such as seawater cooling, industrial canal water cooling, recycled / grey water cooling, stormwater capture and reuse, rainwater harvesting and thermal energy storage. We've designed data centers that don't use water-based solutions, instead using 100% outside air cooling. The point is there's no "one size fits all" model here. Each data center is designed for the highest performance and highest efficiency for that specific location.
Google employees operate our data centers, not third parties
The norm in the industry is for the design and building contractor to drop off a set of owners manuals and drawings along with the keys to the front door and wish the operator of the data center good luck! All too often these operations teams aren't employed by the owner, but rather an outsourced low-bidder. This is not the case at Google. Our employees manage and operate our data centers. If there's one certainty in data center operations, it's that problems and faults will always happen in the middle of the night — typically on Sundays — when nobody else is around to help :-)
Engineering + operation teams are combined
We also take a different approach to the people we hire and how they run our data centers. Our engineers and operations professionals come from very diverse backgrounds but all have a common trait — they're systems thinkers. Many of our team members come from mission critical environments, like the Navy nuclear submarine program, where mistakes can be catastrophic — they understand how systems interact together. Further, we've built regional site teams at all our data center campuses comprised of the engineers responsible for the design and construction, working side-by-side with the operations teams. Together these integrated teams are responsible for building capacity, commissioning the systems and providing 7x24 operations. This gives us an unparalleled level of ownership of our infrastructure.
-
Posted by Joe Kava, VP, Data Center Operations, Google
Google Data Center 360° Tour
Wednesday, March 23, 2016
We’re excited to share the Google Data Center 360° Tour — a YouTube 360° video that gives you an unprecedented and immersive look inside one of our data centers.
There are several ways to view this video:
On desktop using Google Chrome
—
use your mouse or trackpad to change your view while the video plays
YouTube app on mobile
—
move your device around to look at all angles while the video plays
And the most immersive way to view
—
using Google Cardboard (currently supported by the Android YouTube app only, iOS support is coming soon!) Load the video in the YouTube app and tap on the Cardboard icon when the video starts to play. Insert your phone in Cardboard and look around.
A little background . . .
Several months ago, those of us on the
Google Cloud Developer Advocacy Team
had a rare opportunity to tour the Google data center in The Dalles, Oregon. Many of us had seen other non-Google data centers in our careers, but this experience was beyond anything we ever imagined. We were blown away by the scale, the incredible attention to security and privacy, and the amazing efforts to make the data center extremely efficient and green. Additionally, we were proud to meet some of the brilliant people that design, build and maintain these data centers.
If you are a
Google Cloud Platform
customer, then this is your data center as much as it is our data center, so we want you to experience what we experienced.
We hope you enjoy it!
-
Posted by Greg Wilson, Head of Developer Advocacy, Google Cloud Platform
Free Trial
GCP Blogs
Big Data & Machine Learning
Kubernetes
GCP Japan Blog
Firebase Blog
Apigee Blog
Popular Posts
dotCloud provides faster, more reliable PaaS with Google Cloud Platform
Introducing Google Cloud Storage Nearline: (near)online data at an offline price
World's largest event dataset now publicly available in BigQuery
A look inside Google’s Data Center Networks
Google Compute Engine is now Generally Available with expanded OS support, transparent maintenance, and lower prices
Labels
Announcements
193
Big Data & Machine Learning
134
Compute
271
Containers & Kubernetes
92
CRE
27
Customers
107
Developer Tools & Insights
151
Events
38
Infrastructure
44
Management Tools
87
Networking
43
Open
1
Open Source
135
Partners
102
Pricing
28
Security & Identity
85
Solutions
24
Stackdriver
24
Storage & Databases
164
Weekly Roundups
20
Feed
Subscribe by email
Demonstrate your proficiency to design, build and manage solutions on Google Cloud Platform.
Learn More
Technical questions? Check us out on
Stack Overflow
.
Subscribe to
our monthly newsletter
.
Google
on
Follow @googlecloud
Follow
Follow
Follow