Google Cloud Platform Blog: TCP BBR congestion control comes to GCP

Google Cloud Platform Blog

Product updates, customer stories, and tips and tricks on Google Cloud Platform

TCP BBR congestion control comes to GCP – your Internet just got faster

Thursday, July 20, 2017

By Neal Cardwell, Senior Staff Software Engineer; Yuchung Cheng, Senior Staff Software Engineer; C. Stephen Gunn, Senior Staff Site Reliability Engineer; Soheil Hassas Yeganeh, Senior Software Engineer; Van Jacobson, Research Scientist and Amin Vahdat, Google Fellow
Google Cloud Platform—

"BBR allows the 500,000 WordPress sites on our digital experience platform to load at lightning speed. According to Google’s tests, BBR's throughput can reach as much as 2,700x higher than today's best loss-based congestion control; queueing delays can be 25x lower. Network innovations like BBR are just one of the many reasons we partner with GCP." — Jason Cohen, Founder and CTO, WP Engine

From GCP services to cloud users: First, when GCP customers talk to GCP services like Cloud Bigtable, Cloud Spanner or Cloud Storage, the traffic from the GCP service to the application is sent using BBR. This means speedier access to your data.

From Google Cloud to internet users: When a GCP customer uses Google Cloud Load Balancing or Google Cloud CDN to serve and load balance traffic for their website, the content is sent to users' browsers using BBR. This means faster webpage downloads for users of your site.

make TCP fasterChrome web browserQUIC paper describing the BBR algorithm at a high leveldescribingBBRLinux TCPQUIC What is BBR? BB R—— loss-based congestion control

In shallow buffers, packet loss happens before congestion. With today's high-speed, long-haul links that use commodity switches with shallow buffers, loss-based congestion control can result in abysmal throughput because it overreacts, halving the sending rate upon packet loss, even if the packet loss comes from transient traffic bursts (this kind of packet loss can be quite frequent even when the link is mostly idle).

In deep buffers, congestion happens before packet loss. At the edge of today's internet, loss-based congestion control causes the infamous “bufferbloat” problem, by repeatedly filling the deep buffers in many last-mile links and causing seconds of needless queuing delay.

Benefits for Google Cloud customersCUBIC

Higher throughput: BBR enables big throughput improvements on high-speed, long-haul links. Consider a typical server-class computer with a 10 Gigabit Ethernet link, sending over a path with a 100 ms round-trip time (say, Chicago to Berlin) with a packet loss rate of 1%. In such a case, BBR's throughput is 2700x higher than today's best loss-based congestion control, CUBIC (CUBIC gets about 3.3 Mbps, while BBR gets over 9,100 Mbps). Because of this loss resiliency, a single BBR connection can fully utilize a path with packet loss. This makes it a great match for HTTP/2, which uses a single connection, and means users no longer need to resort to workarounds like opening several TCP connections to reach full utilization. The end result is faster traffic on today's high-speed backbones, and significantly increased bandwidth and reduced download times for webpages, videos or other data.

Lower latency: BBR enables significant reductions in latency in last-mile networks that connect users to the internet. Consider a typical last-mile link with 10 Megabits of bandwidth, a 40 ms round-trip time, and a typical 1000-packet bottleneck buffer. In a scenario like this, BBR keeps queuing delay 25x lower than CUBIC (CUBIC has a median round-trip time of 1090 ms, versus just 43 ms for BBR). BBR reduces queues and thus queuing delays on last-mile links while watching videos or downloading software, for faster web surfing and more responsive video conferencing and gaming. Because of this ability to curb bufferbloat, one might say that BBR could also stand for BufferBloat Resilience, in addition to Bottleneck Bandwidth and Round-trip propagation time.

EspressoJupiterAndromedagRPCMaglevCloud BigtableSpannerTCP BBRGoogle innovations provide industry-leading performancejoin the public bbr-dev e-mail group