Donate
Google leaked prefixes – and knocked Japan off the Internet Thumbnail
‹ Back
Mutually Agreed Norms for Routing Security (MANRS) 28 August 2017

Google leaked prefixes – and knocked Japan off the Internet

Andrei Robachevsky
By Andrei RobachevskySenior Technology Programme Manager

Last Friday, 25 August, a routing incident caused large-scale internet disruption. It hit Japanese users the hardest, slowing or blocking access to websites and online services for dozens of Japanese companies.

What happened is that Google accidentally leaked BGP prefixes it learned from peering relationships, essentially becoming a transit provider instead of simply exchanging traffic between two networks and their customers. This also exposed some internal traffic engineering that caused many of these prefixes to get de-aggregated and therefore raised their probability of getting accepted elsewhere.

The incident technically lasted less than ten minutes, but spread quickly around the Internet and caused some damage. Connectivity was restored, but persistently slow connection speeds affected industries like finance, transportation, and online gaming for several hours. Google apologized for the trouble, saying it was caused by an errant network setting that was corrected within eight minutes of its discovery.

This incident showed, again, how fragile the global routing system still is against configuration mistakes, to say nothing about malicious attacks.

What it also showed is a lack of defense – the incident propagated seemingly without any attempt from other networks to stop it.

The Internet Society works to address security in many ways, including the Mutually Agreed Norms for Routing Security (MANRS), which encourages operators to build these defense lines. If one line fails, incorrect announcements will be stopped at the next one. Every operator should be offering such defense – not only for its own security and stability of operations, but for the stability and security of the global communication fabric.

Implementing the four MANRS actions leads to better protection against traffic anomalies caused by misconfigurations; cleaner setups resulting in easier troubleshooting and lower time-to-resolution (TTR); improved peering conditions; and opportunities for valuable collaboration with other operators through a discussion forum and professional network. But, MANRS has a limited scope. It is an absolute minimum that an operator should consider. Its requirements focus on elementary pieces of network topology, such as customer-provider relationships, preventing spoofed traffic for single-homed stub customers.

To be fair, MANRS would not have helped specifically in this case. But if Verizon were a MANRS member, perhaps that would have improved its security posture in more complex situations. If Google were a MANRS member, it could have communicated to its peers more clearly what announcements they should expect.

The more operators implement MANRS, the fewer incidents we will see, and the smaller will be their scope. MANRS is not a one stop solution to all of the internet’s routing woes, but it is an important step in the right direction toward a globally robust and secure routing infrastructure.

As to the leaks, there is currently some work happening in the IETF. The IDR working group has two proposals addressing the route leak problem: “Methods for Detection and Mitigation of BGP Route Leaks”, and “Route Leak Prevention using Roles in Update and Open messages”.

We encourage you to learn more about MANRS and how to sign up, and read more about the Internet Society’s approach to Collaborative Security.

‹ Back

Disclaimer: Viewpoints expressed in this post are those of the author and may or may not reflect official Internet Society positions.

Related articles

14,000 Incidents: A 2017 Routing Security Year in Review
14,000 Incidents: A 2017 Routing Security Year in Review
Mutually Agreed Norms for Routing Security (MANRS)9 January 2018

14,000 Incidents: A 2017 Routing Security Year in Review

How was the state of the Internet’s routing system in 2017? Let’s take a look back using data from BGPStream....

Route Leak Causes Major Google Outage
Route Leak Causes Major Google Outage
Mutually Agreed Norms for Routing Security (MANRS)15 November 2018

Route Leak Causes Major Google Outage

Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that...

Another BGP Routing Incident Highlights an Internet Without Checkpoints
Another BGP Routing Incident Highlights an Internet Without Checkpoints
Mutually Agreed Norms for Routing Security (MANRS)13 December 2017

Another BGP Routing Incident Highlights an Internet Without Checkpoints

Yesterday, there were two BGP routing incidents in which several high-profile sites (Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and...

Join the conversation with Internet Society members around the world