look at seaside and residential buildings next to it

Panama’s IXP, Once Languishing, Has a Renewed Sense of Purpose

Panama’s IXP, InteRed, is now located at a data center, a neutral exchange point.

Panama is home to one of the first Internet exchange points (IXPs) in the Western Hemisphere, InteRed, which was founded in 1997. But despite its early beginnings, InteRed’s growth has been slow. Little traffic was exchanged, and over the years, its limited capacity became noticeable, making it less attractive to potential members. But a change in leadership and the donation of essential equipment and training by the Internet Society in 2021 have reinvigorated the IXP, driving traffic and attracting new members. As a result, it has gone from stagnation, and even the risk of irrelevance, to being considered transcendental to the local Internet ecosystem.

One of the main drivers of this change has been Pablo Ruidiaz, an electronic engineer who specializes in telecommunications and has been the executive director of InteRed since January 2021.

“I’m rebuilding everything,” he says. This has meant modernizing equipment and servers, improving routing practices, negotiating with content delivery networks (CDNs), and relocating the IXP to a neutral point—leaving its initial location at local operator Columbus Networks for the UFINET data center.

Ruidiaz’s extensive experience in both the public and private sector has allowed him to build bridges. He sat on InteRed’s board of directors until 2008, in his previous role as a private wireless operator. He then spent 10 years with the governmental authority for the modernization of the state and universal access services. He is also the founder, and existing board member, of the Internet Society Panama Chapter.

“The [IXP] association was dormant for a long time, without many changes or investments,” says Ruidiaz. “Now, we’re upgrading our critical Internet infrastructure to improve service and ensure that operators benefit.”

“There was a dark period of nine years where nothing happened, from around 2011 until 2020 when the board of directors decided to hire a new guy, which was Pablo. Fortunately, he sort of kickstarted things and renewed the energy of the IXP, which is what was needed,” says Russell Bean. He is the operations manager and innovation incubator at Liberty Latin America, the parent company for Cable & Wireless, one of Panama’s two largest operators. He sat on the InteRed board of directors at one point, as Cable & Wireless was one of the IXP’s first members.

Increasing Its Capacity

Ruidiaz says that because of the pandemic, government data exchanged through InteRed increased by at least 1 to 2 Gigabits per second (Gbps). And since the Panamanian government requires all of its Internet service providers to be connected to the IXP, to keep traffic local, it became essential to expand its capacity.

The first task was the purchase of, and migration to, more efficient and automated infrastructure. InteRed invested in two new switches that were installed in April 2021. This allowed it to increase its capacity and decrease latency.

Beforehand, most of its members had 100 Mb or 1 Gb ports, but there was no room for more connections or ports. Now, the infrastructure allows all members to have 10 Gb ports and there is room for up to 90 ports, with a total cumulative capacity of 3.6 terabits per second.

In early 2021, Cable & Wireless was exchanging 2 Gbps of traffic, well beyond its port’s 1 Gb capacity at the time. With the new switches, it was able to release more traffic, which now reaches 2.5 to 3 Gbps—more than a quarter of InteRed’s total traffic. But Bean says the upgrade almost came too late.

people standing posing for a photo

“We were actually going to peer directly with Claro and Tigo [two of the main ISPs in the country]. And if you lose the three biggest operators in an IXP, it will become irrelevant. We didn’t want that to happen,” he says.

You’ve got to build for the future. By the time someone is asking for more capacity, if you don’t have it, it’s too late. I think they were just in time to get that increase.”
Russell Bean, Operations Manager and Innovation Incubator, Liberty Latin America

In December 2019, before COVID-19, daytime and nighttime peak traffic was around 4 Gbps. By April 2020, it was exceeding 10 Gbps, amid some of the pandemic-related emergency measures. Today, InteRed members regularly exchange around 8 Gbps of traffic during peak hours.

Latency—what users perceive as the delay in downloading content—has also diminished. In March 2021, latency was around 20–30 milliseconds, and by April it was already down to 5–10 milliseconds with the change of infrastructure.

Today, InteRed has 16 members reflecting a mix of small and large carriers, including the two largest in the country: Cable & Wireless and Tigo. But despite having decent membership compared to other IXPs in the region, Ruidiaz says he wants to reach all 47 operators nationwide. For this, he believes the IXP must become more relevant, with more CDNs, root servers, and other services, coupled with education and community building.

New Content Providers and Creators

Among InteRed’s newest members is Edualianza, and it’s one of just two members that create local content. Edualianza offers a complete digital educational platform that serves 12% of the country’s private schools.

“The pandemic made everyone want to connect,” explains Luis Felipe Lince, owner and director of Edualianza. “It put us under more stress than we were used to. All clients were demanding more tools to be able to serve students remotely. It wasn’t so much an increase in clients, but the increase in clients connected simultaneously. We’ve had to invest in more equipment.”

Lince says that the company relied on two ISPs, but one had constant problems and the connection would drop. So, he decided to acquire his own public Internet Protocol (IP) prefix, which enabled Edualianza to join an IXP.

He says joining InteRed has meant spending more per month, between the monthly fee and the fiber optic cables to connect to the IXP—“is quite a lot” for his small business. But Lince says it’s been worth it because his service has improved.

The pandemic made everyone want to connect.”

Luis Felipe Lince, Owner and Director, Edualianza

“When a client logs on, they quickly connect to my servers here in Panama. It was a priority to have this link with InteRed. I’ve done measurements and yes, the connection is better than before. Latency has dropped from 25 to 10 milliseconds and router hops used to be eight and now they’re only six. And I no longer have the problem of losing one of my providers’ connections,” he says. “ The biggest benefit is that customers feel that when they connect, our service is fast and always responsive, regardless of their device.”

To keep InteRed improving, Lince thinks it should increase membership, lower the fees for smaller members, and attract more CDNs.

Ruidiaz agrees, calculating that the current traffic to large CDNs not based in Panama accounts for around 30–40 Gbps. “For me, bringing them in is a priority. Now we can consider it, thanks to our infrastructure. A year ago, thinking about that was a joke. We had to have our house to be able to receive guests.”

He says he is currently in negotiations and he hopes to have at least one new CDN and one root server connected. (The root server will be donated by the Latin American and Caribbean Internet Address Registry (LACNIC) through its +Raíces program). He says this is essential to keep membership growing.

Improving Routing and Domain Name Resolution

Increasing the IXP’s capacity has also enabled other enhancements, such as domain name resolution. In October 2021, InteRed installed new equipment to host LACNIC content, donated by the Internet Society, LACNIC, and LACNIC’s Association of Internet Traffic Exchange Points (LAC-IX).

Ruidiaz is also encouraging good routing practices, through training on Mutually Agreed Norms for Routing Security (MANRS), supported by Internet Society. He’s proud that five operators have already completed the training and hopes all of them will implement the MANRS principles.

MANRS logo and words

“You have to do some hand holding,” admits Ruidiaz. “Everyone is interested in being connected to the IXP, but when you ask something of them, they don’t always respond. One of the things we’ve recognized as important is collaboration [to join] MANRS and the implementation of route servers. [Members] often don’t put routing problems on their books until there’s a problem.”

He says that since they started integrating MANRS within the IXP, there have been two incidents that operators didn’t announce. But they were able to adapt quickly, thanks to MANRS, and according to Ruidiaz, there have been no service interruptions.

Bean says Cable & Wireless was one of the first companies to join MANRS, almost as soon as it existed. Recently a MANRS fellow, he says it’s always important to standardize rules and security.

“That’s definitely helped; we’ve had fewer incidents since then,” says Bean, adding that it benefits all members.

As a big operator, sometimes we can get clumsy and … we realized we had some security issues that MANRS helped us resolve, so it was a good process to go through.”
Russell Bean, Operations Manager and Innovation Incubator, Liberty Latin America

Bean is a great supporter of the IXP, for operators of all sizes, and believes that it ultimately helps ensure greater competition while making networks more robust and dynamic.

InteRed’s new leadership has made the most of good relations with the Internet community. It has also ably identified its own critical areas for improvement and infrastructure-related growth barriers. “Beforehand, the project had latent potential that hadn’t been fully exploited. But because of the changes promoted by its director, it has been able to set itself on a course that is characteristic of IXPs that are on the path towards consolidation,” says Israel Rosas, the Internet Society’s Senior Regional Development Manager.

Image credit: Unsplash, © InteRed, © Klaus Tezokeng Chiha