A Growing Tech Hub in Canada—With the Help of an IXP
Located in the foothills of the Rocky Mountains in western Canada, Calgary is a city of skyscrapers—and it’s fast becoming a tech hub. Yet a little over a decade ago, all of the city’s Internet traffic was being routed through Seattle, more than 1,000 kilometers away in the United States. With their data taking such a circuitous route, Calgary’s 1.3 million residents were feeling the pinch, with slower, more costly connections.
“I was always challenged by not having good enough connectivity here in Calgary,” says Theo de Raadt, a software developer. He’s also a volunteer network manager at the city’s first Internet exchange point, YYCIX.
The YYCIX was conceived in 2010 as a way of keeping Internet traffic local, to lower costs, and improve performance. At the time, there was only one IXP in Canada, TORIX in Toronto, more than 3,000 kilometers away. Calgary needed an Internet exchange closer to home.
Two years later, YYCIX was officially founded, with 14 members in its first year. Since then it’s grown to include more than 70 peers, including Internet service providers (ISPs), network service providers (NSPs), content delivery networks (CDNs), and a number of other Internet users. How did they do it?
Meeting All of Calgary’s Needs
Calgary is more than just skyscrapers and tech innovation, with small towns, rural communities, and farms just a short distance away. NETAGO is a small rural-focused ISP that delivers services away from Calgary’s main hub. It was one of the first to join the YYCIX, in 2013.
As far as benefits for us it really has been about reducing costs on bandwidth and breaking dependence on the larger ISP incumbents. The IXP attracted some of the larger Tier 1 Internet providers and that allowed us to get Internet access at a reduced cost. It brought in the content delivery networks, which took the burden off of our paid connections. It also gave us connections to other IXPs, such as TORIX, which gives us access to other services such as IPTV, which must be delivered over a private network. So, it gave us options that we didn’t have before.“
He estimates that between 20 and 30 percent of the total volume of his company’s data has been going through the YYCIX. This translates into substantial savings: close to 20 percent.
Duchcherer says the YYCIX has also played a role in improving privacy and security by keeping traffic in Canada. “In the United States, their [Patriot] Act allows them to access any traffic going through their networks, so this created a push to make sure websites were hosted in Canada. Even data as simple as access to a Government of Canada website—if that traffic is being routed to the U.S. to get to the servers in Ottawa—there is a risk that it could be spied on.”
Overcoming Early Challenges
From the get-go, getting Calgary’s IXP connected wasn’t easy. Toronto’s TORIX flourished because it had a carrier hotel: a physical point for its networks converge. Calgary, like many large Canadian cities, didn’t have one. Without a carrier hotel, they had to build the IXP in multiple sites, eventually building six.
In addition, YYCIX would have to lay cables to interconnect these sites. “We have 25 kilometer distances between a lot of our members, and fiber optics are expensive,” says de Raadt.
It was a big investment for YYCIX, which is financed entirely through donations. It operates on a shoestring budget of $4,500 per year, thanks in part to its volunteers.
“YYCIX does not have a revenue source and can’t afford to charge because if we do, everyone will go to the Seattle IXP, where they can connect for free,” says de Raadt. (Some data centers charge members a monthly cross-connect fee.)
After many years, YYCIX convinced the City of Calgary to donate free fiber to connect its data centers. This allowed them to consolidate at a key formative stage.
Growing Pains Force a Switch Up
As time went on, and YYCIX’s membership and traffic kept growing—from 9 Gbps peaks in November 2017 to current peaks of 50 Gbps—the fiber was no longer enough. The infrastructure also needed to be upgraded if it was going to stay relevant.
“What happened as we expanded to more data centers is that we had the fiber but didn’t have the correct switches. We exceeded the capacity of 10Gb/20Gb links between data centers. We’re now exchanging 50 Gbps of traffic or more, so we needed ports of 100 Gb. Our backbone must be able to support this,” says de Raadt. “We were afraid we were going to drop traffic due to congestion and our peers would disconnect.”
The Internet Society donated two switches to the YYCIX valued at CAD $32,181—more than seven times YYCIX’s annual budget. One was installed at its data center in November 2020 and the second is awaiting the completion of a fiber build. The switches have allowed YYCIX’s capacity to be upgraded to 100Gb links to its data centers, with the addition of many other 100 Gb ports for future peers.
“We now have a complete fabric: seven switches at six data centers,” says de Raadt. Increased capacity is key to growing the exchange, by attracting new players. “[They} see that we’re doing what we need to improve.”
According to the Internet Society’s Director of Internet Growth and Trust, Hosein Badran, the donation strengthened the IXP environment in western Canada. “It expands their reach and footprint.”
Canada’s IXP Playing Field
The IXP landscape in Canada has changed dramatically since YYCIX was first conceived of more than ten years ago, and the country’s Internet infrastructure is much stronger now. TORIX is no longer the only exchange. According to the Canadian Internet Registration Authority (CIRA), there are now 12 IXPs across Canada, with a total of 432 participants. CIRA touts IXPs for increasing resiliency and reducing congestion, helping stop cyberattacks at the border, reducing transit costs, resolving DNS queries close to the source, and ensuring fast, reliable access to .ca domains plus than 37,000 other second-level domains.
“We need the dot-ca domain at all IXPs and we support them to grow,” says Jacques Latour, Chief Technical Officer for CIRA. CIRA was one of the first peers to connect to YYCIX, and they recently donated another switch. “It’s about increasing performance, speed, and latency and bringing the Internet to the edge.”
YYCIX members are reaping the benefits. Their latency—the time it takes for data to transfer—has improved significantly, dropping from anywhere between 4 and 35 milliseconds to less than 1 millisecond.
Latour says it’s also adding more resilience to the Internet in Calgary, a region that’s seen its share of natural disasters. “Everybody wants to be as close as possible so that if there’s a fiber cut in Canada, say it gets cut by an avalanche in the Rockies, Calgary would be isolated. But with an IXP, the bulk of the Internet can still operate on its own.”
Terry Duchcherer recalls how YYCIX kept NETAGO and its members up and running during the flooding in Calgary in 2013. Other incumbents that weren’t connected to the IXP went down.
De Raadt says mudslides are common in the Rocky Mountains, destroying fiber on a regular basis. In 2017, a rockslide in the Fraser Canyon took out fiber optic cable. Having the IXP allowed some of the region’s Internet traffic to keep working—while other traffic slowed significantly or was cut altogether.
Around the time of the rockslide, de Raadt says western Canada’s two largest operators, Shaw and Telus, only had 1Gb of bilateral peering. “We asked them to expand their connection to 10 Gb. They did and were immediately full. Now, during COVID-19, they have 100 Gb links and they’re not full.”
The COVID-19 pandemic was an eye opener for many. Latour says traffic spikes due to increased Internet consumption made many businesses and institutions realize the need to exchange more traffic locally. He says this led to a 15% increase in IXP membership across the country.
Overall, he says Canada’s Internet exchange ecosystem is “pretty good right now,” but it could capture more traffic and even more peers, like various levels of government. “We’re working with large network operators like the Government of Canada to build its strategy to connect with all the IXPs. Calgary needs more high-profile peers like municipal, provincial, and federal government peers.”
Reaching Calgary’s Potential
YYCIX currently has several content delivery networks, including Cloudflare, Facebook, and Google. But its network manager de Raadt wants more. He hopes existing members will increase the size of their ports, some of which are already at their maximum capacity of 10 Gbps.
De Raadt’s ultimate goal is to interconnect every producer of local Internet traffic in Calgary. YYCIX currently has around 70 members, including one of the region’s largest incumbent operators, Telus. It’s a big vote of confidence in the IXP, but he estimates there are still about 110 potential local networks, and three of the largest ISPs are still not on the exchange: Shaw, Bell, and Rogers.
“If we allow the globals to come in and deliver traffic locally, that will reduce some of the costs that can only be shouldered by incumbents.”
Amazon recently announced that it will open an Amazon Web Services (AWS) region in Calgary in 2023 or 2024. De Raadt expects they will connect to YYCIX.
“You need that leap of faith that the content providers will show up,” says CIRA’s Latour. “Now the peers in Calgary have most CDNs present and the value proposition keeps growing exponentially.”
Still, content delivery networks alone haven’t been enough to lure the incumbents, who have many CDN caches of their own. So, the YYCIX has also been beefing up its domain name system (DNS) servers to attract new players. Having a DNS server within an IXP makes the first lookup of domain names faster, caching the answers for future use, which saves time on searching.
“Telus saw this benefit and joined for that reason three years ago,” says de Raadt. As Telus’ DNS responses continue to improve, de Raadt hopes it will outperform Shaw, Rogers, and Bell and serve as an incentive for them to join YYCIX.
Blazing a Trail across Canada and the Rest of the World
YYCIX isn’t just improving Calgary’s Internet infrastructure—it’s also setting an example for other regions and IXPs. It was the first IXP in the world to perform Resource Public Key Infrastructure (RPKI) filtering, which has since become standard practice for most exchanges. And YYCIX’s longevity, lessons learned, and success have paved the for other exchanges in Canada. The Internet Society is working to support the creation of an IXP nearly 3,500 kilometers north in Inuvik, Northwest Territories. It will help strengthen the IXP ecosystem in Canada’s remote Arctic region—and Canada as a whole.