Transcript* of the IGF Rio Best Practices Forum
Internet Traffic Exchange in Less Developed Internet Markets and the Role of Internet Exchange Points (IXP)
14 November 2007
Rio De Janeiro, Brazil
*Note: The following transcript is based on an audio recording of the IXP Best Practices Forum. While transcribed and edited by a professional transcription service, some errors and inaccuracies may be present. Words or phrases that could not be interpreted by the transcription service are marked with a timestamp (e.g. 20:11).
Sam Paltridge: They possibly know something about IXPs and the terminology that the panellists are going to use. So if you’re from the technical community or you feel you have a good familiarity with the internet exchange points could you raise your hands. And I’m guessing that other people here are here to learn about IXPs. They’re from policy or (0:19) society or business or whoever. We’ve got an excellent set of panellists here, all of whom know a great deal about internet exchange points.
I’d also like to thank ISOC for organising this session at the outset. I’m Sam Paltridge from the OECD. Some of you may be wondering why is the OECD interested in IXPs. About a decade ago, more than a decade ago in fact, we held a workshop on what would today be called internet governance, but it wasn’t then. And many of the panellists mentioned the importance of having internet exchange points which was something new to me. I didn’t know what an internet exchange points were. And I wrote a report saying the internet exchange points IXPs were a good thing and should be rolled out around the world. And a journalist picked up the report and I remember the headline, “OECD says internet should be less US centric”, which probably would be interpreted differently today than it was then. Because the US certainly welcomed that report and recognised that the internet should be developed in all of the world’s countries and I think we’ve gone a fair way to doing that.
Courtesy of Packet Clearing House I borrowed one of their maps and today we can see that countries with IXPs total 79, but there’s still 91 countries without IXPs. And so today we’re going to learn about some of the experiences from people that run IXPs. We’re going to learn whether those 79 countries – how some of them, whether there’s best practices that some of them have, which mean that their IXPs function more efficiently and are more useful to the communities they serve than other IXPs. And to do that we’re going to concentrate at the start on two regions.
Firstly Africa and Michuki who comes from Kenya is going to talk about the Kenyan internet exchange point and the experiences they’ve had. And to broaden that out a little and react to that presentation, Mike Jensen’s going to be the discussant. Then we’re going to shift continents across to Latin America and Gabriel’s going to tell us about the experience in Argentina and then Roque is going to broaden that out to the rest of Latin America. And then our overall panellist discussant is Bill Woodcock.
I’m very pleased he’s here to join us. And Bill’s going to take less time than the other panellists and possibly comment on some of the things that they’ve said but also introduce some work that PCH has been working on in terms of best practice in the area of measurement and IXPs, which is perhaps something that’s a bit new to this area. So without further ado I’d like to give the floor to Michuki and we’re looking forward to hearing what you have to say.
Michuki Mwangi: Thank you Sam. Okay. I don’t know how many of you were just in the last, previous – in this room like the last 20 minutes when I was speaking about the route service. Okay, so very few of you. So I just want to take from the cue where Sam Paltridge has left and in terms of why really the exchange points? Did we actually really need them or was it just by virtue of the fact that it’s according to her when as trying to make the internet more US centric?
Sam: Less.
Michuki: Less US centric. Okay. This works. Okay, I guess I need my presentation to go up. Yes, okay thank you. Okay. So the Kenyan exchange point has one of the most interesting histories you can come across, because it was actually established in November 2000. And just about two weeks into operations the regulator came in and said “No I shut it down.” And so we had to learn that.
The reason being is because they come and tell them who actually had a monopoly of any telecommunications and internet connectivity say that we actually were infringing on that particular monopoly, and so it had to be shut down. It is something our regulator and most of us actually wish to forget, but it actually – I always say it’s a good thing because it shows where we are coming from and it’s actually cleared a way for us to actually start communicating with the regulator.
And so one year down the line we’re actually able to get back on deck with the exchange point. Actually on February 14th 2002 the exchange point came back up, so most of us on Valentines Day were purely with our routers at the exchange point. We had coke and pizza and we actually brought it back up. So it’s usually – Valentines Day for most people is with their spouses for us, its something we toast to this – which has (6:23) we have with the exchange point.
So having brought it back up in 2002 it’s actually been brewing and it was established basically as a non profit entity and since the house has actually been providing the connectivity that we so desire. Now one of the things about the KXPs that it’s probably one of the only licensed ones. As one of the outcomes of actually getting it back on is that it actually had to get – acquire a license. So it’s actually a licensed operator in the country and we actually have to pay license fees to the regulator every year.
Now, well probably a good thing but it actually means that in Kenya if you want to actually provide competition you can actually come and apply for a license and open up your own exchange point. But fortunately is that one is actually seen as a neutral one and so it operates without competition. Initially we only had four ISPs exchanging traffic and basically we got a funding – the funding to actually set up the exchange point from Cisco Systems.
Some members of the ISP also donated the racks and also every member brought on their switches to be able to – their routers to actually exchange traffic. So the question here is did we really need the exchange point or was it a want? And one of the first things I have to say is that by virtue of us having an incumbent telco – and that’s where they were very keen on actually having us shut down – is by the virtue that they felt they were losing something.
Now one of the first things that you have to understand about exchange points means that its sole purpose is to keep the local traffic local.
If you have an incumbent telco any local traffic that you sent to them will always be billed at international rates despite the fact that since everybody goes to them they take it off your pipe and push it down your competitor’s pipe. They save the money and still bill you as if the packet went outside the country.
So that is a loss that for some incumbent operators may not be acceptable, but at the end of it all they still charge you for it. So at that point in time bound with capacity for one Mb in Kenya transit was in excess of $3,000 for a 64K circuit. And the (9:06) of course meant that there were over 700 milliseconds. Now why they were over 700 milliseconds is because it was so expensive to buy the transit through the incumbent.
So what we’ll do is actually buy upstream traffic from the incumbent. The policy in the country was – had a grey area when it came to return traffic. So we had these simple receive only dishes which were set up in every ISP on top of the roof, they had this receive only dishes. So we’d use that for return traffic. Now that return traffic you could buy from anyone and the prices would significantly come lower. So out went through the telco, in came through their foreign operators.
So as a result what would happen is – I skipped a slide, I hope I can go back. Okay. Can you go back for me one slide. Okay, thank you. So as a result of that what used to happen is that if your next door ISP wants to send you traffic, the only way they know how to get back to you is through the receive only operator, because that’s where your preferences are announced. And so that means it will go out through the incumbent and in through some place, maybe from New York or London or basically Europe. So in essence it was quite expensive.
Secondly, we had poor quality of service. That means every time the operator was up and down – was down, we had a problem. I remember one particular incident, and this is back in 1998 I think when the US Embassy in Kenya was bombed, and right next is where they had their old transmit cuts and so they were down for a couple of hours and so was everybody else. We didn’t have an exchange point then so we were actually – everybody sat down and we just called each other and just to find out.
Also, you’re also down? Yes, we’re down, so let’s wait and see. And then when everything came back on they had preference of who was connected back on. So depending on how good or how rude you are to them then it depends that you’ll probably come back on last. So anything that will be local that would have actually continued working did not work. Also that meant that by not having an exchange point it meant we had no opportunity to actually encourage the growth of local content, because then where would we host it?
And at the end of it all looking at all those factors we desperately, I mean this is purely sheer desperation to have an exchange point. Okay. So our policy. What policy did we have when it came to the exchange point? So the first thing we had to emphasise was neutrality. It meant that the facility we’re actually going to set up as the exchange point had to be actually acceptable to all the members.
So how do we go about this? Basically all the members had to be – because the exchange point is run by the ISP Association and part of the license requirement meant that only – the exchange point could only serve licensed ISPs. That was the condition, a licensed condition. Secondly, to ensure sustainability we charge a lot of money. For the first time for you to get connected, you had to pay $1,500 as an initial connection fee and then there was a monthly fee which was charged.
And also to enforce, or rather to make sure that everybody peered against each other we actually had to, what we call a multi lateral peering agreement. So everybody peered with everyone through a route server and we actually prohibited co-location of services of the exchange point. That means you couldn’t bring anything else other than your router. So that was another service we offered. And because very few of us understood anything else BGP was significantly a very new time to us.
So the only thing we knew then was a Cisco router and started routing and so the only thing we knew is an IOS and so we went into BGP in Cisco and that was it. Anything else we couldn’t help you, so we couldn’t want you to bring it on. So what are the learning curves you actually came through by having such a policy? So licensed members, it meant basically we had closed ourselves off only to ISPs. So at that time the regulator felt that only the ISPs – internet came through the ISPs, which was really not the case.
As the internet grew we realised that internet content providers and other people would actually come on board. Also, by virtue of actually having that statement it meant we really really couldn’t get anybody else on board really. And some of the people who actually peer the exchange point to be is the government. And by then they actually did have a license to operate. For instance, the Kenya Revenue Authority peers at the exchange point.
With that kind of policy it meant that the Revenue Authority had to go and apply for an ISP license. They’re the Revenue Authority, they collect taxes. But they had to apply for an ISP license to come and peer the exchange point, which didn’t really make sense. So in essence that particular policy and the role in their license had to actually be relaxed. Membership fees, of course again they were too steep. Too steep for anyone to actually – or rather to attract increased membership and we need the exchange point to be self sustaining.
Then the only way you can do that is by reducing the cost vis-a-vis and then later on go out and market yourself as an exchange point to attract more people to come on board. So that was significantly reduced, it’s currently at $400, initially from $1,500. And the monthly fees are about $280 a month which is billed per port. That means that you pay about $280 per month for a one gig port because we give – the port is capable to support speeds of up to 1GB.
Multi lateral peering. Well so we started off with multi lateral peering because we really didn’t know anything about BGP. Now later on this has actually become a social enforcement tools, because some of the large layers and most of the people who worked in large exchange points will actually understand that.
If you have big players on an exchange point they really at some point want to choose whom they want to peer with. So how do you make sure that all the players being a multi lateral exchange point actually peer with everybody else? So you force that everybody who comes to the exchange point peers with a route server. So they only peer with one device and everybody else is able to see their routes. So that’s one of the tools that we use. So initially it was a technical limitation, now its come actually to work in favour of enforcing the social side of it.
Co-location. Again, this was not a good policy to use and as a result what we’ve done is not entirely relaxed the policy. What we’ve said we could actually allow value added services to be co-located at the exchange point and so for these we’ve been able to host other equipment of their facility. That means that we have route servers now being hosted at the exchange point and so this actually does add value to the exchange point. So the type of vendor, purely fear of the unknown. This has since changed.
Unfortunately no one has brought in a different vendor. We have lets say, one or two different vendors UNIX boxes, but in essence right now there’s more boldness in terms of going out to seek other vendors yet. But we are yet to see anyone bringing in different equipment. And so, something else we’ve also had to learn – I hope I’m not short of time. Three minutes – okay.
Sam: Five minutes.
Michuki: Okay. So the ISP structure. One of the things we’ve had to learn is that over time the exchange point, because it was set up through – by the ISP Association, was seen actually as an ISP club. Now that the government actually has interests in it, the Kenya Revenue Authority is actually peering with them. And for them peering is because of eGovernance. So they’re actually using – they have a custom system that they actually use for imports clearance which is accessed through the network and that’s why they are peering at the exchange point.
They are collecting a lot of money. On average it’s about half a billion Kenya shillings a day, I think I had done that, but basically its half of the revenue that they collect every day through that online system. Now, but they actually have no say when it comes to the policy and day to day operations of the exchange point and that being the case then they would want to participate. But how do they do so if the only body that they can is a lobby group?
So they’re actually insisting that we have to revise the structure of the exchange point. And so as a result of the learning curves we’ve gone through, we’ve had the following. The membership has actually grown; we have 15 ISPs, three internet backbone providers. We have the government peering. Again, the incumbent telco which was actually against us which filed the suite with the regulator to have us shut down is also peering. The ccTLD is also there.
We have the GSM operators, the voice and data. Basically anyone who has internet data is actually peering at their internet exchange point. Value added services because of co-location; we have route service, FNJ. We have gTLD service common net. We have the KE route server for the ccTLD. A public GPS and a route (20:11). So looking at the traffic, basically this is the growth we’ve experienced over the years and you can see over the years 77 per cent, 137 per cent and last year 110 per cent growth. And so one of the things which I had to show here was okay we’ve passed that – so on Valentines Day we usually have unusual traffic, because of free SMSs. People – their free SMS websites, the traffic goes high.
On this year their National Examinations Council results were actually put online as well and for two days we had traffic of over just about 30MB per seconds for two days. And we have challenges, basically margins and acquisitions. That means we lose customers because they margined one provider but they don’t take two ports. So you lose money from people who are peering. Companies wind up; the free SMS website has since shut down. So I’m not sure what happens next year on Valentines Day.
And then the price of the local loop, again it has come down but is not relatively low because there’s limited competition. We only have about two providers of – three providers of local loop infrastructure. And we still have to pay the annual fees to the government. So that comes too. So my conclusion is that it’s really important to create awareness of your exchange point and here have in mind that you all know (21:43) because they’ve gone out and marketed themselves.
So we also need to build this info, the local ISPs. And also we need the governments to promote the growth and a pro-active regulator would also help. Okay. So let me just explain this. So the most important thing about this particular slide I wanted to show is that with an IXP, it shows that there is actually – they’ll only bill for about 11 hours where they’re paying byte per traffic. And they have 13 hours of silence where they actually would be making money if they be able to actually follow the user’s home. Because this just shows the user they’re in the office and not at home.
So they would actually make extra money and so where I have the question marks means who is actually paying for that beat, right? So we have that many number of exchange points in Africa, this is from the NSRC website and that means that there’s a lot which can actually be done in terms of interconnections at transit from the various countries.
That means we can leave Kenya, go all the way to South Africa and all the way to Angola, across through transit or peering if possible. Well I don’t believe exchange points should peer, but transit across to the other part of Africa. So thank you, sorry that took long.
Sam: Thanks Michuki. I propose to go straight to Mike and for questions to the whole panel after we’ve heard all the presentations.
Mike Jensen: Thanks Sam and thanks Michuki for a very good introduction to some of the issues there. I want to react to Michuki’s presentation in terms of some of the questions that the audience might have. And to frame some of the issues that might be relevant to some of the members here in terms of their own planning for internet exchange points. So obviously the first question, especially as we’re looking at the African situation here and the experience there, is what are the barriers to starting internet exchange points? We can see that much less than half the countries in Africa have exchange points and all of those countries bar a couple have multiple internet service providers.
So why haven’t the rest of the countries established them? Well I think one of the big problems is the lack of a functioning Internet Service Provider Association in the country, although it’s a little bit of a chicken and egg situation. We’ve seen in many countries in Africa that the exchange point or the potential cost savings from an exchange point is one of the driving forces behind establishing an internet service provider association.
I think one of the reasons why these associations and the internet exchange points have not taken off in many countries is because of the relative immaturity of the market in these countries. And the internet service providers are all scrabbling to get as many users as they can and there’s quite a lot of lack of trust between the service providers. So it’s important to try and look at ways of building that trust and creating other incentives to getting together to collaborate.
I think another issue is that there is limited technical expertise, especially amongst the smaller ISPs to establish a multi path routing. In many cases it might require paying more for a technician who is familiar with these issues than it does to actually pay for the additional cost of sending the traffic internationally to the other ISP and country. So clearly, there again capacity building amongst smaller ISPs in particular is much required and support for that is desired.
Another issue of course is the lack of local infrastructure or a very high cost for local infrastructure. So if the cost for a national or a lease line across the city is not that much different from an international line or that sort of infrastructure doesn’t exist. Then of course the incentive for the ISP to lease a separate circuit to the exchange point is much reduced. And related to this is the regulations around the use of Wi-Fi or other wireless technologies to connect to the exchange point.
If this type of terrestrial connection independent from the monopoly operator in most of these countries is not allowed, then of course it will be more difficult for the internet service provider to justify connecting to the internet exchange point. Another question that’s come up quite a few times is should the internet exchange point be used to provide international bandwidth to internet service providers connecting to the exchange point?
This is obviously seen by some ISPs on the surface as a good idea. The ISPs could all bank together to purchase a higher bandwidth circuit at lower cost and share it amongst the service providers by the exchange point. But generally we find that most of the internet service providers are not that keen on this because their international connectivities are key differentiated between them in the competitive market in terms of providing service to the end user. So we generally don’t see that taking place.
Another question is, is there a need for multiple exchange point in one country? A key factor here is the cost again of national long distance lease line connectivity or terrestrial connectivity. In many cases where we have this monopoly or duopoly environment where restricted access to the telecom market is in place, national pipes are either very expensive or quite often they may be quite unreliable.
So actually the cost of establishing a terrestrial link between major cities may be prohibitive when compared to the cost of satellite bandwidth. And therefore you may as well just route all your traffic over your single international connection ‘cause you’re going up to the internet cloud by the satellite link anyway. Then what is the role of government in IXPs? We heard in the case of Kenya that there’s quite a rigorous licensing process for the internet service providers and the internet exchange point itself is also approved by government.
My feeling is that in both cases this should not be the case and that internet service providers shouldn’t need a license and internet exchange points shouldn’t need a license either. I think there should be purely a commercial arrangement between the internet service providers that should not require any approval from government. I suppose in the cases where internet service providers are licensed, perhaps one of the things that could be done to encourage the formation of internet exchange points is to include as part of the conditions of that license that they interconnect terrestrially when feasible so that we encourage their formation.
And then finally, there’s a question that comes up quite a lot about the need for a regional exchange point. Clearly it would be great to keep African traffic within Africa and not to have to transit traffic between Tanzania and Kenya via Europe or between Malawi and South Africa via Europe as has been the case at the moment. The real problem here of course is that there’s very little terrestrial inter connection between countries in Africa as yet and there’s a lot of momentum for building back bones between countries.
And unfortunately even where these back bones do exist, such as along the SAT-3 cable and some of the terrestrial cables that are present now in West Africa and Southern Africa, we find that the cost of this connectivity are prices very similar to satellite bandwidth. So there we have the same problem that you may as well just consolidate all your traffic over your satellite link and it’s not really worth differentiating the traffic and trying to move it across a separate satellite link to a foreign country.
So I think until we see competitive fibre back bones more prevalent in Africa we’re unlikely to see much growth in terms of inter connection between IXPs on the continent. Thank you.
Sam: Mike, thanks very much. Gabriel, the experience in Argentina.
Gabriel Adonaylo: Hi. Well first of all I’d like to thank ISOC for inviting me to this workshop. Second, apologise for my English speaking and apologise also for any mistake you can find in my typing of this presentation.
Third, this is just a moment in the life of our IXP in Argentina ‘cause it’s just – it’s like a continuation of a presentation I made last year in Athens when I tried to present the – how our IXP was founded and etcetera. So this is just a moment in the life of our IXP situation. I just want to share with you just in case you may transit a similar situation. And fourth, it’s incredible but Mike just covered in five minutes five years of many discussions that we have been having in our IXPs. So it was pretty good.
So a short introduction of our institution. Our IXP was founded in 1996/97 by 18 participants, 18 ISP companies and after a year we started our operations. We had to transit through a big process to find an operator and to get all the stuff, all the equipment in place. So after a year we started, we began our operations. Now a day currently we have in 2007 more than 40 participants and that is just like a short snapshot. The characteristics of our IXP, it’s a not for profit private sector association.
It’s the ISP and eCommerce and other stuff association in our country. It is not regulated. This means that we do not have the government regulating our activities, but we ourselves developing all the policy process. It is carrier neutral since it is built in the same facilities as the ISP Association. So it’s for every one of us. Its co-operative governance and cost recovery fee structure. This is very interesting. We have a model we’ve been working for long months to develop a cost recovery fee structure, so as to distribute all the expenses into the members.
Not equally but in terms of space, in terms of bandwidths, in terms of if anyone of the players of the participants that were connecting transports to the exchange, other parties. So it’s a little complex but it survived quite a lot of time. During this multi lateral mandatory international traffic filtering and bilateral peering are disallowed and bilateral peering is disallowed because if there’s bilateral peering we are not able to see what kind of traffic is going through that pipe. A very important issue here is that filtering was disallowed and now I will tell you why it is important.
Main drivers for the foundation of our IXP, some of the major participant – oh oh I skipped, sorry. Previous please, no previous, previous yeah yeah that’s it. Main drivers, promotion of local content and eCommerce development. The situation before our IXP — ask almost any IXP — is we had to exchange traffic in the United States before the creation of our IXP. So all the content of our country was hosted outside and not in our territory. So in the creation of the IXP we were bringing a lot of content back to Argentina.
Transferred cost reduction and IP routing performance enhancement for obvious reasons. How are we organised? We have monthly meetings, we have two committees. One in charge of the development of policies and analysing projects and the other is our technical working group that works obviously in the technical detail of our IXP.
So how to become a participant, and here we have a similar situation as Kenya. You have to be a member of the association. You have to hold a telecommunications license. So this is for operators, for ISPs, for carriers and you have to hold an ASN. If you don’t match with this criteria, there’s a special evaluation for participants to become members of the NAP.
We have the equivalent of the IRS connecting to a NAP because we all thought it was very important to have them peering with the ISPs. But after that we have the requirements of banks that wanted to connect to our IXP and so we said no, because we don’t want our IXP to compete with its participants. So this is one of our main rules. Okay. So what’s the theme today of my presentation? Here’s the scenario of what we encountered a couple of years ago. Struggling struggling okay. So thanks.
Some of the major participants de-peered. If they de-peered it was going to be fine for us, I mean no problem with that but the problem was they strangled their bandwidth. So as I told you before to remember that we were not allowed to do any filtering. The problem was that we have BGP sessions up but with bandwidth strangled. So that was a very very serious situation.
There’s still no regulation to protect internet service in these cases and there was no government intervention in the past and in the present also. And ourselves, our lack of appropriate self developed policies in place, so as to avoid these kind of situations. So which were the problems we encounter and which were the consequences of this? Quality of service was seriously affected, decrease of IXP overall traffic, remaining participants, lack of motivation.
Some participants decided directly to disconnect of the IXP, so we got as a result – as a consequence higher costs to access to the peer networks. Before the situation there was almost no cost because just the expenses of the IXP. After the situation when these guys were disconnected, the fee was $350 per MG. Let’s put this in a frame, $350 for a MG for just routing domestic bandwidth. And international, the price market of price referential market for a Meg for international bandwidth those days were $200 for using 10,000 kilometres of sub-c fibre optic rings.
So some players decided to use international pipes to reach domestic networks. So how did we try to recover? I don’t know if we recovered at all, but we’re better than we were in the past. We just set up a looking glass for trouble shooting, for diagnostic purpose. We started with our IPB6 peering service. We implemented DNS route servers. There was a lot of information previous to this workshop about this, so we implemented that route, thanks to (43:31) and BISC. We implemented .com and .net resolution.
The very sign we’re implementing now, see a bunch of ccTLDs resolution, thanks to Bill Woodcock and his team. We are in the way of implementing an NTP quality of service warranties, so as not to repeat the same mistakes as in the past by peering which is in process and traffic flows analysis, plus turning policies more flexible. In this sense for some time we had a limited time offer, no connection charge, so as to bring more participants to our IXP. International traffic routing, we’re thinking of suppressing that barrier and also same thing of course with bilateral agreements.
I don’t know where to point. So which are the results of these issues? More participants currently joining our IXP. There’s a lot of interest for connecting to our NAP. We improved routing performance because we are reaching networks that were customers of these guys that left NAP. Traffic is growing, we are reaching one gig of traffic so we have lower costs, we have more motivation and we are developing new projects.
So conclusions, and I’m finishing. In the life of an IXP such as ours, negative situations may arise seriously affecting internet services in industry, finance, government, the users, research and education. Whatever sector, it affects us all. One way of minimising this impact is to bring as much traffic as possible by having your participants, your services etcetera.
And although not handling 100 per cent of local networks because that’s quite impossible, our (46:14) are still a cost effective solution for ISPs and they contribute to in country internet development. So, and what about regulation or intervention? It may be positive in some cases. We really needed to have probably a government intervention in the past to help us solve this situation, but we were lacking of any intervention. So it may be positive to help us in the near future, so as to prevent negative (46:52) that may affect internet stability. So that’s all for now.
Roque Gagliano: Okay thank you. My name is Roque Gagliano from Latin America and I’m going to talk about the Latin American and Caribbean exchange points. So we’re going to go from the particular case of the Argentine exchange point to a summary of what’s going on in the region. Okay. So an outline of what I’ve been talking about, I’m going to sum it up and show you the location of the different IX in the region. What kind of role, and that’s something that I’ve been asked by the organiser of the meeting – and I want to thank ISOC too for inviting me to this meeting – was to point out what kind of role the government is taking in different countries which is different in the region. Then we’re going to talk about what kind of challenges new IXs are facing and some initiatives and activities are taking place in the region and finally some thought looking forward. So in total regard, depending how you count – around 20, 21 IXs in the region.
Some of them we don’t really know exactly the operation standards, but they are – those 21 are basically in 12 countries and we’re going to show them there’s a large amount of IXs in Brazil for example. So we do have countries with more than one IX. The pioneer work started in 1998 with Panama and Argentina and Brazil and then it’s growing the number of IXs and many countries are willing to set up their own.
Most of them just deal with local and metropolitan traffic and that’s something we’re going to address to that. There’s some revelation in some IXs that do not allow regional traffic to go through them. And IX models that we see, many of them are ISP associations and as we just saw in the Argentinean case that we can see that in several countries. Also we have commercial IXs which are basically run by companies. Okay next.
This is a pretty picture we did for the last project, so we can see that 21 IXs that I was mentioning from the Cuban in the north to the Argentinean and Chilean in the south. A particular comment about the Chilean IXs actually is showing us one IX, but actually in Chile there’s around six what is called BIT or internet point of – actually can be translated literally to internet exchange point.
And there’s a regulatory framework in Chile where those IXPs are requested by the authority to be interconnected. So even though that connection is only on what we call the level three connection, if you as a customer connected to any one of those IXPs by the authority you would be shown in any other. Okay. So you only need to be connected to one of them. So we can see on this plaque that Brazil has nine IXPs and the reason for that is the project that is taking place at the CGI which is what’s called the PTT project.
Basically they’re setting up a new IXP in every major city in Brazil. So what kind of service have we find at different IXPs in the Latin American region? Well normally what we find is layer two IXP or typical level two of IXP. We got a route server for multi lateral agreement. In some of them this multi lateral agreement is mandatory and in others it’s just voluntary. We’ve got a looking glass service, we got some traffic measurement tools and quality of service checking. The (51:24) server, ccTLD hosting, gTLD hosting, NTP servers and others. We decide the typical one.
So we can see how the internet exchange point as a meeting place helps the hosting of some critical infrastructure we’ve been talking about during this meeting and so they’re very useful infrastructures. The role of the different governments in the Latin American region, we can split in different categories. We’ve got an example in Chile of very high government regulation and even though not an expert in that, what basically the regulation says that the local traffic in Chile has to stay local and that translates that every single ISP has to be connected to an IXP. And those IXP are forced to be connected between themselves. And not only that but they have to publicly show the plot and the statistics of the connection. They have to show the traffic, the latency and the packet loss. Then we get to Brazil where there’s a government involvement we can say because the CGI is financing and involving the government of this project are called PTT, where they’re setting up IXPs in different cities.
So we can say it’s not a regulation issue but it’s definitely an involvement. And then we got other experiences which is similar to what’s called in Kenya when the government is participating on the IXP, but not as a regulator but as a member. And the government adds, as any other customer can ask for the same number, and participate as a member. So one of the issues that I mentioned was the same regulation of the IX in the region. As I say most of the IXPs where many of them were setted up as part of – as an ISP association.
So we can see that several IXPs in the region, they have several regulation policies and most of the policy what they involve, for example multi lateral agreements or control on the routes that they are allowed a quality of services, licenses, association fees etcetera. Can you go back? So one of the questions the ISOC people asked me was, what kind of challenges people that are trying to set it up new IXPs are facing?
So what I did is I went back to the main reason we have and I’m going to show that later and ask the people okay, what kind of challenges you have? And this is summary of the response I got. The first one is to try to get a critical market. Trying to get people on board. To try to convince people that this is a good idea, this is something that is needed. And particularly in many places what they try to get the big players on board. And that’s not always necessary in order to set up your IXP.
The second issue that was mentioned was if the problem of finding a neutral location which is appropriate to set up an IXP. And I’m talking there a location with power, air conditioning etcetera. Finally also there was mention of transit cost. The cost of getting traffic through. For example your transit in Miami is getting lower and lower.
So again the IXPs had to compete with that and also has to compete with private peering that happens between especially bigger players. So I want to mention now some of the initiatives I have in the region and the first of all is the NAPLA mailing list and forum. NAPLA basically it’s a forum where all the IXs of the region, we get together and once a year and co host it with a (55:58) medium. Once a year we get together and discuss IX issues and try to get some knowledge discussions.
Also we do have a mailing list where IX issues in the region are discussed and you can find them – the URL where you can join the mailing list and our next meeting’s going to be in Southern Brazil. We’re trying to get more people involved so you are all invited to our meeting. Also there’s several government forums where the IX issue has been raised for example, we are participating the eLAC 2007 and also in the (56:43) for example, these issues mentioned. Okay.
So looking forward – so still the biggest recurrent course of our network are the international links. IXPs help reduce this cost by keeping local traffic local. The increasing of the traffic is ultimately the driver for the IX, and traffic is increasing, our broadband is getting wider everywhere. And that also increases our inter regional traffic because we share the same culture. People in my country watch the same TV as in my neighbouring country etcetera. So if we want to follow this pattern with more and more inter regional connection, IXs are going to be fundamental.
So if we got to mention that IXs as a meeting point to get everybody together we can say that as it was borne in the access forum, then we can say that’s supply and demand meets. So strong and neutral IX (57:52) are going to be really dreamers to host critical infrastructure and also very fundamentally to bring contour provided to the region. Okay and that’s everything, thank you.
Sam: Thank you very much for that.
(General discussion about microphones).
Sam: Well it’s okay anyway because Bill’s the next speaker. I just wanted to give Bill about five minutes to comment and particularly if he thinks this experience is pretty much a global experience to mention the measurement project and any other quick comments you’d like to make and then I’ll throw it open to question and answers.
Bill Woodcock: And are there any terms of what we know at this point?
Sam: Well we’ll wait and see I think.
Bill: So I think by and large what you heard was reasonably representative. The Latin American environment is a little bit different than most because there are fewer participants and its a little bit more (59:03) than most of the rest of the world. But aside from that I think you saw firmly representative things there. Mike’s comments particularly I think generalised or were a good generalised description of the issues.
One thing that the other panellists I think were very politic or diplomatic about which when these things go wrong I’m often the one that has to go deal with it. So I’m going to be a little bit more blunt about this one. There are occasionally really well meaning, well intentioned naive people who think that interconnecting IXPs with each other is going to be a good idea. Like if one is good and two are better than one really big one, if you jam those two together we even better.
But in fact this is not the case, in fact if you do that it puts a growth cap on all the ISPs that connect to any of them because it removes their growth model right. It removes the business model for them to grow beyond connecting to a single exchange point, which is how ISPs go to become multi national an eventually global. So it’s really really important that you not try to force IXPs to interconnect with each other.
Like for instance (1:00:25) they did because this guarantees that (1:00:26) and ISPs will grow to be regional beyond that scope or if they’re to do so they’ll lose money doing so. So to move on from that what you see on the screen here is live webpage of PCH’s website. This is a view of our database of exchange points sorted by region here. You can see 17 exchange points in Africa and then down at the bottom here the Asia Pacific region starts with 63 and another little bit of diplomacy there was the question of whether 20 or 21 exchange points in Latin America, whether you count Cuba or not.
So basically the statistics here, number of participants and the amount of traffic for each one. You see there question marks in some of the spots there. The reason for those question marks is because we don’t have live data coming out of those exchange points and that’s something that Sam’s urging and that of the urging of the FCC International Bureau. We’ve worked with IXP operators and with IXP operations associations to put together a global best practices document for IXP documentation.
That is if you’re running an internet exchange point, what should you be publishing in terms of documentation about the exchange point? And it’s really a pretty simple document it says you should publish how many participants and who they are. You should publish how much traffic is going through the exchange point but not on a per user basis, just the aggregate. And you should document who has what IP addresses in the exchange point.
So between these three things these are enough for us to start to get a handle on how much internet traffic there is globally. That’s a number that we haven’t had since the NSF net backbone was taken apart in 1992 and 1993. So governments know that the internet is a critical resource but they have no handle on how big it is or how critical it is or how much Chilean traffic goes to Argentina versus how much to Peru for instance.
These are all things that government, industry, NGOs, grant making organisations, development aid agencies, would all love to know because it would help them quantify their efforts. So that’s something that Sam is largely responsible for pushing us to do and it’s something that we’ve been working with IX operators to make it more complete over the course of this year. So we hope in another year or so to have that much more complete, get rid of a lot of those question marks.
So anybody who’s interested in exchange points is welcome to go to our website www.pch.net/ixpdir exchange point directory and there’s drill down from all of these. You can go into each of these and see who the participants are and what the traffic’s been like historically and so on and so forth. So a lot of information there and we urge people to participate in the documentation.
Sam: Excellent. Time for question and answers and thank you for your patience. Gentleman here, could you please identify yourself. Perhaps you come here or we have a roving mike? Okay. That’s all right, that one’s working.
Radesh: Good evening, I’m Radesh, I’m from India, President ISPI Association of India, as well as the Director in National Internet Exchange of India. My question to the panel is that you’re talking of the regional connectivity between the exchanges. How successful it is and how they are going to compete with their national long distance player of that country. Like into Latin America you are saying that a lot of countries are connecting to have a lot of internet exchange into their respective city. Whether the countries are being connected internet each of it each other are the regional connectivity’s there?
Sam: Two volunteers. Who’d like to volunteer for that? Roque.
Gabriel: Two.
Bill: Three.
Sam: Okay.
Gabriel: I will keep talking about Argentina because on my left hand I have the person who will speak about that in America. Our IXP in Argentina is not connected to any other in Latin America and the answer is what you just said. I mean it’s – that will compete with its participants. So my point of view lets – I mean IXPs in Argentina or Latin America – well lets keep talking about Argentina, is to resolve traffic in Argentina and also to facilitate traffic from abroad, but not for interconnecting NAPs. That’s my point of view.
Roque: Oh hi. Well my – when I was talking and the original traffic’s increasing I was not mentioning any kind of regional interconnection IX or something similar. I was just saying that there’s sometimes some regulation that makes difficult for ISP from a different country to connect to several IX which are not from their home country. So what I mean is we see what’s going on right now, most of the traffic in different countries are just local routes.
We are going to see from now on more and more regional routes originated from interregional players, and that’s going to make sense for them. So multi lateral -mandatory multi lateral agreements. That kind of set regulation is against, or there’s no enhancement the participation and the meeting of all this regional traffic in the exchange point. I don’t know, so what I see is going to happen, and many of these regulation may soften in order to increase the participation of other players.
And we’re going to see more and more regional traffic on those fabrics by not connecting the fabrics. Okay. The only – today what we’ve seen is that in those IXs there’s just local traffic or metropolitan traffic. There’s one case also in Ecuador and the same organisation has two IXs in two different cities. And that’s something they’ve been discussing, about what they’re going to do between the two cities. And again the problem they’re having is that interconnecting those two fabric means competing against their own members, which are the ISPs.
So yeah, when I mentioned an increase of regional traffic, I meant that more and more – there was going to be more and more regional interest in being in those fabrics. Surely this mandatory agreement is not what those players are willing to deal with.
Bill: Yes, so the previous two speakers had it right. I will again be a little bit more blunt. What you mentioned is exactly the problem, I just described. (1:09:01) is the reason why there was a four year period which India had no growth in the internet sector whatsoever, because (1:09:01) was trying to inter connect exchange points and was trying to make the exchange points mandatory and was trying to run transit through the exchange points and was essentially re-creating the SNL.
So there was no growth in India until that stopped and the ISPs were able to start growing again, because again the business of ISPs is hauling traffic between exchange points or between a customer and an exchange point and another customer right. So if you create some other entity that glues exchange points together it removes the growth path and the business model for ISPs and there’s no way for them to grow or make money.
Sam: Next question? Sir, and then yes – this gentleman first and then this gentleman.
Male Speaker: Actually I have a couple of questions so.
Nichelle: Thank you both. You sort of stole part of what my question was going to be because in the presentations that we’ve heard, we’ve heard the ISP operators say that you’ve been forced multi lateral peering. And my first reasonable question was going to be, do you think that’s a good idea and do you plan on keeping that for the life of your – for your IXP?
The second question I have and this is a more general one and anybody can feel free to answer it is the issue about? I mean you mentioned international traffic. But I’d like to extend that concept of international traffic to transit traffic, where transit traffic could be through a region, it could be through a country, it could be through anything else that you want. Is there a model for building that?
Is that something that you would suggest to somebody starting up an ISP? Oh and my name is Nichelle from Internet Solutions in South Africa.
Sam: Michuki.
Michuki: Okay. So to answer your first question. I would like to enforce personally, I would like to enforce multi lateral peering, but also leave room for private peering.
Sam: Bilateral.
Michuki: Oh bilateral peering or private peering depending on the way you choose to look at it. So currently are the (1:11:27) internet exchange point, it relaxes the fact that by default you have to do a multi lateral peering. But that does not mean that if you have private arraignments or bilateral arraignments that you cannot go ahead with them, but still you have to do the multi lateral.
Again, the moment you have too much of private peering or bilateral peering then in my opinion or are they now peering? We see the value of the exchange point is actually brought down, especially for the smaller players who actually carry the majority of the traffic. It’s not necessarily all the time but the larger players are the ones with larger traffic. They may have the more custom ones but if you put together all the small players together, the aggregated traffic is probably worth more than the larger players.
So it will lose value for the IXP as a whole and then you will see they established – they’re moving on to maybe a much more acceptable better exchange point that actually adds value to them by allowing them to peer generally with everyone else.
Male Speaker: Regarding this…
Sam: It’s okay.
Male Speaker: I need the microphone because I would address the question to Kenya. When a provider wants to peer just bilateral and not multi lateral to prevent them to be a member of the NAP, I mean are they obligated to do both, to also be multi lateral?
Michuki: No. We – by default you have to be a multi lateral. So we’ve not had a request where has come and said I do not want to peer multi lateral, I want to do bilateral. So they understand the policy that when they come in, they have to do multi lateral and if they choose to peer with others for other reasons, let’s say transit for instance. And they want to get transit through a bilateral peering agreement, then that is allowed. But by default you have to announce your preferences to everyone else.
Male Speaker: You cannot connect only to the bilateral fabric?
Michuki: Nobody has asked yet.
Bill: Nobody has asked yet?
Michuki: Nobody has asked yet.
Male Speaker: What would be the answer if a big provider wants to connect only, because this can be a solution for cases like the one described for Argentina when the big providers are not interested on peering with all the small ones. So the second question would be what do you think, I just need to peer for example. What do you think about regulating too much the IXP and making a lot of restrictions on the way the (1:14:29) is advertised, brought in the filtering on the receiving preferences obligating to make the announcements of all the preferences and things like that?
Bill: In general I would say that no regulation on what it is that ISPs are required to do at an exchange point or sorry, participants because I don’t think you should require participants to be an ISP right as we’ve seen. I don’t think you want any requirements on them. They self govern very very well in this regard. The one exception to that is that I think it is reasonable for a national regulator to require that carriers within their country have some means of delivering traffic within the country.
That is requiring carriers to keep domestic traffic within the borders of the country, not dictating that they use an exchange point because they could go and buy service from another ISP to achieve the same end and that’s just as good. The difference that you and Michuki were talking about is between multi lateral peering and mandatory multi lateral peering.
And I would say that mandatory multi lateral peering, that is trying to compel large providers to enter into an open ended contract with unknown other signatories is not likely to fly with any large carrier. And I think those of you who’ve worked for large carriers know what trying to get that past your legal department would look like. So effectively what that does is it precludes for participation of any large pre (1:16:02). It’s a feel good thing for small IXs that still have just small participants, but it doesn’t work to compel anyone.
Roque: I also about multi lateral, mandatory multi lateral is that I was – I’ve been talking in the last year that I’ve been working as a moderator of the Napla mailing list with several content providers interested in coming to the region and the peering co-ordinator told me that they don’t like that. And they – it actually prevented them from connecting to the IX access. So I think it’s not necessary if a big transit provider for example, you prevent them from connecting to the IX because you don’t – you force them to do mandatory multi lateral.
That’s not necessarily good for the small players because in several countries the access network to get the last mile is very expensive too. And I remember when I was in the bush and they were talking the Nigerian case and the IX as a meeting place and the fabric as a meeting place were I can choose the access that I get. But I can get access to one place and I can choose between three or four transit providers just by BGP configuration and he knows and I can change. Now it’s going to make my transit much lower in price because there’s going to be competition.
And I cannot do that through mandatory multi lateral agreement. I can do that with bilateral agreement. So if I choose the location of the exchange point to be neutral and to have relatively cheap access in the local loop that’s going to be very good for small players.
Male Speaker: My last question would be…
Sam: Very quickly ‘cause we’ve got about five minutes left.
Male Speaker: Are there any plans to improve the regulation in the NAP for Argentina?
Gabriel: Oh we’re just thinking about that. The plans are – we currently have those plans but that’s the cliché, I mean it’s almost confirmed that we will have international traffic. But the key issue now is we don’t know yet if multi lateral peering will be mandatory or not. And let me ask Bill one question. Why do you think ISPs would want – ISPs that are currently peering in the IXP, would want mandatory multi lateral peering so as to implement also to have also bilateral peering in the IXP? Why would…
Sam: And why would Bill’s…
Gabriel: I mean what’s the reason why because probably myself and also Christian, we do not have that very clear.
Sam: Let’s take this question and then is there any other businessman who has a burning question? Okay. Let’s take this question, then I’ll start with Bill and then a final comment from all the panellists.
Gabriel: Okay thank you. I have one comment and one question. The comment is that I think that’s while the multi lateral, bilateral IXPs are preferable. So it has a withering pact in the – because in the region and not only in a single country. I think that it is important to remark that existence of an IXPs very important, even if the IXP is evasive and multi lateral limit. So I think that its not – we cannot fold in the position of criticising very much the model of the IXP, because the XPs very important anyway.
I think one thing that is very important in the IXP and probably most important and the fact if it is based on multi lateral or bilateral (1:20:25) is the openness of the IXP. Because us, we have in the same way that we have IXPs in which the big ISPs have left because they don’t find it very interesting. We have also some other experience in which we have some experience in which only the big ISPs participate. And so it’s also and they have some barriers like the fees or sometimes the memberships closer.
That’s one point and so one question we asked for velocity. You say that in Kenya, I think then I understood that the government mandate or ask the ISPs to keep their local traffic – domestic traffic within the country I understood. You said Bill but you are referring now a different country?
Sam: I’ll get Michuki to take that in his final comment if that’s okay?
Bill: I was saying that that is a good – I was saying that that is a good model for national regulation and there are many countries that have done that.
Gabriel: Okay. So I…
Bill: I would be not pressed to give you that.
Gabriel: …you were not speaking about any country in this specifically?
Bill: No. I’d have to go and look.
Gabriel: Okay thank you thank you, for the verification. But in that case I don’t think that it would be a regulation with (1:21:56). So in fact I think that it should bring them – oh I don’t like – what I’m saying is that I don’t like this regulation because it can give the government some power in the country of the traffic for deciding which traffic should not remain within the (1:22:24). Thank you.
Sam: Thanks very much. So in the final comments perhaps Bill you could adjust Gabriel’s question. I also have one final question. I note there’s one OECD country on the map of Latin America that doesn’t have an IXP. There’s one OECD member country that doesn’t have an IXP and that was Mexico.
And perhaps Roque or someone who’s free to be blunt could comment on that and see if there’s something that we should be highlighting, because if there is no need, if that’s the reason that’s fine and we’re very happy. But let’s go in order down, we’ll start with Bill and then we’ll go right down to Mike for a final comment.
Bill: So to address the question of why it is that many – well not many, some small IXs with small participants like the idea of mandatory multi lateral. It’s because the participants are small and they have never had a legal department to worry about and they’ve never had the constraints on their operation that a large corporation would.
They don’t have insight into the mindset of a big phone company and so they believe that they can compel the big phone company to do what they want by requiring them to sign an agreement. And in the fact of the matter the big company simply won’t sign the agreement. They won’t sign an agreement that compels them to do something they don’t want to do.
Michuki: Okay. I think there was a question we just would do if somebody came to KXB and asked I wanted to bilateral. And I think that’s one bridge which have not been very – I have said I will want to cross when I get there, but the chances are that we’ll probably let them peer on a bilateral with whoever they agree to peer with.
I would be probably very insistent that they try and announce their own preferences cause then most of the time when you have people who own bilateral is because of transit purposes more than providing more transit. But when someone just says their content is expensive, so they actually want to do bilateral.
Gabriel: Well yes, so how would I enforce that? So we do know what the preferences are so we’ll insist to see them on the looking glass that’s – we can see their preferences. So that’s a way of going about it and then say if anything in transit you can actually go into bilateral peering. That’s one approach I would look at but in essence for content providers who are large enough then we’ll say all my preferences actually is value, I wont’ do bilateral. Then at that point in time we’ll look at that differently.
Sam: Roque.
Roque: Well it’s the last comment just to mention that again exchange points a meeting point where and also as an enabling place where a lot of things can happen where a country can then decided to host critical infrastructure or has a place where to put it and not even think about it. A place where content providers then for example as I mentioned in yes, the section; 15 per cent of traffic is Google, YouTube. So when they come you know where to connect.
A place where even you can buy cheaper perhaps transit, it’s a good idea and perhaps something anybody who want to take home and think about it and there’s a lot of material on the web. We have a mailing list for the Latin American region but there’s also a mailing list for the African region that perhaps and I don’t know if you had an exact link for that. But there’s several several mailing lists about exchange points and people are really willing to help anybody who has any idea on want to start setting up and are share bonding their countries.
Sam: Oh okay sorry.
Roque: I’m not really familiar with the Mexican situation but I can tell you that what I heard and what has been said is that very very strong private peering between the bigger players and bigger players, they are really really big. But and the other thing is, is there’s a very huge exchange point in Dallas and Dallas basically is where most of them get together and fibre to Dallas is not so expensive as it could be for anybody souther than Mexico. So I would say that probably Dallas is the Mexican exchange point.
Sam: Thank you. Gabriel.
Gabriel: Yep.
Sam: No no.
Gabriel: Just to wrap up the conclusions or what?
Sam: Just a final comment and then…
Gabriel: Okay yeah, yeah.
Sam: Just to finish that.
Gabriel: I think well almost everything has been said. I express our very important internet assets for any country. I also think that there’s maturity process that probably an IXP can begin by be regulated, by be multi lateral, mandatory or whatever. And in the process of developing the IXP in the life of the XPs and during some – any situations that can facilitate to mature and to develop new issues that may be beneficial for all the internet community in that country.
Sam: Thank you. Mike, any final comment?
Mike: Just a brief one to reinforce what Gabriel said. I think internet exchange points are going to grow in importance as we see more and more movement towards packet based – our service providers in the voice sphere as well and we move away from circuits switched networks. So internet exchange points are going to really become a really critical internet resource and we should do whatever we can to make sure they flourish in every metropolitan area nevertheless.
Sam: And my take away message would be to underscore that and the challenges to build awareness of the benefits really, because that’s one of the main barriers I think to getting IXPs to those 90 odd places that don’t have them. And perhaps some of those places that do have them making them work more efficiently. So thank you to The Internet Society, thank you to the panel and could you join me in thanking the panel and thank you for your patience.
End of Meeting.
Venue:
Carnegie Mellon University in Qatar
Education City
PO Box 24866
Doha, Qatar