France-IX – Lessons learnt

The French context was very specific. We had many IXPs in France when we decided to launch France-IX in 2010. So, before starting it was decided to run an enquiry to make sure there was a real demand for a change. Most of IXPs were run by operators (which may represent an issue regarding neutrality and independence), and were based on free business-model (scalability issue): Paris was not seen as a strategic peering place, as the peering market was very fragmented. The consequence was that many local players were peering abroad on wider and more efficient IXPs and content providers didn’t come to Paris to exchange traffic.

The market study shown that there was a global agreement within the French Internet community to have a federative IXP in France. A workgroup has been conducted to foster the change. Then France-IX was born.

Neutrality and independence

Since the 1st day, it was very important to make sure we would run an independent and neutral organization. So that it was decided to setup a dual core entity: an association “France IX” and a private company “France IX Services”, with the association as the unique shareholder of the private company. We could have run either the association or the private company but it was decided to run both: customers sign a contract with the private company and at the same time they become members of the association with one right of vote for annual meetings (to vote for closure of accounts, elect board members…), and the private company brings more flexibility for operations, commercial aspects, marketing…

To guaranty neutrality, the rules voted and approved are the same for all the members, independently of the size of the member.

Importance of board and board members

The founding members did constitute the first board of directors, to make sure that the founding members were able to help and bring either human or financial resources to France-IX. Some founding members also made equipment donations to help France-IX to start and avoid spending too much money. To maintain independence, it was decided to reimburse by the time the board members who had decided to support financially France-IX when it started.

After 3 years, it was planned to renew the board by third, with members being able to apply for a board seat: to avoid any dispute, the board members to be renewed have been designated by a draw. The very first board was composed of 6 seats, only with people representing their company, and since the board was upgraded to 8 seats, with still 6 seats for companies and 2 seats for individuals.

Partnerships

It was very important to consolidate the French peering market and solve the issue of the fragmented market. It was decided to setup partnerships with some other IXPs: partnerships have been established then with other French IXPs or IXPs very close to France (setting up cross-border interconnections).

Large footprint

When France-IX was launched, it was decided to setup directly 6 PoPs around Paris, to make sure this footprint would maximize our capability to collect the existing networks. We decided to run directly the PoPs interconnections using dark fibres and then integrate progressively WDM (using DWDM passive equipment) for scalability. Choice of passive WDM was particularly pertinent (versus active DWDM equipment) as we were running a metropolitan infrastructure where the distance between PoPs is less than 30km usually. It was very cost efficient.

Building a backbone and added-value Services

An Ethernet layer2 infrastructure was built initially, and it evolved to a VPLS backbone in order to deploy easily new services.

We decided to start with basic services (Unicast IPv4/IPv6 services, NOC support), to avoid complexity at the very beginning, but as soon as we grew it was important to be able to provide additional added-value services such as routes servers (to ease peering for new members), develop a nice an friendly portal for members and build an information system, make sure to ease access to DNS resources (national DNS extensions, TLD’s, DNS root servers), provide private VLANs (to allow private interconnects), provide a looking-glass, setup a reseller program… Now, after 5 years, we are about to release 100Gbps ports offers, start to provide SLAs…

Operating an IXP

We use to hear that it is easy to build an IXP from the technical side. It is partly true, but when you grow (by adding additional PoPs) engineering may become more complex, and your members will ask for more professional services.

Nonetheless, if you want to operate in an efficient way an IXP, it is important to define some rules/procedures to be able to grow fast and smartly:

  • Of course, when running services, you need to have a real NOC, available 24/7. In our case, we decided to outsource level-1 support and keep internally level-2 support (based on very skilled engineers having experience on WAN operators backbones or previous IXPs).
  • Pre-cable positions between the customer patch-panel and the switching platform in order to save time and avoid DC human works.
  • Automate configurations whenever possible.
  • Define/write procedures to connect members and apply it. In our case, we defined a very clear procedure allowing a new comer in the technical team to connect a member easily.
  • Define the Network Technical Specifications and share them with the customers. This will permit to have clean configuration on customer side and will facilitate the operation of the network.
  • Create a knowledge database and share it with the connected members.

Keep in mind that an IXP can grow very quickly and the time you invest initially is time you will save in the future.

Cooperation and community

When you run an IXP, you must always keep in mind 2 things: you can get support from the community (useful to start with), and when you grew it may be appreciated that you share your experience. We decided to join Euro-IX association, which allow us both to get support and share at the same time. However, as soon as the IXP is growing the best-effort model for human resources is not scalable. In the case of France-IX, at the very beginning human resources were brought by founding members and we were also getting some help from the community. Now we have a strong technical and commercial team, and all the people are employed by France-IX. However, the number of employees is not directly linked to the number of members connected to the IXP (even if of course when you have more members it does generate more work): the staff growth is also driven by the diversity of services we propose by the time.

Community is very important. When running an IXP, you must listen to your community and meet the people also. That’s why we participate in many events: any NOGs meetings (FrNOG, MENOG, NANOG), but also peering events like EPF, or events directly concerning IXPs (AfPIF, Euro-IX), global community events (Africa Internet Summit, Afrinic, RIPE…), and we also run our own events.

As mentioned above, we are now in a phase of sharing our experience: that’s why we are happy to contribute in project like AXIS (with great cooperation with ISOC) to help African countries (especially French speaking countries) to get their own national IXP.

Designing the network

The very first France-IX design was a pure layer-2 backbone, very simple. Since came a second round when we decided to deploy new PoPs and to integrate core PoPs to have a dual-core networks to optimize the mesh (and the cost of backbone links) and still propose a resilient infrastructure. During this second phase we also generalized the use of passive DWDM equipment to bring scalability for interconnecting PoPs. The very last phase we deployed in summer 2014 consisted in introducing a third core PoP and make sure then that all others PoPs were connected with fully redundant dark fibres paths to 2 of the 3 core PoPs and to generalize VPLS everywhere. We also did select equipment bringing much more density for 10Gbps ports and allowing 100 Gbps connections.

Last time we decided to renew the full backbone to have equipment with much more port density and also integrate more flexibility within the network by generalizing VPLS, we did setup a lab which did simulate the whole new core network.

However running a Proof of Concept before deployment does not prevent from bugs or outages, even by generating some traffic during a large period of time. So it is very important in our case to get a direct and efficient support from the equipment vendor (avoiding intermediaries), as we are concentrating a very large numbers of connected networks. In such context, when we run our tender to select the equipment vendor, a focus was not only done on equipment features but also on support.

In order to optimize equipment life duration, when we renew equipment from core PoPs, we globally try to re-use them on edge PoPs or keep them for brand new PoPs (and then we upgrade the equipment when a new PoP grows and does reach a critical mass).

We also decided to deploy our latest backbone based on 2 vendors. For the choice of optical modules we also work with 2 vendors.

At the very beginning of France-IX, we were pretty flexible regarding connections of new members. However flexibility may please a new member but it does not scale. Now, we apply very strict engineering rules (allow only single MAC address, filtering rules, check for unauthorized protocols…), but to make it work it is important to publish them so that everyone knows the rules. Strict rules could displease some members, but it’s the way to ensure the stability of the IXP when you have hundreds of connected customers.

Transparency

The IXP community covers a large amount of Internet players. It is important to be transparent with this community providing information about how France-IX works on the technical side. For example, a network weathermap and traffic statistics per PoP are available on our website. We also send to our members (trough a dedicated members mailing-list) a quarterly report detailing the main actions undertaken during last months.

How to make sure the IXP will last?

There are key points to be matched, to make sure your IXP will last:

  • Organization
    • Keep and guarantee neutrality and independence
    • Keep diversity of representatives and members type within the board (CDNs, operators, Carrier Neutral datacenters, etc)
    • Remain transparent about the way the IXP is run and managed (including accounts presentation to members)
  • Services
    • Provide a professional service as soon as the IXP is increasing (and not only a best-effort service), with a real 24/7 NOC
    • Listen to your members and their expectations, but also be proactive to propose new services
  • Business-model
    • Avoid free models (scalability issue)
    • Apply fair and transparent prices (public prices)
    • Keep in touch with the market and the real players
    • Identity new players/prospects
    • Integrate a marketing/communication process (public relations, website, social networks, events…)
  • Think about extensions
    • Check about possible partnerships with other IXPs
    • Development of new PoPs
    • Check about cables infrastructures (especially submarines cables when there are)
    • Develop reseller program to bring more members

Future of France-IX

France-IX is now 5 years old (we did celebrate our 5th year a few weeks ago). When France-IX was launched the idea was to have a strong footprint on the 1st day (by starting with 6 PoPs) and to gather French Internet community, and of course bring back France as a visible international peering place. Since, we have more than 275 connected members, around 400 Gbps of traffic, 10 PoPs in Paris and 2 PoPs in Marseille.

Significant developments have been done on France-IX backbone infrastructure during the last few months, in order to prepare for important growth of traffic and number of members. During summer, we will insert 100 Gbps ports linecards in several of our chassis. This offer will be available on core PoPs in Paris, as well as in Marseille. Paris and Marseille will continue to be run as 2 differentiated IXPs, even if a gateway is available for small members who need to peer on both by running one single connection. For larger members they will need to go through one of our resellers if they want to get connectivity to Paris when connected in Marseille, or even simpler connect directly to both Paris and Marseille.

We are also working on new services, like SLAs, market place… to make sure we match our community expectations. They will be available by the end of the year. Of course we will develop new services by keeping fair pricing, because of the model of our organization (not for profit association owning a private company).

We also keep in mind to either develop additional PoPs in other French regions, or establish partnerships with new IXPs in France when developed outside Paris or Marseille, as our goal is of course to continue gathering French community.

We have received requests to setup PoPs outside France. However, we also keep in mind the reason why France-IX was created, and the aim is not to develop everywhere in the world just because some others are doing it. By default, we prefer to work in a cooperation mode, in which we help and support other IXPs initiative and potentially partner with them. Even when setting up new partnerships we also consider the mutual benefits rather than only looking for new commercial opportunities. Access to French contents and partnering with French speaking countries is also a key point in our strategy. We receive many requests for partnering or even setup interconnection with us, but the point is not to collect as many partnerships or interconnections as possible. You need to have a global strategy and serious reason to do it.

Meet France-IX

France-IX will be present in Mozambique for AfPIF meeting, and will be one of the sponsors of the event. This is a great event to share experience and talk to other people who are willing to setup IXPs in Africa. Our team will be happy to meet you there!

More information about next AfPIF meeting

Additionally, France-IX is organizing its annual meeting in September in Paris. Anyone interested in France-IX can attend the meeting.

More information about this meeting and registrations