Our digital lives are awash with dynamic, rich and ground-breaking services; many of which we could not have imagined it possible to consume, buy or view just a few years ago. At Equinix, we see first-hand that private interconnection is a foundational piece of the modern plumbing that supports this technological age and a key enabler of many of these new services. The fundamental reason for this, is that the private interconnection gets removed from the public internet and falls under the direct control of the provider and consumer. Unfortunately, not all digital services can be supported with a private end-2-end model in this way. What are those use cases and why does private interconnection not help?
Private interconnection is great, but not applicable to all use cases. Take for example a Massively Multi-Player Online Game (MMPOG) that has a following across Eastern Europe and Russia. Our fictional game is played in real-time by thousands of users, geographically distributed across a large region. Content Distribution Networks (CDN) can help, but ultimately, we need to rely on the local Internet Service Providers (ISP) or ‘eyeball networks’ across many countries to deliver the content to the users. In summary, if the point of consumption is a highly distributed group of users, private interconnection cannot support the entire scope of service delivery. With private interconnection removed from the toolbox, the heavy-lifting of content distribution for these services is therefore fixed to the same internet architecture that we have been living and working with for the last 25 years. With the popularity of gaming, streaming services, eCommerce and critical apps needed by home workers exploding in recent years, many are now asking: Is the core architecture of the internet fit for purpose?’ The answer is no.
Before we look at one of the key challenges facing the core of the internet, let’s have a recap:
In the diagram above, an end user is attached to ISP A and content is hosted within ISP B. In the early days of the public internet, traffic between the two networks would go through at least 1 of the so-called Core/Tier-1 internet backbone providers. Very quickly, the ISP communities in each country happened upon the idea of direct connectivity between their networks. This was and still is through a) direct Private Network Interconnects (PNI) or b) Internet Exchange Platforms. This concept of direct peering offered significant benefits of cost reduction, security and performance and has thrived over the intervening years. However, the internet is a global domain and inevitably users in one country will request access to content stored in another. This is illustrated in the diagram above through the user connected to ISP C. He or she may also need to access the content within ISP B, but because a direct connection does not exist, the traffic hops through a series of networks until it reaches the destination. These connections, whether they are one or many hops, form the fabric of the internet. Each individual network domain uses an Autonomous System Number (ASN e.g.: AS1) to identify itself and exchanges routing information through Border Gateway Protocol (BGP). Thus, in a highly simplified form, you have the workings of the internet core.
Lack of visibility and control has always been the challenge with such an architecture:
- Path lengths can change,
- Congestion that you are not aware of can occur upstream
- Security incidents may interrupt normal operation in a partner network
In previous years, the impact of this was more limited as the internet was treated as a best effort delivery tool. Now however, the requirements are different, and the stakes are much higher. The public internet must be robust, secure and highly available. Thankfully, the last 18 months has seen a concerted effort to fix things and a lot of the efforts have been in one area: internet security.
In the internet model, each network advertises ranges of IP addresses that are assigned to users and content. In the diagram above, ISP C (AS7) trusts that the BGP advertisement received from AS6 is accurate and sends traffic destined for those IP addresses to AS6. What happens if AS6 has made an honest mistake and advertises large IP address ranges that do not belong to them? In that case, traffic can erroneously be diverted to AS6 with potentially huge service disruption. Just 2 months ago we saw a major example of such case scenario when a BGP misconfiguration at a large public company caused a series of outages. The same result can occur when an incorrect BGP route advertisement is knowingly made. This is called a BGP Hijack. Many of the BGP Hijack occurrences are connected to advanced cyber-criminals acting for commercial, political or other reasons.
Thankfully, the industry began to act, initially through the MANRS (Mutually Agreed Norms for Routing Security) organisation. By 2018 the need for change was clear and by the end of that year most of the largest Internet Exchange Providers and other ISP had signed up to the MANRS charter to implement and promote ‘internet security best-practice’. One of the main drivers was to have the industry look again at validating BGP route advertisements before they are accepted, as the acceptance of invalid or incorrect routes is the fundamental flaw that is being exposed. IP address validation had existed to some degree in various industry databases known as the Internet Routing Registries (IRRs), where an IP route is associated with the ASN that will announce it. Whilst useful, the IRR model on its own is not sufficient. The records are sometimes incorrect or missing and are not supported by any form of cryptographic signing. Something else is needed – enter Resource Public Key Infrastructure (RPKI).
RPKI offers owners of IP addresses a means of crypto signing a record held in the IRR through the creation of a Route Origin Authorisation (ROA). This ROA is in turn held by the local Certificate Authority, known in this case as Trust Anchors (TA). If the reader would like further details on RPKI and the implications of its use – a great starting point is this 2018 blog by Martin Levy from Cloudflare. RPKI is slowly being implemented across the internet community and offers huge benefits to those who rely on the internet for crucial services. As a user (ISP or Enterprise), what can you do?
If you are an ISP, there are some common BGP Security Best Common Practices that can be followed:
- Implement a clear routing policy that ensures correctness of BGP announcements
- Operate a robust method of validating IP announcements that are received (eg IRR and RPKI)
- Deploy security controls to prevent activity such as IP Spoofing
- Ensure that you work closely with your router vendor or systems integrator to make sure you are deploying correct software versions and security features for the internet edge
- Facilitate an open communication and co-ordination between Operators. Keep your details up to date in all public registries and be available to your peers and the community
- Be the agent of change in your organisation – raise the subject, ask the questions – what are we doing, how are we protected, what can we do better?
- Press your Internet Exchange Providers for their position on RPKI and other security models
If you are a customer of an ISP then you are relying on them as your provider to be the custodian of your business on the internet. The same questions are valid: “As an enterprise that leverages the internet for services delivery, how are you as my ISP helping to ensure maximum availability for my traffic. How are you working with your partners to support the stability of the internet and protect traffic, users and applications?”
The internet is not perfect, but as new services come on stream, we need it to be secure and reliable. The current initiatives in areas of internet security are a great first step and will have a lasting positive impact on users. Let’s get them implemented!
Learn how Equinix enables networks, content providers and large enterprises to exchange internet traffic through the largest global peering solution across more than 30 markets. Download the Equinix Internet Exchange Data Sheet.