The 3 L’s of Gaming – Location, Location & …. Latency !!!

Those who work within the technology and internet space have surely noticed the unstoppable rise and march of gaming in recent years. Sophistication, detail and player engagement have reached levels unimagined by those who grew up with Pac-Man, Pong or even the more recent Mario and Sonic adventures. The latest information from Fortune predicts that in 2019, an estimated 2.5 billion gamers will generate US$152 billion in overall global revenue for the gaming sector—a number that exceeds global film and music revenues combined.[1] On the back of such growth, the industry is going through another evolution with the arrival of cloud gaming, a topic already covered by our colleagues Diego Julidori & Jim Poole in their recent blog posts.

In those two posts, the authors outlined the recent changes to the gaming world and the importance of both interconnection and content placement to the new models. Using their work as a starting point, let’s take a closer look at some of the mechanics that are at work. Even though the gaming industry can win by adopting interconnection and proximity as key pillars to success, there are still some challenges to overcome in providing the universal experience that is required. The good news is that based on our position at the digital edge, Equinix is very well positioned to work in partnership with this sector to deliver the infrastructure and service layers that are needed.

A reminder: major shifts impacting gaming

The previous posts discussed three major trends impacting gaming today:

• First, we are moving to a cloud delivery model. For gaming, this means streaming content delivered to any device that can leverage a browser or thin client. Cloud economics will begin to drive even greater innovation—and most importantly, barriers to participation will continue to fall. With friction reduced, playing a game will be as easy as starting a video on YouTube; something Google Stadia is counting on.
• Next, games will evolve further to incorporate augmented reality and virtual reality and deliver more immersive experiences where appropriate.
• Finally, as shown in the chart below, the rise of eSports will add further to the ecosystem. VentureBeat has forecasted that the global eSports audience will reach 600 million in the next two years.[2]

Growth and more traffic are on the way due to these three factors and others not called out here. What does this growth mean for the infrastructure and service layer?

Supporting the best user experience

It’s understood that a robust and resilient internet is needed to ensure maximum experience for internet users and gamers. Network uptime is key. While most of us experience a very reliable internet service, in recent years the industry has seen a large increase in some types of network instability. In particular, outages occur in the internet core, caused by BGP route leaks and BGP hijacks. Gaming is a key use case affected by such incidents. We examined this in detail in a previous post and looked at the various mitigations that are being deployed. So security is key to supporting the future of gaming. What other factors are involved?

The previously mentioned posts looked at the importance of content placement and interconnection. Gaming organizations and their partner CDN providers know that to create proximity between users and content, you need to deploy a distributed edge and peer as much as possible on services such as Platform Equinix® and the Equinix Internet Exchange™. While this strategy is sound, sometimes further analysis is needed to support deployment decisions. Why? It turns out that a straight line is not always the fastest way to a destination, as the internet still employs “scenic routing” when connecting two points on a map. In other words, getting your content close to eyeballs is not always a matter of deploying POP and IX services that are in the closest geographical hub to your targets. At Equinix we have been working closely with some of our largest gaming customers and some of our industry partners to dig deeper into these vagaries.

Scenic internet routing

Acceptable latency for gamers is dependent on game type, but according to recent testing by Richard Leadbetter from Digital Foundry, a target for total interface/keyboard-to-pixel delay of about 166 ms is a good start. Within that 166 ms we have some unavoidable delay, and that leaves around 76 ms for the actual network delivery (RTT).[3]

If you are a committed gamer in a market such as London, Singapore or New York, your experience is likely consistent, smooth and of high quality. However, the gaming sector has several peculiarities, one of which is the location of hotspots. Because the demographic of gamers is skewed away from the normal population, we see pockets of great gaming consumption in atypical locations. These are major markets for the largest gaming organizations, and they currently experience significantly poor performance due to excessive latency. Two of the largest gaming organizations in the world are also Equinix customers, and were sufficiently concerned that they suggested we join forces and analyze the situation.

How do gaming organizations typically address these secondary markets? While you may not be able to deploy in all locations, the common model is to get your compute as close as possible to these markets. Geography is often the metric that organizations use. A few examples:

• There are major gaming markets in Western and Central Russia. If you are not deploying in Russia, then from a geographic perspective, Stockholm is a good candidate for an edge POP.
• Tel Aviv is a major market. Istanbul is close and in theory a good location for an edge POP.
• Kuwait City is a major market. Dubai is the regional interconnection hub and a logical choice for an edge POP if you are not in Kuwait.
• Finally, in LATAM, Lima is a big market. Sao Paulo is the major hub in the region and usually where organizations will establish a POP first with a goal of serving the remainder of the region from there.

This approach is the one to take. It builds on the idea of creating proximity and is in line with the concept of a distributed digital edge that was mentioned in previous posts. Unfortunately, “scenic routing” can disrupt even the best laid plans, and in our research with our gaming customers we have pinpointed some of the most worrisome cases, as illustrated below.

In order to complete the testing, we worked with toolsets that include the RIPE Atlas probes and also network analysis and BGP AS-Path tools from Oracle Dyn. We specifically tested the latency toward users situated on two ISPs in the destination city. Remember, the target network latency is 75-80 ms or less. The results are in:

• We tested two major ISPs in Russia. For one, the result and latency was always good (46 ms). For another, traffic from Stockholm to Russia almost always hairpinned through Amsterdam and AMS-IX. This more than doubles the overall RTT latency and takes it to the point of unacceptable latency. Not a good start, and not good news for those gaming organizations that elected to serve Russia out of Stockholm.
• For one major ISP in Tel Aviv, traffic from Istanbul intermittently headed to northwest Europe, again transiting AMS-IX. This more than doubles the network route, interrupting quality to a significant degree.
• Dubai is only 1,760 km from Kuwait City and should be a great location to serve its neighbor. Unfortunately, when accessing one particular major ISP in Kuwait from Dubai, 50% of the time traffic hairpinned to India for a total route of 18,800 km and latency of around 140 ms—well outside the required margin.
• But spare a thought for our friends in Lima who are served by one of the country’s largest providers. Almost half of the traffic emanating from Sao Paulo has its network route tripled to 34,000 km by way of an extremely scenic route that takes in Miami and Texas. Network latency of 221 ms will lead to lots of frustrated users!

How can the industry respond?

The solution to these issues must start with the network and peering community.

• First, peer more; maximize your existing edge nodes and leverage multiple internet exchanges and more private network interconnects (PNI).
• Next, in emerging locations, we must work together to demand and institute more local peering ecosystems that do away with these hairpin models.
• Industry players should make a cooperative effort to achieve the greatest levels of interconnection.  Employing tactics to limit competitors’ access have detrimental long-term impacts and outcomes to emerging regions and cities, and contribute to such sub-optimal network routing.
• Finally, more research is needed to establish a baseline of the current predominant routing topologies so that action can be taken. Equinix is already working with major content and gaming partners in this area and we will continue these efforts.

Cooperation will solve the current challenges. Gaming will continue to grow and reach new markets. Interconnection and openness hold the key.

[1] Jonathan Vanian, “Cloud Gaming Is Big Tech’s New Street Fight,” Fortune, June 19, 2019.

[2]Andrew Paradise, “The Rise of eSports as a Spectator Phenomenon,” VentureBeat, November 30, 2018.

[3]Richard Leadbetter, Google Stadia Specs Analysis & Exclusive Performance Testing, Digital Foundry YouTube Channel, March 19, 2019.