Paranormal activity in data centers is assumed to be folklore, invented by weary late-night technicians and spread on Internet Relay Chat (IRC) channels by DevOps engineers anxious to explain common outages. But with more than 220 data centers around the world, we’ve got a unique vantage point, and this kind of weird activity is certainly on the rise. In particular, we’ve noticed a disturbing increase in organic behavior across large clustered systems.
“That’s right,” said Peter Venkman, Jr., Senior VP of Forensic Services at Ghostbusters, Ltd. “You get a bunch of computers clustered together, throw in a bit of Terraform and there’s no telling what havoc will ensue. It’s like they are programmed to create mischief as soon as nobody is watching!”
After reviewing thousands of tickets from distraught customers — usually arriving in the middle of the night, during events like Black Friday or code releases — here are some of the cluster glitches recently uncovered by the Platform Equinix operations team.
The cloud edit Cluster-Geist
After watching too many reality organizational shows on Netflix, a Cluster-Geist in an autonomous multimedia cluster on a hybrid multicloud infrastructure started its own “cloud edit.” It began tidying up by analyzing every virtual partition in the cluster, asking the other cluster nodes, “Does this instance give you joy?” Overwhelmed by the flood of application workloads, the Geist frantically started containerizing everything, tagging them with exquisite calligraphy and organizing them by the colors of the rainbow. Weighed down by data gravity, the Cluster-Geist held a “last chance” data egress sale to purge old data from its storage pools. Exhausted by all the merging and purging, the Cluster-Geist retreated to its quorum, likely never to be seen or heard from again.
The DDoS Cluster-Geist
After a number of distributed denial of service (DDoS) attacks on its supercomputing cluster in a university think tank, a DDoS Cluster-Geist trolled the perimeter along its network firewall waiting for malicious packets to arrive. As soon as a mob of suspicious packets started to gather outside the firewall, the Geist emerged, and shouted, “If you want to pass by me, you must answer these questions three!” The packets snickered, challenging the Geist. “Bring it on,” they chanted.
The Cluster-Geist proceeded with the first question. “What is your header?” The botnet zombie army laughed at him, and one by one stated their header addresses. “Come on,” one of them yelled. “Make this worth our while.” “Question 2,” sneered the Geist, “What is your favorite protocol?” “Really?” they replied, “You don’t recognize an IP ping of death attack when you see one?” The Cluster-Geist smiled and launched his third and final question, “What is the estimated global installed interconnection bandwidth capacity of the Government and Education industry in 2023, as projected by the fourth annual Global Interconnection Index?” The packets were stymied. Not being able to answer the question, they were dropped into the dark fiber ether and were lost forever.
The bare metal Cluster-Geist
A SaaS company deployed on our new Equinix Metal service recently started noticing strange behavior appearing on its Grafana monitoring dashboards. A spike in traffic was followed by an alarming percentage of dropped packets. With alerts firing, the PagerDuty calling tree was activated and began waking up the Site Reliability Engineering (SRE) team from their slumber. The bare metal Cluster-Geist, which had been pestering the production cluster for weeks, had struck again!
After quickly logging in via remote console using a laptop placed next to her bed, the lead engineer, Jillian Holtzmann, was able to catch the Cluster-Geist in action. With a wicked smile on its face, the Geist was busy changing the internal domain name server (DNS) records within the cluster in an attempt to confuse the SRE team. Finally, caught red-handed!
Unsurprisingly, the Geist was proud of its work, and was wearing a hoodie with the following haiku emblazoned on its front:
What can customers do to protect themselves?
While the jury is still out on paranormal activity in data centers, there are a few precautions every customer should consider taking:
- Invest in Immutable Deploys – Nothing keeps your clusters safer like wiping out an entire production deployment every so often and starting from scratch. With our new fully automated Equinix Metal service, this is as simple as a few commands. It just might be time to get on the cloud native bandwagon!
- Wiggle All the Cables – Sometimes a Cluster-Geist will try to fool a technician by loosening random cables instead of swapping them. A simple fix is to wiggle each cable before locking your colocation cage or cabinet. What could go wrong?!
- And remember, never deploy on Fridays!
These Cluster-Geist computer glitches are stranger than truth, because in fact, they are figments of our collective imagination. But with advances in artificial intelligence and machine learning ecosystems on Platform Equinix, who knows what’s possible!
Have a safe and healthy Halloween!
Subscribe to the Equinix Blog