Hackers Can Slip Invisible Malware into ‘Bare Metal’ Cloud Computers
Credit to Author: Andy Greenberg| Date: Tue, 26 Feb 2019 15:57:40 +0000
The security world's paranoiacs have long cautioned that if a computer falls into a stranger's hands, it shouldn't be trusted again. Now one company's researchers have demonstrated how, in some cases, that maxim applies just as strongly to a class of machine that never touches your hands in the first place: cloud servers.
On Tuesday, researchers at the security firm Eclypsium published the results of an experiment in which they showed that they could, for a certain class of cloud computing servers, pull off an insidious trick: They can rent a server from a cloud computing provider—they focused on IBM in their testing—and alter its firmware, hiding changes to its code that live on even after they stop renting it and another customer rents the same machine. And while they made only benign changes to the IBM servers' firmware in their demonstration, they warn that the same technique could be used to plant malware in servers' hidden code that persists undetected even after someone else takes over the machine, allowing the hacker to spy on the server, alter its data, or destroy it at will.
"When organizations use public cloud infrastructure, they’re essentially borrowing equipment, like buying it used off of Ebay, and it can be pre-infected before they start using it," says Yuriy Bulygin, Eclypsium's founder and a former head of Intel's advanced threat research team. "In a similar way, that equipment can be infected if the cloud service provider hasn't sanitized all its equipment at the deepest level, including the firmware."
That cloud sanitization problem, Eclypsium's researchers were clear to point out, doesn't effect all cloud servers. A typical cloud computing setup generates every customer's computer as a so-called "virtual machine," a kind of sealed aquarium within the computer isolated from the server's actual hardware and other customers' virtual machines on the same box. But everyone from Amazon to Oracle to Rackspace also offers so-called bare metal servers, in which a customer rents and fully control an entire computer in an attempt to improve performance or, ironically, security. IBM has thousands of enterprise customers who use bare metal machines for everything from video conference hosting to mobile payments to neurological stimulation treatments.
By renting a machine in a bare metal set-up, an attacker can get far more dangerous levels of access to components that can carry malware on to that server's next renter. "The problem is definitely worse and much easier to exploit on bare metal services," Bulygin.
Hackers, both in research and real-world intrusions, have for years demonstrated that the firmware in little-considered chips that control everything from USB drives to hard drives can provide a hidden foothold for malicious code. Those infections can evade all antivirus, and even survive a complete wipe of a computer's storage.
In their experiments, Eclypsium's researchers homed in on the firmware of a powerful component in the Super Micro servers, which IBM offers customers of its bare metal cloud computing service, known as a baseboard management controller. The BMC is used to remotely monitor and administrate the server, and it's capable of everything from accessing the computer's memory to altering its operating system. In previous research, Eclypsium has even demonstrated that a corrupted BMC can be used to rewrite the firmware of other components, bricking computers or paralyzing them for a potential ransomware attack.
"Once the firmware is infected, there’s really no way to know if it is still infected or to recover from it."
Karsten Nohl, Security Research Labs
In their experiments, Eclypsium's researchers would rent an IBM bare metal cloud server, and then make a harmless alteration to its BMC's firmware, simply changing one bit in its code. Then they'd stop renting the server, releasing it back into IBM's pool of available machines for other customers. A few hours later, they'd rent enough servers to find the same exact machine again, identifying it by the serial number of its motherboard and other unique identifiers. They found that despite the fact that they were supposedly being handed a "fresh" machine, the BMC firmware alteration remained.
"The infection of the firmware is persistent, it’s not reimaged when you reimage the whole software stack," Bulygin says. And although the researchers made only a benign change, they say it would be easy enough to hide truly malicious firmware with the same trick.
In response to Eclypsium's research, IBM posted a statement downplaying the vulnerability as "low severity," but promising that it now carefully wipes its servers' BMC firmware between different customers' uses: "IBM has responded to this vulnerability by forcing all BMCs, including those that are already reporting up-to-date firmware, to be reflashed with factory firmware before they are re-provisioned to other customers," the statement reads. "All logs in the BMC firmware are erased and all passwords to the BMC firmware are regenerated."
As of Monday night, Eclypsium's researchers said they could still perform their catch-and-release trick, implying that IBM's fix wasn't in place yet. But an IBM spokesperson told WIRED that a "fix has been implemented and we are working through the backlog."
Even so, other firmware-focused researchers are skeptical of both IBM's "low severity" label on the vulnerability and its supposed fix. Karsten Nohl, who developed the so-called BadUSB attack that invisibly alters the firmware of USB sticks, pointed out that BMC firmware could be altered to both offer hackers control and to "lie" to administrators when they try to reflash it—essentially telling the update mechanism that it's been updated without removing the hackers' code. "Once the firmware is infected, there’s really no way to know if it is still infected or to recover from it," Nohl says. Another well-known firmware hacker, H.D. Moore, argues that only adding a piece of hardware to the server to check the firmware's integrity would fully solve the problem.
IBM, for its part, didn't respond to a question from WIRED about the difficulty of trusting firmware updates. And since Eclypsium only tested IBM's bare metal offerings, it's not clear if the same firmware issue applies to other companies as well.
The good news, Nohl argues, is that the bare metal servers are only a small minority of cloud setups, and that virtualized servers would be far harder to attack with the firmware trick. But that's little comfort to anyone using those vulnerable setups now. "It's a niche. But niche or not, it doesn’t matter," Nohl says. "Even for a niche, it’s a very relevant attack. And there’s no easy way to prevent it."