Meltdown and Spectre Patches From Intel and Others Have a Rough Start
Credit to Author: Lily Hay Newman| Date: Tue, 23 Jan 2018 21:44:59 +0000
The Meltdown and Spectre vulnerabilities, first revealed at the beginning of the year, affect pretty much anything with a chip in it. That ubiquity has made the process of releasing patches understandably arduous. Every type of impacted hardware and software requires its own specially tailored solution, and even a fix that works as intended may slow down system processes as a side effect. The bigger issue so far, though, is that some patches have done more harm than good, requiring recalls and sowing general confusion.
A lot of the focus has fallen on Intel, because all of the company's modern chips are impacted, and the company's attempts to patch the vulnerabilities have seen mixed results. Intel shares the hot seat, though, with fellow chipmakers ARM and AMD. Operating system developers including Microsoft, Apple, and the Linux Group have also been on the hook for providing patches. These fixes, though, can inadvertently cause serious problems beyond processing slowdowns, including random restarts, and even the blue screen of death. Spectre in particular is also more of a class of vulnerability than one easily resolvable bug, so it's proven especially difficult to create one-size-fits-all patches for the flaw.
"We've never seen such an expansive bug like this that impacts literally every major processor," says David Kennedy, the CEO of TrustedSec, which does penetration testing and security consulting for corporations. "I was on at least 10 calls last week with big companies and two yesterday explaining what's happening. They have no idea what to do when it comes to patching. It's really causing a mess."
It doesn't help that processor companies downplayed the challenges at first.
Intel memorably said in its first statement about Meltdown and Spectre that, "any performance impacts are workload-dependent, and, for the average computer user, should not be significant and will be mitigated over time." Sounds great, right? In practice, Intel has had to repeatedly step on this initial nonchalance, revealing that its newer processors are also susceptible to patch-related slowdowns, and that it pushed out some patches too soon. On Monday, Intel retracted one of its Spectre patches because of random reboot issues, and suggested that system administrators roll it back or skip it if they haven't installed it already. "I apologize for any disruption this change in guidance may cause," Intel executive vice president Neil Shenoy said in a statement.
'All of this is pure garbage.'
Linux Creator Linus Torvalds
Intel's problems have trickled down to other manufacturers and developers as well. For example, the cloud infrastructure company VMWare said on Thursday that it would delay microcode—fundamental code that coordinates between hardware and low-level software—updates because of problems with Intel's firmware patches. Similarly, Lenovo announced last week that it had to withdraw some of the firmware patches it had issued because of stability concerns. Dell joined the fray, pulling certain Spectre firmware patches on Monday. "If you have already deployed the BIOS update, in order to avoid unpredictable system behavior, you can revert back to a previous BIOS version," Dell said in an update to customers.
Linux creator Linus Torvalds criticized Intel's patches for the Linux kernel in a public message board on Sunday. "All of this is pure garbage," Torvalds wrote. "The patches are COMPLETE AND UTTER GARBAGE. … They do things that do not make sense." (Emphasis his.)
Microsoft, too, has gradually admitted to more vulnerability-related Windows slowdowns. The company also had to pause distribution of its Meltdown and Spectre patches for certain AMD processors two weeks ago, because the updates were causing fatal errors in some machines. For its part, Apple recently had to walk back some of its claims about protections for older operating system versions. On Tuesday, the company released various combinations of Meltdown and Spectre patches for High Sierra, Sierra, and El Capitan.
Some chipmakers that initially stayed quiet eventually admitted that Meltdown and Spectre left at least some of their processors exposed. AMD, for example, originally said in a statement on January 3 that, “Due to differences in AMD’s architecture, we believe there is a near zero risk to AMD processors at this time,” but the company was forced to revise its assessment a day later, admitting that many of its chips are impacted. Similarly, Qualcomm didn't confirm that its chips were affected until days after the public Meltdown/Spectre disclosure.
The open-source enterprise IT services group Red Hat knew about Meltdown and Spectre as part of industry collaboration before the public disclosure, and the company worked ahead on developing and testing patches. But on Thursday it, too, withdrew certain Spectre patches based on Intel's microcode updates, "due to instabilities introduced that are causing customer systems to not boot."
"It’s very frustrating for our customers when services say 'well we have a fix for X chip and Y chip, but not A, B, or C chip'," says Christopher Robinson, who runs Red Hat's product security program management team. "So we want to make sure that we can have a consistent answer… At some point in the future we’ll revisit rereleasing this software, but right now it’s just too much in flux."
Though other critical and ubiquitous vulnerabilities have certainly required massive coordinated response over the years, the mitigation efforts for Meltdown and Spectre are unprecedented in just how many devices, users, and organizations are involved. Developing stable patches for every processor, every firmware stack, and every operating system adds up to a tall order. While Meltdown has been a fairly straightforward bug to patch, Spectre mitigation requires more sweeping, conceptual changes in how processors manage data flows, making it more likely that early versions of proposed fixes will have problems.
But in many cases, initial attempts at damage control may have done more harm than good, by downplaying the risks of patching issues. Meltdown and Spectre are critical enough vulnerabilities that they certainly needed to be patched quickly, even if this meant moving forward with imperfect fixes. But as chipmakers and other developers attempted to save face and calm investors, misplaced optimism may have ultimately misled customers about how much patch testing to do, and what to rush to apply.
Now both individuals and organizations continue to struggle with understanding whether they have the right updates installed to actually protect their systems without causing more problems. "This has probably been some of the most confusion I've ever seen on an exposure," TrustedSec's Kennedy says. "It wasn't well coordinated."
Read the inside story of how for teams of security researchers discovered Meltdown and Spectre independently—all within a few months of each other
Meltdown and Spectre are as devastating as they are complicated. Here's how they work, and why they're such a menace.
Fortunately, some important steps have already been taken to fix the problem—but a full solution is still years away.