Meet Cliff Stoll, the Mad Scientist Who Invented the Art of Hunting Hackers
Credit to Author: Andy Greenberg| Date: Wed, 18 Dec 2019 12:00:00 +0000
Thirty years ago, Cliff Stoll published The Cuckoo's Egg, a book about his cat-and-mouse game with a KGB-sponsored hacker. Today, the internet is a far darker place—and Stoll has become a cybersecurity icon.
In 1986, Cliff Stoll’s boss at Lawrence Berkeley National Labs tasked him with getting to the bottom of a 75-cent accounting discrepancy in the lab’s computer network, which was rented out to remote users by the minute. Stoll, 36, investigated the source of that minuscule anomaly, pulling on it like a loose thread until it led to a shocking culprit: a hacker in the system.
Stoll then spent the next year of his life following that hacker’s footprints across the lab’s network and the nascent internet. In doing so, he revealed a vast web of similar intrusions into military and government agencies carried out by a group of young German hackers, eventually revealed to have been working in the service of the Soviet KGB. The story that Stoll unraveled from that tiny initial clue, which he published in late 1989 as a kind of digital detective memoir, The Cuckoo’s Egg, turned out to be the very first known case of state-sponsored hacking—a tale far bigger than he could have ever imagined when he began hunting those three quarters missing from his lab’s ledger.
Today, that story has taken on a larger life still. As The Cuckoo’s Egg hits its 30th anniversary, the book has sold more than 1 million copies. And for a smaller core of cybersecurity practitioners within that massive readership, it’s become a kind of legend: the ur-narrative of a lone hacker hunter, a text that has inspired an entire generation of network defenders chasing their own anomalies through a vastly larger, infinitely more malicious internet.
Stoll asks people who have interviewed him to sign his personal copy of The Cuckoo’s Egg.
As for 69-year-old Stoll himself, he talks about the entire series of events as if he still can’t believe all the fuss he’s caused. “I thought it was a weird, bizarre hiccup I’d stumbled into,” Stoll told me when we first spoke last year, after I called the home number he lists on the very eclectic website for his business selling klein bottles—blown-glass oddities that, topologically speaking, have only one side, with no inside or outside. “I had no idea this would become a multibillion-dollar industry. Or essential to running a large business. Or that the CEO of a credit reporting company could lose his job because of computer security. Or that thousands of people would have careers in the field. Or that national institutions in many countries around the world would devote themselves to exploiting security holes in computer networks.”
In fact, Stoll is an unlikely legend for his cybersecurity industry admirers. On the day I visited Stoll in his Oakland home last month, just a few days after the 30th anniversary of The Cuckoo's Egg’s publication, he had spent the morning watching Mercury transit the Sun with his telescope. Stoll has a PhD in planetary astronomy and had intended to make stargazing his career before Lawrence Berkeley transferred him—not entirely voluntarily—into the IT department.
When I arrive, he takes me to his workshop in the back of the house, a room with one wall covered in printed pictures of inventors, mathematicians, and scientists who inspire him: Felix Klein, Alan Turing, Emmy Noether. Then he flips up his desk on a hinge to reveal a door in the wall beneath it.
Inside is a small, homemade forklift robot, which lives in the crawlspace beneath his house. Using a remote control and watching several screens that show a feed from the robot's cameras, he wheels his little bot across the cramped storage space under his home, its walls lined with cardboard boxes, to delicately retrieve a crate full of beautifully crafted klein bottles wrapped in paper.
Stoll is still curious about hacking too. A couple of months earlier, he mentions, he decided on a lark to reverse-engineer some hackers’ malware-laced Excel file to see where it hid its malicious code. “I said to myself ‘Oh, here’s how they’re hiding it.’ It was very sweet and a useful lesson,” Stoll says, sitting on the floor of his workshop next to his forklift bot. “Having said that, I’m not very interested in cybersecurity today. I wish I was more interested. I wish I could help people defend their systems. Instead, I went back to figuring out how to make a klein bottle that can sit without wobbling.”
Royalties from The Cuckoo's Egg paid off Stoll’s mortgage years ago. Today, klein bottles sales provide him another—very modest—income stream. As for cybersecurity, beyond a few conference talks, he hasn’t worked in the industry for decades. The same omnivorous curiosity that drove him to chase his hacker for a year eventually led him to devote the next 30 to his other interests like mathematics, electronic music, and physics—none of which he claims to be an expert in. “To a mathematician, I’m a pretty good physicist,” Stoll deadpans. “To a physicist, I’m a fairly good computer maven. To real computer jocks, they know me as somebody who’s a good writer. To people who know how to write … I’m a really good mathematician!”
“To a mathematician, I’m a pretty good physicist,” Stoll says.
"To people who know how to write," he says, "I’m a really good mathematician!”
But if Stoll is a cybersecurity amateur, few experts have had as much influence on the field. Stoll’s fans in the industry point out how, in hunting his hacker 30 years ago, he pioneered techniques out of necessity that would later become standard practice. Stoll slept under his desk at the lab and programmed his pager to alert him when the hacker logged into the network in the middle of the night. He also set up dozens of printers to transcribe every keystroke the hacker typed in real time. All of that added up to something like the first intrusion detection system.
When Stoll traced the hacker’s intrusions to the Department of Defense’s MILNET systems, an Alabama army base, the White Sands Missile Range, Navy shipyards, Air Force bases, NASA’s Jet Propulsion Laboratory, defense contractors, and the CIA, Stoll was mapping out an intrusion campaign just as threat intelligence analysts do today.
When he planted hundreds of fake secret military documents on his network that tricked his hacker into staying logged into the Lawrence Berkeley system long enough for a German telecom employee to trace the intrusion to the hacker’s location in Hanover, he was building a “honeypot”—the same sort of decoy regularly used to track and analyze modern hackers and botnets.
“The Cuckoo's Egg documented so many of the methods we now use to deal with high-end intruders,” says Richard Bejtlich, a well-known security guru and author of The Tao of Network Security Monitoring: Beyond Intrusion Detection, who has worked on incident response and network monitoring at companies like Corelight and FireEye. “You can see in the book almost everything you need to do in an incident. The mindset, the thoroughness, the commitment to it. It’s all there.”
Even before his book was published, Stoll’s hacker-tracking work at Lawrence Berkeley National Labs inspired its sister institution, Lawrence Livermore National Labs, to try to develop more systematic, automated defenses against hackers. An engineer there, Todd Heberlein, was given a grant to build the world’s first network security monitoring software. “You could literally say that Cliff Stoll kick-started the entire intrusion detection field. We essentially automated in software much of what Stoll was doing,” Heberlein says. “Once I had our tools turned on, we saw people every day trying to hack our network and sometimes succeeding. An entire crime wave was happening and no one was aware of it.”
Eventually a version of Heberlein’s network monitoring software was deployed to more than 100 Air Force networks, including the ones Richard Bejtlich found himself working on during his time in the military in the late 1990s. As a high school student, Bejtlich had been captivated by a paperback copy of The Cuckoo's Egg, and he reread it during that time in the Air Force. “Every element of what Stoll did, we were doing,” he recalls.
Around 2010, when he was working as director of incident response for General Electric, Bejtlich says he read it again, and found dozens more lessons for his team. He’d later pull them together for a talk about those lessons, "Cooking the Cuckoo's Egg,” that he gave at a Department of Justice cybersecurity conference.
Just as much as its technical lessons, The Cuckoo’s Egg captures a deeply personal side of the job of hacker tracking too. The long hours, friction with bosses, federal agents who demand to be briefed on discoveries without sharing their own information, and tensions with loved ones—Stoll’s then-girlfriend (now ex-wife) didn’t always appreciate his nights sleeping under his desk to hunt an invisible white whale. “There are still incident responders who sleep under desks and are awoken at weird times. You’re at the mercy of the intruder,” Bejtlich says. “Anyone who has done this can relate to being away from the family and working crazy hours. it’s completely familiar even 30 years later.”
But there’s a thrilling side to Stoll's story as well: an ideal for aspiring network defenders, many of whom hope to someday find themselves the protagonist in a detective story like the one Stoll wrote about. “People who get into cybersecurity dream they’ll work on something like this,” says Chris Sanders, a security consultant who created a course based on The Cuckoo's Egg called "The Cuckoo's Egg Decompiled." “They imagine finding the thing that becomes the bigger thing. We all want to live that. Some live it and some don’t. But we all get to live it vicariously through Cliff.”
Stoll makes and sells blown glass klein bottles that, topologically speaking, have only one side, with no inside or outside.
That fantasy version of Cliff Stoll is hard to make out in the mad scientist, klein bottle-selling Cliff Stoll of today. But, it turns out, underneath 30 years of layered polymath whimsy, the obsessed hacker hunter is still there.
After he finishes giving me a tour of his workshop, Stoll sits me down in his cluttered dining room lined with books, including a full 20-volume set of the Oxford English Dictionary, one of the first things he says he bought with his Cuckoo's Egg advance. He starts reminiscing, telling a story about his hacker hunting that isn’t in the book.
After Stoll helped German police trace the Lawrence Berkeley National Lab’s hacker to an address in Hanover, they arrested the intruder—a young man named Markus Hess. The police found that Hess, along with four other hackers, had together decided to sell their stolen secrets to the Soviets.
What he didn’t mention in the book is that he later met Hess in person. When Stoll was called to the German town of Celle near Hanover to serve as an expert witness in the case, as he tells it, he ran into Hess in the courthouse bathroom, coming face to face with the hacker he’d chased online for a year. Hess recognized Stoll, and began asking him in English why he had so doggedly pursued him. “Do you know what you’re doing to me?” Hess asked, according to Stoll’s 30-year-old memories. “You’re going to get me sent to prison!”
Stoll says he simply told Hess, “You don't understand,” walked out of the bathroom, and testified against him. (That telling of events couldn’t be confirmed with Hess, who has no contact information available online and hasn't commented publicly on The Cuckoo's Egg in decades. Even Hans Hübner, one of Hess’ co-conspirators at the time, told me he had no idea about how to reach him. Hübner also noted that his own primary motivation in hacking had always been exploration and technical discovery, not Russian money. He believes Hess, who was given a 20-month suspended sentence for his intrusions, likely felt the same.)
At this point in the story, Stoll becomes silent and his face twists into a pained expression. Slowly, I realize that he’s angry. Then Stoll tells me what he really wanted to tell Hess: “If you’re so smart, if you’re so brilliant, make something that will make the internet a better place! Find out what’s wrong and make it better! Don’t go screwing with information that belongs to innocent people!” Stoll says.
He startles me by pounding his fist on his dining room table. “Don’t think you’re licensed to break into computers because you’re clever. No! You have a responsibility to those who have built those systems, those who maintain those networks, who built the delicate software. You have a responsibility to your colleagues like me to behave ethically.”
This is the other ingredient to Stoll’s hacker-hunting obsession, and the same drive in so many others in the cybersecurity world who followed him—not just curiosity, but a kind of low-burning moral outrage. For Stoll, it seems to stem from a time few other internet users remember, a time before the World Wide Web even existed and when most denizens of the internet were idealistic academics and scientists like him. Before the hackers—or, at least, the criminal and state-sponsored ones—arrived.
“I remember when the internet was innocent, when it crossed political boundaries without a care, when it was a sandbox for intellectually happy people,” Stoll had told me in our first phone call. “Boy, did that bubble burst.”
He never imagined, 30 years ago, that the internet would become a medium for dark forces: disinformation, espionage, and war. “I look for the best in people. I want to live in a world where computing and technology are used for the good of humanity,” Stoll says. “And it breaks my heart.”