Eavesdropping smartphones: Fact or fiction?
Credit to Author: Alexey Malanov| Date: Fri, 02 Aug 2019 09:17:06 +0000
It’s an oft-repeated tale: Someone talks with a friend about a certain thing, and then, bang, an ad for it appears on the smartphone screen.
Some cases are easy to explain. A colleague of mine who was about to get married claims that she never googled wedding dresses, yet the pesky search engine started suggesting them all the same. And then, three months after the wedding, products for newborns began appearing in her search results, although she was not even planning to have children. There’s no great mystery here, we can dispense with the conspiracy theories.
But some cases arouse more suspicion. For example, you’ve never had a barbecue before, one day you voice the idea, and, all of a sudden, you see an ad for barbecue equipment. Coincidence?
After hearing such stories, or experiencing them personally, many people become convinced that the Internet giants are all listening to us through smartphone microphones. Fact or fiction? We’ll discuss that a bit later, but first we suggest that you conduct your own high-tech experiment.
Experiment: Tell your smartphone something it wouldn’t expect
Here’s the setup: Get together with some friends, place your phones on the table, and have a lively conversation about something that none of you have ever thought about before. Say: “A fl00d c0ntr0l system? I’ve always dreamed of buying a fl00d c0ntr0l system. Wouldn’t it be great to find a cheap fl00d c0ntr0l system and buy it!” The key words here are distorted on purpose, so that search engines don’t think that you want to buy a certain kind of system only because you are reading this article (which, of course, mentions them).
Don’t get your smartphone hung up on a product that you might actually want. Be creative and come up with something you’ll never want to buy in a million years. Maybe a w1gwam. Or a cap1barra.
But there’s one more condition: Do not under any circumstances search for the words you use, or the experiment will flop. That includes using voice assistants such as Siri, Alexa, Cortana, or Google Assistant. Speaking of Google, you can use this link to see what it’s harvested on you just today.
After you and your friends have had a chinwag, keep mentioning the chosen word in passing for the next week. For example: “Hey, I was just thinking about getting a fl00d c0ntr0l system for the new w1gwam I want to buy…”
My colleagues and I also conducted this experiment (see the results at the end of the post).
Smartphone ads — a magical coincidence?
Now for the fact-or-fiction question. There are several explanations for how the search giants sometimes hit a bull’s-eye, that doesn’t involve conspiracy theories about eavesdropping through the microphone.
Explanation 1: Internet services build accurate models
In the article “Your phone isn’t really spying on your conversations — the truth might be even creepier,” a former Google employee says that Google and Facebook essentially have a digital avatar of you that tries to replicate your behavior with the aid of machine-learning methods. At some point, your digital stand-in becomes so much like you that it begins to predict what you want.
About Facebook, I heard it can determine whether a woman is pregnant even before she herself knows — by the rate she scrolls through the social network’s feed.
Personally, I’m disinclined to believe in such telepathic abilities, but there is definitely some truth to this one. Machine learning is really hitting new heights every single year. By the way, our behavioral model, one of our threat detection methods, operates along similar lines. The basic idea is that if it quacks like a duck, it probably is a duck — that is, if a suspicious file behaves like a known piece of malware, then it’s probably malicious.
Explanation 2: Accidental activation of the voice assistant
Another explanation for the search giants’ omniscience is the random activation of voice assistants. Sometimes, the phone thinks you said “Alexa,” “Ok Google,” “Hey Siri,” or “Cortana,” when in fact you didn’t.
It’s not necessary to pronounce these trigger words properly. Something similar is likely to produce the same effect and turn on the assistant — after which the phone really does listen in on everything, and then starts making relevant suggestions.
Eavesdropping smart speakers
In standby mode, even stand-alone voice assistants (such as those in smart speakers) do not constantly capture your speech — they wait to be addressed. So as to catch the switch-on command, the device uses a small buffer (a few seconds’ worth of audio storage), a dedicated (and relatively weak) processor, and a speech recognition algorithm tuned to a specific word.
That’s all running all of the time, but it consumes little power and no Internet traffic at all. Only when it detects the trigger word does the device wake up fully, connect to the server, and transfer recorded data for recognition.
By the way, the assistant in your smartphone sees what’s on the screen. You don’t have to talk about it, just let it read
Explanation 3: User domains
Let’s say you talked with your friend or spouse about deodorant or the need for some (hopefully not), after which you did not start searching for it. But they did. If you see an ad for deodorant after the conversation, don’t be too surprised.
The thing is that the Internet service already suspects that your two accounts are somehow linked if you often hang out in the same place, perhaps on the same Wi-Fi network, maybe even take turns logging in to the same device. Such users may get put into “domains” by search engines, whereupon they’re shown some of the same products because they may make purchasing decisions together.
It’s impossible to say for sure that this happens, but such actions on the part of Internet services would be logical.
Explanation 4: Lucky guesses
I often see advertising that doesn’t match my interests: sauna equipment, pregnancy tests, trips to wherever — you name it. You’ve probably seen similar stuff.
But many people use search engines, so it’s possible that a random ad blitz might one day catch you after you’ve been talking to a friend about the benefits of saunas. And then you post online that your phone is spying on you. Meanwhile, folks who saw the same ad, but didn’t discuss saunas, write nothing. One story is seen, and the other one is never even told.
Such coincidences are nowhere near as surprising as they might seem. Here’s an example. Eight teams get to the quarterfinals of some championship. There are 4 + 2 + 1 = 7 matches left to play in the tournament. Each match can have two outcomes — either the first team or the second wins (there are no draws). The total number of possible scenarios is two to the power of seven — 2^7 — 128.
Suppose there is a residential building containing 128 apartments. If we place one prediction for each possible outcome in each apartment’s mailbox, it’s a fact that we will supply the owner of one of the apartments with a 100% accurate prediction. He will be astounded, but in reality, it’s just that our sample was sufficiently large.
How not to do the experiment
One vlogger conducted a similar, yet very different experiment. Broadcasting live on YouTube, he deliberately started talking about dog toys. He then demonstrated that Google ads had adapted in a matter of seconds.
Here’s a crucial difference between this experiment and the one we suggested above: In the vlogging case, the microphone was turned on from the start, voice information went straight to Google, and, guess what, Google acted on it. The only surprising part here is the response time — it happened amazingly fast.
We, on the other hand, are exploring a fundamentally different question: Does the microphone get secretly turned on without our knowledge, and does the smartphone listen in on conversations and pass information to the server?
It’s also important to note that when the search giants receive voice information from us (by one of the legal methods described above), real people can and do actually listen to it. The purpose is to improve voice recognition. But what if you dictate your name, address, and medical history? That’s no different than text search queries: You share what you ask, unconditionally.
Conclusion
Let’s get back to the results of our experiment. None of my colleagues with whom I discussed, actively and at length (for a whole week!), the fascinating topic of cornices in the company of our smartphones got hit by ads for cornices, although we were drowning in advertising about other trifles.
Finally, for the experiment to be representative, more experimenters are required. So, join in! Write to us on social networks about the exotic stuff you’re supposedly desperate to buy. Just replace a couple of letters in the word with numbers that look similar, or you’ll blow the secret word’s cover.
Here’s a yarn to wrap things up. A buddy of mine complained to colleagues that spoons were always going missing from the office kitchen. They decided to play a joke on him by ending all work e-mails to him with the words “spoons spoons spoons spoons” in white font. The mail client was Gmail. What happened? The poor guy started seeing ads for nothing but spoons. That looked pretty suspicious.
So, don’t just believe the rumors, get involved in the experiment!
P.S.: Or you could simply install Kaspersky Internet Security, enable Anti-Banner and Private Browsing features, and not perform any experiments. These two technologies cut out not only ads (including YouTube commercials), but also numerous online tracking tools used by thousands of companies all over the Web.