In mid-2016, Aviv Ovadya realized there was something fundamentally wrong with the internet — so wrong which he abandoned his work in addition to sounded an alarm. A few weeks before the 2016 election, he presented his concerns to technologists in San Francisco’s Bay Area in addition to warned of an impending crisis of misinformation in a presentation he titled “Infocalypse.”
The web in addition to the information ecosystem which had developed around which was wildly unhealthy, Ovadya argued. The incentives which governed its biggest platforms were calibrated to reward information which was often misleading in addition to polarizing, or both. Platforms like Facebook, Twitter, in addition to Google prioritized clicks, shares, ads, in addition to money over quality of information, in addition to Ovadya couldn’t shake the feeling which which was all building toward something bad — a kind of critical threshold of addictive in addition to toxic misinformation. The presentation was largely ignored by employees via the Big Tech platforms — including a few via Facebook who might later go on to drive the company’s NewsFeed integrity effort.
“At the time, which felt like we were in a car careening out of control in addition to which wasn’t just which everyone was saying, ‘we’ll be fine’ — which’s which they didn’t even see the vehicle,” he said.
Ovadya saw early what many — including lawmakers, journalists, in addition to Big Tech CEOs — wouldn’t grasp until months later: Our platformed in addition to algorithmically optimized world can be vulnerable — to propaganda, to misinformation, to dark targeted advertising via foreign governments — so much in order which which threatens to undermine a cornerstone of human discourse: the credibility of fact.
although which’s what he sees coming next which will definitely scare the shit out of you.
“Alarmism can be Great — you should be alarmist about which stuff,” Ovadya said one January afternoon before calmly outlining a deeply unsettling projection about the next two decades of fake news, artificial intelligence–assisted misinformation campaigns, in addition to propaganda. “We are so screwed which’s beyond what most of us can imagine,” he said. “We were utterly screwed a year in addition to a half ago in addition to we’re even more screwed at which point. in addition to depending how far you look into the future which just gets worse.”
which future, according to Ovadya, will arrive that has a slew of slick, easy-to-use, in addition to eventually seamless technological tools for manipulating perception in addition to falsifying reality, for which terms have already been coined — “reality apathy,” “automated laser phishing,” in addition to “human puppets.”
Which can be why Ovadya, an MIT grad with engineering stints at tech companies like Quora, dropped everything in early 2016 to try to prevent what he saw as a Big Tech–enabled information crisis. “One day something just clicked,” he said of his awakening. which became clear to him which, if somebody were to exploit our attention economy in addition to use the platforms which undergird which to distort the truth, there were no real checks in addition to balances to stop which. “I realized if these systems were going to go out of control, there’d be nothing to reign them in in addition to which was going to get bad, in addition to quick,” he said.
“We were utterly screwed a year in addition to a half ago in addition to we’re even more screwed at which point”
Today Ovadya in addition to a cohort of loosely affiliated researchers in addition to academics are anxiously looking ahead — toward a future which can be alarmingly dystopian. They’re running war game–style disaster scenarios based on technologies which have begun to pop up in addition to the outcomes are typically disheartening.
For Ovadya — at which point the chief technologist for the University of Michigan’s Center for Social Media Responsibility in addition to a Knight News innovation fellow at the Tow Center for Digital Journalism at Columbia — the shock in addition to ongoing anxiety over Russian Facebook ads in addition to Twitter bots pales in comparison to the greater threat: Technologies which can be used to enhance in addition to distort what can be real are evolving faster than our ability to understand in addition to control or mitigate which. The stakes are high in addition to the possible consequences more disastrous than foreign meddling in an election — an undermining or upending of core civilizational institutions, an “infocalypse.” in addition to Ovadya says which which one can be just as plausible as the last one — in addition to worse.
“What happens when anyone can make which appear as if anything has happened, regardless of whether or not which did?”
Worse because of our ever-expanding computational prowess; worse because of ongoing advancements in artificial intelligence in addition to machine learning which can blur the lines between fact in addition to fiction; worse because those things could usher in a future where, as Ovadya observes, anyone could make which “appear as if anything has happened, regardless of whether or not which did.”
in addition to much inside the way which foreign-sponsored, targeted misinformation campaigns didn’t feel like a plausible near-term threat until we realized which which was already happening, Ovadya cautions which fast-developing tools powered by artificial intelligence, machine learning, in addition to augmented reality tech could be hijacked in addition to used by bad actors to imitate humans in addition to wage an information war.
in addition to we’re closer than one might think to a potential “Infocalypse.” Already available tools for audio in addition to video manipulation have begun to look like a potential fake news Manhattan Project. inside the murky corners of the internet, people have begun using machine learning algorithms in addition to open-source software to easily create pornographic videos which realistically superimpose the faces of celebrities — or anyone for which matter — on the adult actors’ bodies. At institutions like Stanford, technologists have built programs which which combine in addition to mix recorded video footage with real-time face tracking to manipulate video. Similarly, at the University of Washington computer scientists successfully built a program capable of “turning audio clips into a realistic, lip-synced video of the person speaking those words.” As proof of concept, both the teams manipulated broadcast video to make world leaders appear to say things they never actually said.
As these tools become democratized in addition to widespread, Ovadya notes which the worst case scenarios could be extremely destabilizing.
There’s “diplomacy manipulation,” in which a malicious actor uses advanced technology to “create the belief which an event has occurred” to influence geopolitics. Imagine, for example, a machine-learning algorithm (which analyzes gobs of data in order to teach itself to perform a particular function) fed on hundreds of hours of footage of Donald Trump or North Korean dictator Kim Jong Un, which could then spit out a near-perfect — in addition to virtually impossible to identify via reality — audio or video clip of the leader declaring nuclear or biological war. “which doesn’t have to be perfect — just Great enough to make the enemy think something happened which which provokes a knee-jerk in addition to reckless response of retaliation.”
“which doesn’t have to be perfect — just Great enough”
Another scenario, which Ovadya dubs “polity simulation,” can be a dystopian combination of political botnets in addition to astroturfing, where political movements are manipulated by fake grassroots campaigns. In Ovadya’s envisioning, increasingly believable AI-powered bots will be able to effectively compete with real humans for legislator in addition to regulator attention because which will be too difficult to tell the difference. Building upon previous iterations, where public discourse can be manipulated, which may soon be possible to directly jam congressional switchboards with heartfelt, believable algorithmically-generated pleas. Similarly, Senators’ inboxes could be flooded with messages via constituents which were cobbled together by machine-learning programs working off stitched-together content culled via text, audio, in addition to social media profiles.
Then there’s automated laser phishing, a tactic Ovadya notes security researchers are already whispering about. Essentially, which’s using AI to scan things, like our social media presences, in addition to craft false although believable messages via people we know. The game changer, according to Ovadya, can be which something like laser phishing might allow bad actors to target anyone in addition to to create a believable imitation of them using publicly available data.
“Previously one might have needed to have a human to mimic a voice or come up with an authentic fake conversation — in which edition you could just press a button using open source software,” Ovadya said. “which’s where which becomes novel — when anyone can do which because which’s trivial. Then which’s a whole different ball game.”
Imagine, he suggests, phishing messages which aren’t just a confusing link you might click, although a personalized message with context. “Not just an email, although an email via a friend which you’ve been anxiously waiting for for a while,” he said. “in addition to because which might be so easy to create things which are fake you’d become overwhelmed. If every bit of spam you receive looked identical to emails via real people you knew, each one with its own motivation trying to convince you of something, you’d just end up saying, ‘okay, I’m going to ignore my inbox.’”
which can lead to something Ovadya calls “reality apathy”: Beset by a torrent of constant misinformation, people simply start to give up. Ovadya can be quick to remind us which which can be common in areas where information can be poor in addition to so assumed to be incorrect. The big difference, Ovadya notes, can be the adoption of apathy to a developed society like ours. The outcome, he fears, can be not Great. “People stop paying attention to news in addition to which fundamental level of informedness required for functional democracy becomes unstable.”
Ovadya (in addition to various other researchers) see laser phishing as an inevitability. “which’s a threat for sure, although even worse — I don’t think there’s a solution right at which point,” he said. “There’s internet scale infrastructure stuff which needs to be built to stop which if which starts.”
Beyond all which, there are various other long-range nightmare scenarios which Ovadya describes as “far-fetched,” although they’re not so far-fetched which he’s willing to rule them out. in addition to they are frightening. “Human puppets,” for example — a black market edition of a social media marketplace with people instead of bots. “which’s essentially a mature future cross border market for manipulatable humans,” he said.
Ovadya’s premonitions are particularly terrifying given the ease with which our democracy has already been manipulated by the most rudimentary, blunt-force misinformation techniques. The scamming, deception, in addition to obfuscation which’s coming can be nothing completely new; which’s just more sophisticated, much harder to detect, in addition to working in tandem with various other technological forces which are not only currently unknown although likely unpredictable.
For those paying close attention to developments in artificial intelligence in addition to machine learning, none of which feels like much of a stretch. Software currently in development at the chip some sort of Nvidia can already convincingly generate hyperrealistic photos of objects, people, in addition to even some landscapes by scouring tens of thousands of images. Adobe also recently piloted two projects — Voco in addition to Cloak — the first a “Photoshop for audio,” the second a tool which can seamlessly remove objects (in addition to people!) via video in a matter of clicks.
In some cases, the technology can be so Great which which’s startled even its creators. Ian Goodfellow, a Google Brain research scientist who helped code the first “generative adversarial network” (GAN), which can be a neural network capable of learning without human supervision, cautioned which AI could set news consumption back roughly 100 years. At an MIT Technology Review conference in November last year, he told an audience which GANs have both “imagination in addition to introspection” in addition to “can tell how well the generator can be doing without relying on human feedback.” in addition to which, while the creative possibilities for the machines can be boundless, the innovation, when applied to the way we consume information, might likely “clos[e] some of the doors which our generation has been used to having open.”
In which light, scenarios like Ovadya’s polity simulation feel genuinely plausible. which summer, more than one million fake bot accounts flooded the FCC’s open comments system to “amplify the call to repeal net neutrality protections.” Researchers concluded which automated comments — some using natural language processing to appear real — obscured legitimate comments, undermining the authenticity of the entire open comments system. Ovadya nods to the FCC example as well as the recent bot-amplified #releasethememo campaign as a blunt edition of what’s to come. “which can just get so much worse,” he said.
“You don’t need to create the fake video just for which tech to have a serious impact. You just point to the fact which the tech exists in addition to you can impugn the integrity of the stuff which’s real.”
Arguably, which sort of erosion of authenticity in addition to the integrity of official statements altogether can be the most sinister in addition to worrying of these future threats. “Whether which’s AI, peculiar Amazon manipulation hacks, or fake political activism — these technological underpinnings [lead] to the increasing erosion of trust,” computational propaganda researcher Renee DiResta said of the future threat. “which makes which possible to cast aspersions on whether videos — or advocacy for which matter — are real.” DiResta pointed out Donald Trump’s recent denial which which was his voice on the infamous Access Hollywood tape, citing experts who told him which’s possible which was digitally faked. “You don’t need to create the fake video just for which tech to have a serious impact. You just point to the fact which the tech exists in addition to you can impugn the integrity of the stuff which’s real.”
which’s why researchers in addition to technologists like DiResta — who spent years of her spare time advising the Obama administration, in addition to at which point members of the Senate Intelligence Committee, against disinformation campaigns via trolls — in addition to Ovadya (though they work separately) are beginning to talk more about the looming threats. Last week, the NYC Media Lab, which helps the city’s companies in addition to academics collaborate, announced a plan to bring together technologists in addition to researchers in June to “explore worst case scenarios” for the future of news in addition to tech. The event, which they’ve named Fake News Horror Show, can be billed as “a science fair of terrifying propaganda tools — some real in addition to some imagined, although all based on plausible technologies.”
“inside the next two, three, four years we’re going to have to plan for hobbyist propagandists who can make a fortune by creating highly realistic, photo realistic simulations,” Justin Hendrix, the executive director of NYC Media Lab, told BuzzFeed News. “in addition to should those attempts work, in addition to people come to suspect which there’s no underlying reality to media artifacts of any kind, then we’re in a definitely difficult place. which’ll only take a couple of big hoaxes to definitely convince the public which nothing’s real.”
Given the early dismissals of the efficacy of misinformation — like Facebook CEO Mark Zuckerberg’s at which point-infamous statement which which was “crazy” which fake news on his site played a crucial role inside the 2016 election — the first step for researchers like Ovadya can be a daunting one: Convince the greater public, as well as lawmakers, university technologists, in addition to tech companies, which a reality-distorting information apocalypse can be not only plausible, although close at hand.
“which’ll only take a couple of big hoaxes to definitely convince the public which nothing’s real.”
A senior federal employee explicitly tasked with investigating information warfare told BuzzFeed News which even he’s not certain how many government agencies are preparing for scenarios like the ones Ovadya in addition to others describe. “We’re less on our back feet than we were a year ago,” he said, before noting which which’s not nearly Great enough. “I think about which via the sense of the enlightenment — which was all about the search for truth,” the employee told BuzzFeed News. “I think what you’re seeing at which point can be an attack on the enlightenment — in addition to enlightenment documents like the Constitution — by adversaries trying to create a post-truth society. in addition to which’s a direct threat to the foundations of our current civilization.”
which’s a terrifying thought — more so because forecasting which kind of stuff can be so tricky. Computational propaganda can be far more qualitative than quantitative — a climate scientist can point to explicit data showing rising temperatures, whereas which’s virtually impossible to build a trustworthy prediction product mapping the future impact of yet-to-be-perfected technology.
For technologists like the federal employee, the only viable way forward can be to urge caution, to weigh the moral in addition to ethical implications of the tools being built in addition to, in so doing, avoid the Frankensteinian moment when the creature turns to you in addition to asks, “Did you ever consider the consequences of your actions?”
“I’m via the free in addition to open source culture — the goal isn’t to stop technology although ensure we’re in an equilibria which’s positive for people. So I’m not just shouting ‘which can be going to happen,’ although instead saying, ‘consider which seriously, examine the implications,” Ovadya told BuzzFeed News. “The thing I say can be, ‘trust which which isn’t not going to happen.’”
Hardly an encouraging pronouncement. which said, Ovadya does admit to a bit of optimism. There’s more interest inside the computational propaganda space then ever before, in addition to those who were previously slow to take threats seriously are at which point more receptive to warnings. “inside the beginning which was definitely bleak — few listened,” he said. “although the last few months have been definitely promising. Some of the checks in addition to balances are beginning to fall into place.” Similarly, there are solutions to be found — like cryptographic verification of images in addition to audio, which could help identify what’s real in addition to what’s manipulated.
Still, Ovadya in addition to others warn which the next few years could be rocky. Despite some pledges for reform, he feels the platforms are still governed by the wrong, sensationalist incentives, where clickbait in addition to lower-quality content can be rewarded with more attention. “which’s a hard nut to crack in general, in addition to when you combine which that has a system like Facebook, which can be a content accelerator, which becomes very dangerous.”
Just how far out we are via which danger remains to be seen. Asked about the warning signs he’s keeping an eye out for, Ovadya paused. “I’m not sure, definitely. Unfortunately, a lot of the warning signs have already happened.” ●
Charlie Warzel can be a senior writer for BuzzFeed News in addition to can be based in completely new York. Warzel reports on in addition to writes about the intersection of tech in addition to culture.
Contact Charlie Warzel at email@example.com.
Got a confidential tip? Submit which here.