A Guide to What We Now Know About the NSA's Dragnet Searches of Your Communications

Charlie Savage of The New York Times confirmed this week what we have been warning about for years, including to the Supreme Court last fall: The National Security Agency (NSA) is "searching the contents of vast amounts of Americans' e-mail and text communications into and out of the country, hunting for people who mention information about foreigners under surveillance . . . ." The rub: If you've sent an international email or text since 2008, chances are the government has looked inside of it. In other words, the same NSA surveillance dragnet that government officials have consistently dismissed as speculative and far-fetched is very, very real.

The Times's front-page story raises questions akin to those advanced by a report in The Guardian last week revealing that under a program codenamed "XKeyScore," NSA analysts use dropdown menus and filters — just like the ones we all use every single day on the web — to gain instant access to "nearly everything a typical user does on the Internet." Essentially, XKeyScore is the NSA's very own, very powerful surveillance search engine.

These reports are particularly alarming in light of government officials' emphatic public statements denying that the NSA ever peeks inside the contents of Americans' communications without a warrant. We've explained why those disavowals have been misleading; now, we know they're simply untrue.

There are important details about these programs that still need to be filled in. But a large amount of information is already in the public domain. What follows is an attempt to tie together many of the threads of the last two months' worth of disclosures, in order to get a big-picture look at what the NSA is doing, and how.

Prism or Upstream (or both)?

As a starting point, it's useful to review the various ways in which the government is watching us.

In early June, The Guardian and The Washington Post reported the existence of "Prism," a program authorized under the FISA Amendments Act (FAA), now located in Section 702 of the FISA, that allows the NSA to receive data directly from U.S. companies like Google and Facebook and thereby collect the contents of foreign targets' emails, text and video chats, photographs, and more. As we've explained, Prism sweeps up the communications of many Americans in the process.

But another component of the government's omnipresent surveillance of global communications that has escaped sustained attention — until this week's Times bombshell — is likely even more extensive. In The Guardian's first follow-up report on Prism, it published a slide that identified "Two Types of Collection" under the FAA: Prism, and Upstream. In contrast to Prism, "Upstream" collection involves the collection of communications — both their metadata and their content — as they pass through undersea fiber-optic cables. According to renowned surveillance journalist James Bamford, Prism was designed to fill "gaps in the coverage" provided by the NSA's Upstream collection. Those gaps exist both because certain communications are not available through Upstream, and because Prism collection avoids encryption that may protect certain upstream-collected communications. (A new story in The Guardian reveals that once the government has this data, it can search through it for individual Americans' communications without a warrant.)

The NSA's data collection from undersea cables is not entirely new. Since the 1970s, at least, the NSA has been tapping cables on foreign soil to conduct surveillance abroad. But over the past ten years, that changed.

In 2005, The New York Times reported on a vast expansion of the NSA's role in intelligence gathering at home. First, James Risen and Eric Lichtblau disclosed a secret program of warrantless eavesdropping targeted at communications inside the United States. One week later, the same reporters revealed that the NSA had also been capturing American communications on a much broader scale by "tapping directly into some of the American telecommunication system's main arteries" with the cooperation of U.S. telecommunications companies. (The NSA's access to U.S.-based fiber-optic cables was recently revisited by the Associated Press and The Washington Post.)

The "Upstream" interception of domestic cables — which, Bamford says, carry "about 80 percent" of the world's telecommunications — allows the government to conduct pattern analysis on a huge trove of metadata, and to eavesdrop on foreign-to-foreign communications that travel over U.S. networks. But it also has another serious and inevitable consequence: the warrantless acquisition of metadata and content concerning purely American communications.

Savage's story in the Times advances the ball by explaining, at least in part, what the government does with the things it's accessing via U.S.-based cables. Intelligence officials told the Times that the spy agency casts a net that ensnares virtually every electronic and text communication passing through U.S. borders (in either direction) over fiber-optic networks. The government makes a copy of everything; it then searches inside every message and retrieves, for longer-term storage, any message that includes content matches to "selectors," or approved "targets" under Section 702 of the Foreign Intelligence Surveillance Act.

But the government's dragnet is not confined to the fiber-optic cables running across domestic boundaries. Recent reporting by Brazil's O Globo, Germany's Der Spiegel, and Australia's Sydney Morning Herald — all based on documents obtained from Edward Snowden — paint a picture of an interconnected network of communications cables and facilities across the world to which the United States has access. (Globally, roughly 99 percent of the world's communications travel over undersea fiber-optic cables.) The government apparently gains that access through its own taps, arrangements between American and foreign telecommunications companies, or agreements with foreign intelligence services. When NSA Director Gen. Keith Alexander says he intends for the agency to "collect it all," this is exactly what he means.

How does XKeyScore fit in?

The bottom line: However it does it, the United States is accumulating an almost incomprehensibly massive trove of information, in the form of both metadata and content. But the sheer comprehensiveness of that data also presents the government with a confounding practical problem: If you're looking for the proverbial needle in the most colossal digital haystack the world has ever known, how do you even begin to go about finding it?

One way, as Savage's reporting tells us, is by "copying and then sifting through the contents of" the communications that pass over U.S. borders, and then marking some of those messages for longer-term storage in the government's content databases.

But another — though perhaps related — way to sort the world's data is, apparently, by using XKeyScore.

In typical Fort Meade-ese, the agency's training presentation defines XKeyScore as an "Exploitation System/Analytic Framework" that "[p]rovides real-time target activity." In plainer English, XKeyScore is an integrated set of software interfaces, databases, and servers that allows analysts to find and access the online activities of anyone in the world, in real time. As Marc Ambinder — who has previously written about XKeyScore — explains, it "is not a thing that does collecting," but a cohesive system that lets the NSA sift and sort through the enormous pile of data the agency is vacuuming up every minute around the world.

In other words, XKeyScore is something like a search engine.

Of course, it's a uniquely powerful one; a slide published by The Guardian indicates that it can both search deeper into data and process it faster than the NSA's other tools. In part, that's because XKeyScore involves the use of 700 servers deployed at approximately 150 sites around the world—mostly U.S. and allied military and intelligence facilities, as well as U.S. embassies and consulates. Servers at each site permit the retention of one months' worth of metadata and three-to-five days' worth of content at a time. Ambinder writes that XKeyScore is "useful because it gets the ‘front end full take feeds' from the various NSA collection points around the world and importantly, knows what to do with it to make it responsive to search queries."

Thus, XKeyScore is a sort of clearinghouse for data collected around the world, as well as an interface for homing in on particular information. (The worldwide server network receives this data from various sources: the Upstream-collecting undersea cables, government-satellite collection, and the Prism program. How all of the NSA's sources of information relate to how and where they are stored is a very important hole in what is already publicly known.) Data flows into XKeyScore collection sites; it is stored on the system's servers, with content remaining there for between three and five days, and metadata for as long as a month; and NSA analysts search those servers to identify the communications of its targets.

Who exactly is the NSA targeting?

But XKeyScore is not just a practical solution to the problem of collecting all of the world's data. It's also supposed to implement restrictions on whose data the NSA is lawfully permitted to examine; effectively, it is the NSA's proxy for the full protection of constitutional privacy rights to which American citizens are entitled.

Since the FAA prohibits the targeting of U.S. persons, or even foreign persons inside the United States, the NSA is not supposed to be able to indiscriminately seek out data in its streams. Thus, as The Guardian's slides reveal, NSA analysts must select "foreign factors" in order to conduct searches through XKeyScore. The NSA's XKeyScore "foreign factors" appear to be the same as those disclosed in an earlier Guardian story about NSA targeting procedures, that impose limited rules on the government's warrantless surveillance authority under the FAA.

As our explainer from last month details, those procedures seriously threaten Americans' privacy—even if they are being properly followed by everyone in the NSA—because the protections for American are weak, and riddled with exceptions. Moreover, the prohibition on "targeting" Americans has only limited purchase when it comes to ensuring that Americans' communications will not be swept up in the NSA's global dragnet. Indeed, as my colleague Patrick Toomey discussed last month, "there is nothing inadvertent or accidental" about the size of these loopholes; they were written into the law for the very purpose of allowing the government to gain access to Americans' communications.

And this week's Times story confirms yet another reason that the NSA's procedures are inadequate. The new revelations make clear that the government's use of Section 702 goes far beyond what it has previously confirmed to the public — that its foreign-intelligence surveillance only captures the communications of U.S persons or others inside the United States who are in direct contact with the NSA's targets. Instead, the NSA can search the communications of everyone for information "about" its targets — at least when those communications travel across American borders. Capturing communications "about" approved targets means that no matter who you are or who you talk to, the content of your emails and texts are fair game for NSA surveillance.

Greenwald's reporting presents a different threat as well. The XKeyScore slides show us how an analyst might have the technical capability to breeze past targeting limitations by clicking through filters the way we all search for the lunch we want to order.

According to the slides released by The Guardian and Greenwald's reporting, XKeyScore allows analysts to search the metadata and content residing on any XKeyScore server by different selectors — name, email address, telephone number, IP address, keywords, and even language or type of Internet browser — "without prior authorization." Searches might either turn up content — which analysts can then view or listen to — or help an analyst identify an anonymous user who is, for example, searching for "suspicious stuff" or speaking a language "out of place" for a given region. Some slides also suggest that the XKeyScore searches can also query data stored in other NSA databases. And Ambinder insinuates that anything targeted through XKeyScore can be directed into one of the government's more permanent metadata and content databases, some of which the Post's Barton Gellman described in June.

In an online video shortly after the revelations began, Snowden claimed that "sitting at [his] desk," he could "wiretap anyone, from you or your accountant, to a federal judge or even the president, if [he] had a personal email." The government's reaction was both harsh and emphatic. Gen. Alexander went so far as to testify to the Senate Appropriations Committee not only that Snowden's claim was "false," but that "I know of no way to do that."

The Guardian slides give us good reason to believe that Snowden was telling the truth, and that Alexander was being coy. Greenwald reports that Snowden has stated that supervisors often find ways to circumvent the very targeting procedures that are meant to confine the NSA's searches: "It's very rare to be questioned on our searches," he said, "and even when we are, it's usually along the lines of: ‘let's bulk up the justification.'" As far back as 2009, members of Congress were raising alarms about the NSA's collection under the FAA, including Rep. Rush Holt's ominous warning that "[s]ome actions are so flagrant that they can't be accidental."

* * *

We'll have to wait for additional revelations — or, preferably, official disclosures by the government — to fully understand how XKeyScore, and the government's other FAA surveillance capabilities, actually work, and how much power NSA analysts have to circumvent the agency's own procedures. But what we already do know gives us plenty of important reasons to reject it as a model for surveillance that is compatible with the Constitution.

Learn more about government surveillance and other civil liberty issues: Sign up for breaking news alerts, follow us on Twitter, and like us on Facebook.