OPEN SOURCE IT ALL:

The most reliable way for the intelligence community to gauge the effectiveness of its various transparency programs would likely be periodic public polling, but any attempt by today's intelligence agencies to measure public opinion in the U.S. would stir uncomfortable memories of Cold War-era projects. In this context, the Texas National Security Network fielded a nationally representative survey of 1,000 Americans with the survey market-research firm YouGov. The first round of the poll is intended to establish a stable baseline that measures Americans' overall perception of the American intelligence community, its effectiveness, regard for civil liberties, and democratic oversight.

Our goals in gathering and analyzing this information this year and in the future are: 1) to inform scholarly and popular debate on the proper role of intelligence in our democracy; and 2) to help intelligence community officials design public-facing programs that respond most directly to the actual knowledge, beliefs, and concerns of the American people. We view this project as a useful complement, and not a challenge, to Lawfare's monthly polling on public confidence in U.S. national-security institutions, including the intelligence community.

Here are several key takeaways from the initial report:

56 percent of respondents indicated that the U.S. intelligence community "plays a vital role in warning against foreign threats and contributes to national security" while only 7 percent of those questioned thought our intelligence agencies were "no longer necessary". Large differences were noted in this pool based on respondents' general knowledge of foreign affairs;

75 percent of respondents regarded the intelligence community as effective in "preventing terrorist attacks against the U.S.", but only 43 percent judged the intelligence community equally effective at "protecting the privacy an civil liberties of Americans". More Democrats than Republicans credited the intelligence community with protecting their rights (see figure above);

More than 90 percent of Americans agreed with the longstanding charge by U.S. presidents that the intelligence community employ "[a]ll reasonable and lawful means" to ensure the government receives the best possible intelligence, but only 38 percent believe that in gathering this information the U.S. intelligence community should "respect the privacy rights of foreigners to the same extent as Americans". Democrats were more likely than Republicans or Independents to extend privacy protections to foreigners;

63 percent of Democrats believe the intelligence community can share more information with the public without compromising its effectiveness, while only 42 percent of Republicans hold that view;

From a shortlist, 24 percent of respondents selected the NSC as the institution primarily responsible for monitoring U.S. intelligence, while only 11 percent thought the president played this role. Only 20 percent of those polled associated the Congress with intelligence oversight, while 21 percent (incorrectly) believe federal courts and judges play a central role in regulating U.S. intelligence.

We see this as a preliminary point of departure for our assessment of the public's view of the U.S. intelligence community. 

Two overlapping developments in particular have greatly influenced the growth of open-source intelligence.

First, the explosion of social media has given us instant access to a wealth of user-generated content. From Facebook to Twitter to Google+, we are now only ever a few keystrokes away from a potentially global audience. And as these tools increase global connectivity, people seem increasingly willing to project their thoughts, opinions and observations into cyberspace. The process of information generation has produced what has been described as "new digital commons of enormous size and wealth".

Second, and on a larger scale, the scope of open-source intelligence has been completely changed by the rise of big data. The meaning of this term is contested but it is commonly used to describe "data that exceeds the processing capacity of conventional database systems". According to some estimates, some 1,200 exabytes of data now exists in the world and 90% of it was created in the last two years alone.

Social media is a good example of big data in practice. Users are generating 500m tweets and 90m blog posts on Tumblr per day. Every minute, 100 hours of video is uploaded to YouTube. The social media universe is expanding at an astronomical rate.

Little surprise, then, that this networked society has spawned new analytical approaches and opportunities. From everyday policing to counter-terrorism to civil war, information gleaned from open sources can provide insights into world events like never before. Finding what you want in this sea of information can be like looking for a needle in a haystack - but it can be done.

Eliot Higgins pieced together his analysis of weapons use in Syria by trawling through hours of video footage uploaded to YouTube, refining and cross-referencing what he found with information drawn from other websites and platforms.

But the Brown Moses approach represents just one end of a spectrum. At the other end, at the cutting edge of informatics and computing, researchers are developing complex algorithms aimed at automating the process. This will enable them to make big data work as a source of intelligence too. In between are those who combine the manual searches with freely available and often very useful analytical tools.

A recent report by Demos hinted at the potential for social media analysis to support police investigations, for example. Tweets and status updates can provide eyewitness testimony and pictorial evidence of incidents or be used by police to "measure groundswells of emotion". Left unchecked, bubbling tensions could develop into episodes of collective violence. Monitoring them online could enable a faster response, or even preventative measures.

This is not open-source intelligence as it has traditionally been understood. The move towards automated analysis hints at something more akin to signals intelligence. It is perhaps for this reason that researchers active in the area of social media analysis have coined their own term: social media intelligence or SOCMINT. Fundamentally, however, this is open-source intelligence in the sense that publicly accessible information is being exploited for intelligence purposes.

Technology has breathed new life into open-source intelligence. Whether it is derived from the work of internet investigators like Brown Moses or complex algorithms, it is more valuable than ever before.

Tweet   @brothersjudd