February 28, 2004


Open Source Intelligence (RICHARD S. FRIEDMAN, Summer 1998, Parameters)

Ninety percent of intelligence comes from open sources. The other ten percent, the clandestine work, is just the more dramatic. The real intelligence hero is Sherlock Holmes, not James Bond.
--Lieutenant General Sam Wilson, USA Ret. former Director, Defense Intelligence Agency

Former Ambassador to Algeria L. Craig Johnstone (presently State Department Director of Resources, Plans and Policy) recently told a Washington conference that during his assignment in Algeria, he bought and installed a satellite dish enabling him to watch CNN so he could have access to global news. He recalled:

The first week I had it running was the week of the Arab League summit in Algiers and, for whatever reason, the Department was interested in finding out whether Yasser Arafat would attend the summit. No one knew, and the day of the summit Washington was getting more frantic. We in the Embassy were banned from the summit site so there was no way we could find out whether or not Yasser Arafat would show. Finally, at about noon I was home for lunch and watching CNN when the office of the Secretary of State called. The staffer on the other end asked if there was anything at all he could tell the Secretary about Arafat's participation. And just then, on CNN I saw a live picture of Yasser Arafat arriving at the conference. "He is definitely at the conference," I reported. The staffer was ecstatic and went off to tell the Secretary. The next day I received a congratulatory phone call from the NEA bureau for pulling the rabbit out of the hat. How did you find out, they asked? The secret was mine. But I knew then and there that the business of diplomacy had changed, and that the role of embassies, how we do business in the world, also had to change.

Ambassador Johnstone's story provides an example of the value of information from open sources. Allen W. Dulles, when he was Director of Central Intelligence, acknowledged to a congressional committee, "more than 80 percent of intelligence is obtained from open sources." Whether the amount of intelligence coming from open sources is 90 percent, 80 percent, or some other figure, experienced intelligence professionals agree that most information processed into finished intelligence may be available from open sources. This essay explores the significance of a trend toward increased recognition of the role of open source information and discusses what this may mean for intelligence consumers at every level. [...]

Enthusiastic proponents of open source intelligence argue that the information revolution is transforming the bulk of any nation's intelligence requirements and reducing the need to rely upon traditional human and technical means and methods. But Robin W. Winks, distinguished Yale University historian who served in the Office of Strategic Services during World War II and in its successor, the Central Intelligence Agency, concluded, "Research and analysis are at the core of intelligence . . . . [Most] `facts' are without meaning; someone must analyze even the most easily obtained data."

The emerging debate between investing in technology and developing competent analysts concerns itself basically with the value and role of open source intelligence. To understand some of the forces that are shaping the debate, we need to weigh the relative benefits of primary and secondary sources, two discrete subsidiary classes of open source material. Primary sources, generally taken to include print and electronic media, have always provided information of value to the intelligence community in current intelligence, indications, and warning as well as background information used by analysts in their work. What the so-called information revolution has done is to increase the ability of users to gain access and to manipulate the information, and although most intelligence managers do not believe that the number of primary sources has expanded greatly, the number of secondary sources has increased exponentially. To compound the analyst's problem, the objectivity and reliability of many secondary sources are often questionable. We will need more experience before we can accept expansion of secondary sources as a benefit to the management of national security.

The largest general open source collection in the world is the Library of Congress. To replace the original library, which was destroyed during the War of 1812, Congress in 1815 purchased the private library of former President Thomas Jefferson, greatly increasing the collection's size and scope. The Library of Congress now includes works in more than 450 languages and comprises more than 28 million books, periodicals, and pamphlets as well as manuscripts, maps, newspapers, music scores, microfilms, motion pictures, photographs, recordings, prints, and drawings. The library's services also include research and reference facilities, which coordinate with or amplify local and regional library resources.

There are also several thousand databases available from commercial organizations; LEXIS/NEXIS, Dialog, Reuters, and The New York Times come to mind. Any discussion of contemporary open sources must now include the Internet and the World Wide Web (WWW). The World Wide Web (developed in 1989) is a collection of files, called Web sites or Web pages, identified by uniform resource locators (URLs). Computer programs called browsers retrieve these files.

The term "Internet" describes the interconnection of computer networks, particularly the global interconnection of government, education, and business computer networks, available to the public. In early 1996, the Internet connected more than 25 million computers in more than 180 countries. The Internet provides an immense quantity and variety of open source information and must be increasingly looked upon as a source for intelligence purposes.

The Internet and the World Wide Web exemplify technology that is not yet mature. One hallmark of immature technology is an underlying anarchy and a potential for disinformation. In October 1938, when radio broadcasting was emerging as a reliable source of information, producer-director Orson Welles, in his weekly radio show Mercury Theater, presented a dramatization of an 1898 H. G. Wells story, War of the Worlds. The broadcast, which purported to be an account of an invasion of earth from outer space, created a panic in which thousands of individuals took to the streets, convinced that Martians had really invaded Earth. Orson Welles later admitted that he had never expected the radio audience to take the story so literally, and that he had learned a lesson in the effectiveness and reach of the new medium in which content was struggling to catch up to technology.

Recent examples with the Internet and its spin-offs suggest that e-mail abuses, careless gossip reported as fact, and the repeated information anarchy of cyberspace have become progressively chaotic. This does not mean that the Internet and the Web cannot be considered seriously for intelligence work, but it does mean that intelligence officers must exercise a vigilant and disciplined approach to any data or information they acquire from on-line sources.

Part of the brilliance of Admiral Poindexter's idea for an intelligence market is that you could then take this mass of information and have a narrower group of folks place their bets on what's most likely to come of it all. It wouldn't render perfect answers, but it couldn't possibly do worse than our intelligence services historically have.

Posted by Orrin Judd at February 28, 2004 12:51 PM

Or you could hire a posse of deaf leftists.

I.F. Stone, when he became deaf and couldn't do interviews, started reading government reports. He concluded that governments -- ours anyway -- eventually publishes everything it knows. And he did a good job of showing it.

In my estimation, the "James Bond stuff" represents virtually 0.0% of useful intelligence.

Doesn't matter anyhow. Even perfect intelligence is useless if the policymakers don't know what to do with it. Which has been -- even according to you -- the norm.

Posted by: Harry Eagar at February 28, 2004 1:44 PM

About 15 years ago I remember reading that some anti-nuclear groups had decided they would protest nuclear weapons by finding and publishing all the non-classified weapons-related material they could find. (*That* makes sense, right?) They found quite a bit, and no doubt the Pakistanis, North Koreans, Iraqis, etc. found this research to be invaluable.

Posted by: PapayaSF at February 28, 2004 2:14 PM

The futures market would work best if it were anonymous and open to all, like a good bookie joint. Ideally it should accept all definite propositions on a parimutual basis, with a small take out for the cost of administration. I can only imagine the sancitmonious howling.

Posted by: Robert Schwartz at February 28, 2004 5:06 PM

I seem to recall reading somewhere that a lot of "intelligence" work, in real life, is really just reading the newspapers, watching the TV news shows, and listening to the radio. An astute analyst can glean a _heck_ of a lot if he knows how to read between the lines, or so I've always been told.

Posted by: Joe at February 28, 2004 7:23 PM

Valerie Ahl & T. F. H. Allen identify why searching for additional data is generally less fruitful than rechecking the basis for one's position. Inadequate presumptions are much more costly than are deficiencies in information -- "If only we can find that last piece in the puzzle."

Check your premises.

Posted by: Larry H at February 28, 2004 9:17 PM

Joe: You've nailed it pretty well. I work as a political risk analyst in the energy industry using open sources almost exclusively (we get some industry-specific stuff from our own scouts and such). Doing this sort of analysis involves sifting a lot of data, and knowing what to make of it. And having the freedom to make a gutsy call every one in a while.

Unfortunately, the CIA's bureaucracy tends to drive a lot of bad analysis. There are many good individual analysts. Hell, I went to grad school with some of them and will vouch for 'em! But by the time the consensus the CIA requires is achieved, what would be most useful is inevitably stripped or sanitized. Those gutsy calls just don't exist. But those are what are the most useful!

Incidentally, a futures market of some sort would be useful not because majorities would get the analysis right -- that's part of the CIA's problem, the reliance on majorities and consensus -- but that you would flush out the scenarios that are most helpful to policymakers.

Posted by: kevin whited at February 28, 2004 11:48 PM

As I thought about it, I decided that the best way to create an open intelligence market would be under Deep Cover in some southeast asian place (Macau?) where gambling is legal and nobody would think that it was a CIA operation.

Posted by: Robert Schwartz at February 29, 2004 5:30 PM

Why would this yield better assessments about changes in the future than the financial markets, which also operate on open sources and whose members are notoriously unable to catch even big swings in advance?

Markets are Darwinian. They do not control the future, they just assess rewards and penalties for good and bad guesses.

Posted by: Harry Eagar at March 1, 2004 1:49 AM


Is this some kind of zen koan: the swings are the market.

Posted by: oj at March 1, 2004 8:31 AM

How could you protect the market from manipulation? The terror masters could spread misinformation by "shorting" the very strategies that they are going long on. Or they could use the market to see where the betting is at any one time, and use it to "time" their attacks at the moments of greatest complacency. After all, they are the ones who know what every one else is guessing at.

Posted by: Robert Duquette at March 1, 2004 8:52 PM

That's who you're trying to lure in--it's all disinformation--games within games--riddles wrapped in enigmas....

Actually, anyone can post info but only a select group plays the market.

Posted by: oj at March 1, 2004 9:38 PM

Actually, Harry, the market is the most reliable leading economic indicator.

Posted by: David Cohen at March 1, 2004 11:20 PM

Yeah, the market's predicted nine of the last six recessions.

Posted by: Michael Herdegen at March 2, 2004 12:29 PM

I don't want to get into the realm of imperfect information, since we're talking about a specific area here.

Among the many, many problems with intelligence (from gathering to using), one of the most dangerous is trusting it.

It takes a stronger mind than we usually elect not to reason from "I have good intelligence" to "since I have good intelligence, the only thing I have to plan for is what I know is coming."

This point is similar to what Robert was getting at, I think.

Let's say you detect the next brewing attack and deploy to resist it. The attacker, who has his own intelligence, detects this and prepares a different attack. Think Nimitz at Midway.

Anyhow, that is why sound military planning says that you do not plan for what you expect your enemy to do but for what he is capable of doing.

Posted by: Harry Eagar at March 2, 2004 3:20 PM

Plan for what you want to do and screw your enemy.

Posted by: oj at March 2, 2004 3:29 PM

Worked great for the Poles.

Posted by: Harry Eagar at March 2, 2004 8:37 PM

If the Poles had nukes in 1939 the world would be a better place today.

Posted by: oj at March 2, 2004 8:44 PM
« 60-40 VISION FILES: | Main | DUH?: »