INTELLIGENCE ANALYSIS QUALITY

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP81B00401R002400210022-2
• Release Decision: RIPPUB
• Original Classification: T
• Document Page Count: 24
• Document Creation Date: December 16, 2016
• Document Release Date: July 11, 2005
• Sequence Number: 22
• Publication Date: April 23, 1979
• Content Type: MEMO

Approve
25X
25X1
25X1
MEMORANDUM
d For Vease 2005T/@6/& iib-RDP81B00404002400210022-2
SUBJECT: Intelligence Analysis Quality
A memorandum on managing the improvement of intelligence analysis
asserted that defining good intelligence analysis was a necessary step
in the management process. The attached rambling essay (of which I'm
not particularly proud) is a first cut at such a definition.
It will not satisfy anyone, but those who have analyzed successfully
will see some of their instincts reflected. It is,'however, abstract.
I ask the forbearance of those who cannot provide their own examples to
support the generalizations and ask them to accept on faith for a time
that each point speaks to a real and important problem in intelligence
analysis. Examples can be provided, but at considerable cost in length
and classification. For the moment a more or less philosophical discourse
is needed. If we are impatient with this and so limit outselves to
"real" problems like, "How do we fix our assessment of Taoist resurgence?"
or "Who will make up the next class of our analytic methods course?" we
doom ourselves to,inefficient and often counterproductive gropping.
If we are to ever get far enough ahead of the problem of improving
intelligence to control it instead of having the problem drive us, we
must step back, decide what we want to achieve (which the attached paper
addresses in part), then decide how to work toward that (which the
memorandum on management addresses in part), and finally set out to do
it, necessarily piecemeal.
The attached paper is intended to serve as a target for attacks.
Therefore, ample space is left throughout for marginal notes. Hopefully
it will be circulated for comment and criticism, e.g. to the NIOs and
Office Directors. One could then accumulate comments and criticisms,
discuss the issues revealed by them, and revise the paper. Some such
process is needed if this is to be more than an academic exercise because,
if any program is to succeed in improving intelligence analysis, it must
fully recognize that we are trying to change human beings. Proclamations
don't change people. People change only if they are persuaded and
people are persuaded most readily if they are genuine participants in
defining goals and how to achieve them.
THIS MEMORANDUM IS UNCLASSIFIED
I
Approved For Release 2005/,r008,(O j-CC~Y RDP81 B00401 R002400210022-2.
 25X1
Appro~
ed For .ease 2009/U8/0?~'CIA-RDP81 80040,02400210022-2
Some copies of this memorandum will have attached copies of an
interagency intelligence memorandum to support the premise that the
ideas expressed in the paper are not pie-in-the-sky. If this IIM could
be produced in the absence of clear guidance, we can reasonably expect
that much more and better analyses can be produced if sufficently explicit
guidance existed, and was accepted.
Pleas for a philosophical discourse notwithstanding a reasonable
question for a hard-nosed manager to ask is, "If I believe all this,
what do I do?" Several answers are suggested in the memorandum on
management. Some answers are alluded to in the attached paper. No
answer is given either place with sufficient specificity to be implemented
straightaway. This is because the underpinning of both papers is that
successful solutions can only be devised by those who must implement
them. Therefore, the papers only sketch what such solutions might look
like.
However, to close on a concrete note one can envision a group of
managers and analysts debating the proposition (made in the attached
paper) that review processes would be more efficient if they operated on
outlines rather than finished essays and concluding that the following
procedural rules should be implemented.
-Each analysis will be prepared in an outline format.
-After the analyst has completed the outline no editing
is permitted until the review process is complete.
-All reviewers must provide marked up copies of the basic
analysis. The rationale for each indicated change must
be provided.
-The final review authority will examine the basic analysis
together with the comments of intermediate reviewers and
will indicate which changes are to be incorporated.
-Then the analyst incorporates all authorized changes
into the final analytic paper which becomes the official
record of the analysis.
-Thereafter an editor writes essays in various forms,
styles and lengths (subject to the analyst's review)
to satisfy various consumer needs.
One could imagine this group arguing that, no matter how stilted this
seems, we must cut the Gordian Knot if we are to hope to abolish the
25X
Approved For Release 20051 I tkJ~-RDP81 B00401
 Approve
For ease 200 ?8/b?CRTA-RDP81600401&02400210022-2
endless nit-picking and wordsmithing that now characterize our review
25X1 processes and that stifle creativity and bog down production. Then
again, the group might produce a totally different plan. The one thing
that can be said with confidence is that if they themselves produce a
plan that addresses the problems, it will be better as a practical
matter than anything I could conjure.
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
TOP SECRET
25X1
25X
 Approved ForWase 2005/08/03: CIA-RDP81B0040602400210022-2
INTELLIGENCE ANALYSIS QUALITY
"Let these examples also serve to inform the
politician of vast projects that the human mind
can never penetrate however extensive it may be,
because their minute combinations must be developed
in order to foresee or regulate events that depend
on future contingencies. We can explain past
incidents clearly, for their causes are now dis-
covered, but we always deceive ourselves about the
future which is concealed by secondary causes from
our rash and prying inspection. That the expectations
of politicians should be disappointed is not a
phenomenon peculiar to the present age. It has
been the same during all the ages in which
human ambition gave birth to grand projects."
Fredrick, The History of the Seven
Years War, I, viii-ix, II, 369
Fredrick the Great of Prussia serves to start
us on the first task, defining what good intelligence
analysis is not. The second task, defining what it is,
will follow and will be harder.
WHAT GOOD INTELLIGENCE ANALYSIS IS NOT
Fredrick pointed out that good intelligence
analysis is not the same as consumer satisfaction.
Consumers haven't changed in two hundred-odd years;
they'll never be satisfied. Failures will be heralded
and successes taken for granted, the latter because (at
least in part) the timely anticipation of events allows
a decisive policymaker to preempt them. This is not to
argue for ignoring serious policymaker critiques of
intelligence; it does argue that each critique should
be judged on its own merits. Also it argues more
generally that good press will not be ours.
Second, good intelligence analysis need not mean
good intelligence. Intelligence consists broadly
of three phases, collection, analysis and dissemination.
For intelligence to be successful all three steps must
be taken; a misstep anywhere can cause failure. Absent
the basic information, the most finely tuned analytic
machinery cannot produce useful intelligence. Absent
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
 Approved For 40 ease 2005/08/03: CIA-RDP81B0040it 02400210022-2
the ability to provide assessments in forms intelligible
and persuasive to the people who should act, the most
insightful analysis is worthless. In any case, we will
not address here questions of collection and dissemination
in spite of the fact that they are important, we will
limit ourselves to the middle stage.
Next, when our microscope focuses on the middle
stage we find that there are finer gradations in the
process of intelligence analysis itself. We also s.ee that
intelligence analysis is a seamless web so that any cut
for purposes of taxonomy does violence to the whole.
Nonetheless for our purposes here it will be useful to
cut the web into four sections that we may call the
functions of intelligence and describe by the following
catch phrases:
FACTS & FIGURES: Who's where? When does how
much rain fall? How big is it? How much does it
cost? etc.
RECKONING & REPORTING: What happened yesterday or
did nothing happen? What is the military capability
of an assembly of men and equipment? What is the
productive capacity of an assembly of men and
machines? etc.
PREDICTION & PROGNOSES: What will happen tomorrow
or next year? What are the critical factors
influencing developments? What are the key
uncertainties? etc.
WATCH & WARD: Among all the scenarios that might
unfold, which ones are both sufficiently likely
and sufficiently important to the US to merit
special attention?
Each of these functions, make somewhat different demands
on the analyst. Here we will not attempt to address all
of these demands, rather we will concentrate on the
demands of the PREDICTION & PROGNOSES function with only
peripheral mention of the others.
Approved For Release 2005/08123 : CIA-RDP81 B00401 R002400210022-2
 Approved For 0ease 2005/08/03: CIA-RDP81 80040,02400210022-2
Fourth, we will ignore questions of relevance even
though it is clear that when all is said and done superb
analyses of irrelevant problems are not better than shoddy
analyses of urgent problems. Ignoring relevance only
recognizes that we cannot address all of the problems of
the world at once.
Fifth and finally, we should distinguish between
intelligence officers as "analysts" and as "seers." We
are not talking here about the latter function, valuable
as it may be. In discussions of these matters there is a
tension, usually not articulated or even recognized, between
two views. On the one hand some think of good intelligence
as the result of certain methods of reasoning or information
processing which if properly applied to any situation will
produce truth. Others think that truth emerges from a
mental process which is too subtle and complex to be
codified so it can be achieved only by long and intimate
involvement with the subject area. Reality surely
lies in the middle. I know that years at sea gives a sh.ip's
captain a "feel" for his vessel so that he can often sense
when things are not right, but it is only the ability to
mentally break down the myriad of sights, sounds, smells, or
their absence into mechanical, electrical and electronic
systems (i.e. analysis) that allows him to convert that
sense into a program of troubleshooting that will guide his
crew to a defective diode or a stuck valve which can then
be repaired. A similar, but more complex situation, must
exist when one is trying to comprehend the workings of a
political and economic system in another culture. And
success can only come from a proper, and perhaps fortuitous,
marriage of intuition and science.
We must and do try to emerse people so fully in the
culture of another country that they will be able to sense
what is going to happen even if they cannot articulate why.
This is an essential objective of the United States
Intelligence Community, and while it is not the subject of
our investigation here, two points about it are relevant.
First, we seek for seers most urgently in political analysis
because this is the area which we understand least; we
intuitively recognize that we have no immediate hope of
Approved For Release 2005/08/%3 : CIA-RDP81 B00401 R002400210022-2
 Approved For ease 2005/08/03: CIA-RDP81 B0040is 02400210022-2
modeling how the Lower Slobovians operate, but we trust that
the bright people we emerse in the culture of Lower Slobovia
for years will assimilate it in some undefined way and
will thereafter be able to predict the course of
Lower Slobovian politics. The second relevant point follows
from the first. That is, compared to the paucity and
inadequacy of the criteria for judging analysis, we have
no criterion for judging prescience. So in the end
pinning our hopes on area specialists is an act of faith.
WHAT IS INTELLIGENCE ANALYSIS?
Having stated several things intelligence analysis is
not, we must define what we mean by analysis in general
and intelligence analysis in particular. One philosopher
might define analysis as the separation of a complex whole
into its constituent parts in order to study them and
their interconnections individually. Another might, with
equal support in philosophical literature, define analysis
as testing the validity of an hypothesis by tracing out its
implications; its validity is tentatively accepted with
increasing confidence as more and more implications are
traced out and matched with reality, but as soon as one
falsehood is found the hypothesis is declared invalid.
Each of these views captures some of what we mean even
though we do not and never have used the term "analysis"
rigorously in the Intelligence Community. Indeed, in some
cases we have used "analysis" as a synonym for "writing
style"; this clearly has been an inappropriate use of the
term, but even so there remains ample room for differing
usages.
Having said this and further having accepted for
purposes of discussion some combination of the two standard
philosophical definitions of analysis, a reasonable question
is, "What sets intelligence analysis apart from analysis
in general?" The quick answer is, "Not very much.." This
is valuable because it suggests that we do not have to
confine ourselves to post mortems of intelligence failures
or even studies of intelligence successes. We can learn
from the experiences of all the sciences and humanities.
However, the quick answer deserves amplication lest
we ignore two critical points that distinguishes
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
 Approved For 49 ease 2005/08/03: CIA-RDP81 B0040a02400210022-2
intelligence analysis from analysis in general. First,
intelligence is different in the degree to which it
deals with information that belongs to knowledgeable
thinking people who have reason to conceal or distort
it. Therefore, far more than even social scientists,
intelligence analysts must treat their raw material
with a skepticism just short of paranoia. Second,
intelligence analysis, like other relevant analysis in
government and business, often must meet deadlines.
Such deadlines rarely exist to the same degree in the
academic world so the writings in the academic literature
on how to do good analyses should be taken with a grain
of salt. The fact not recognized in some of this
literature is that in the real world, "the excellent is
the enemy of the good," if being excellent
instead of just good in an analysis means that decisions
will have been made before the policymakers see the
excellent analysis or if it means that other areas are
being neglected while fine-tuning one.
We now turn to the subject at hand, "What constitutes
good analysis?" with a final warning. Our discussion
will be disappointing because there's not very much
definitive to be said. (That's why it's taking so long
to say it!) We need to admit at the outset that quality
in intelligence analysis is fundamentally an aesthetic
judgment. We cannot let this be an excuse for doing
nothing, but we should not hoist ourselves on our own
yardstick just for the sake of having some formula or
code that looks scientific. We will never have an
explicit quantitative measure of analytic quality, but
we can and must have explicit qualitative statements
of our aesthetic criteria. The remainder of this paper
is an attempt to state them in the spirit of providing
targets at which to shoot rather than of carving
scriptures on tablets. So we really are talking about some
desirable features of analysis rather than defining
good analysis.
For purposes of discussing analysis we will break
it into half a dozen parts which can be called:
-5-
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
 Approved For ease 2005/08/03: CIA-RDP81 80040902400210022-2
Models - the framework within which we
view the part of the world under study.
Assumptions - those things which are taken
for granted.
Hypotheses - things we don't take for
granted and, indeed, among which we want to
be able to judge.
Facts - those things that we see as
relevant to distinguishing among hypotheses
and as bearing on our comfort with our
assumptions and models.
Reasoning - the process by which we combine
assumptions and facts within our models to
distinguish between our hypotheses.
Conclusions - our assessments of the
hypotheses under consideration.
As we talk about each of these in turn we can describe
a number of pitfalls that have been discovered, usually
inadvertently and always painfully, by analysts down
through the ages. And having done so, we will conclude,
not surprisingly, that good analysis consists of having
handled all of these six components of analysis well.
THE COMPONENTS OF ANALYSIS
Models are the central feature of good analysis.
They are the bones on which the flesh can be hung.
The word "model" is deliberately used recognizing that
it curls many a humanist lip in disdain. This is
because the issue is fundamental and cannot be evaded
if there is to be any hope of dramatically improving the
quality of intelligence analysis. Whatever the unpleasant
connotations of being too mechanistic or too systems
analytic, the issue must be faced squarely. Until we
require ourselves to lay out explicitly our understanding
of how the systems we study work, we have no hope of
improving our institutional ability to understand year by
year. Lacking models because there is nothing sufficiently
solid to criticize, correct, restructure and observe, i.e.
nothing concrete enough to grasp, and change or
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
 to hold fixed to see if it works. On occasion I have
worked with analysts of strange cultures over a period
of months during which time we radically altered our
view of how people in that culture function at least
three times, sometimes passing through old models
without fully realizing it. Only when time permitted
digging through the records of old work-did our
oscillations become apparent. Without explicit mental
models intellectually we are like ants riding a
pendulum; we think we're traveling, but we're only
oscillating in minor intellectual fads.
What is a model? It is an explicit statement
of the process at work. It need not be a huge
mathematically derived computer-based thing, although
in some cases (such as understanding missile performance)
that is appropriate. A model may also, for example, be a seminal
paper on how the Saudi government and royal family
interact, or it may be only an outline. The point is
that when the model is explicit we can critique it and
improve. We can go back to previous work and say,
"These developments are not consistent with what we
said." We can then either modify the model or prove
that the developments are, after all, consistent. But
had the original work simply laid out the Delphic
prophecy that "The Saudi royal family will see its
future as intimately tied to developments in the Arab
world and will react so as to protect its fundamental
interests"...How many times have we read that? ...we
couldn't critique or improve on it. We couldn't even
get hold of it!
In addition to being explicit, what should a model
be? It should be elegant! Elegance, that is, in the
sense of the mathematicians to whom elegance means the
proper balance between brevity and coverage. A model
should be brief and yet capture the essence of reality.
Thus the simplest explanation need not be elegant; it
may be merely an appeal to a slogan, e.g. explaining
French foreign policy as arising from a French vision of
grandeur. On the other hand, convoluted arguments are
noxious even if they are carefully crafted and succeed
Approved For Release 2005/08703 : CIA-RDP81 800401 R002400210022-2
 Approved For Sase 2005/08/03: CIA-RDP81 B0040V02400210022-2
in explaining reality. Copernican astronomical theory
triumphed over the Ptolemaic not because it predicted
the movement of celestial bodies more accurately, but
because it was simpler and yet still predicted.
If one believes this about models, what does one
do? First one doesn't worry very much about OSI, OWI
or even OSR whose people, by and large, have grown up
acclimated to the use of models. One worries
moderately about OER. One worries a lot about OPA where
the whole concept of formal (i.e. explicit) mental models
is anethema. In this environment a right first step
is to require that everything must be outlined instead
of written in the customary essay format. The outline
provides a better framework for evaluation than the
essay where verbage and adverbage can conceal a host of
sins. If the review process operated on outlines
(with the essays being written only after the substantive
analytical work and review was done) it would be much.
more productive.
Models play another role when fleshed out with
assumptions, hypotheses, facts, reasoning and conclusions.
Typically we become more confident of a theory or
model as more and more facts accumulate in support of
it and we reject it when too many "facts" are inconsistent
with it. A model, by being explicit, gives us the ability
to do more than simply wait helplessly to see
what facts fate dumps in our bucket. It allows us to
conceptually vary the components of the model one by one
to purposefully see if our theory is adequate. A
particularly fortuitous combination of insight and
circumstances in this exercise will occasionally allow
us to formulate what some philosophers of science call
the crucial experiment. In an intelligence officer's
terms this means the ability to direct collectors in
search of those crucial pieces of information that will
allow us to determine if our whole theory is supportable.
More often the use of explicit models will not lead us to
such grandiose heights, but it will often give us the
ability to guide the collectors to more fertile fields
than they would otherwise cultivate.
Approved For Release 2005/08/D3 : CIA-RDP81 B00401 R002400210022-2
 Approved For ease 2005/08/03: CIA-RDP81 B0040 is 02400210022-2
A final comment on models risks taking this
discussion far into esoteric areas. It is that the
explicit statement of the relations which are assumed
to exist in our subject area allows us to consider in
an orderly fashion what things are constant and hence
reliable and what things are variable. The degree to
which we can do this will be the major determinant
of our predictive power. For example, the reason that
physics theories generally predict well while
economic theories predict poorly is that physical models
contain "conservation laws" at critical points while
economic models do not. For example, Newtonian physics
is largely based on the conservation of momentum and
energy while microeconomic theory is based on the
concept of individual economic wants or values which
do not obey conservation laws, i.e. a person's desire
for a good is not invariant, but increases or decreases
in anticipation of future price changes.
The existence of this difference between physics and economics
would not even be apparent to us unless the two sciences
had laid out their respective theories explicitly so that
we could study the differences between them and recognize
in both cases the limitations on prediction that are
implicit in the strength of their conservation laws. Few
intelligence models can include conservation laws as strong
as those in economics, let alone.those in physics. But
the explicit enunciation of how we think things work will
allow us to intelligently judge the circumstances under
which habits will remain operative, i.e. crude
conservation laws will be obeyed. Thus to say, "the
Vietnamese are bloody minded" is useful only if it defines
how the senior decisionmakers in the Vietnam government
will respond to various pressures and defines the limits
within which those pressures may vary without causing a
fundamental change.
Assumptions. Analysis in any field must proceed
on the basis of assumptions. In fact, to a large extent,
the degree to which.a set of assumptions commonly held
by all practioners in a field are adequate to provide
the basis for fruitful work is a measure of the advance-
ment of the field. The two major requirements we should
impose on our assumptions is that they be explicit and
that they be consistent across our hypotheses.
Approved For Release 2005/08103: CIA-RDP81 B00401 R002400210022-2
 Approved For0ease 2005/08/03: CIA-RDP81 B0040it 02400210022-2
A major defect of much intelligence analysis is that
too many assumptions are implicit. As a consequence
errors creep into the process. Consider, for example,
assumptions about US policy. Intelligence analysts
read the newspapers; thereby they absorb certain per-
ceptions about US policy which are commonly held by
Americans concerned with foreign policy. On the other
hand they are rarely privy to the thinking of senior
policymakers and therefore do not have an accurate
picture of the options that are or could be considered.
Thus their analyses can be based on inadequate under-
standing and will be read by people whose frame of
reference is different from theirs. It would be
easy to argue that policymakers should tell intelligence
analysts what they think are viable options, but this
line of argument is pointless. First, because policy-
makers often don't know well enough themselves to be
able to articulate the range of options
any better than intelligence officers could.
Second, they are often too busy to do so even if they
could articulate options better than we. And finally
because,for a variety of reasons, this is a matter outside
the control of intelligence officers. Therefore, as a practical
matter, what intelligence officers can do is to establish
a rule that their work will contain an explicit statement
of what they assume reasonable US policy options are
and then to trace out the impact of those options on
the situation under study. It may well be that the
result will be a conclusion that US policy will not
impact on unfolding events. However, it is also
possible that the explicit statement of such assumptions
will lead to further questioning of the analysts by
policymakers who see other options that should be
explored. Laying out options and exploring their
implications need not imply endorsement of one or
another option.
Definitions also take on the character of assumptions.
For example, political analysts typically (and subconsciously)
define as equivalent the names of a government leader,
government building, capital, country and nationality.
Thus Brezhnev, Kremlin, Moscow, USSR, Russians and
Soviets are used as synonymous. Such shorthand is an
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
-10-
 Approved For ease 2005/08/03: CIA-RDP81 B0040002400210022-2
insidious perpetrator of the homogeneity syndrome,
i.e. treating foreign nations as if they were homogeneous.
The syndrome often also involves the unwritten assumption
that heads of state and leadership elites make decisions
solely on the basis of a careful and completely rational
calculus of world politico-strategic facts rather than
emotion or internal power struggles.
To root out this form of implicit assumption analysts
should be forbidden for one full year to employ this
shorthand. Furthermore until examination of internal
diversity becomes commonplace a statement like,
"I have examined all important groups in Brazil and
find unanimity with respect to this matter." should be
required in all analyses that do not explicitly
examine the diversity in the situation under study.
Consistency of conclusions from year to year is
not a virtue, but consistency of assumptions among
hypotheses within a particular analysis is essential.
I cannot illustrate this point directly because those
analyses I know which lay out hypotheses explicitly
seem to keep assumptions consistent but most intelligence
analyses don't lay out hypotheses and so the question
of consistent assumptions doesn't arise. However,
the flavor of the problem can be captured by looking
at analyses of different questions pertaining to the
same issue. For example, how often upon probing do we find the
assumptions behind analyses of India's foreign policy
to be different than the assumptions behind analyses --on
the self-same issues--of Pakistan's foreign policy?
All too often if one believes that Pakistan-India
relations are a crucial part of the two nations'
foreign policies.
Hypotheses. An analysis should explicitly
evaluate at least three viable alternative hypotheses each
of which purports to explain what we see. "Viable" means that a
patently false strawman, set up only to be knocked down,
won't do. If there is only one viable hypothesis the
analyst should abort his project and turn to a worth-
while topic. "Alternative hypotheses" should be
meaningful in the sense that which hypothesis is true
should have a significant impact on the future.
Otherwise the matter is not worth pursuing.
Setting three as the minimum number of
hypotheses reflects the experience of people who have
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
-11-
 Approved Forivease 2005/08/03: CIA-RDP81 B0040a02400210022-2
tried to explore decisionmaking in groups of people.
The rule of thumb that has emerged from working with
committees and decision papers is that 3-5 alternatives
represents the range that can be handled when spoken
and written English is the main analytic tool. Fewer
than three alternatives rarely can cover the range
of options credibly, and as one moves beyond five
alternatives people tend to become
confused. In the abstract, however, our goal should
be to examine alternatives which for
practical purposes exhaust the possibilities and yet
are limited to those that are significantly different
from one another in the sense that whether one
hypothesis or another is true matters. In some OSI,
OWI, OSR and OER analyses we already regularly
employ many hypotheses, but these are cases where
the main analytic tools are mathematical. To the
extent that we improve our analysis in general we can
perhaps handle more hypotheses.
But for the moment we should walk with three
and defer consideration of running. Lest we let our
ambitions soar too high at once, we should note that
the high, low and best estimates of foreign military
forces produced in various intelligence documents
exemplify one way to cope with the problem of
alternative hypotheses. Anyone who doubts the
difficulty of changing habits of doing business
should retrace the agonies of the mid-60s when
senior Defense Department policymakers dragged the
Intelligence Community kicking and screaming into
this now commonplace practice. It will be no easier
to get analysts to formulate alternative political
or economic scenarios, alternative motivational
explanations, etc.
This example, however, does serve to illustrate
the practical importance of the use of alternative
hypotheses because the projection of high, low and
"best estimate" forces is in large part a matter of
differing hypotheses of foreign intentions. The
Approved For Release 2005/08/92_ CIA-RDP81 B00401 R002400210022-2
 Approved Foroease 2005/08/03 : CIA-RDP81 B0040002400210022-2
intentions vs. capabilities debate frequently arises
in military intelligence. Clearly intentions
unsupported by capabilities are not important,
except perhaps in the case of a foreign government
which is so oblivious to consequences that it
might act in defiance of the discrepancy between
its intentions and capabilities. On the other hand
focusing totally on capabilities (which is generally
equivalent to focusing on "high" projections)
violates a central purpose of intelligence, to
approximate the truth. The use of three
hypotheses leaves a lot to be desired, but it is
a better approximation than any one hypothesis.
A final comment on hypotheses is that even
in the 3-5 range, the "right" number of hypotheses
to be subjected to detailed examination will vary
with circumstances. Initially various hypotheses
must be subjected to an informal evaluation
in which the probability of each hypothesis
is weighed with its importance if correct. In this
way both very interesting but highly improbable
and likely but trivial hypotheses will be eliminated
from further consideration in order to increase the
efficiency with which a smaller group of operationally
important and reasonably likely hypotheses can be
studied.
Facts. A common saw, especially at middle
levels, on the policy side of the government is,
"Give me the facts, I'll draw my own conclusions."
This naive view fails to realize that what is a fact
is a matter of judgment, but a similar naivete often
afflicts intelligence officers. The intelligence
officer's version of this error is failure to
realize that facts and theory are not separable.
The accuracy of one's mental model of the process
under study determines to a large extent whether
one recognizes a fact when one sees it. To
illustrate the point imagine ourselves as Eskimo
schoolchildren reading a Post "Style" section
Approved For Release 2005/08IQ3~_ CIA-RDP81 B00401 R002400210022-2
 STAT Approved For Release 2005/08/03 : CIA-RDP81 800401 R002400210022-2
Next 1 Page(s) In Document Exempt
Approved For Release 2005/08/03 : CIA-RDP81 800401 R002400210022-2
 Approved For Vase 2005/08/03: CIA-RDP81 B0040is 02400210022-2
"Are we sure we were looking hard enough for
evidence of a deal to buy the Soviet military's
support of a move to depose Krushchev that we
would not have missed a piece of evidence?"
Eliminating irrelevant facts in order to
economize effort is also important. Because
connections are especially fuzzy, political
analysis is particularly susceptible to the
intrusion of irrelevant information. The
basic guideline for this is that a piece of data
is not a "fact" (worthy of attention) unless it
serves to:
-establish whether or not the mental
model in use is adequate,
-determine if the hypotheses under study
are mutually exclusive and exhaustive, or
-distinguish among the hypotheses under study.
Facts should be presented reasonably consistently.
Apples are rarely compared to oranges in OWI and such
comparisons are reasonably easy to detect in OSR,
but in OPA it is sometimes hard to decide what's an
apple and what's an orange.
Earlier we noted that intelligence analysis is
unique in the degree to which it deals with
information that belongs to knowledgeable thinking
people who have reason to conceal or distort it.
Therefore, we asserted that intelligence analysts
must treat their raw material with a skepticism just
short of paranoia. This applies particularly to
those classes of "facts" which are susceptible to:
-Shifts of meaning in transmission and
translation, a problem that can be
especially significant in dealing with
human reporting.
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
-16-
 Approved For f ease 2005/08/03: CIA-RDP81 B0040 02400210022-2
-Deliberate falsification with intent to
deceive. The history of the intelligence
business is full of examples of this.
-Fabrications without malicious intent, e.g.
a common practice in journalism is to
report as fact that which seems plausible
in light of other information, ethnocentricity
can be especially damaging here. This is
not to argue that press reports should be
treated with particular skepticism. It is
to argue that the intelligence analyst
should always be questioning the motives,
knowledge and proclivities of every source
of facts.
Reasoning. The lines of reasoning in an
analysis should be arranged so that a literate person
can trace clearly the connections between facts,
assumptions and conclusions for each hypothesis.
Each step of the reasoning should be clearly
identifiable as implication or intuition. There
is no sin in the latter. Indeed, as long as it
is labeled as such that is what we want to pay area
specialists to be able to do. One of the great
intellectual beauties of good analysis in any field
is the use of innovative techniques to carefully
reason across, to bridge, the unknowns in the
problem. In this respect, the intelligence
analyst's problem is not radically different
from the archaeologist's. However, the praise
of intuition and imagination in any field should
not be a warrant for sloppy reasoning. Again at
the level of detail appropriate to distinguishing
among the hypotheses under study it should be
possible to follow the analyst's thought processes
and to critique them. The earlier discussion of
the utility of models for these purposes applies
to reasoning as well and need not be repeated.
Approved For Release 2005/08/43_ CIA-RDP81 B00401 R002400210022-2
 Approved For *ease 2005/08/03 : CIA-RDP81B0040002400210022-2
25X1A
Conclusions. Six years ago told
analysts that the least important element in drafting
an estimate is the conclusions. This remains true
although 0 might not agree with the reasons stated
here. If everything else in an analysis is done well
the conclusions will follow relatively easily as long
as we recognize that "conclusions" is plural. The
most important part of presenting conclusions is
developing this plurality. Doing so consists of
some or all of the following points.
Differing views of truth can be held by
reasonable men who examine the same analysis. These
differing views should be presented clearly and
objectively. It is not sufficient to say that CIA
holds one view and DIA another. The analysis must
be retraced to show how different weights put on
different assumptions and facts and how different
reasoning lead to the different assessments of the
debated hypotheses.
A part of the conclusions of an analysis is
identifying uncertainties and evaluating their
importance, a process usually called sensitivity
analysis. Virtually everything that enters into
an intelligence analysis is uncertain so as a
practical matter one cannot agonize over every
uncertainty. The key in sensitivity analysis is
to select those factors in the overall analysis
which are both uncertain enough and influence
the determination of which hypothesis is correct
enough to merit detailed examination. Having
made our selection we must ask how the evaluation
of the hypotheses would change as the factors
themselves change individually and in combination.
There is no place in analysis that the intelligence
officer's understanding of his subject is put to
a harder test than in this elucidation. It is so
hard that sins of omission must surely be more
forgivable here than anywhere else. There are
no rules except perhaps the thesis that it is
better to be roughly right than precisely wrong.
Approved For Release 2005/081q,%: CIA-RDP81 B00401 R002400210022-2
25X1
 Approved For Spase 2005/08/03: CIA-RDP81B0040002400210022-2
Having dwelled on the need to illuminate
uncertainty and disagreement, we must recognize
that at the end we have a moral obligation to
provide the policymaker with our judgment. This
is an awesome responsibility because we are not
dealing with trivia. One refuge from this respon-
sibility has been sought all too often in the past,
that is to state Delphic conclusions which can
never be disproved by events. The obvious solution
of requiring the analyst or team to state cate-
gorically which hypothesis is true is not, however,
workable because analysts simply do not (and
cannot) know.
A reasonable (and workable although painful)
compromise is to require stating the likelihood
of each hypothesis being true. This can be done
with at least three degrees of precision.
-Approximate probabilities. Recognizing
that precise probability statements, say
to the second decimal place like 93%, will
normally not be worth the effort to produce,
we should only aspire at most to
approximate probability statements like
90% or 35%.
-Odds. One can give a somewhat less
precise assessment by using odds based on
whole numbers. A mathematician
would say that this means little in
principle. However, some people feel
more comfortable saying
"3 out of 4 chances"than "75% probability,"
and for whatever reasons, readers do
seem to take statements like "even odds"
or "50-50 chance" as being less precise
than "50% probability."
-19-
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
 Approved Forlsease 2005/08/03 :
CIA-RDP81B0040002400210022-2
-Adjectival statements.
For
analysts who are simply petrified with
fear at the prospect of using numbers
words must suffice. The Sherman Kent
chart is an attempt to codify such
statements and to relate them to more
precise measures. If we do no more than
agree to use some such standard
terminology we will have made progress
because now an adjective like, for example,
"probable" can mean anything from
"just better than equally likely" to
"virtually certain."
CLOSING REMARKS
When one has laid out an analysis in accordance
with these guidelines what does one have? Some very long and
pretty dull reading to be sure. However, one has
the basis for a reasonably competent writer (who
needs to know little about the subject) to repackage
the product ad infinitum to serve the needs of
high-level and low-level policymakers, and general
thrill seekers. If the analysis is properly done,
packaging the product for dissemination is relatively
easy. Our current agonies over repeated reviews
and debates ad nauseam over turns of phraseology
arise in part because we are papering over obvious
cracks in our analytic walls.
To repeat an assertion made in the context of
models, one also has a vehicle which will facilitate
future critiques of the analysis for everything is
laid out naked inviting any would-be kibitzer to
take pot shots. Even folk who do not understand the area under
study will be able to detect the standard errors
that have plagued men's intellectual endeavors in
many fields. For example "good analysis," as seen
here,. makes such fallacies as "Achieving maximum
gain at minimum cost" stand out more starkly
to would-be critics.
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
 Approved For0ease 2005/08/03: CIA-RDP81B0040002400210022-2
Similarly, critics are invited to proclaim
that the analyst has used too few indicators.
Why will this be the case? Because people
always use too few indicators whether they're
betting on horses, planning personal estates or
trying to detect the outbreak of war. Unconsciously
we all pick a few, no more than half a dozen, things
we look at in determining trends. This is inherent
in our mental machinery. The analytic process
can extend our horizons to perhaps tens or hundreds
of things, depending upon how we go about it.
This assertion about the role of good analysis
in facilitating criticism is repeated
because it is fundamental. We must produce work that
can be criticized! This goes very much against the
human grain. No one likes to be "torn apart" by
critics; everyone subconsciously seeks to avoid it
by a variety of devices, some of which have been
discussed above. Accepting this view implies
that we will shift our thinking about good
analysis away from the finished products of the
Intelligence Community toward the intermediate
products and processes. However, without a view of "good
analysis" similar to the view here we have no way,
consistently year after year as people come and go,
to build a better and better understanding.
We should, however, end on a humble note.
The Earl of Halsbury, drawing on his experience
in World War II, lectured a group of analysts
twenty-five years ago and pointed out that we
will never reach the end of the process. He said,
"The content of a human social or
historical situation is always richer than
verbal analysis can display. Reality is like
a tangled skein of threads. With infinite
patience the analyst dissects one thread
out of the bundle and is delighted to observe
a causal nexus between its parts. We thus
reach truth, but never The Truth. The
reality inevitably bristles with more
detail than can be apprehended as a whole."
Approved For Release 2005/08/03 : CIA-RDP81 B00401 R002400210022-2
-21-
 25X1 D Approved For Release 2005/08/03 : CIA-RDP81 800401 R002400210022-2
Next 36 Page(s) In Document Exempt
Approved For Release 2005/08/03 : CIA-RDP81 800401 R002400210022-2