SEMINAR ON NATIONAL INTELLIGENCE ESTIMATES

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP81B00493R000100020010-0
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 16
• Document Creation Date: December 16, 2016
• Document Release Date: October 7, 2004
• Sequence Number: 10
• Publication Date: February 26, 1980
• Content Type: MIN

Approved For Rease 2004/10/28 ? CIA-RDP81 B0049300100020010-0
SE'CRE
SEMINAR ON NATIONAL INTELLIGENCE ESTIMATES
Center for the Study of Inte:'ligence
Central Intelligence Agency
Washington, D. C. 205))5
26 February 1980
25X1
S'1CR1"T
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved For Release 2004ATCIA-RDP81 80049300100020010-0
Seminar on National Intelligence Estimates
Perennial questions about the validity and utility of
National Intelligence Estimates (NIEs) again surfaced from
the recent inquiry by the Center for the Study of Intelligence
into the impact of intelligence on the foreign policy review
and decision process. Policy makers have been critical of
the relevance of NIEs and analysts have expressed concern
that the goals of the estimative process are no longer
clear. The establishment of the National Intelligence
Council (NIC) as a collegial body to prepare and issue NIEs
provided a timely occasion to take a. critical look at estimative
intelligence.
On February 26, 1980 a group of 15 people met under the
auspices of the Center for the Study of Intelligence to
share their views and to exchange their ideas on the NIE
process and its role for the future. Participants were
drawn from Intelligence Community specialists who have been
or who will be actively involved in estimative intelligence.
The agenda. was designed to provide for a'discussion of the
purpose, theory and practice of estimates. Particular
emphasis was placed on the following questions:
--What role should NIEs play in making foreign policy?
--Is there an ideal methodology for writing an estimate?
--Can an evaluation process to judge estimates be
establish-cd ?
I . Historical Persct-ive
At its inception, the National Intelligence Estimate
was designed to serve as a. comprehensive analytic product:
--The NIE was to be a paper in which all of the
available and relevant resources within the
Community were brought to bear on the question(s)
at hand.
--By projecting forward, the NIEs were to shed
light not only on the situation as it existed,
but also to make judgments about the likely course
of events. This was not meant to be a prediction
but rather a means of bringing out a range of
SECRET
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved For R4ease 20044M9 CIA-RDP81 B00493WO0100020010-0
possible developments and the relative probabilities
that they might occur.
NIEs were to provide a process for uncovering and
illuminating real differences of opinion where
they existed among the various agencies. They
were-to provide, for example, an opportunity for
the NFIB principals to dissent from the main
paper. The fact that there were differences
among the agencies or among analysts within a
single agency was not important; rather it was-
an explanation of just what the arguments entailed
which mattered. most.
Early Days
In the early days of the Office of National Estimates, the
original role and purpose of estimates was clear. Under the
Eisenhower administration, estimates played an integral part
in the policy formulation process. Each policy paper was
required to have an estimate attached to it. The process
also included an extraordinary effort to find out on a daily
basis exactly what was bothering the President and what he
really wanted to know. The NIEs were then carefully tailored
to address those concerns.
Viewing this 1950s period, in hindsight, one participant
felt that it would be more appropriate to label it as the
"Golden Age of Estimates That Never Was." It was really
during the Kennedy administration that estimates had an
important policy impact, although'it was more unstructured
and less deliberate. The DCI himself was actively
involved in the estimative process during this period and
had the confidence of the President. In addition, senior
intelligence officers had access to the NSC principals, and
were able to provide useful and timely products directly to
them.
Losing the Path
Whether due to organizational or personality problems,
it was the consensus of those present that we have somehow
"lost the path" toward effectual estimates in recent years.
Three reasons were suggested for this:
--The vertical structure of analytic elements:
2
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved For M ease 2004/'IUl28I cIA-RDP81B00493R6O0100020010-0
Policy makers want papers that provide a synthesis of
materials reaching across diverse disciplines. They want
intelligence analysis to put into perspective the various
elements of a complicated situation. Such papers must
provide the policy maker with an understanding of how these
factors--political, economic, and technical--interact, and
they must be written so that they can be rapidly read and
absorbed. With the exception of the annual strategic and
conventional forces estimates, there seems to be no real
mechanism within the intelligence community today that can
bring about this synthesis of facts and ideas. Instead
the Intelligence Community provides "1000 bits of information
stapled together--and thus 1000 pages to be rea.d."
--The United. States as reactor:
Within a rapidly changing international environment the
United States has become less an initiator of action than a
reactor to events. The structured style of policy making
that predicates estimates on how others will react to U.S.
action is thus obsolete. In such a "catch-up" situation
there is some confusion about what the Intelligence Community
should. provide. Are policy makers so concentrated on the
immediate, tactical problems that they miss or ignore the
larger implications of their actions? This problem is
reflected in the-difficulty of getting anything written and
read beyond current intelligence.
--The dispersion of skills formerly residing in ONE:
Although there were many pluses and minuses to ONE,
there seems little doubt that estimates it produced were used
by policy makers during its existence. There is no single
production office today that has the analytic skills the ONE
Staff and Board were able to bring together during their
heyday.
One participant concluded this portion of the discussion
by reminding his colleagues of the danger in generalizing
too much or of putting the past into neat, well-defined
little categories. Even in the past there was no instant or
automatic line-up between an NIE and policy. Sometimes NIEs
ran counter to policy; sometimes they had a potential for
impact but were ignored by policy makers.
SECRET
3
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved For Release 2004/ ' d/f$Z! IA-RDP81 B00493MD00100020010-0
A. NIE Role in the Policy Process Today
What role should the NIEs he playing in the foreign
policy process? Is it possible to construct them so that
17
they fit the requirements of any given set of policy makers?
For the present administration, tactical (short-range)
decisions made within a critical mode seem to have taken
precedence over those with a more global (long-range) scope.
If NIEs are intended to play a role in the more global
aspects of decision-making, can they still he used effec-
tively in crisis?
Several participants felt that NIEs have an important
role to perform even in tactical situations. They cited the
recent example of Panama as one in which an intelligence
estimate provided important support in dealing with what was
essentially a tactical policy problem. Yet many recognized
that, as policy makers deal with short-range problems,
traditional estimates are not always suitable. It appears
that administrations become increasingly short-sighted as
their time in office proceeds, tending to concentrate
more on the immediate problems that develop.
Perhaps the tendency to shift from long-range to tactical
policy making is one reason why the group could generally
agree that Special National Intelligence Estimates have had
the best track record over time. SNIEs were designed to
address a narrow question over a finite period of time, to
bring the issue into sharper focus and to use the best
people available to accomplish this. The direct question/specific
answer format did much to get the policy maker personally
involved in the process. There was some concern among the
participants that the SNIEs had become eroded over time and
overridden by other systems. These "others"--the NIO system,
for example--have not as yet been brought into the process
sufficiently, and this allows the policy maker to go about
his business without directly bringing in the Intelligence
Community. One participant suggested that we should think
about. returning to the SNIE in the future in order to draw
the Commun:it back into the process and not just one individual
or agency.
It was the consensus of those present, however, that to
apply NIEs only to the actual decision stage of policy making
is a much too narrow view of their usage. With the many
potential intelligence inputs available, the NIB provides a
SECRET
Approved For Release 2004/10/28 :4CIA-RDP81 B00493R000100020010-0
 Approved For Release 2004/'10"/ZS' CIA-RDP81 B00493Q00100020010-0
conceptual framework which has been argued. out within the
Community. It provides a carefull.y thought-through backdrop
to help policy makers address problems as they occur. This
instructive quality of estimates is important.
The formal process of doing an estimate itself serves
a twofold purpose: organization of substantive material and
elimination or a "factoring out" of that which is unlikely
to occur. If people. across the Community have done or are
doing their jobs, the policy makers will be given:
--a systematic view of the problem in all its aspects,
--a checklist, and
--an examination of relevant factors perhaps not
thought about before.
Whether the judgments to be drawn from this information
are right or wrong is another question. Nevertheless,
estimates ought to provide some assurance that the whole
question has been thought about in depth.
III. Usefulness of NIEs
After a general acknowledgment that NIEs still do have
a viable role to perform in the policy process, the seminar
discussion turned its attention. to their actual usefulness
to the policy maker. Some consumers have complained that
estimates are not relevant to a discussion of policy options
for they do not provide clues to the options policy makers
must eventually decide upon. Others complain that when
NIEs do directly address policy options, it is in terms of
"assailable generalities."
In a potential crisis situation, such as El Salvador,
to state in an estimate that the chances for the survival of
the present junta are well less than SOo is an irrelevant
judgment for the-policy- maker who is already committed to a
particular policy. He cannot readily abandon this policy
orientation without losing something of his credibility in
the process. A judgment which says that his policy may be
in serious trouble only, reminds the policy maker of the
risks he has taken.
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 SECRET
Approved For eIease 2004/10/28 CIA-RDP81 B00493P000100020010-0
It is important to keep in mind that military estimates,
notably NIE 11-3/8, are really quite different from other
estimates where a political or economic context is the primary
focus. The efficacy of the military estimates is less in
dispute, but in fact their role is different from that of the
"country/problem" estimates. In codifying a view of the state
of adversary strategic or conventional forces, the predominant
impact of these estimates is not at the higher policy making
levels, although they may have an indirect effect there.
Instead, they provide data to be used as basic reference
material for all-level briefings and future planning.. One
participant commented. that one could corner the Secretary
of State with a military estimate and get him to listen to
it. Such technical, numbers-oriented estimates are more
readily accepted because desk officers and policy makers do
not know how to deal with such information on their own.
Considering themselves to be accomplished political analysts,
policy makers feel capable of making just as good a judgment
on the broader issues as an intelligence estimator.
Intelligence by,Osmosis
Information appears to move through the national
security structure almost by osmosis. There is no stan-
dardized way by which people are informed, particularly
senior officials. Instead there are the informal luncheon
conversations, the more formal staff-prepared papers and the
oral briefings. The result of all this is the welling up of
vast amounts of intelligence as it moves through the structure
in its various forms. Intelligence papers shape people's
views often without their knowing it. Even if policy makers
do not read a particular paper, they may still obtain its
information from their colleagues or staffers with whom they
talk. Such is the problem of assessing the usefulness of
estimates--policy makers' criticism that NIEs "do not tell
me anything that I already don't know" may in fact stem from
this acquisition of NIE information via other sources.
In remarking on the disrespect shown to NII"s in recent.
years one participant related. the following story:. A junior
policy maker once told an intelligence officer "I do not get
my estimates from you, I get them from The Washington Post."
Whereupon the intelligence analyst responded, "Ali, but where
do you think The'Washington Post got their estimates!"
SECRET
6
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 "6A
Approved For Release 2004 L :-RDP81 B00493W00100020010-0
Based on his experiences, one discussant stated that
many NIEs are being used at least in the sense that they are
read, considered or modified with other inputs. Whether or
not a policy maker agrees with them is another matter. He
may consciously or unconsciously extract bits from them and
put these into his own products. There are countless inputs
into the policy process, and intelligence is only one of
them. Within intelligence, NIEs are only a single component.
Nevertheless'. NIEs may have a secondary influence through
their effect on the opinions of those who are to participate
i.n the policy meetings.
Making Estimates Relevant
There are no easy solutions to making judgments about
the impact of intelligence on policy makers and in determining
whether an estimate is relevant to a policy maker's needs.
Intelligence estimators can be more certain about the
relevance of military estimates, but in the realm of politics,
making such judgments is more difficult. Some discussants
thought that we do not know as much as we should about what
policy makers really want.
One means of finding out is to increase our contact
with consumers, particularly through the NIOs. Active
dialogue must be established with the policy makers:
---to detect the need for an estimate,
--to make the estimate policy relevant, and
-to enable the estimate to forecast to the point
where it can give clues as to what policy makers
should be considering in the future.
The interaction between policy makers and intelligence
is a. critical. part of the estimative process. But how are
we to get policy makers more directly involved? Both INR
and DIA make an effort to bring policy people into the
estimates process by getting their inputs to the kinds of
questions estimates should be addressing, by discussing with
them their concerns and by including them in the framing of
the terms of reference. It is said that. policy makers are
always "too busy." But' is this the case?. INR, for example,
has routine access to policy makers through regular briefing
hours. It is important to get as many as possible on the
policy side involved in the terms-of-reference stage. If
7
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved For lWease 2004/ U/zS CIA-RDP81 B00493M00100020010-0
policy makers believe that they have a. stake in the estimate,
it is more likely that they will be interested in the final
product.
The Quality of Estimates
Although admittedly a. generalization, it was felt that
there had been a decline in the quality of the estimative
product relative to the ability of the consumer to do his
own estimating and intelligence work. In part this has been
clue to a growing sophistication of policy makers as well as
their more direct access to raw intelligence traffic. If
"knowledge is power," then perhaps the Intelligence community
has indeed lost something.
There was some disagreement, however, concerning a
second broad. generalization, i.e. with the breakdown of ONE
there has been a decline in the quality of people responsible
for producing estimates. One participant felt that in
actuality the reverse is more likely true. In its latter
years, the Board. and ONE included people who had nowhere
else to go. Sent "out to pasture" in effect, they were not
taken seriously by the senior management. Today our analysts
are better trained, even though they have more limited
experience at estimate writing. The important point is how
the available people are used. in constructing an estimate:
one can follow the old "Greco-Roman" style with its rigid
staff system or the freestyle, ad hoc arrangements found in
more recent years. Despite complaints by consumers of
irrelevance and length, our estimative system is unique in
its ability to handle competing sources of information.
IV. ovements for NIBS
It was suggested. that perhaps the production of NIEs
should follow more closely that of SNIEs, only placed- in a
broader context. As with SNIEs, policy people should
be brought in to help form the terms of reference. As in.
the SNIEs, the questions to be addressed should be rigorously
defined and focused.. Such rigor is especially important
where the consequences of U.S. policy deci~i.ons would be
catastrophic if. wrong. Estimates should attempt to. give
alternative views and to include the implications of bad
judgments. The NI1" me'cha.nism should provide to the policy
maker not an answer but a range of possible answers.
Approved For Release 2004/10/28: CIA-RDP81 B00493R000100020010-0
 Approved For Tease 2004IS1b/ I CIA-RDP81 B00493ii000100020010-0
Is there some. methodological way to provide this range
which is more "right" than others? Sherman Kent posited an
analogy of the estimate process as a pyramid--the known and.
accepted. facts making up its base with the point at the top
being the agreed judgment about what is going to happen. In
reality, however, this final, single point is rarely reached;
perhaps we ought not to try. Perhaps it would. be more
useful to outline a scope of possibilities ranging from more
than likely potential problems to those which can safely be
ignored as unlikely. ?
NIE production currently tends to focus on assessments
only of what the most likely course of action is or would be
in the face of a. crisis; policyinakers are left unprepared
for any others. There needs to be more attention to assessing
the implications for the United States of alternative courses
of action.. The cluest.ion arose as to whether or not such an
assessment would in fact be an estimate or some completely
different paper. Most participants seemed to agree that we
have the opportunity to do a. lot more and still have it
considered. a legitimate estimate.
An excellent case study is a recent paper on the
situation in El Salvador. As an estimate this paper goes
beyond. the basic. judgment that the junta will fail. The
paper states that barring any major outside intervention, it
is inevitable that the revolutionary process will be triumphant
and will profoundly alter the existing structures and relation-
ships. It is the nature and duration of the revolutionary
process itself which will determine the final outlook:
--if the process takes a long time (e.g., two years)
the. final composition will be more extremely radical.
--if the process is completed quickly (e.g. six months)
the final picture will be more moderately left-wing.
But nothing in the way of U.S. policy will make any
difference.
The question of what impact an estimate will. have is an
important consideration. NIEs can have the most impact where:
--There isJestablished an interactive network of policy,
and intelligence contacts,
--The DCI is actively involved., and
Approved For Release 2004/10/289 CIA-RDP81 B00493R000100020010-0
 Approved For11ie1ease 20041LQ.jCIA-RDP81 B00493W00100020010-0
--the estimate does for the customer what he cannot
do for himself.
The policy maker's first reaction is "what can we do?"
It is the job of NIEs to present the situation as it exists
or will exist. It is.however up to the policy maker to deal
with it. NIEs can posit responses to U.S. policy options,
but they cannot formulate those options.
Caught up In daily operational routines, the policy
maker may not be aware of some potential or developing crisis.
For example, Iran did not receive early attention because
policy makers and their staffs were concentrating on Arab
and Israeli difficulties. NIEs must serve to keep the
policy maker from getting side-swiped by unexpected events
while attention is being diverted elsewhere. Here an added
premium is placed on the personalities of the NIOs involved--
whether they are energetic indealing with policy makers,
whether they have the respect of their policy counterparts
and whether they are effective in presenting the estimates,
especially ones which may be uncongenial to the policy
maker's plans.
Problems in Coordination
There was some discussion about the El Salvador example
as an "estimate." Although speciFic i_n scope like an SNIE,
it was produced as an NFAC product. This led directly into
a comparison of the impact of NFAC versus interagency papers.
The question here centered on the value accorded to our
community coordination procedures. There is a common feeling
among NIP consumers that coordination results in watered-down,
rather useless consensus judgments. To be considered more
useful, interagency estimates must not only lay out the existing
disagreements but explain them as well.
Part of the dissatisfaction with coordinated interagency
papers derives from their relative timeliness. Community papers
are slower, more protracted in their production. Agency papers,
by comparison, are thought to be more up-to-date. This may,
however, be attributable to reader prejudice, as community
Alert Memorandums are produced in a day.
Regardless of whether or not an important paper is termed
an "estimate, it is up to the Community to see that the
right people are made aware of its existence. Unfortunately we
tend to drop the ball at this point, relying almost on chance
to see that it is read.
SECRET
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
10
 SECRET
Approved For$eiease 2004/10/28 : CIA-RDP81B00493W00100020010-0
V. Methodology
Is the policy maker better served if we do alternative
analysis or if we seek to reach agreed judgments? While
acknowledging policy-maker criticism of coordinated judgments,
a few of the seminar participants felt that there was a
certain amount of myth to the merits of creative disagreement.
Alternative judgments do have valise but where a consensus
can be reached it is good. to say so. It need not, however,
be an either/or situation. It is possible to do alternative
analysis and still reach some agreement.
The 1973 Middle Fast war was cited as one example. It
was the consensus of the intelligence community that the
Arabs would not attack. Nonet.hless it was thought valid to
consider the alternative hypothesis that they would attack.
But the system did not operate to bring out this second
"less probable" alternative. Perhaps this is in part due to
the hammering out of the various positions by each group
before the interagency session is convened. It is highly
unlikely that real differences among analysts of one agency
will surface in the process unless they also exist as
differences of view among agencies.
There was sonic disagreement as to the use of footnotes
in estimates. One participant thought that the footnote
procedure was pejorative, implying that there was some
"rebel" out of phase with the group. Another participant
felt that the footnote was useful. in that cli ff-erences of
view not directly expressed in the text of the final estimates
would stand out and be read. Appropriate footnoting would
prevent a reader from otherwise missing these differences of
opinion.
It was generally agreed among the participants that
alternative analysis should be used when possible and that
it should be a. mandatory exercise to at least explore all
alternatives. Where alternatives are rejected, they should
still be listed with an explanation. for their dismissal..
VI. Evaluation of NIEs
Without some means of evaluating the NIE process, it is
difficult to tell how well we-are doing. Looking at estimates
after the fact can often present a skewed view. Asking the
intended policy makers for their opinions on the quality of
the estimate does not provide a. consistent standard, especially
SECRET
1.1
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved For Tease 2004/IW1t#GIA-RDP81 B0049O00100020010-0
when quality is equated with utility. One participant
pointed out that it was important to make distinctions about
the type of estimative judgments being evaluated---lumping
estimates which dealt with softer, more difficult information
together with those using relatively tin-questioned data
distorts their respective contributions.
One problem with postmortems conducted in the past was
that the persons doing them were most likely to have been
the analysts of the original estimates themselves. It is
good to go back periodically to review an estimate or-a
series of estimates to see exactly what they did or did not
contribute, but it should be done by small teams, not the
author.
Postmortems of estimates whose original purpose was to
undertake some kind of prediction do not help the policy
maker. Such an evaluation will show only that the predicted
event did or did not happen. Most policy makers already
have some chosen objective in mind. What they most want to
know from the estimate are the elements in the situation
which would make the desired. outcome more probable. It
would be useful for policy makers if these typos of estimates
could. be evaluated for future reference.
Postmortems of 11-3/8 estimates are relatively easy to
do-they are either right or wrong. The :judgments contained
in political estimates, however, are extremely difficult to
evaluate. For one year a running box score was kept on the
forecasting ability of those latter estimates. The results
proved futile:
--500 of the events were never resolved, and
--in a substantial number of the remaining 50%, things
predicted did indeed happen but not quite in the way
described by the estimates. There were also a number
of estimates which forecast such events as the sun
will rise tomorrow."
This is not to say that postmortems of political estimates
are useless. There are lessons to be learned from hindsight.
One study analyzed existing estimates up to. the 1960.s. Pre-
dictions dealing with more quantitative analysis, such as
technology or gross weapons capabilities, proved to be
adequate. But the thread of political reality in the more
general estimates also proved to be surprisingly good: 75-
80% right for the world as a whol.c.
12
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
 Approved ForQ&1ease 2004I1912 -RDP81 B00493 000100020010-0
In the case of political estimates perhaps the question
to be asked is not Did we predict accurately? but Did we
understand the dynamics of the situation? It is., however,
very difficult to sit down and read a paper five years after
it was written and not compare it to the way we understand
the situation now and judge its value accordingly. If the
reader is able to get a sense of who is doing what to whom
and the pressures involved, that is the best non-quantitative
judgment that we can ask for.
Much can be ].earned from an evaluation of a successful
estimate---how it was drafted, by whom it was written and its
terms of reference. Is there a. certain constancy involved
or is each one different? It may also help in hiring people
to write future estimates if it is determined that a certain
"type" of author is responsible for those judged to be most
successful.
VII. Summary
In the discussion of the role, theory and practice of
NIEs, three major conclusions were reached by the seminar
participants:
1. It was agreed that the NIE may play a greater role
in the foreign policy process than policy makers realize or
admit. The role of the NIE is not just as a vehicle for
prediction but
--it can educate the policy maker.,
--it can organize information for the policy maker, and.
--it can do for the policy maker what he does not want
or cannot do for himself.
2. It was agreed that not all consumer complaints
about estimates are valid. There will always be some problem
in the.way policy makers remember how they used, the estimates
as compared to how they in fact may really have used them. In
other words, "Intelligence officers remember their analysis as
being more useful than it was while policy makers remember. it
as being less useful than it was." There are limited ways to
overcome such complaints.- One crucial factor is to involve
the policy maker in framing the terms of reference. This
helps to make, the estimate more policy relevant while
establishing for the policy maker some level. of interest.
SECRET
Approved For Release 2004/10/22 CIA-RDP81 B00493R000100020010-0
 Approved For Tease 2004/1 fU IA-RDP81 B004932000100020010-0
3. It was agreed that alternative methodologies are
useful where they make sense. A consensus where it can be
justly reached is good, but where these hypotheses are in
competition alternative analysis is very useful. Footnoting is
still a viable means of presenting alternative views or
"thought objections."
Approved For Release 2004/10/2i'} CIA-RDP81 B00493R000100020010-0
 25X1 Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0
Approved For Release 2004/10/28 : CIA-RDP81 B00493R000100020010-0