WORKING GROUP EVALUATION OF NIES

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80M01133A000900100002-7
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 17
• Document Creation Date: December 14, 2016
• Document Release Date: September 20, 2002
• Sequence Number: 2
• Publication Date: January 1, 1972
• Content Type: MF

Approved For ease 2002/10/10 _:CIA-RDP80M0113*0900100002-7
DCI/IC 72-0961
?4E' FORANDt ?` FOR: "embers of the tISCIC Uorki ng Group
SUBJECT: Working Group Evaluation of ?'l I Es
1. This memorandum sets forth a proposal by which the Working
Group would become involved in the evaluation of published National
Intelligence Estimates in response to the Presidential memorandum of
5 November 1971 and NSCI!) No. 1.
2. Both of these documents call for improvement in the "quality,
scope and timeliness of the community's product." Because of the
importance of the MIE documents and because these estimates are
intended for the use of high-level consumers, it appears appropriate
that attention of the `Working Group be directed to an evaluation of the
extent to which "CIE nublications are responding to the needs of the
policy-making levels of the federal government.
3. At Tab A is a description of past surveys of user reactions
to National Intelligence Estimates, including the 1969 request of the
DCI to the Secretary of State, Secretary of Defense and Assistant to
the President for National Security Affairs for their views or
suggestions as to how NIEs on Soviet military subjects night be made
most useful" to them. The responses contributed to a marked revision
in the format of these estimates.
4. The success of this effort suggests that involvement of these
same top-level officials, or their representatives on the NSCIC Working
Group, in the evaluation of PiIEs on a fairly broad basis could provide
valuable inputs for the improvement of the National Intelligence
Estimates oro cess.
following manner:
5. Accordingly, I am recommending that the Working Group approve
this as a project for a six-month period and carry it out in the
Approved For Release 2002/10/10: CIA-RDPB
25X1
 Approved For ease 2002/10/10 :CIA RQP80M0113*0900100002-7
a. The Working Group will select the particular NIEs
which are to be evaluated.
(1) Twenty-eight "lIEs, including one Memorandum
to Holders, have been published in 1972 as compared with
28 in 1971 and 26 in 1970. The frequency of publication
in 1972 ranged from a high of five in May to none in
February and one each in January, July, August, September
and November.
(2) Should the Working Group elect, for instance, to
initiate the project with a review of some of the NIEs
published in the last quarter of 1972, the selection
would be from the following:
NIE 11-3-72
NIE 11-8-72
NIE 12-72
NIE 20-72
NIE 32.1-72
NIE 36.1-72
NIE 36.2-72
NIE 75/76-72
NIE 89-72
Soviet Strategic Defenses
Soviet Forces for International Attack
The USSR and the Changing Scene in Europe
Problems in U.S.-West European Relations
Bangladesh
Eqypt after the i lssian Ouster
Iraq's Role in Middle Eastern Problems
The Horn of Africa
Venezuela: The Politics of Oil
(3) Estimates projected for publication in the first
quarter of 1973 (including several originally scheduled
for completion in the fourth quarter of 1972) are:
tiIE 3-72
NIE 11/20-72
International Petroleum Prospects
Soviet Posture on Mutual Balanced 25X6
Force Reductions
PIIE 91-72
Argentina
NIE 11-10-73
The Uses of Soviet Military Power in
tiIE 13-8-73
Distant Areas
China's Weapons Program for Strategic Attack
NIE 32-73
Pakistan
NIE 36.7-73
Syria
NIE 85-73
Cuba
b. The Working Group chairman will forward a copy of each
NIE to be evaluated, and two copies of an evaluation questionnaire
(proposed format of which is at Tab B) to the Working Group
representative of each NSCIC principal. It is intended that the
evaluations be prepared only by personnel who are not a Dart of
the intelligence production community.
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 Approved For ease 2002/10/10 : G DP80M01I3,*0900100002-7
c. The evaluation is to be accomplished within 30 days,
and a copy of each completed questionnaire returned to the
Working Group chairman. The.other copy of each completed
questionnaire is to be retained by the Working Group member
as basis for his comments when the responses of the entire
group are discussed at the next meeting.
d. Once all evaluations have been received, the chair-
man's staff will prepare a summary of the comments for
distribution to all members and to the Office of National
Estimates.
e. The evaluations of each estimate will be discussed
at the next following Working Croup meeting, at which time
the Director, Office of National Estimates, will be invited
to bring with him two O/NE Board and Staff representatives
responsible for the particular estimate or area of estimating.
f. Minutes of the discussion on each NIE will be made
available to the CIA Office of National Estimates. As appropriate,
the Chairman of the NSCIC and the Director of Central Intelligence
will be provided any recommendations which may arise in the
course of Working Group discussion of the estimates.
6. Discussion of the evaluations of individual NIEs at the
Working Group meetings is expected to be a valuable element of the
project. The intelligence community has never before had a forum in
which a broad spectrum of the representatives of high level consumers
can meet with the chiefs of intelligence production organizations to
voice their views with respect to specific estimates or types of
estimates. The NSCIC Working Group provides such a forum.
7. In the process of completing the evaluation, Working Group
members will be encouraged to solicit inputs from those user elements
of their organizations which they consider are most concerned with tie
subject matter of the estimate. If additional copies of the question-
naire are desired, they will be provided. It may well be that individual
members of the Working; Group will desire to have complete questionnaires
filled out by different elements of their departments or agencies.
Such a response is highly desirable if it will contribute to an
expression of consumer reaction to the NIE.
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 Approved For Reoe 2002/10/10 cIa- 80M01I33A 900100002-7
B. I recommend that the Working Group approve initiation of
this project at its next meeting, now tentatively planned for shortly
after mid-January. At this meeting, I hope we can reach agreement
as to the particular UIEs to be evaluated, and can decide upon any
changes to the proposed questionnaire which any member considers will
enhance its utility for our 'purposes.
Bronson Tweedy
Chairman, NSCIC Working Group
Attachments :
1. Tab A
2. Tab B
DCI/IC/PRG 19 Dec 72
Distribution:
orig,- PRG subject (filed PRG Project 6-72)
Mr. Tweedy
1- - CfVroiio
1 - PRG chrono ~'"? cL,
l'- each member with attachments
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7
 Approved For RSse 2002/10/10"' '"F [)P80M011331.900100002AT
SURVEYS OF USER REACTIONS
TO NATIONAL INTELLIGENCE ESTIMATES
I. The expansion of the U. S. intelligence community since World
War II has resulted, among other things, in a marked increase in the
output of intelligence products- -estimates, serial publications, reports,
bulletins, memoranda, etc. Wide variation exists in the audiences to
which individual products are addressed and in the number of copies
which are disseminated. Because of concern as to whether the right
audience was being reached or whether the products themselves were
responsive to the needs of.consumers, various intelligence organizations
have conducted surveys of one type or another to measure consumer
reactions, usually in the interest of obtaining comments which would
provide basis for improvement of the product. The following paragraphs
describe the more important of such efforts relating to National Intelli-
gence Estimates, and an assessment of the results achieved.
2. Survey of 1955
a. In February 1955 the Director of Central Intelligence asked
the Board of National Estimates to conduct an informal survey of
the use which consumers made of National Intelligence Estimates
(NIE's). The survey, which was undertaken after four years
experience with the national estimate system, was intended to shed
light on (1) whether NIE's were circulated appropriately; (2) who,
besides the National Security Council, were the ultimate consumers;
(3) how extensively NIE's were actually used by these consumers;
and (4) the extent to which, as then written, the NIE' s met such needs.
b. The survey was conducted in two stages. The first was a
written questionnaire addressed to all offices and agencies receiving
NIE's and intended to indicate NIE dissemination within each agency:
The second stage was a series of interviews with a sample of policy-
maker consumers to ascertain what use they actually made of NIE's.
The survey was not intended to obtain opinions regarding the
adequacy of NIE content or judgments on changes to make them
Approved For Release 2002/10/10 ?CIA-Fj
25X1
A-1
 Approved For ease 2002/10/10 Cli'4.F b'80M01133100900100002-7
more useful. The primary goal was to establish the distribution
pattern of NIE's and to indicate the kinds of uses to which they
were put.
c. The survey indicated that an average of 249 NIE's were
distributed regularly to the White House, the NSC, the Operations
Coordinating Board and the major departments and agencies with
national security responsibilities. Many of these were further
distributed to major commands and missions outside Washington.
d. Only a relatively few NIE's (selected by intelligence and
screening officers) were seen by the president and department
or agency heads, and in such cases these top-level individuals
usually read only the conclusions, briefs, or selected portions.
The' exceptions were most likely to be NIE's dealing with crisis
situations requiring urgent and high-level policy decisions.
NIE's were, however, found to be extensively read by second and
third echelon officials in major departments and agencies and by
various staffs responsible for national security planning and
execution. It appeared that some NIE's did not reach all of the
departmental policy or planning levels or other key officials who
might have found them useful. In some cases, this appeared to
result from failure of subordinates to pass the NIE's to their chiefs,
and in others the criteria for distribution appeared to be overly
restrictive.
e. NIE's were found to be most generally used for background
purposes, although some were used in working on specific planning
and policy problems. The majority of users indicated that the
format and problem coverage of NIE' s were adequate, but many
expressed dissatisfaction with particular NIE's. There were
complaints that the NIE's were too long to be read in full by top-
level individuals; they were said by some to be too general for
detailed use in policy planning and execution; and some were said
to be not relevant or sufficiently timely for use in consideration of
specific policy problems. Overall importance of such complaints
was not considered sufficient to warrant action to alter the format
or content of NIE' s. Each Intelligence Advisory Committee
member was asked to reexamine the pattern of NIE distribution
so as to improve the usefulness of NIE's to the policy, planning
and executive branches of his agency. Each IAC agency was
Succeeded by the U. S. Intelligence Board on 15 Sept,rnber 1958
A-2
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 Approved For ease 2002/10/1O" dA-RRP80MO113300900100002-7
requested to brief new key officials within its department on the
nature of the NIE and the means at their disposal for requesting
NIE's responsive to specific intelligence problems they might have.
3. Conference with the Assistant Secretary of State for Policy
Planning (1957)
The DCI, Director of National Estimates and his BNE members
met on 25 July 1957 with Robert W. Bowie, Assistant Secretary of State
for Policy Planning to discuss the form and content of the NIE program.
Mr. Bowie' s suggestions to enhance the usefulness of NIE' s to the
policy planning process were as follows:
a. There was undue emphasis on precise statements of
probability, with too little discussion of underlying factors.
b. In many cases, it would be useful to discuss more
possibilities, even if they had only a 20 percent chance of
eventuating.
c. The estimate should show the degree to which predictions
were based on evidence as opposed to judgment.
d. Estimates should put more stress on the degree to which
U. S. action could affect the predicted line of development, and
what kinds of U. S. action could have impact.
. e. NIE's were not very readable, even on interesting
subjects--the conclusions were "dehydrated" and lacked flavor.
He also suggested that the. IAC promulgate as "think pieces" a
series of papers written by a single person or a small group (in
the manner for Foreign Affairs articles) for consideration by
policy-makers.
4. Survey of NIE Distribution (1959-1960)
a. This survey effort had two aspects: (1) the circulation
of a questionnaire enclosed in seven selected NIE' s published
during the period July 1959 to February 1960 to obtain certain
detailed information regarding the readership of NIE's, and
(2) an inquiry to the distribution offices of the USIB agencies to
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7
 Approved For Pease 2002/10rt& CiA -~2DP80MO1133*0900100002-7
determine how NIE distribution is controlled. The latter is not
relevant to the present study. The survey was undertaken to
investigate the nature of and causes for an apparent large increase
in NIE distribution from 250 in 1956; by 1960 some 365 copies of
NIE's were regularly being sent to recipients according to a
standard distribution list, and about 85 more were held in reserve
for special distribution. The survey ascertained that the substantial
increase since 1956 was attributed to the following factors:
(1) Natural expansion of readership as the utility and
significance of NIE's became more firmly established
throughout the government.
(2) The growing number of senior officials who required
NIE's in carrying but their responsibilities.
(3) An increasing number of users in the intelligence
community itself.
(4) The succession of world crises which generated
increased interest in national intelligence on the part of
commanders and p].a.nner s .
(5) A general increase in the number of people in the
government concerned with the communist threat throughout
the world, developments in modern technological warfare,
and the capabilities and intentions of the Soviet Bloc.
(6) The addition of new readers overseas --political
advisors, ambassadors,and military planners and commanders.
(7) Agency reorganizations, including changes in locations
of offices.
(8) An increase in the size, responsibility, and complexity
of agencies served by NIE's.
(9) The establishment or expansion of reference and
library facilities in the USIB agencies.
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7
 Approved For ease 2002/10/10x 4=P80M01 13,*0900100002-7
(10) An increase in the number of NIE's released to
foreign governments. (From about 26 to 45)
(11) An apparent increase in readers whose need to know
might be marginal.
b. An analysis of the returns from 636 respondents of the
questionnaire addressed to NIE recipients indicated that about
50 percent were engaged in intelligence, 19 percent had responsi-
bilities in policy planning and coordination, and 8 percent were in
operations. Only 7 percent listed themselves in the area of
research. However, a substantial number of readers (34 percent)
indicated that they read only the NIE conclusions. The questionnaire
(at Tab 1) did not solicit ideas on how NIE's might be improved.
5. Queries to Secretaries of Defense and State re Soviet Military
Estimates (1969)
a. In November 1969 the DCI asked the Secretary of State,
the Secretary of Defense, and the Assistant to the President for
National Security Affairs for their views or suggestions as to how
National E stimates on Soviet military subjects might be made
"most useful" to them. Questions included: (1) is the division
by subject (Strategic Attack, Strategic Defense, and General
Purpose Forces) the best way of presenting the estimates; (2) is
the amount of detail presented excessive and should there be
different types of estimates for different readers; and (3) do existing
procedures provide intelligence judgments of most vital interest,
together with adequate evidence or argument, and are dissents
presented adequately?
b. The Secretary of State offered no "fundamental" criticism,
but his suggestions for "refinements" were as follows:
There could be "summary estimates" for the top
policy makers, more comprehensive than the
"conclusions'! sections of the then current estimates,
but more condensed than the then current estimates.
This would permit the basic NIE to be more detailed
and complete with a fuller presentation of the evidence.
Annexes could provide background information on major
issues, and there could be greater use of maps, charts
and graphs.
A - 5
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 Approved For ease 2002/10/10: CIA-RDP80M0113*0900100002-7
There should be a greater effort to evaluate Soviet
doctrine and strategy together with discussion of the
fundamental differences of opinion within the
intelligence community. These could be incorporated
in the "basic estimate" and this might obviate the need
to issue a separate NIE 11 -4 as frequently as in the
past.
The "summary estimates" would be written after the
basic estimates were completed and would give an
over-view of major Soviet weapons systems and
describe significant changes in Soviet forces. They
would also include a discussion of the general nature
of Soviet military expenditures. The "summary
estimates" for 11-3 and 11-8 should be'
combined into a single paper so policy makers would
have in one document an assessment of the main
elements of information on Soviet strategic forces
which have to be taken into consideration in calculating
the US-Soviet strategic balance. This would be useful,
for example, in discussing US options for the SALT
negotiations.
The estimates might contain a brief statistical
summary of US forces, including forward projections
drawn from the OSD FYDP, to give the reader a basis
for comparing US and Soviet forces.
c. The suggestions of the Secretary of Defense were as
follows:
Consideration should be given to relegating much of the
detailed quantitative and qualitative analyses of
Soviet weaponry to appropriate appendices. This
would leave in the main body of the NIE only the
more critical aspects of strategic weapons systems.
The main body should also contain changes, dis-
agreements of the intelligence community, and levels
of confidence inherent in the principal judgments.
"Main Trends in Soviet Military Policy"
"Soviet Strategic Air and Missile Defenses"
"Soviet Capabilities for. Strategic Attack"
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 Approved For ease 2002/10/10: CIA-R~P80M0113.0900100002-7
- The estimates, especially the strategic attack and
defense papers, should be revised and updated more
often, perhaps quarterly. (The NIE on an annual
basis was said to be of questionable usefulness a
few months after its issuance. )
- It would be useful to include five-year estimates in
the NIE', perhaps less precise than the short-term
estimates but more definitive than the ten-year trends
of the then current estimates.
Each NIE should have a section comparing major
judgments of the previous NIE with the current
judgment. Divergencies could be explained anfi
tendencies to consistently over-estimate or under-
estimate should be brought to the attention of the
policy-maker so that he could determine the "hedge"
factor he must introduce in decisions regarding
defense programming and planning.
d. The change which both the Secretary of State and the
Secretary of Defense recommended- -that a condensed statement of
the principal findings of Soviet military estimates be prepared for
the top policy-makers-- was adopted and NIE's on these subjects
now have a lengthy summary and conclusions section at their
beginning. Another suggestion of the Secretary of State was also
adopted and resulted in a deeper probe of Soviet military strategy.
NIE 11-4 (now "Issues and Options in Soviet Military Policy") has been
retitled and considerably expanded in scope and depth. The Secretary
of Defense's suggestion to include five-year estimates of Soviet
force levels was also adopted, but in the form of several alternate
projections for force levels, depending on assumptions as to the
success of SALT talks, US alternate force levels, etc.
6. Meeting with the White House Staff in 1970
The Board of National Estimates and its staff members
conducted a series of meetings in 1970 with the staff of the Assistant
to the President for. National Security Affairs concerning the form and
content of military estimates. As a result of these meetings, a great
deal more factual material was included in the estimates, together with
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 Approved FoSlease 2002/10/10 : CIA-RDP80M011,0000900100002-7
an expanded discussion of the factors leading to the conclusions of the
estimate. On 8 March 1971 the President commended the DCI and "the
entire intelligence community" for NIE 11-8-70 which the President said
was a considerable improvement over the previous version. He stated
that he found particularly useful:
"--The frequent sharply-defined, clearly argued
discussions of the various contested issues.
"--The attempt to incorporate a wide range of
sources, such as clandestine reports and Soviet
SALT statements.
"--The alternative force models based on explicit
differences in underlying assumptions and the
attempt to define which were the more likely
models.
"--The quantitative detail for each model which
illustrates the differences between the models
and gives an operational meaning to some of the
general statements. "
Approved For Release 2002/10/10 : CIA-RDP80M0l133A000900100002-7
 TAB B
Approved For*ease 2002/10/10: CIA-RDP80M011*00900100002-7
CONSUMER EVALUATION OF NATIONAL INTELLIGENCE ESTIMATES (NIE)
PURPOSE
This evaluation is an element of the National Security Council
Intelligence Committee (NSCIC) Working Group program to provide
explicit feedback to intelligence producers from consumers.
This evaluation should not be prepared by intelligence personnel. What
is.sought is a user evaluation of an intelligence product.
INSTRUCTIONS
Evaluators are requested to complete, this form within 30 days
of receipt.
Security classification of the completed form will be based on
the classification of the estimate being commented upon and classi-
fication of the comments included in the evaluation..
The completed evaluation is to be forwarded to:
D/DCI/IC
ATTN: PRG/IC
Room 6E18
CIA Headquarters
Washington, D. C. 20505
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7
 ? 0,Approved FoOlease
A. NIE No. and Title
Date of Publication
AT,
00900100002-7
B. Evaluator Address
C. NIE Utility and Timeliness
1. Primary use to which NIE is being or will be put:
a. Policy planning d. Military R&D planning
b. Operations planning e. Background info.
c. Military force planning f. Other (specify)
2. Timeliness of the estimate
a. Very timely b. Timely c. Not timely
If c is checked, why is the estimate not considered timely?
3. Utility of the NIE for ongoing or anticipated policy/operations problems:
a. Absolutely necessary d. Nice to have
b. Very useful e. Little or no use
c. Useful
If a, b, or c is checked, which sections or portions of the estimate
were particularly useful (i.e., provided specific and unique information
or judgments directly applicable to policy/operations)
4. If some other source currently satisfied the need for this estimative
information in an equally useful or comparable fashion, identify such
source:
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7
 Approved Fo4ease 2002/10/10: CIA-RDP80MO110000900100002-7
D. Scope and Format
1. The scope and format of the estimate are:
a. Excellent b. Satisfactory c. Unsatisfactory
2. If c. was checked, how could the scope and/or format of the
estimate be improved? To accomplish what purposes?
E. Factual Data
1. For purposes of your organization, the degree to which
factual data is presented is:
a. Excessive b. Sufficient c. Not enough
2. If a..or c. was checked, for what purposes would more/less
detail. improve the utility of the estimate?
F. Expression of Uncertainties
1. The uncertainties relating to the data or the situations
described are brought out:
a. Excellently b. Satisfactorily c. Inadequately
2. For purposes of your organization, the;estimate treats the
full range of uncertainties:
a. Very well b. Adequately c. Inadequately
3. If the answer to 1. and/or 2. is c., how could the treatment
of uncertainty have been improved?
Approved For Release 2002/10/10: CIA-RDP80MO1133A000900100002-7
 Approved Fooolease 2002/10/10 : CIA-RDP80MO11000900100002-7
G. Expression of Alternatives
1. For purposes of your organization, the range of alternative
projections and the description of each alternative are:
a. Quite comprehensive b. Adequate c. Inadequate
2. If c. was checked, in what manner could the presentation
have been improved?
H. Quality of the Estimative Logic
1. The extent to which the analysis logically ties together
the factual. and estimative elements of the paper is:
Excellent b. Satisfactory C. Inadequate
2. If c. was checked, in what manner might the quality of the
estimate have been improved?
I. Overall Reaction to the Estimate
('Note: The purpose of this section is to encourage you to summarize
your reactions to the estimate and to present such additional comments
as you consider useful as feedback to the intelligence community. If
you have made hand-written notes on the estimate itself, it would be apprec-
iated if the estimate is returned with the evaluation form.)
1. The following words or phrases characterise this estimate:
a.
Exceptionally valuable
d.
Rehash of conventional
b.
Stimulating and persuasive
wisdom
c.
Imaginative no'4 vi ew.wlpo i nt,- .
e.
f.
Shallow and unconvincing
Dull
3
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7
 Approved Fololease 2002/10/10 : CIA-RDP80M011,000900100002-7
2. Basis for the foregoing characterization of the estimate
is:
3. In summary, your overall comments as to the extent to which
the estimate met the requirements of your organization
are as follows:
4
Approved For Release 2002/10/10 : CIA-RDP80M01133A000900100002-7