EVALUATION OF INTELLIGENCE PRODUCTS

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80M01133A000900040005-1
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 6
• Document Creation Date: December 12, 2016
• Document Release Date: August 14, 2002
• Sequence Number: 5
• Publication Date: July 20, 1972
• Content Type: MF

Approved For ,se 2002/08/21L .801VI01133P900040005-1
g1
11 25X1A
20 July 1972
MEMORANDUM FOR: Mr. Tweedy, D/DCI/IC
SUBJECT: Evaluation of Intelligence Products
1. These comments and my request for your approval of the
policy set forth in this memorandum result from consideration of
NSCIC Working Group activities to date and my reading of the
report, "CIA Evaluation Systems - Preliminary Observations,"
which the Inspector General made available to me on 19 July.
The attached 12 June memorandum for the Inspector General from
Mr. Colby, which you may already have seen, also pertains.
2. I recommend that the Product Review role with respect to
the evaluation of intelligence products be as follows:
a. To participate as appropriate in such reviews of the
responsiveness of intelligence products to consumer needs
as may be sponsored by the NSCIC Working Group. It is
anticipated that these reviews will focus primarily on the
assessment of intelligence responsiveness in crisis situations
(as in the case of the India-Pakistan crisis of 1971) and
on the review of the adequacy of intelligence inputs to specific
problems of high policy interest (as in the case of inputs to
NSSM-69).
b. To arrange, where appropriate, for consumer evaluation
of specific intelligence products which are of high consumer
interest. The proposal to have NSCIC Working Group members
assume responsibility for completion of evaluation forms on
NIEs and SNIEs for a period of time is an example.
c. To identify specific problem areas which appear to call
for corrective action. These will pertain more to identification
of gaps in the production program (or, perhaps, "overkill")
than to the specific evaluation of production output. My pending
proposal for the application of special effort to the Soviet
R&D effort is an illustration.
d. To conduct research in areas identified as posing
problems related to production. The proposed project with
OMS/PSS to improve the quality of future production by
Approved For Release 2002/08/21 ~~A-F DP80M01133A000900040005-1
 Approved For ReWe 2002/08/21 : Cam: ' M01133AIS900040005-1
developing better recruitment criteria for the selection of
analysts is one example. Another is the proposal that PRG
look into the impact which delays in the processing of certain
types of data are having on the production effort.
3. Your approval of the foregoing would mean that the development
of product evaluation systems for use within the production elements
of the intelligence community is recognized as an in-house responsibility.~i'
The program which Mr. Colby and the Inspector General are pursuing
within CIA , as per Mr. Colby's memorandum of 12 June, would be
recognized as an Agency responsibility, supporting the aims of the
DCI in response to the Presidential memorandum of 5 November 1971 and
to the new NSCID No. 1. PRG would have no role in this activity
except that, once the CIA is operating, I suggest it would be
appropriate for PRG to draft a letter for the DCI to send to 0 25X1
citing the CIA actions and requesting that they
advise him as to the internal measures they are taking for quality
control and evaluation of their products.
25X1A
C ief, PRG/IC
Attachment.:
as stated
Distribution:
orig - addressee and return (PRG subject- filed
1-
Approved For Release 2002/08/21:,:TIA=RDP80M01I33A000900040005-1
 Approved For it ase 2002/08i t :eEAa-06R8(iMOeH 3.00900040005
12 June 1972
MEMORANDUM FOR: Inspector General
SUBJECT : Evaluation and Productivity
1. Thanks to the report on "CIA Evaluation Systems -- Pre-
liminary Observations" submitted by the IG, a useful discussion of
productivity and evaluations took place at the Director's Annual
Conference. While there was some discussion, if not dissent, as to
your appraisal of what systems actually exist, there was general
agreement that the subject requires more and continuing attention.
The Director is particularly concerned as to the ink ela ionslAp
between motivation of our ersonn-el and roductivit of t_ it ei orts,
and the -degree to which evaluations can contribute to maintaining
these at a high level. He charged the Deputies with examining the
actual status of their evaluation systems i., their directorates (and
he called for a discussion of this subject in detai a a ox ncoming
Deputies Meeting. The evaluation assessment should include all
sorts of post-morterris and assessments of effectiveness, in addition
to any specific evaluation systems which may exist. -
2. The Inspector General is directed to nrm -th,~ 51-
vision of the assembly his study and the develonr. ent of thy. a,
f 1 t' on
a
zl
priate age= a and backup material for a future discussion o eva u
systems at aITepu ies Meeting. This should be carried out in close
coordination with the directorates through whatever officer is nominated
by each Deputy as a point of contact for his directorate. The meeting
agenda should include the following subjects:
a. Inventory of current evaluation systems;
b. Utilization of evaluation systems in program, personnel
and other decision-making;
f1 T9 iS "'.tea .. ~.?''^6- ^ ((f ~..... _.'GI:J..~ /'/ '..
Approved For Release. 2002/08/21 : CIA-RDP80M0l133A000900040005-1
 Approved For lease 2002/06121:. CIA-RDP$OMO 113 00900040005-1
-2-
c. Interrelationships and compability of evaluation systems;
d. Criti,,,, A ~,f C?rrant Eu^ _73,s~4_;.rilF36~r"6~e~:+iiG~= ,rLrS~ _Wlth
reference to. the needs of unit, office/division, directorate and
Agency levels of management respectively;
e. Recommendations.
3. While we' should move with all deliberate speed on this action,
I think it important that the discussion and resultant 'recommendations
be fully informed and responsive to CIA's unique requirements for flexi-
bility, security, and adjustment to customer and policy needs. Thus, no
date is established for this Deputies' discussion, but you and the Deputies'
representatives should recommend a date which will give the time needed
to do a thorough and sensitive study responsive to the Director's desires
and appropriate for presentation to him.
.25X1A
y
Executive Director- Cofnptroller
cc: DD /I
DD/P
DD IS &T
DD /S
.a4 1 :,i
Approved For Release 2002/08/21 CIA-RDP80M01133A000900040005-1
 25X1 A Approved
2002/08/21 : CIA-RDP80 M 011330900040005-1
U 6. Andy also said he was working with DIA where a project
is underway to produce an experimental product using Bayesian
techniques. My discussion of the effort underway in PRG emphasized
my feeling that production managers would have to have considerable
control over the effort if they were to be receptive to the results,
but Andy is of the belief that attention also needs to be given to
a community mechanism outside DDI, DDS&T, and DIA. He has an idea
that the use of outside consultants and contractors working with key
elements of the DCI/IC staff might prove feasible. He also said that
Pat Parker of ASD/I has advised-him DIA already is using outside
consultants to train selected analysts in advanced analytical
methodologies.
7. Since Andy had mentioned the need for quality control in -
his 27 July memorandum, I told him I was aware of an effort already
underway to survey evaluation methods and by close of the year was
charged with preparing a study recommending how an improved evaluation
system could be brought into effect. Pickett asked if he could be
put in touch with this group and I declined on the grounds that the
1 1 activity was being closely held and should be so regarded at least
until its report had been completed. I did not identify the office A.
which was preparing the study.
8. We also discussed briefly the discussions PRG has had with
OMS/PSS focused on ways and means of identifying the attributes of
good and bad analysts. I reported, however, that current indications
were this would be a rather expensive project and decision had not
been reached as to whether to proceed with it.
9. Andy indicated most of his present activities relate to
formulation of a net assessment program. Commander Robin Pirie,
who has had experience in OSD/SA prior to recent sea duty, has joined
Andy's staff and will work on net assessment matters.
10. With respect to on-going projects, Andy is still reviewing
notes on the proposed revision of the India-Pakistan study; the
NSSM-69 study is in second redraft; Bob Baraz of State is heading
the team on the Arab-Israeli cease-fire study;E_ :1 5X1A
of CIA is heading the Jordan/Fedayeen civil war study; and 25X1
of DIA will do the LAMSON-719 study. 0 wants JCS participation 25X1
in this.
11. Andy's office is being moved next week to Room 302, E.O.B.
The telephone number will remain the same.
25X1A
hief, PRG/IC
Distribution: '~.p,
orig - PRG subject-bid PR,{L P ?-L
1 - Mr. Tweedy
1 - chrono
1 - PRG chrono
2
Approved For Release 2002/08/21- CIA.RDP80M01I33A000900040005-1
 Approved For Rose 2002/08/21 : 6 0M01133A0900040005-1
Excerpts from MOR of
with Andrew Marshall, NSC Staff
"4. With respect to the expression of community judgments on
intentions and objectives, I suggested he might consider proposing
that during crisis sutations, ONE establish an ad hoc group, including
community representation, which would function full-time and prepare
SNIEs as necessary to maintain the currency of community judgments
as the situation evolved. This would not put ONE in the current
intelligence business but would provide a means by which policy
makers could be kept aware of intelligence community judgments
on major facets of the crisis situation."
"5. Andy said he did not consider the ONE style of writing was
adequate for such purposes. In his view, ONE writing is deliberately
fuzzy and does not convey the -Z ear an T is -inc expression of views
e ed by decision makers.
"11. We discussed the matter of feedback from users to intelligence
producers, and Andy's view was that his attempts to increase feedback
had not shown much receptivity at the intelligence end. He said
the primary reaction has been a 'bristling' or a comment to 'get it
in writing from Mr. Kissinger' without much else happening."
12 July 1972, Subj: Conference
Approved For Release 2002/08/21 : CIA-RDP80M01133A000900040005-1