INTELLIGENCE DIRECTORATE EVALUATION SYSTEMS

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80B01495R000400080023-8
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 10
• Document Creation Date: December 19, 2016
• Document Release Date: October 20, 2005
• Sequence Number: 23
• Publication Date: September 26, 1972
• Content Type: PAPER

Approved `wwr Release 2005/11/23: CIA-RDP80B0W5R000400080023-8
INTELLIGENCE DIRECTORATE
EVALUATION SYSTEMS
`1:'rlw *ua.rpose of this paper is to describe the various ways and
means by wig i r- evaluative judgments are developed and used in the
decision- makinf mss of Directorate management.
The Intelligence Directorate functions primarily as a producer
of finished national intelligence. It also provides some community
services of common concern in the overt collection of raw intelligence
information and in the processing and exploitation of imagery collected
by overhead reconnaissance programs. Ancillary to both of these func-
tions is the production of intelligence guidance which reflects the needs
for information and the intelligence usefulness of information acquired
by community collection resources.
The Directorate evaluative processes, therefore, concern (a)
the adequacy of its analytical publications in providing intelligence
support to the formulation and implementation of national policy; (b)
the usefulness to the community of the information derived from its
collection and exploitation programs; and (c) the degree to which its
guidance to collectors enhances their effectiveness in providing a con-
tinuing flow of information needed in the production process. Inherent
in all of this, of course, is the continuing process for evaluating and
motivating the personnel who perform all of these functions and for
judging the level and apportionment of Directorate resources which
will provide greatest productivity at the most reasonable cost.
Intelligence Analysis and Publication
The Directorate publishes annually over 500 in-depth analyses of
various foreign, political, economic and military affairs related to U. S.
strategic policies and actions. In addition, the President's Daily Brief
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
 Approved For Release 2005/11/23 : CIA-RDP80B045R000400080023-8
and the Central Intelligence Bulletin reflect the day-to-day dynamics
of many of these same foreign affairs in consonance with the current
concerns of U. S. policy levels. Finally, the Directorate submits
contributions to responses for various National Security Study Memo-
randa as appropriate.
What to Produce and Publish? Unfortunately, intelligence has
never succeeded in having the highest policy levels of government spell
out, in an orderly and comprehensive form, the substance of its advance
needs for intelligence support. Although we have lived for years with
the PNIOs and CNIOs and are now still trying to find a more meaningful
way of expressing intelligence objectives in DCID 1/2, the intelligence
production process responds principally to a highly informal but con-
tinuous flow of a variety of "indicators. " A. large part of the Directorate
production process, therefore, does not lend itself to programming and
scheduling in accordance with a static list of intelligence objectives;
production is dependent upon the dynamics of world situations, the con-
cerns of U. S. policy makers (as expressed in such things as NSSMs),
and U. S. reaction to or involvement in world affairs.
Many of the indicators find their way into the Directorate because
the DDI and the Director of Current Intelligence attend the DCI's morning
meeting. The Director's attendance at Congressional briefings, and
his meetings with the NSC and its panels and other groups provide a
constant feel for what is required to keep the intelligence production
process in tune with policy-level needs for intelligence support. Simi-
larly, involvement of the DDI, the ADDI and other key Directorate
managers in the affairs of the NSC substructure provide a further flow
of indicators which are the bases for the continual reshaping of both the
current intelligence and the in-depth analysis efforts of the Directorate.
As a mechanical aid to the review and management of the substan-
tive profile of in-depth intelligence studies, the intent to produce a
study (Intelligence Memoranda and Reports) is recorded in a centralized
Directorate production schedule. These entries describe the studies to
be produced within the next eight weeks and indicate, where appropriate,
the intended recipient and the purpose to be served. This descriptive
-2-
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
 Approved Vii- Release 2005/11/23 : CIA-RDP80B04A05R000400080023-8
schedule aids both the DDI and the ADDI in ensuring that proposed
studies are timely and appropriate to principal policy and opera-
tional concerns of government. which need intelligence support.
How Good is the Product? The centralized production schedule
enables all managers to know what is in the pipeline, who requested it
and when it will be available for delivery. As manuscripts are com-
pleted, they are subjected to a number of division and office-level
quality controls during the editorial process. These quality controls
include a rigid evaluation of whether the conclusions of the study are
consistent with and supported by the evidence at hand. Following
coordination with appropriate Agency components, the study receives
a final review and approval by the DDI or his Assistant before it is
published.
Should subsequent events prove that the conclusions of a study
were not on the mark, the effectiveness of our analytical process is
examined by means of a post-mortem process. For example, following
publication of intelligence studies on shipping through Sihanoukville,
newly available data indicated inaccuracies in our original judgments.
As a result of a post-mortem, methodologies for analysis of data were
changed to safeguard against similar errors in the future. So far, over-
all marks for objectivity and accuracy in intelligence publications have
been good and there is every expectation that they will continue to be so.
Of late, the Directorate has begun trial use of a process to
regularize scoring of the accuracy of judgments contained in finished
intelligence issuances. As of now, the Office of Current Intelligence
records each of its published predictions and at a later date, the accuracy
is checked against actual events. The results are reported to the DCI
on a monthly basis; the system is being tested as an aid to evaluation
of (a) the effectiveness of the current intelligence analytical process and
(b) the accuracy and validity of the sources of the information on which
the prediction was based. If, over a time, this method proves to be a
useful aid to Directorate management, it may be applied to other pro-
duction components of the Directorate.
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
SECRET
 Approved, pr Release 2005/11/23: CIA-RDP80B0. 95R000400080023-8
As to determining the degree to which Directorate products
meet the intelligence support needs of policy-level recipients, it is
unproductive to ask consumers of intelligence to grade that which he
is receiving. This is especially true when checklist forms are used
and the measure of effectiveness is a machine listing of the stylized
responses. An informal system of constant liaison with consumers
may provide the most meaningful feedback on the adequacy of Direc-
torate intelligence publications.
Information Collection and Imagery Exploitation
Two Directorate components are concerned with the overt
collection of information--one from and the
other from coverage of the foreign radio and press (FBIS). Two others,
NPIC and IAS, are concerned with the exploitation of imagery derived
from overhead reconnaissance programs. All, except IAS which is
CIA's departmental center for imagery analysis, are community ser-
vices of common concern. Consequently, the adequacy of the products
of three components are judged in terms of usefulness to all member
agencies of the intelligence community. IAS effectiveness is judged
on the basis of its responsiveness to direct support requirements of the
Agency.
All of these collection and exploitation components base their
respective efforts on (a) general intelligence guidance reflective of the
principal concerns of community production components, (b) specific
intelligence requirements to guide exploitation of selected sources and
source materials and (c) evaluative feedback on the importance and
usefulness of information derived from the sources.
All of the above factors are brought into play through both formal
and informal mechanisms. Representative of formal guidance used by
these components are: the DCI-approved National Tasking (Imagery)
Plan; USIB-approved COMIREX requirements for mission exploitation;
the CIA Current Intelligence Reporting List; and the formal Specific
Information Collection Requirement system in use in the community.
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
25X1
 Approved r Release 2005/11/23 : CIA-RDP80B05R000400080023-8
To measure the adequacy of their efforts in meeting the
requirements of consumers, all components regularly seek formal
appraisals of the intelligence value of selected aspects of their
respective reporting. Such appraisals may be tailored to measure
effectiveness of a selected , FBIS bureau, or group 25X1
of photointerpreters. In other instances it may measure overall
component effectiveness in making a meaningful contribution to an
important topical intelligence problem. In addition, I 25X1
Isends to consumers a formal evaluation form along
with selected individual reports as a means of gauging the quality
of its source-selection process.
The informal mechanism for substantive guidance and feedback
to the collection and the processing managers of the Directorate con-
sists of constant liaison with the recipients of their products--namely,
the analytical elements of the community. This system probably pro-
vides the now of evaluative data most useful to the managers in shaping
the thrust of their respective efforts in order to satisfy the most
important and urgent needs of their customers. Also, as a means of
self-evaluation, both NPIC and IAS will do ad hoc post-mortems of the
accuracy of their readouts of the past as seen in the light of new dis-
coveries provided by subsequent reconnaissance missions.
One of the Directorate's continuing responsibilities in support of
the DCI is to provide guidance to community managers and program
review authorities in the planning and programming of intelligence re-
sources, especially those related to high-cost collection efforts.
Additionally, the Intelligence Directorate provides such guidance as
a means of ensuring that the information received from all collection
systems satisfies the needs of its production process.
The system for providing intelligence guidance generally consists
of the requirements process for expressing our information needs to
collectors and the evaluation or assessment mechanism for gauging the
value of collectors' products. The latter measure is intended to influence
program reviews and resource decisions that affect the nature and
-5-
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
 Approved- Release 2005/11/23 : CIA-RDP80BO4A95R000400080023-8
quality of the raw information flowing into our intelligence production
process. We use a variety of analytical methodologies and systems
to arrive at judgments concerning the effectiveness of all sources
which feed information to the production process. Among these is a
system for maintaining a running account of the contributions and
importance of designated collection programs to the conclusions in
the daily Central Intelligence Bulletin, as well as each of our in-depth
studies.
The principal question, however, is how do we evaluate the
effectiveness and adequacy of our intelligence guidance- -in short,
does it influence the community collectors to allocate, task and operate
their respective resources so as to satisfy the information needs of
the CIA production process? There is no simple way to determine
the answer to this question.
25X1
-6-
Approved For Release 2005/11/23 : CIA-RDP80BO1495R000400080023-8
SECRET
 Approved pr Release 2005/11/23 : CIA-RDP80B095R000400080023-8
It is readily apparent that it would be most difficult to attempt
to weigh the effectiveness of CIA inputs to such intelligence guidance
in terms of its singular impact on collection resources. Once again
we can rely only on continuing informal feedback and subjective
judgment of our effectiveness in this sphere of activity.
Information Handling
The Directorate's role in support of Agency programs also is
reflected in the handling of the massive flow of information which we
must manage effectively. The Central Reference Service is responsi-
ble for the dissemination, storage and retrieval of all-source informa-
tion in support of production and related activities. The effectiveness
of information handling processes are under continual surveillance,
and is particularly meaningful through the feedback from Agency elements
which CRS serves. For example, the adequacy of CRS' responsiveness
to production needs is tested on a regular basis by consulting consumers
on the effectiveness of CRS' machine-assisted dissemination system.
CRS also manages the community-wide program for procurement of
publications needed in the intelligence production process. Specific
judgment is provided on the level of expenditure against the stated needs.
Should components deplete their allotted quotes, moreover, they are
required to review their publications requirements and subsequent
requests must be approved by supergrade-level authority.
Intelligence Personnel and Resource Management
The strength of this Directorate lies in the quality of its people.
Concerted efforts are made at the recruiting stage to make sure that
we are getting what we need in terms of both the attitude of the person
towards intelligence and his qualifications to serve in the field. Since
universities are the best source of new professional personnel, the
Directorate seniors are in close touch with the academic world. This
contact is of multibenefit: (1) it provides for the identification of people
who might be interested in serving with the Agency; (2) it affords an
opportunity for intelligence analysts and managers to exchange ideas
with their academic counterparts, this of course within the bound and
-7-
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
SECRET
 Approved#pi- Release 2005/11/23: CIA-RDP80B0495R000400080023-8
constraints of security; (3) it provides the source for external
contracts for projects that the Agency might want to have undertaken
but for a variety of reasons not with its own resources.
All things considered, the contacts with the academic community
can be considered as part of the Directorate's quality control. It
offers needed checks and balances on the recruitment and hiring of
personnel as well as contributing to the development of the Agency's
research effort.
Once people have been brought on board and assigned to the
various production shops, a continual assessment effort is made to re-
late their abilities to the responsibilities of the Directorate. Only one
part of this assessment consists of the production of annual fitness
reports. The Directorate, through its supervisory chain and its Career
Boards constantly looks at its personnel and evaluates effectiveness and
potential. Training and other educational opportunities are afforded to
people to make them more skillful in analytical techniques as well as
to help them to acquire substantive knowledge and managerial know how.
At the same time, the Directorate engages in a constant review
of the resources allocated to its programs. Management studies examine
the level of effort directed towards specific tasks and weigh the alterna-
tive means of achieving the expected results. Moreover, the impact of
anticipated man power and financial restraints are projected against
changing requirements to ensure that the allocation of resources are
most effectively responsive to the increasing demands for sophisticated
intelligence production. .
In summation the people who are the best qualified and most
effective in this Directorate are those who have been created by an
organization which remains alert to its resource needs relative to the
dynamics of the demands being made upon it.
-8-
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8
 Approved For Release 2005/11/23 : CIA-RDP80B04&95R000400080023-8
In conclusion, evaluative systems used in the Directorate
rely principally on the human mind to process data received in
many different forms and through various means. At times it is
necessary to support this primary evaluative system with more
mechanical subsystems but only where (a) numerical or other
symbolic value codes can represent an accurate and meaningful
portrayal of all factors bearing on the value judgments, and (b)
the system provides the manager with information he does not
already have available through use of the human brain A.DP system.
-9-
Approved For Release 2005/11/23 : CIA-RDP80BO1495R000400080023-8
 Approver Release 2QO51t1L223.z CIA-RDP80B 95R000400080023-8
SUBJECT: Evaluation and Productivity
Distribution:
Orig & 1 - Addressee
1 - DD/I
1 - C/DDI Planning Staff
1 - C/IRS/PGG
1 - C/IRS/Chrono
1 - C/IRS/Misc Projects
DDI/ IRS / HJJenne: rrh
(9/26/72)
-10-
Approved For Release 2005/11/23 : CIA-RDP80B01495R000400080023-8