CHALLENGE PROCEDURES

• Document Type: CREST
• Collection: General CIA Records
• Document Number (FOIA) /ESDN (CREST): CIA-RDP80M01133A000700150005-1
• Release Decision: RIPPUB
• Original Classification: S
• Document Page Count: 11
• Document Creation Date: December 15, 2016
• Document Release Date: April 9, 2004
• Sequence Number: 5
• Publication Date: January 1, 1973
• Content Type: NOTES

Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
SUBJECT: Challenge Procedures
Rather than comment on paper of 19 November
on the same subject, I have written a new paper which deliberately
avoids a rehearsal of the various arguments against challenge pro-
cedures. I take as a point of departure the DCI's instruction to the
J[CS of December 973 to develop such w procedures, (The concept,
contrary to assertion, was not first advanced by the D/DCI/IC
on 2 January 1974--the D/DCI/IC at that time was responding to the
JDCI, who himself was responding in part to our Middle East Post
Mortem.) I have also deliberately avoided a discussion of various
proposed alternatives which, in my view, are impractical or
undesirable (e. g. , idea of an NIO-size office of
professional challengers).
Instead, I have sought here to advocate one specific approach
to the problem--the soundest and most sensible I could come up
25X1
with. (And it may be that others, including
will wish to play
25X1
the devil with my advocacy. ) In addition to providi
ng broad considera
-
tions, I have used an idea broached by
in a br
ief paragraph (on
25X1
the bottom of p. 1S in his memo), developing this into a general
proposal and suggesting some specific procedures.
I suggest that this memo be given to the DCI for his consideration
(presumably after we have shown it to the D/DCI/NIO). If the DCI
concurs, I would then urge that he circulate it for later discussion
at USIB.
Chief, PRD IC
25X1
25X1
Approved For Release 2004/05/05 ,194>~0M01133A000700150005-1
'kn 1~3 ~1 A~
 C SECRET
Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
SUBJECT: Challenge Procedures - A Proposal
Some General Considerations
The DCI first proposed the establishment of challenge
procedures in October 1973, when, in the aftermath of the
failure of intelligence to warn of the impending Arab attack
on Israel, he and others suggested that a means be found to
introduce some form of devil's advocacy into the Community's
analytical proceedings.'.
Briefly, what seems to be needed is a challenge procedure
which would, inter alia, assist production analysts to overcome
three occupational hazards to which, according to our post-mortem
:reports, they are occasionally subject:
--Preconceptions: the tendency to discount information
that runs counter to long-held convictions;
"Specifically, the DCI stated: "The IC Staff.. . will develop regular
systems to be implemented by the NIOs to ensure.that serious
divergent points of view and conflicting elements of information
not be submerged by managerial fiat or the mechanism of reinforcing
consensus.... Such systems will also be charged with ensuring the
establishment of means to provide the views of devils' advocates,
adversary procedures, and the use of gaming techniques as
appropriate. " ("The Performance of the Intelligence Community
Before the Arab-Israeli War of October 1973: A Preliminary Post-
Mortem Report, " submitted by the DCI, December 1973, p. 22. )
25X1
Approved For Release 2004/05/05A - 80MO1133 000700150005-1
~ Mfll ff,
 Approved For Release 2004/05/DP80M01133A0 0700150005-1
--Reinforcing consensus: the tendency for divergent
views of individual analysts to be submerged in a sea of
conventional collective wisdom;
--The current intelligence syndrome: a myopic view
of the forest because of forced focus on current intelligence
trees.
The notion that some way should be found to challenge conventional
si:bstantive judgments has by now itself become quite conventional. ,-
But perceiving the wisdom of establishing challenge procedures does
not lead easily into an appreciation of precisely what form such
procedures should take. It is the who, when, how, and why of
challenge procedures which confront those who seek improvements
in analytical performance:
o just who in the community (and with what credentials)
is entitled to challenge whom?
o just when (and vis-a-vis chat) should the challengers
perform their art, i. e. , challen e ?
' Conventional enough, at any rate, (0 elicit challenges from those
who see only problems in a devil'- advocate procedure.
Approved For Release 2004/05/05i`:` 4'k6P80M01133A000700150005-1
 4" r~ \
Approved For R( elease 2004 1 IEIA-RDP80MO1133AO00700150005-1
o precisely how should the challenger perform, i. e. ,
through what system or medium should he present his contrary
views ?
4) and, finally, why should the challenger risk his (and
perhaps his sponsoring agency's) reputation for sagacity for
the sake of representing, by definition, a singular and probably
unpopular point of view?
Some Guiding Principle s
Clearly, some general rules of the game should be developed.
A challenger should possess substantial substantive competence in
the area under exploration, but he should not be so close to the subject
that he lacks perspective and suffers from the analytical disabilities
listed above. He should be familiar with Community processes and,
to a degree, Community "politics." And he should be articulate and
persuasive. (He need not believe his own advocacy, but he should
be devilish enough to convince others that he does.) Finally, and
obviously, he should have the time to do the job properly.
When should this paragon exercise his persuasive talents in this
strange manner? Certainly not as a matter of routine concerning
ordinary subjects. He should confine himself to matters of high
PE,
Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
 Approved For Release 2004/05 . tA-RT
DP80M01133A000700150005-1
moment (those of long-term significance as well as of immediate
concern) and papers of particular consequence (NIEs, Alert Memo-
randa, etc. ). And he should deal, full-time, only with specific
circumstances, problems, and/or papers for a stated period (weeks
or months). A more or less permanent assignment as a devil's
advocate would soon sap the vitality and demolish the credibility
of even the most enthusiastic and skillful practitioner. And the
devil probably would, over time, withdraw from the process.
There already is, in being, Community machinery, the NIO
system, which could accommodate--effectively, if not easily--
the establishment of challenge procedures. No other component
of the Community performs so many significant substantive functions
for the DCI and USIB; no other component is so close to the consumers;
and no other component is so involved in the production of important
Community assessments. And from the point of view of the advocate
himself, no other component could so readily provide him with the
papers, contacts, forums, and general support necessary for the
practice of devils' advocacy.
Indeed, in our canvass of alternatives, we could find no other
office or mechanism which could properly support a regular devil's
Approved For Release 2004/05/05 : CIA-RDP80M0l133A000700150005-1
 C
Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
advocate process, and we do not--for a variety of reasons--favor
the establishment of a new institution to perform this task.
And what of the risk of serving as a champion of unpopular
causes? Perhaps the devil's advocate--assured that he would
serve only a short tour, and guaranteed a respectful (if not
sympathetic) audience--might find the exercise of his imagination
and the influence he brought to bear on the weighty judgments of
the Community reward enough. And, at the conclusion of his
sentence, he might find some solace in the drink which the NIO
will buy for him at Sans Souci.
And Some Specific Proposals
Assuming, then, that, challenge procedures are to be instituted
within an NIO-sponsored system, we would propose the following
specific measures:
A. w The DCI or the D/DCI/NIO, in consultation with
USIB, if appropriate, should determine if a given subject and
paper seem sufficiently important to warrant the institution of
official challenge procedures. (A standard country paper on, say,
Argentina would probably not so warrant; an SNIE on, say, Soviet
reactions to specific US courses of action, altnost certainly would. )
Is ,
Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
 C
Approved For Release 2004/05/05 CIA-RDP80M01133A000700150005-1
USIB principals (and perhaps other senior figures in the Community)
should be encouraged to nominate candidates for devil's advocate
positions. In consultation with the D/DCI/NIO, the D/DCI/IC,
and others, as appropriate, the NIO responsible for a given paper
should then appoint an intelligence officer as a substantive challenger,
responsible as a devil's advocate for the effective presentation of
contrary substantive views.
1. The possible appointment of a challenger should
be discussed early on (perhaps during Terms of Reference
meetings) with the various agency representatives and with
2. The designated devil's advocate should be an
intelligence officer who has appropriate substantive credentials,
experience, and seniority. He could be drawn from any
element of the Community and might, in some circumstances,
be another NIO, perhaps one with related substantive responsibilities.
3. The devil's advocate should play his role for the
life of the paper concerned- -in the case of an NIE, from the TR
stage through USIB consideration; in the case of an Alert Memo-
randum, from its inception through the DCI's approval. He
should, in effect, serve a temporary tour in the devil's advocate
capacity.
Approved For Release 2004/05/05: ?> 0M01 133A000700150005-1
yy ?:ri
 Approved For Release 2004519-CIA-RDP80M01133A00)(0700150005-1
4. There are of course practical obstacles to
such a procedure. Few if any components will feel free
to spare a senior officer for devil's advocate duty. The DCI
may thus wish to ask USIB principals to consider this
procedure to be a necessary part of USIB's set of substantive
and estimative responsibilities---and, in effect, as significant
as providing, for example, representatives to attend meetings
on NIE s.
5. In some instances (e.g., NIE 11-3/8), the NIO
might wish to consider the appointment of more than one
devil's advocate, depending on the size, complexity, and
diversity of the paper under preparation.
B. The challenger would be charged with: developing and
presenting plausible arguments against the conventional wisdom
and against any or all of the papers' major judgments; expressing
disbelief or skepticism about certain specific pieces of evidence
and/or discrete conclusions based on them; challenging the logic
and coherence of given lines of argument; and identifying any gaps
in information and "holes" in argumentation -vyhich he perceived
as affecting major judgments.-.
Approved For Release 2004/05/05 CIA-RDP80M01133A000700150005-1
 ':ia w'AA
Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
1. The devil's advocate would be responsible for
eliciting the views of other responsible officers in the
Community who held views contrary to those offered in
the paper under review (contrary to those offered in
expressions of dissent as well as in the main text).
2. The devil's advocate should present his
advocacy orally, during meetings on the paper, and, when
appropriate, in writing. The NIO in charge would be duty-
bound to respond to the challenger's case, though would be
empowered to accept or reject his advice after due considera-
tion. The degree to which the devil's advocate was able to
influence substantive judgments in the paper would thus
ultimately be determined by the NIO.
3. All papers subjected to this form of challenge
procedure would bear a notation to that effect somewhere
(in an introduction, opening statement, footnote, whatever)
in the paper. In some instances, the burden of the challenger's
argument (especially if otherwise ignored) could also be
succinctly stated. For example:
Approved For Release 2004/05/05 : CIA-RDP80M0l133A000700150005-1
 Approved For Release 2004/05/05 : CIA-R -
Qi
The principal conclusions of this estimate
were subjected to a critical review by the Com-
munity's "devil's advocate. " The contention of
the advocate that, in the circumstances described
in the paper, the Arab governments would almost
certainly sue for peace (because they would be
principally concerned about their own survival)
was carefully considered. This argument was,
however, rejected, in the main because of reasons
discussed in paragraphs 14-16 of the text.
4. Occasionally, however, the NIO might wish to
reproduce the devil's advocate's case in extenso If so, that
case should be interwoven with the main text. This would
minimize repetition and confusion and place arguments in a
proper context. (Most NIEs, etc., already march in this
manner, and should continue to do so, though the pros and
cons of course should not be presented in a way likely to
obscure the conclusions. )
,Indeed, it has been suggested that many papers take the form of
straight advocacy but provide a separate annex setting forth the
paper's pros and the devil's advocate's con arguments; or that the
main text itself should interweave these pros and cons. Adoption
of the first suggestion--in effect the publication of a devil's annex--
would, with perhaps very rare exceptions, probably only irritate
and confuse the consumer.
HT
SE
Approved For Release 2004/05/05 : CIA-RDP80M0l133A000700150005-1
 Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1
5. Whatever the impact of his role on a given
paper, the devil's advocate should make his case available
to other elements of the Community (e. g. , current intelligence
components) so that they, too, could be informed by his
advocacy.
25X1
Chief, Product: Review Division
Approved For Release 2004/05/05 : CIA-RDP80M01133A000700150005-1