A DELPHI EXAMINATION OF FUTURE SOVIET STRATEGIC MILITARY OBJECTIVES

• Document Type: FOIA
• Collection: FOIA Collection
• Document Number (FOIA) /ESDN (CREST): 0005479581
• Release Decision: RIPPUB
• Original Classification: U
• Document Page Count: 48
• Document Creation Date: June 24, 2015
• Document Release Date: January 15, 2010
• Case Number: F-2010-00126
• Publication Date: August 1, 1971

DIRECTORATE OF
SCIENCE & TECHNOLOGY
Scientific and Technical
Intelligence Report
A Delphi Examination of Future Soviet Strategic
Military Objectives
CIA
DD 1101 USiOY
APPROVED FOR RELEASE DATE:
OSI-STIR/71-17
August 1971
 iECR
A DELPHI EXAMINATION OF. ,
FUTURE SOVIET STRATEGIC MILITARY OBJECTIVES
Protect Officer
OSI?STIR/71.17
August 1971
CENTRAL INTELLIGENCE AGENCY
DIRECTORATE OF SCIENCE AND TECHNOLOGY
OFFICE OF SCIENTIFIC INTELLIGENCE
smatter-.
 This report describes and presents the results of a Delphi (question-
naire) examination of the of the relevance tree
.* The Delphi participants, CIA employees of
diverse backgrounds (see appendix C), were as e to assign numerical
values representing the priority that they felt the Soviet Union would
place on obtaining significant improvement, or upgrading, of the various
strategic alternatives in the relevance tree. The time frame of interest is
5 to 15 years in the future. The results of this exercise, conducted from
November 1970 to April 1971, form requisite inputs to subsequent
studies and analyses concerned with the forecasting of
possible. future Soviet weapons and strategic systems. This report was
produced by the Office of Scientific Intelligence. It was coordinated
within CIA, but the reader is cautioned that its conclusions result from
the application of a particular methodology and do not represent co-
ordinated CIA opinion on the strategic military objectives of the USSR.
"Relevance tree: a structured array of objectives and alternatives, arranged as
aggregates of activities, which a decision-maker might consider for future emphasis,
improvement, upgrading, and/or investment. A vertical relevance tree starts with
objectives and contains a hierarchy of sub-objectives.
 Q8 EV_11E7nrEM_1rr_
TECHNICAL FOREWORD
The of future threat identification involves a
blend of inductive and deductive techniques and relies heavily upon
the systematic use of experts to make quantitative judgments in specifi-
cally defined areas. The general philosophy Fonsists of breaking down
a large number of complex evaluations into relatively simple numerical
assessments that can be stored, manipulatedi analyzed, and easily re-
vised. The methodology is based in part on the PATTERN (Planning
Assistance Through Technical Evaluation of Relevance Numbers) sys-
tem of relevance network analysis developed by Honeywell, Inc.-I-5 The
relevance tree portion is designed to assist in the identifica-
tion and evaluation of Soviet military Specific Objectives (working-level
tasks that may be performed to satisfy one or more national goals). The
general relevance tree, beginning with Soviet national goals and ending
with Specific Objectives, is shown schematically in Figure 1.*
The original PATTERN technique involved the development of
specific criteria for each node, the asignment of weights to each of these
criteria, and the assignment of relevance numbers to each alternative
vis-a-vis each criterion. An overall weight attached to each alternative at
a node is then obtained by forming a linear combination of the relevance
numbers. * *
Recent theoretical and experimental work indicates the feasibility
of a "holistic" approach to the assignment of the overall weights to the
nodal elements of the relevance tree. This variant to PATTERN was
used in this exercise and required an individual to assign overall nodal
*For the purposes of this exercise, the term element will be used to designate
any component entry in the relevance tree. The elements of the relevance tree are
denoted by the boxes in Figure 1. A node will refer to a collection of elements,
consisting specifically of the alternatives and the superior objective that they
support. A node is often referred to as a branch point. An alternative at a node is
one of the elements which represent means for attaining the objective. The alterna-
tives at one level of the relevance tree become the objectives at the next lower
level. The alternatives at a node are not necessarily mutu 'y exclusive and
exhaustive.
"Consider a node having n alternative elements, el, j- 1, 2, , n, and a set of
m criteria, CR, k =1, 2, ... , in. Then the nodal relevance number associated with the
element el is given by
m
rr = 7, Wk ro,
k=1
where W. is the weight (measure of importance) associated with criterion C%, and
rik is the relevance number of the element ej based on a consideration of criterion
 weights directly, taking into account implicitly the various criteria or
application of that combination. It is of interest also to note that the
gested before,? 7 but this paper describes what is probably the first real
combination of the Delphi and relevance tree techniques has been sug-
formal application of the Delphi method, described in Appendix A. The
bases for evaluation. Moreover, these weights were arrived at through
holistic simplification of PATTERN has been suggested.7
those for the Specific Objectives, from which a priority ranking may be
element of interest,4 " In the desired weights are primarily
of the tree. The usual prescription is that the branch weights are formed
from the product of the nodal wei hts in the levels superior to the branch
mathematically to obtain weights associated with the various branches
In PATTERN the nodal weights are combined
 NATIONAL
GOALS
Figure L General Relevance Tree Structure for the
NATIONAL ACTIVITIES
SPECTRUM OF CONFLICT
MISSIONS
TASKS
SPECIFIC OBJECTIVES
 CONTENTS:;
Page
PREFACE ....... .......... iii
TECHNICAL FOREWORD .. v
SUMMARY AND CONCLUSIONS ... ....... 1
DISCUSSION OF RESULTS AND RANKINGS ................. 4
Examples of results at nodes ....... ........ . 4
Rankings of tasks and specific objectives .. . ................ 7
Comments on correlations and arms-control ................... 14
REFERENCES ..................... .... ....... ... ..... 41
APPENDICES
Page
A. Nature of the Exercise ........... . .... ....... 21
The Delphi Technique ....... .. . .... ..... 21
The Premise for Priority Assessment . ......... 22
Criteria or Bases for Evaluation .. . . ..... .. 22
B. The Relevance Tree for Strategic (General) Warfare ........ 24
Definitions of Relevance Tree Elements ... . ............ 25
C. Organization and Conduct of the Delphi Exercise ... . ........ . 33
Assignment of Priority Numbers--Scaling Procedure ........ 33
Non-Numerical Responses and Feedback Materials .......... 34
An Investigation of Sensitivity to Arms-Control ....... . ..... 34
D. Tabulation of Nodal Results ........ .................. 36
PF-10-11S
 A DELPHI EXAMINATION OF FUTURE SOVIET
STRATEGIC MILITARY OBJ 4 CTIVES
SUMMARY AND CONCLUSIONS
A defined set of future Soviet strategic military
objectives has been examined with a combination
? of the Delphi technique for elicitation of expert
opinion and the relevance tree method for manipu-
lation and aggregation of that opinion. The rele-
vance tree, containing the objectives which were
evaluated and ranked according to the priority
which the Soviet Union might place on obtaining
a significant improvement, or upgrading, over the
next 5 to 15 years, is shown in Figure 2.
The opinions elicited in this exercise place high
emphasis on defensive areas, with anti-ballistic mis-
sile (ABM), anti-aircraft, and early warning sys-
tems generally assigned top priority. Expected So-
viet concern about enemy strategic forces as targets
was exhibited by the high ranking attached to the
counterforce capabilities against submarines, land-
based missiles, and strategic aircraft. Anticipated
future Soviet interest in the anti-submarine warfare
(ASW) problem is also manifest by a high priority
on ocean surveillance and naval command, control,
and communications.
A limited examination of the sensitivity of the
results to arms control indicated in the view of the
participants an emergence of emphasis on intelli-
gence/Warning/Control objectives in the case of
a fairly comprehensive restriction on offensive and
defensive weapons.
Analysis of the data has suggested certain revi-
sions or modifications in the methodologies em-
ployed to aggregate opinions in the relevance tree
network. The results indicate areas of future work
on such methods which might be fruitful and per-
tinent.
RANKINGS AND RELEVANCE NUMBERS
Rankings for the Task and Specific Objectives
levels of the relevance tree have been obtained
using various types of mathematical procedures for
manipulation of the data collected in the Delphi
process. Considering these variations and the ob-
served sensitivities of the rankings with future arms
control environments, the following (partial) rank-
ordered list of Specific Objectives is a good repre-
sentative summary:
411 Active Air Defense/ABM Systems
112 Destruction of Forces/Submarines
331 Strategic Intelligence/Early Warning
412' Active Air Defense/Anti-aircraft . Systems
111 Destruction of Forces/Land-based Missiles
322 Operational Intelligence/Ocean Surveillance
312 Command, Control, Communications/Naval
Forces
114 Destruction of Forces/Strategic Aircraft
321 Operational Intelligence/Satellite Surveillance
422 Air Defense/Mobile Basing
423 Air Defense/Deception Techniques
122% Destruction of Resources/Transportation
214: Strategic Mobility/Spacelift
432 Civil Defense/Protective Shelters
431 Civil Defense/Evacuation and Dispersal
Only the Specific Objectives near the top and
near the bottom have been listed, since interest in
the results will tend to focus on the most important
items and those which may be safely ignored rela-
tive to the others. The top 10 Specific Objectives
listed above are in the top 10 in almost every
variation investigated, and the bottom five are al-
most always on the bottom, with variations gener-
ally in exact position or rank in both groups. In
selecting a single set of relevance numbers or
numerical weights and an associated ranking, it is
 111 LAND BASED MISSILES
DESTRUCTION
112 SUBMARINE FORCES
OF
113 SURFACE NAVAL FORCES
FORCES
114 AIRCRAFT 6 CRUISE MISSILES
115 ORBITAL & SPACE WEAPONS
STRATEGIC
(GENERAL)
WARFARE
LOGISTICS/
SUPPORT
INTELLIGENCE/
WARNING/
CONTROL
DESTRUCTION
OF
RESOURCES
311 AEROSPACE OFFENSIVE CCC
312 NAVAL FORCES CCC
313 SPACE CCC
314 AEROSPACE DEFENSIVE CCC
315 NATIONAL COMMUNICATIONS
MISSION
LEVEL
CONFLICT
LEVEL
DESTRUCTION
OF DEFENSES,
CONTROLS,
INTELL.
STRATEGIC
MOBILITY
FORCE
PROJECTION
COMMAND,
CONTROL, AND
COMMUNICA-
TIONS
32 OPERATIONAL
INTELLIGENCE
STRATEGIC
INTELLIGENCE
ACTIVE
AIR DEFENSE
PASSIVE
AIR DEFENSE
CIVIL
DEFENSE
121 CITIES & POPULATION CENTERS
122 CIVILIAN TRANSPORTATION
123 CIVILIAN COMMUNICATIONS
124 IND. & MIL. VALUE TARGETS
131 DEFENSIVE SYSTEMS
132 MILITARY CCC
133 STRATEGIC WARNING & INTELLIGENCE
134 SPECIAL TARGETS
1211 STRATEGIC AIRLIFT
212 STRATEGIC SEALIFT
1 213 STRATEGIC LAND TRANSPORT
214 STRATEGIC SPACELIFT
221 SPACE/ORBITAL BASES
222 SUBSURFACE SEABASES
223 SURFACE SEABASES
224 LANDBASES
321 SATELLITE SURVEILLANCE
322 OCEAN SURVEILLANCE
323 GLOBAL SURVEYS
324 METEOROLOGICAL SURVEYS
331 EARLY WARNING
332 POST-ATTACK ASSESSMENTS
333 LONG-RANGE ESTIMATES
334 FORCE DEPLOYMENTS
1411 ABM SYSTEMS
412 ANTI-AIRCRAFT AND CRUISE MISSILES
1413 ANTI-SATELLITE AND SPACE SYSTEMS
1421 STRATEGIC HARDENING
422 MOBILE BASING
1423 DECEPTION TECHNIQUES
1431 STRATEGIC EVACUATION & DISPERSAL
432 PROTECTIVE SHELTERS
1433 POST-ATTACK MEASURES
TASK SPECIFIC OBJECTIVES
LEVEL LEVEL
Figure 2, Relevance Tree for Strategic (General) Warfare.
Complete definitions of the nlomonts are contnmed in nppendi. B. The number to the
upper loll of enCh olemnnt will be used ns a form of shorthand to designate the element
the nndc number IS dnfrted as the number of the nblerbve element in the node, for
example' Node 4 in the node in which Defense, (Element 4) is the objective and the Alter-
natives are Active Air Defense (41). Passive Air Defense (42). and Civil Defense (43).
2 SEC-REf
 probably best to choose the Round 2 product of
the non-normalized (nodal) averages in table 7.
The ranking for this case, shown in a staggered
format which illustrates the broad (mission-level)
categories of dominance, is shown below.
Although only five of the 16 nodes in the rele-
vance tree were examined in Round 3 of the Delphi
process under the three hypothetical Arms-Control
Specific Objectives
Rank Defense
Intel]/
Warning/
Control
Logistics/
Strike Support
1
.......... 411
2
..........
3
.......... 412
4
..........
312
5
331
6
..........
7
..........
322
8
..........
321
9
..........
314
10
..........
311
12
.......... 422
13
....
334
14
..........
15
315
16
.......... 413
17
..........
132
18
..........
113
19
..........
133
21 .......... 421
22 ..........
313
23 ..........
124
24 ..........
26 ..........
27 ..........
28 ..........
29 ....... .,.
332
30 ..........
324
31 ..........
213
32 ..........
222
33 ..........
34 ..........
35 ..........
30 .......... 423
37 ..........
221
38 ..........
223
39 .
40 . .. .... 433
41 ..........
42 .......... 432
43 .......... 431
Environments, it is believed that the results would
not be significantly different if the remaining 11
had been included. Node 0 is probably the domi-
nant factor in variation with arms control, with
lesser influences from subordinate nodes. The one
node examined at the task level, 41, did not exhibit
great variations, even though it has ABM as an
explicit alternative and ABM was specifically re-
stricted in two of the hypothetical environments
considered. Much more significant differences are
apparently obtained from the manner in which the
numerical assignments are manipulated and ag-
gregated than from such substantive origins.
Two major lessons were leaned in the course of
se that impinge on future work with
phisPxP 1) it is workable and beneficial to col-
lect non-normalized assignments of an abstract
quantity such as "priority" via the Delphi technique;
2) important assumptions or potential uncertainties,
such as SALT and future arms control environ-
ments, should be considered explicitly. It has been
found feasible to handle this by explicit definitions
and a repeated evaluation of the given question
for each assumption. For Delphi studies of the type
considered here, two rounds appear sufficient, un-
less some significant fact turns up or initially un-
known uncertainties appear and require further
definition and iteration.
FURTHER ANALYTICAL POSSIBILITIES
Considering the richness and amount of data that
was collected in the Delphi exercise, additional
analyses might be performed. From a quantitative
point of view, use could be made of various com-
binations of nodal assignments and normalization;
more elaborate sensitivity analysis might be car-
ried out; and, in particular, some effort might be
expended in the analysis of the results
correlation studies might be attempted to better
understand the results obtained in the third round
(with the hypothetical Arms-Control Environments)
as compared to, say, the second round. Finally, since
a democratic procedure was used in the analysis
reported here, that is, all results obtained were
used and each respondent's assignments were
weighted equally, it would be interesting to investi-
gate the effects of restrictive selection of the results
on each node, for example, by considering only the
sub-set of respondents common to all three rounds
 and/or an attendant weighting of those assignments
(e.g., by using the self-ratings, which have thus
-Jar only been exploited in the original assignment
process) . ,
From 'a qualitative point of view, much might
be done with the wealth of commentary provided
by the respondents. in Part B of the Delphi Ques-
tionnaires. At this time, the commentary has only
been used by the participants themselves in the
feedback process. The comments might well be
studied more thoroughly and possibly interpreted
in non-numerical context. Such a study could be
compared with the numerical results to answer
such questions as whether the numerical procedures
preserve accurately the gist of the opinions ex-
pressed non-numerically.
Finally, from a strictly methodological point of
view, the state of the art in both Delphi and
relevance tree technology might be enhanced by a
careful examination of the data obtained in this
exercise. The non-normalized assignments which
were collected at each node may lend themselves
to a rigorous treatment of tha question concerning
the validity of multiplication between levels of the
relevance tree; further experimentation is warranted
to ascertain the validity of various corrections to
the PATTERN results to counter the bias due to
the actual structure of the tree. The qualitative
analysis described above might well give some in-
sight into Moth of these questions.
DISCUSSION OF RESULTS AND RANKINGS
A tabulation of the results obtained in the Delphi
examination of the 16 nodes in the relevance tree
is contained in Appendix D. This tabulation forms
the basis for the results and analyses below. The
distributions at each node are portrayed by the
quartiles, medians, and means. The PATTERN
process requires normalized data, and the usual
procedure is to normalize the individual assign-
ments at each node and then compute the (nor-
mali''.ed) average of these.OThese results are
shown in the far right-hand column. For compari-
son, the (non-normalized) averages have been
normalized and tabulated in the second column
from the right. It should be noted that there is
general agreement between these last two columns.
A simple measure of dispersion in assignments is
obtained from the so-called Inter-Quartile Range
(IQR ), the difference between the upper and
lower quartiles (Qn--Q, ). Note that, by definition,
the IQR contains the middle 50% of the assign-
ments. In the following sections some examples of
results at individual nodes are presented, followed
by the rankings obtained for the Specific Objectives
(and Tasks) by combination of the nodal assign-
ments.
EXAMPLES OF RESULTS AT NODES
Node 0 is probably one of the most interesting
and important nodes in the entire relevance tree.
As a consequence, this node merits some additional
attention and description. The distribution of pri-
ority number assignments, in the form of the so-
called Delphi houses, is shown in Figure 3. Here,
one can observe the reduction in IQR on the second
round as compared with Round 1. Also, it appears
that the specification of an Arms-Control Environ-
ment in Round 3 results in yet further improvement
in consensus for most of the elements at this node.
The investigation in Round 3 of the Arms-Control
sensitivity is described in Appendix C. Expansive
definitions wer given for the three hypothetical
Arms-Control Environments (ACES) considered.
In simplest terms, the three environments are given
in the table below.
Table 1
Short titles for the three hypothetical Anns-
Control Environments (ACEs) investigated in
the third round of the Delphi Exercise
(see Appendix C)
ACE-0
...
Terminntion of negotiations
ACI -1
Limited (AI1Nt) ngreenient
ACE-2
...... Full (offensive+defensive) agreement
An alternative way of presenting the results at
Node 0 is that shown in Figure 4, in which fre-
quency histograms for rank of the four elements
are given. Such a representation emphasizes the
 LEGEND
Element I Strike
Element 2 logistics/Support
Element 3 Intelligence/Warning/Control
Element 4 Defense
Arms-Control Environments (ACE):
ACE-0 Termination of Negotiations
ACE-1 Limited (ABM) Agreement
ACE-2 A Full Agreement
2
i
3
ROUND 3 ACE-O
0
ROUND 3 ACE-1
0
ROUND 3 ACE-2
0
Figure 3, Representation of Nodal Assignments at Node 0.
The housee ae a ehatlhand s qy of denobnp the dtatttbutMn of asstQmenfL
to which the peel denotes the m.tdian and the wells denote the quartiles to 1.03).
 ACE-1
Agreement)M
ACE-2
(A Full
Agreement)
1 2 3 4
Strike
LN
1 2 3 4 1 2 3 4 1 2 3 4
1 2 3 4 1 2 3 4
topistics/ intelli
Support Warn n%1
Control
1 2 3 4
Defense
Figure 4, Frequency Histograms of Rank of the Elements at N'pde 0 for Round 2
and the Three Arms-Control Environments (ACEs) Considered in Round 3 (ACE
Definitions In Appendix M.
$11570 7.71 CIA
CI CRET
 relative positions of the elements as indicated by
the respondents and conveys a measure of cer-
tainty to be associated with the rank of a given
element. In this figure, for example, one clearly sees
the inferior position accorded to Logistics/ Support
(Element 2) in all cases-it was placed in fourth
position by some 60% of the respondents on Round
2 and by as high as 79% in Round 3 (assuming
ACE-0, termination of SALT negotiations). The
emergence and dominance (in the case of a full
Arms-Control Environment, ACE-2) of Element 3
(Intelligence/ Warning/ Control) is to be observed
and probably more clearly so than was indicated
in Figure 3. This Mission was ranked third by
nearly half of the respondents in ACE-0, second
in ACE-1, but first by an overwhelming 78% of
the respondents in ACE-2.
The nature of the distributions of normalized
assignments are of interest, particularly because
the averages of such distributions are generally
used in the PATTERN procedure to obtain rank-
ings of the elements across an entire level of the
relevance tree.2 4 The few nodes with three alter-
natives lend themselves to a graphic illustration
of such assignments in the form of barycentric, or
triangular, coordinates, a technique well-known to
physical chemists and metallurgists. This method
uses the property of an equilateral triangle that
the sum of the distances to the three sides from
a point within is a constant. In this representation,
a point in the very center of the triangle denotes
an equal-weight assignment on all three elements,
and a point at a vertex denotes a weight of 1 at-
tached to the element cited at that vertex and a
0 to each of the other two elements.
Figure 5 illustrates the reduction in dispersion
between rounds but with only slight shifts in the
mean assignments. Figure 6 shows the distribution
of assignments for the same node as shown in Fig-
ure 5 for each of the hypothetical Arms-Control
Environments. It should be noted that the disper.
sion Is slightly greater for ACE-2 than for either
ACE-0 or ACE-1, a phenomena which appears to
be the general case for the nodes investigated in
Round 3. Figure 7 Illustrates a case in which the
dispersion was reduced between rounds and in
which there was also a noticeable shift in the mean
assignments. This figure also reveals an example
of a hold-out from Round 1 which ended up quite
removed from the consensus position on Round 2.
Figure 8 illustrates a rather unusual result in that
there was a rather definite lineal distribution formed
on the second round. Most of the distributions ob-
served in this exercise tended to be circular, or
only slightly distorted from such a pattern, as
shown in the preceding figures.
A very rough estimate was made of the overall
reduction in dispersion as the number of rounds
increased. On Round 1, the average IQR was about
39, whereas on the second round it dropped to 31.
This is similar to the behavior observed by Dalkey
in his experiments at Rand, in which the dispersion
will be reduced successively between rounds, in
the absence of relevant facts, but with no improve-
ment in error (as measured by the mean or median
estimate).9-10 It is interesting to note that the
average IQR on Round 3 was reduced to 29, with
a range from 26 on ACE-0 (Termination of negotia-
tions) to 32 on ACE-2 (Full agreement). One
concludes that there is less of a consensus in the
case in which a fairly full offensive and defensive
agreement is effected.
RANKINGS OF TASKS AND SPECIFIC
OBJECTIVES
Suppose we let u,?v denote the priority number
for the with alternative at the Ath node, regardless
of the normalization procedure used. In general,
we shall assume a multiplicative rule of combina-
tion for the nodal assignments in order to obtain
overall weights (relevance numbers) for the Task
and Specific Objective levels. Thus, the relevance
numbers for the Task ij, ij=11, 12, , .. , 43, are
given by
raj-"xi . ',ni 1 (1)
and the relevance numbers for the Specific Ob-
jective ijk, ijk =111, ... , 433, are given by
r1Ik 0w, . 'a,t . 't*u~~rtI . '~xu~ . (2)
In the PATTERN scheme, the relevance numbers
are computed by using the average of the nor-
malized assignments at each node (the far right-
hand column of data in the tabulation in Ap-
pendix D). The relevance numbers and rankings
obtained from this data for the Tasks and the
Specific Objectives are shown respectively in Table
2 and Table 3.
Inspection of Tables 2 and 3 reveals a very
interesting and disturbing result, namely that Tasks
 FRET
LEGEND:
31 Command, Control
and Communications
32 Operational intelligence
33 Strategic intelligedce
31 32
Figure 5. Barycentric Coordinate Representation of the Normalized
Assignments at Node 3, The Tabulation Gives the Mean Assignments
for the Entire Set of Data Collected and the Means for the Sub-set
of Respondents Common to Round 3 (See Figure 6)
SECT
 LEGEND:
31
Command, Control
ACE-0 Termination
and Communications
of Negotiations
32
Operational
ACE-1 Limited
Intelligence
(ABM) Agreement
33
Strategic
ACE-2 Full Agreement
Figure 6. Barycentric Coordinate Representation of the Normalized
Assignments at Node 3, Round 3, for Each of the ACE's Considered.
811878 7.71 CIA
-SEEM-
 SEGRE;
Round 1
(21) .400
.420
.180
(17) .405
.442
.152
0
0 0
0
422 Mobile Rasing
423 Dacaption
a
0
0
Round 2
(17) .966
.4R1
2 are Shown Shaded in Round 1.
511570 7.71 CIA
Figure 7. Barycentric Coordinate Representation of the Normalized
Assignments at Node 42. The Four Responses Missing from Round
421 422
SECRET
 LEGEND:
431 Strategic Evac
A Dispersal
432 Protective Shelters
433 Post-Attack
Measures
(14) .293
.258
.451
Round 2
(14) .256
.279
%a& 432
Figure 8. Barycentric Coordinate Representation of the Normalized
Assignments at Node 43. The Four Respondents Missing in the
11-
 Table 2
Relevance Numbers and Rankings for the Tasks as Obtained by the PATTERN Procedure
11
0.1188
2
0.1127
2
0.1282
2
0.1335
I
0.09157 6
12
0.0750
8
0.06521
10
0.07632
0
0.06878
10
0.0562 10
13
0.0756
9
0.08606
7
0.08352
6
0.0867
7
0.06022 9
21
0,10512
3
0,1037
3
0,09861
3
0.1037
4
0.1120 3
22
0.07488
10
0.07826
9
0.07439
10
0.07826
8
0.08514 7
31
0.0858
5
0.093515
4
0,08117
7
0.09349
5
0.1077 5
32
0,0832
6
0.08374
8
0.07968
8
0.0929
6
0.112 4
33
0.001
4
0.08772
6
0.08815
4
0.1076
3
0,1393 1
41
0.1682
1
0.1624
1
0.1053
1
0.1274
2
0.1189 2
42
0.0812
7
0.08059
5
0.08468
5
0.07168
9
0.0739 8
43
0.0406
11
0.03699
11
0.04002
11
0.03596
11
0.03322 it
*Round 3 :omputations used the nodal assignments from Round 2 for the eleven nodes not investigated explicitly in the third
(ACE) round of the Delphi process.
21 and 22 and the Specific Objectives beneath
them, particularly 211 and 212, seem to be rated too
high. This over-rating is somewhat unexpected,
considering the emphasis given to Element 2 at
Node 0 (particularly exhibited in Figure 4) and
the comments made by the respondents. An obvious
source of this over-rating is the relevance tree
structure itself: Node 2, in which the Tasks 21 and
22 arc compared, is the only binary node in the
tree and it seems reasonable, considering the nature
of the PATTERN prescription, to suspect an auto-
matic bias in the results. One notes that the Tasks
under Element 2 will have a weight in the neighbor-
hood of 0.5 whereas those elsewhere automatically
have a weight near 0.33.
Suppose one were to have assigned equal weights
to each element at each node in the tree. This no-
choice, no-preference situation would, for the given
tree structure, automatically result in a ranking or
sorting of the Tasks and Specific Objectives as a
result of the non-uniform splitting or branching of
the tree. Although not completely correct, one
might at least expect that a nearly impartial result
would have obtained for the Tasks and Specific
Objectives, given impartiality everywhere else. This
train of thought leads one to attempt'to correct the
results obtained by PATTERN using some sort of
comparison of those results with those which would
be obtained in the no preference situation. In other
words, it might be appropriate to consider rankings
based on the magnitude of the ratio of the PAT-
TERN relevance numbers to a set of numbers ob-
tained by impartial (uniform) assignments at each
node.* Although the viewpoint here is slightly dif-
ferent, the motivation for, and the results obtained
from, this heuristic correction for structural bias
are essentially identical to those given elsewhere."
The ratios and associated rankings obtained by
a modification of the PATTERN results are given
in Tables 4 and 5. Some shifts from the previous
results are to be observed, particularly the sup-
pression of Tasks 21 and 22 and their subordinate
Specific Objectives. On the other hand, violence
has not been done for the most part to the positions
of the other elements. No attempt has been made
as yet to justify in detail this modified ranking, an
analysis which might well be based, however, on
the extensive Delphi-derived commentary of the
participants.
*For the Tasks and Specific Objectives, these ratios are
given respectively by
r,r'=r,r/(1/4)(1/n,)-4n,r,r,
r,rti'=4n,n,rr,rti= n,rr,r',
n, = the number of alternatives at node i,
no l =the number of alternatives at node if.
12
 Table 3
Relevance Numbers and Rankings for the Specific Objectives as Obtained by the
PATTERN Procedure
Round I Round 2 Round 3 ACE-0 Round 3 ACFr 1 Round 3 ACE 2
SO Relevance Rank Relevance Rank Relevance Rank
Relevance Rank Relevance Rank
111
0.02851
10
2.829471(2
13
3.2179,'-2
0
3.3511(2
7
2.200/(2
18
112
0,03208
7
3.021111''2
8
3.4351(2
6
3.5781(2
4
2.4541(2
17
113
0.0202
23
1.848741'`2
26
2.10212
22
2.1901'`2
22
1.5021;"2
33
2.5779,x'2
17
1.7671(2
29
115
0.01544
34
1.397831'2
34
1.58912
30
1.65612
30
1. 1361(2
38
121
0.01739
27
1.408491``2
32
1.6491(2
28
1.4861''2
33
1.21412
36
122
0.01058
42
8.08579E 3
43
9.46413
43
8.529E-3
43
6.9691'3
43
123
0.01663
31
1.330241`-2
36
1.5555712
31
1.4039-2
34
1.147E-2
37
124
0.031
8
2.973489.x2
9
3.4801(2
5
3.1361(2
11
2.5631' 2
16
131
0.02268
17
2.62495E-2
15
2.5471-2
I'll)
2.6441(2
15
2.1111(2
23
132
0.02419
16
2.556101'2
16
2.4811x'`2
16
2.55751-2
18
2.05612
24
133
0.01814
25
2.168819-2
21
2.10512
21
2.1851(2
23
1.74412
30
134
0.01058
43
1.255655:39,"2
37
1.2191-2
36
1 .2661'2
37
1.01 11(2
40
211
0.02838
11
2.925471-2
10
2.7811'2
11
2,925E-2
12
3.1839.x' 2
10
212
0.03574
4
3.5506411x'2
5
3.3339 -2
7
3.5061`2
5
3.8159.-2
5
213
0.02733
13
2.738741(2
14
2.603E-2
14
2,7399-2
14
2.080F,-2
12
214
0.01367
37
1.203381'2
38
1.1349x'2
39
1 .1931'` 2
38
1.309R-2
35
221
0.01647
32
1 .63556312
30
1 .55559`2
32
1.6361`2
32
1.77912
28
222
0.02172
19
2.410411( 2
18
2.291 F 2
19
2.4101(2
21
2.6221(2
15
223
0.01647
33
1.674701-2
28
1.59212
29
1.6781,'`2
28
1.822E-2
26
224
0.02022
22
2.105191(2
22
2.00112
23
2.10512
24
2.290K-2
19
311
0.01716
28
1.9176712
25
1.66412
27
1.9171'2
26
2.20812
21
312
0.02059
21
2.4602312
17
2.13551(2
20
2.45501(2
19
2.83312
13
313
0.01201
39
1.403171'`2
33
1.2189,''2
37
1.4021(2
35
1 .6161'2
31
314
0.01888
24
1.9:36381(2
24
1.6801'2
26
1.93512
25
2.22912
20
315
0.01716
29
1.637049.' 2
29
1.42112
34
1 .0361'2
31
1 .8851"2
25
321
0.02662
14
2.838791(2
12
2.7011.'`2
13
3. 149K-2
10
3.79712
6
322
0.02746
12
2.889031(2
II
2.749E-2
12
3.20551(2
9
3.86412
4
323
0.01747
26
1.5582691(2
:31
1.5061'2
33
1.75612
27
2.11712
22
324
0.01165
41
1.063501'-2
40
1.0121'2
42
1.1801'2
39
1.4231'2
34
.331
0.03003
9
3.2717712
7
3.2881(2
8
4.0141(2
3
5.19612
1
332
0.01456
36
1.1315212
39
1.13712
38
1.38812
36
1.79712
27
333
0.02093
20
1.082361'2
23
1.99212
24
2.43212
20
3.14812
11
334
0.025548
155
2.385851.x-2
19
2.3891(2
18
2.91612
13
3.7891(2
7
411
0.07905
1
7.48747K" 2
1
7.50512
1
4.94212
I
4.63612
2
412
0.05214
2
5.246101x-2
2
5.20712
2
4.5221(2
2
4.04212
3
413
0.037
3
3.5082312
4
3.81812
4!
3.27312
8
3.21012
9
421
0.03248
6
3.278991'`2
6
3.09912
10
2.6231"2
16
2.7051(2
14
422
0.0341
5
4.309281(2
3
4.07312
3
3.4481(2
6
3.0551'-2
8
423
11.01462
35
1.370731'-2
35
1.2961(2
35
1 .0971`2
40
1. 1311''2
39
431
0.01218
38
9.460951'8
42
1.02512
41
9.2041" 3
42
8.5051`3
42
432
0,01177
40
10320R1.''2
41
1.1171(2
40
1 .0031x"2
41
0.2691(3
41
433
0.01665
30
1,720131(2
27
1.8611(2
25
1.67212
29
1, 0451(2
32
13
 Ratios and Modified Rankings Obtained for the Tasks by an
Application of Equation 3
Ratio
Rank
Ratio
Rank
Ratio
Rank
Ratio
Rank
It
1.4313
2
1.358
2
1.544
2 .
J.600
l
1.103
5
12
0.0108
7
0.7856
9
13
0.9108
8
1,037
6
1.000
5
1.045
6
0.834
8
21
0.8409
9
0.8200
8
0,7880
9
0,8299
8
0.9029
6
22
0.599
10
0,6261
10
0.5951
10
0.6261
10
0.6811
9
31
1.033
4
1.127
3
0.978
6
1.126
4
1.298
4
32
1.002
5
1.009
7
0,96
7
1.119
5
1.349
3
33
1.096
3
1.057
5
1.0(32
3
1.296
3
1.678
1
41
2.026
1, 057
1
1.992
1
1.535
2
1.432
2
42
0.978
6
1.079
4
1.02
4
0.8636
7
0.8904
7
43
0.489
11
0.4457
11
0.4822
11
0,4332
11
0.4003
11
Suppose we postpone the normalization of the
data and apply the multiplicative formulae using
the non-normalized averages at each node (Ap-
pendix D, column A). The tabulation of the re-
sultant products, normalized after multiplication,
and the associated rankings for the Tasks and
Specific Objectives are given in Tables 6 and 7
respectively. The significance of these results and
the. procedure employed to obtain them will be
discussed in the following section, in which the
various rankings are compared.
COMMENTS ON CORRELATIONS
AND ARMS-CONTROL
Let us consider, as a simple means of com-
parison of the various rankings, the rank-difference
correlation coefficient, given by *
N
-6 (X,-YI)' /N(N'--1), (3)
i=1
where N is the number of ordered pairs (XI, YI ),
and X, and Y, are the rankings of the ith element
in the two rankings considered. Perfect agreement
is indicated by p = -I-1, while p = --1 indicates
exactly opposite rankings.
*Dixon, W. J., and F. J. Massey. Introduction to Statis-
tical Analysis, New York, McGraw-Hill, 1957, p 294-295.
Round 3
ACE?-O ACT,-1 AC1 -2
It is of interest to compare the non-PATTERN
rankings introduced in the previous section to the
PATTERN rankings and to one another. The cor-
relation between PATTERN and its modification
varies, depending on the round and the tree level,
but typically is of the order of 0.85. The correla-
tion between PATTERN and the third approach,
the product of the non-normalized averages
(PNNA) is worse, typically about 0.70. In the final
possibility, we find the correlation between the
modified PATTERN (ratio) and the PNNA to be
quite high, on the order of 0.92. This final result
is very interesting because it means that com-
parable results are obtained from the modification
of PATTERN to account for tree structure and the
perfectly straight-forward results obtained from
multiplication of the non-normalized nodal averages.
Suppose the modified PATTERN results are cor-
rect. Then it appears as if normalization at a node
results in a loss of information (reflecting on tree
structure) which has to be reintroduced math-
ematically. On the other hand, collecting and com-
bining non-normalized data with no corrective ac-
tion is appealing in its simplicity and very attractive
in terms of its correlation with modified PATTERN.
Non-normalized nodal assignments obviously con-
tain more information content than normalized
assignments, and it is possible that accommodation
to the tree. structure is part of this content. Using
 111 1.71 1 4 1.69769E0
112 1.925 2 1.81267EO
113 1.212 14 1.10924E0
114 1.354 7 1.30539E0
115 0.9266 23 8.38697/c1
121 0.8346 28 6.760771x` 1
122 0.508 39 3.881181" 1
124 1.488 5 1.42727E0
131 1.089 19 1.2599880
132 1.161 16 1.2260380
133 0.8709 26 1.0410380
134 0.508 40 6.03137K 1
211 0.0082 24 9.36150E" 1
212 1.144 17 1. 1220.510
213 0.8746 25 8.76306h," 1
214 0.4373 42 3.850838'1
221 0.5272 36 5 . 2340:3K
222 0.6949 32 7.71331/(1
223 0.5272 37 5.359241: I
224 0.647 33 6.736021,.'- 1
311 1.03 26 1 ,15060/x0
312 1.236 11 1 .4 76141.'0
313 0.7207 30 8.410061.'- 1
314 1.133 18 1.1618:380
315 1.03 21 9.822231' 1
321 1.278 10 1,36262/;0
322 1 , 318 9 1 .38(374 /x0
323 0.8387 27 7.596908`1
324 0.5591 35 5.1048081
331 1,441 6 1.57045K0
332 0.6989 31 5.431328 1
333 1.005 22 9.515331'` 1
334 1,223 13 1.1452180
411 2.846 1 2.605491x0
412 1.877 3 I .888001'0
413 1.332 8 1.20296E0
421 1.169 15 1.1804480
422 1.228 12 1.55134K0
423 0.5262 38 4.934631'-1
431 0.4385 41 3.409108'1
432 0.1239 43 3.715481''1
433 0.5993 34 6, 102478-1
Ratios and Modified Rankings Obtained for the Specific Objectives by an
Application of Equation 4
Round 3 ACE-?0
Round 3 ACN-1
Round 3 ACE -2
Ratio
Rank
Ratio
Rank
4
1 .93080
3
2,0111'0
2
1 .379/x70
10
3
2.06:H'0
2
2.14780
I
1.47380
8
20
1 .26110
13
I .3141'0
12
9.01 11(1
26
it
1.184K0
7
1.546K0
6
1,0601x0
18
27
0.5351:1
23
0.934/(1
22
6.81:381
32
30
7,913P:- 1
27
7. 1311'-1
30
5.8271(1
34
40
4.5431.'-1
40
4.0941(1
39
3.345E-1
41
32
7,47:31'-1
28
6.7351,("1
32
5.50381
37
8
1 .67080
5
1 .50680
9
1.2301x0
14
1:3
1 .2231'0
14
1 .2691,0
13
I .013E0
21
1.1
1.191 EO
15
1 .236K0
15
0. 8688 y' 1
22
21
1.(11(1110
11)
1,049R0
21
8.3731(1
20
34
6.8538 1
34
6.0768-1
34
4.8511(1
38
24
8.8()01,' 1
24
9 3618: l
25
I .0181'0
to
I 9
I .00 7 K0
18
1. 122EO
20
1 .22180
16
25
8.331/(1
26
8.7641(1
20
9.5:34 K'1
25
?II
:3.6291x 1
43
3.8188-1
41
4.1898'1
39
:37
4.975/(1
37.
5;.2348`1
38
5.6941'1
35
28
7.3:321(1
29
7.7138 1
21)
8.:3911` 1
28
36
5.0941'' 1
36
5.3508(1
37
5.8301' 1
33
31
0.4031(1
33
6.7378' 1
31
7,3291,;- 1
30
17
9, 084 1(1
21
1 , 15080
19
1 .32580
12
7
1 .28180
12
1.4758(1
10
1,700E0
5
20
7.3061i'- 1
30
8.4141,'-1
28
9.6031(l
24
16
1 .0081'0
20
1 . 1611,0
18 .
I .338 80
11
22
8.5231' 1
25
9.8171?; 1
23
1 .1:31 E0
17
10
I .29780
I I
1.5i 12E0
8
1 .823E0
3
9
1 .320EO
10
1 .5:38K0
7
1 .855580
2
29
7.22981
31
8, 4281' 1
27
1 .016E0
20
38
1:8571(1
'38
5.0631.''1
36
6.8281'1
31
5
1.5781:0
6
1.92780
3
2.49480
:35
5.4581(1
35
0.6638' 1
33
h.025/( I
27
23
9.56281
22
1 . 16780
17
1 .611 /;0
7
18
1,1471'0
16
1.400EO
11
1.81980
4
1
2.702E0
I
1.77980
4
1.669E0
6
2
1.87510
4
1,62880
5
1.45;580
9
12
1 , 37580
9
1. 17880
16
1 . 155E0
16
15
1.1101'0
I?
9.4441(1
24
0,7371'1
23
6
I .4601'0
8
1, 2411'()
14
1 , 28010
13
311
4,66.18 1
39
3,9.18K' 1
40
4,0711"1
40
43
:1.6888-1
42
3.3141,'1
43
3.0621'"1
43
42
4,0'201'?1
41
:3.6111' 1
42
3.3371.'`1
42
33
6,6991x'"1
32
6,0191' 1
35
5.5611' 1
36
CEGRr,r -
15
 Normalized Products and Rankings for the Tasks Using Non-Normalized
Averages as the Nodal Assignments
ACE. 0
Round 3
- - - - - - - - - - - - - - -
AC)& I ACE.-2
Product
Rank
11
0.120
2
0.116
2 .
0.142
2
0.120
2
0.071
5
12
0.076
8
0,071
8
0.090
7
0.074
7
0.051
10
13
0.089
6'
0.096
6
0.004
6
0.085)
6
0.062
8
21
0.059
0
0.056
9
0.052
9
0,058
9
0.068
6
22
0.051
10
0.051
10
0.046
10
0.052
10
0.061
9
31
0.112
4
0.115
3
0.103
4
0.127
3
0.155
2
332
0.107
5
0.110
4
0.100
5
0.124
4
0.155
33
0.114
33
0,108
5
0.110
3
0.143
1
0.194
1
41
0. 153
1
0. 1533
1
0.142
1
0.10f)
5
0.091
4
42
0.079
7
0.088
7
0.082
8
0.068
8
0.065
7
43
0,040
II
0.037
11
0,036
11
0.030
11
0,027
11
the PNNA scheme is certainly interesting, if not
rigorously supportable at this time. If such a pro-
cedure were justified, its use would represent a
significant departure from the PATTERN prescrip-
tion.
It is doubtful that the results will vary signifi-
cantly, for the raw data obtained in this exercise,
if meaningful statistical quantities other than the
average are used at a node. For example, in Round
2, if one selects the normalized medians of the
distributions as the characteristic nodal assignments
and computes rankings by the PATTERN process,
a correlation of about 0.96 obtains between this
ranking and that of the PATTERN processing of
averages of normalized assignments. As another
example, correlations of better than 0.98 are ob-
served for rankings obtained from the latter pro-
cedure and those derived from PATTERN process-
ing of the normalized averages (the figures in Ap-
pendix D, column NA).
Inspection of the tables of rankings reveals that
in a large proportion of the cases, the rank of a
Task or Specific Objective on Round 2 agrees or
falls within the range of ranks observed in, the three
ACE cases. In part, Round 2 tends to correspond
with ACE-0 or ACE-1, as might have been ex-
pected. Using the rank correlation coefficient and
considering, as an example, the modified PATTERN
or PNNA results, the correlations (for the Tasks)
between Round 2 and ACE-0, 1, and 2, respectively,
are about 0.94, 0.90, and 0.72. For the Specific
Objectives, the corresponding correlations are about
0.98, 0.96, and 0.85, where the shift to higher values
is due to the fact that the Round 2 values were
used on the eleven nodes not specifically examined
under the three ACE's in order to compute overall
rankings.
On any given round or ACE, it does not matter
for ranking if normalization of the PNNA's is per-
formed, although this has been done for the results
given in Tables 6 and 7. Normalization is preferred
if a single set of numbers is desired for a given
round or ACE. On the other hand, normalization
may be inappropriate when comparing the magni-
tudes of the various numbers used to obtain the
rankings of the elements for various ACE's, since
this normalization may be destroying information
content. Figures 9 and 10 illustrate the behavior
of the non-normalized products of the non-nor-
malized averages for the Tasks and a portion of
the Specific Objectives, as a function of ACE. In
both figures, the ordinate is the fraction of the maxi-
mum possible value. Since there are three levels
in the tree and each element at each node can
have a value of 100, the maximum possible score
for a Specific Objective PNNA is 108. A better
feeling for the behavior of elements in the tree
may accrue from inspection of Figures 9 and 10.
 Normalized Products and Rankings for the Specific Objectives Using Non-Normalized
Averages as the Nodal Assignments
so
112
113
114
115
121
122
123
124
131
132
133
134
211
212
213
214
221
222
223
224
311
312
313
314
315
321
322
323
324
331
332
333
334
411
412
413
421
422
423
431
432
433
Round 1
Round 2
Round 3 ACH,O
Round 3 ACF--1
Round 3 AC.-.
Product
Rank
Product
Rank
Product
Rank
Product
Rank
Product
Rank
4.89512
3
4.420/(2
4
2.4461(2
15
4.3791.-2
2
4.321 K-2
2
5.2221:'-2
2
4.7161(2
2
2.6091,``2
12
2,8001(2
15
2.728A-2
18
3.2971,'2
10
2.977/'2
14
1.648P,-2
26
3.2221(2
13
3.1811` 2
11
3.8441(2
6
3.4711`2
10
1.921E-2
20
2.2111.-2
24
2.1031(2
25
2.3421(2
20
2.2951(2
22
1.2701(2
32
1.3151.' 2
31
1 . 1501 2
33
1.4331(2
29
1.1761(2
33
8.0901(3
36
8.87113
39
7.6091(3
39
9.4231(3
36
7.7231(3
38
5.3111(3
40
1.1761(2
34
1.06612
35
1.3201(2
30
1.0821, `2
35
7.4399-3
37
2.2841(2
23
2.2251(2
23
2.7551,'`2
17
2.2.581(2
23
1.5541(2
29
2.5751-2
17
2.9(381(2
14
2.888h-2
15
2,7091(2
16
1.8841.'`2
21
2.5731(2
18.
2.7401(2
17
2.6661(2
18
2.5011'2
18
1.7401.'-2
23
2.1841:2
25
2.555012
19
2.482/'2
22
2.3281'x"2
19
1.8191(2
27
1, 314R_2
32
1.5421,'2
27
1.301 F-2
27
1.408/'2
30
9.7891(3
35
1.4239-2
29
1.4321 2
28
1.291 1(2
31
1, 4301(2
29
1.7121,'2
25
1.6601(2
27
1 , 6861(2
26
1.5191,'2
26
1.6841(2
27
2.01515 2
18
1.3371`2
30
1,346E-2
31
1.213/"2
33
1.34415"2
31
1.609/x'2
28
7.1961,'`3
40
6.033/'3
41
5.4381(3
41
6.0261(3
40
7.2121(3
39
1,025E-2
37
0,9791 3
37
8.9951(3
38
9.9671(3
36
1.1931(2
33
1 . 1831(2
33
1.2711(2
32
1.1461,''2
34
1. 270K-2
32
1.5201(2
30
9.6271(3
38
8.63551(3
38
7.7838-3
39
8,6241(3
37
1.03212
34
1 . 1671(2
35
1 .1121" 2
3.1
1.0021:"2
35
1.1101:2
34
1.3291(2
31
3.4241(2
10
3.3621(2
10
2.9731(2
14
3.6201(2
9
4.4791(2
7
3.9781(2
4
4.2311 2
4
3,7411'2
7
4.5561(2
3
5.637/"2
2
2.3401(2
21
2,4001(2
22
2.1221(2
24
2.5841(2
17
3.1981(2
10
3.6361(2
8
3,409/'`2
9
3.0141(2
13
3.6701(2
8
4.34112
6
3.242E-2
12
2.9171.'' 2
15
2.5801.'2
19
3.141/"2
13
3.8871(2
9
3.4441(2
9
3.6941."2
8
3.3091x'`2
9
4.06412
6
3.14112
5
3.6631(2
7
:3.8471(2
7
3.4471`2
8
4.2331(2
5
5.3551'2
4
2.3801(2
20
2.1141`2
24
1.8941"2
25
2.3261(2
20
2.9431?2
11
1 , 5791(2
28
1.41 18- 2
30
1.2641(2
32
1 .5531." 2
28
1 .9651(2
19
3.8421(2
6
4.0861(2
3
4.1061.'-2
4
5,2801."2
1
7.2601(2
1
1 .903K_2
26
1 .4 281(2
29
1.4351"2
28
1.8461(2
25
2.5381 2
13
2.6661(2
16
2.3071(2
20
2,5191(2
21
3.24012
12
4.455K-2
8
3.3011'"2
11
3.0511"2
13
3,0661(2
12
3.9431(2
7
5.4211(2
3
5.6541'2
1
5.7101(2
1
5.7 75 r 2
1
3.2731` 2
II
2.5371(2
14
4.0101(2
3
4,2525-2
3
4.0411'2
5
2.9661"2
15
2.2421-2
16
3. 18.51'"2
14
2, 0101.`2
16
3,1001"2
11
2.1991`2
24
1.7281`2
24
2.3271(2
22
2.4161'2
21
2.2378-2
23
1.8291"2
26
1.7701'2
22
2.4201-2
19
3.0521?2
12
2.8231."'2
16
2.3101(2
21
2.2351(2
17
1.092/'2
30
9.9891(3
36
9.2481'"3
37
7.5611"3
39
7.3181,'"3
38
4.584 R-3
43
3, 3481`3
?13
:3.2191"3
43
2,6681(3
43
2.349K-.1
43
5.0431-3
42
4 , 1 181(3
42
3.9591?1
42
3.2821?3
42
2.8901`3
42
6,5651(3
41
7.5321'3
?10
7,24;31'3
40
6,0031(3
41
5.2871"3
41
For example, one notes from Figure 10 that Ele- is implied by the normalized products in Table 7.
ment 331 may not be in as strong a first-place The truth is probably somewhere between the nor-
position in ACE-2 as is Element 411 in ACE-0, as malized and non-normalized cases.
 18
ACE-0 ACE-1 ACE-2
Figure Q. Non-normalized Products of the Non-Normalized
Averages for the Tasks, as a function of Arms-Control
Environment (ACE).
611561 7,71 CIA
 411\
312
334
322
321
314
311
333
315
313
323
ACE-0 ACE?1 ACE-2
Figure 10. Non-normalized Products of the Non-normalized Nodal Averages
for a Selection of Specific Objectives. For Clarity, Only the Upper 17 and
Lowest 3 Specific Objectives are Shown; the Remaining 23 Would Fill the
Apparent Gap.
19
 APPENDIX A
NATURE OF THE EXERCISE
Some important background material was pro-
vided to the participants in the Delphi Exercise
in the instruction booklet prepared specifically for
this.^ 4aterial describing the Delphi technique
itself is reproduced in this Appendix, along with
the discussion provided as the rationale for the
selection of priority as a basis for evaluation, and
suggested criteria for evaluation. Appendix B con-
tains the complete set of definitions for the ele-
ments in the relevance tree, also given in the book-
let. The material which discussed the scaling
procedures for the numerical assignments is given
in part of Appendix C.
THE DELPHI TECHNIQUE 8-10 13
Delphi is the term given to the process developed
at Rand for eliciting and systematically refining
the opinions of expert individuals. Because it is
based on the intuition and speculation of individ-
uals, it appears to be especially useful in problems
of forecasting. The main features of the Delphi
technique are:
Anonymous response
Iteration and controlled feedback
Statistical group response.
Anonymous Response
In the Delphi technique, the opinions of the par-
ticipants are obtained by formal questionnaire.
Debate is replaced by a program of sequential in-
terrogations, At no time are the members of the
expert group in direct contact with one another.
All communication is through a central clearing-
house or a project director, The degree and nature
of the interactions in the Delphi panel can be con-
trolled, whereas a face-to-face group interaction
tends to introduce uncontrollable factors. Group in-
teractive procedures necessarily require that each
person explain, defend, or promote his position
with possible attendant grandstanding, prestige-
seeking, and a hardening of positions. Groups may
also exhibit a band-wagon effect with opinions
based on those of the ranking or dominant individ-
uals. The advantages derived from the Delphi
method are that it requires less time than a con-
ference and each respondent works essentially at
his own speed and at his convenience.
Iteration and Controlled Feedback
The interactions in a Delphi exercise are effected
by conducting one or more iterations with con-
trolled feedback between rounds. The feedback
may be in the form of (i) the median and quartiles
of the group responses on the preceding round, or
(ii) relevant facts, information, or opinion. In each
round, the participants are invited to revise their
previous opinions and may be asked to provide
reasons in support of them.
Statistical Group Response
The opinion of the group is defined or repre-
sented as an appropriate aggregate of the individ-
ual opinions obtained on the final round. Typically
the median of the individual responses is used.
Such a definition of group response admits a spread
of opinions without coercive conformity, i.e., it
reduces group pressure and insures that every mem-
ber is represented in the final response.
The above aspects of Delphi help to minimize
the biasing effects of dominant individuals, irrele-
vant communications, and group pressure toward
conformity. Adoption of the Delphi technique is
advantageous in view of indications that the pro-
cedure leads to an enhanced acceptance of the
group response over that obtained through more
conventional procedures.
21
 e r r
An additional feature" of the Delphi work at
Rand is the self-rating.? It appears that a meaning-
ful estimate of the accuracy of the group response
to a given question might be obtained by combining
the individual self-ratings of the competence of an
individual into a group rating.
Expertise and Accuracy
On the question of accuracy, a few remarks are
in order, based on the Rand experience with Delphi.
It has been observed that the accuracy of the
Delphi estimates is generally improved as the num-
ber of respondents is increased. The accuracy of
the estimates sometime decreases, however, as the
time allowed for a response increases, Finally, it
has been noted in some Delphi experiments that
"the most significant parameter in the experiment
is likely to be how much the individual members
know about the subject matter." K
THE PREMISE FOR
PRIORITY ASSESSMENT.
An illuminating overview of the intelligence prob-
lem that the methodology is intended to
address has been set down by the late Allen
Dulles." His statement forms a kind of manifesto
for the development of this methodology:
One general range of subjects that receives constant
attention and very frequent, regular estimates is the
development of what we call military hardware, par-
ticularly by the Soviet Union. This means Soviet programs
and progress in missiles, nuclear warheads, nuclear sub-
marines, advanced type aircraft and anything that might
approach a breakthrough in any of the sectors of this field,
as well as in the field of space. This is one of the most
difficult tasks which face the intelligence estimator.
Here one has to deal with Soviet capabilities to pro-
duce a given system, the role assigned to the system
by the military, and its true priority in the whole military
field.
It is always difficult to predict how much emphasis
is given to any particular system until the research and
development stage has been completed, the tests of ef-
fectiveness have been carried out, and the factories have
been given the order to proceed with actual production.
As long as a Soviet system is still in its early stages, our
estimates will stress capabilities and probable intentions;
as hard facts become available, it is possible to give all
actual estimate of programming the system.
The phrase "priority in the whole military field"
is considered relevant to this exercise, forming a
succinct statement of the role of the relevance tree
portion of the methodology. It is intended
that an evaluation of oviet priority will include
not only an evaluation of needs and incentives but
also a measure of the likelihood that action would
be taken to carry out a program or a development.
CRITERIA OR BASES FOR
EVALUATION
In the holistic approach employed in this exercise,
the balloter is not given explicit criteria or bases
for an evaluative comparison of the alternatives or
elements at a node. The balloter is expected to
assign weights to the elements taking into account
simultaneously the various possible considerations
that might obtain. The procedure does not require
an independent evaluation of the alternatives on a
criterion-by-criterion basis. The participants in the
exercise were reminded, however, of some possible
criteria that might be reviewed before assigning
weights to the alternatives.
One possible general set of criteria that might be
used in the tree for Strategic (General) Warfare is:
Military considerations
Diplomatic considerations
Economic constraints
Psycho-cultural considerations
Here, one simultaneously considers the priorities
of the alternatives at a node from the military,
diplomatic, economic, and psycho-cultural points
of view and assigns an overall priority number to
each alternative. It may be assumed that various
irrational factors which might be involved in So-
viet decision-making, as well as factors dictated
by history or tradition, are considered within the
psycho-cultural criterion.
Alternatively, a collection of criteria might well
be cast in the form suggested in a paper on trade-off
studies."" Criteria suggested for the trade-off of
forces implementing national policies are:
Versatility
Deterrent capability
Expandability
National acceptability
International acceptability
These criteria might be appropriate at about the
Mission level of the relevance tree.
22 CI I2ET
 Yet another suggested set of criteria that may that could be taken into account in the assignment
be involved in weapons procurement decisions is: of priority numbers. It was suggested also that
Cost-effectiveness and economic impact some attempt be made to factor SALT into the
Strategic uncertainties considerations at some of the node, possibly in the
Technological obsolescence form of assumed agreements.*
. Objectives and policy desiderata
The suggested criteria or potential bases for evalu-
ation were not required to be applied strictly at
every node of interest. They were cited only to
remind the balloter of the myriad of considerations
This casual approach to the uncertainties concerning
SALT and future arms control environments was felt to
be quite unsatisfactory, however, and a modification was
made on the third round of the Delphi exercise, as described
elsewhere (Appendix C).
23
 APPENDIX B
THE RELEVANCE TREE FOR STRATEGIC (GENERAL) WARFARE
The relevance tree is intended to assist in the
identification and evaluation of areas of Soviet
priority to improve significantly or upgrade an
existing capability. Thus, the emphasis is on isolat-
ing and defining significantly improved systems or
novel weapons concepts that might obtain in a
future Soviet strategic force posture.
The relevance tree was constructed with a num-
ber of subordinate desiderata in mind. First, it was
desired that the number of nodes and alternatives
at a node be kept reasonably low so as to make
implementation tractable and reasonably econom-
ical. Second, because of theoretical reasons, it was
desired that approximately the same low number
of alternatives or branch elements occur at each
node on a given level.""" Third, the members of
the Specific Objectives level should all be reason-
ably compatible with one another in terms of their
complexity and content. Fourth, it was hoped that
reasonably neutral descriptions and definitions could
be employed. Finally, it was desired that as many
nodes as possible be constructed such that their
evaluations might have some significance and
interpretability independent of the overall process.
The relevance tree for Strategic (General) War-
fare is shown schematically in Figure 2, and the
definitions used for its elements are given below.
The definition of this element in the Spectrum of
Conflict is intended to encompass the strategies
referred to in the United States as `assured destruc-
tion' and `damage limitation'. This element em-
braces weapons of mass destruction which are gen-
erally intercontinental in terms of capability. The
weapons and countermeasures may have deterrence
as their primary purpose and thus may not ever
be used.
,24 &CEGRET
 -&EeRET_
DEFINITIONS OF RELEVANCE TREE ELEMENTS
Strategic (General) Warfare - Node 0
Armed conflict between major powers in which the total resources of the
belligerents are employed and the national survival of one or more of the
belligerents is in jeopardy.'
ELEMENT #
The destruction, incapacitation, or neutralization 1
of enemy strategic offensive and defensive forces
and resources, including strategic missiles, aircraft,
naval forces, orbital systems, defensive measures,
warning and intelligence functions, transportation,
communication, military and industrial value tar-
gets, and population centers.
Logistics/ The deployment of strategic forces and materiel to 2
Support remote or distant areas for purposes of support for
offensive or defensive missions, to perform stra-
tegic interdiction of continental areas, to permit
"third world" operations, and to obtain such psy-
chological and strategic benefits associated with
"conspicuous deployment."
Intelligence/ The acquisition, processing, analysis, and dissem- 3
Warning/ ination of information (operational or national,
Control t .ctical or strategic, basic or topical) concerning
the resources, forces, and actions of extant and
potential enemies prior to and during general or
strategic conflict; warning of attack; and the di-
rection and control of the movement and applica-
tion of strategic forces and resources.
Defense The protection, by active or passive means, of 4
selected point and/or area targets (civilian or
military) by destruction, incapacitation, degrada-
tion, or neutralization of attacking weapons, ex-
clusive of the defensive or protective contribu-
tion of the warning, intelligence/reconnaissance,
and command/control/communications systems.
' Joint Chiefs of Staff, Dictionary of United States Military Terms for Joint Usage, Washing-
ton, D.C., JCS Pub. 1, 1 Jan 66, U.
 Node 1
The destruction, incapacitation, or neutralization 11
of enemy strategic offensive forces (missiles, stra-
tegic bomber aircraft, naval forces, and orbital
weapons).
The destruction of enemy resources and value 12
targets, including cities and population centers,
military and industrial value targets, civilian trans-
portation systems, and civilian communication
systems.
The destruction, incapacitation, or neutralization 13
of enemy strategic defenses, command, control,
communications, early warning, and intelligence
systems.2
SPECIFIC OBJECTIVES DEFINITIONS ELEMENT #
The destruction, incapacitation, or neutralization 111
of land-based strategic missiles. The missiles may
be fixed or mobile and may be based in the polar
ice caps and possibly the shallow territorial waters.
The destruction, incapacitation, or neutralization 112
of enemy strategic weapons located or based be-
low the surface of international waters and/or
beneath the polar ice caps.2
The destruction, incapacitation, or neutralization 113
of enemy surface naval strategic forces (aircraft
carriers, missile launching vessels, and strategic
transport vessels)."
The destruction, incapacitation, or neutralization 114
of enemy strategic aerodynamic threats, such as
manned strategic bombers and cruise missiles.
The destruction, neutralization, or incapacitation 115
of enemy strategic threats based on, or consisting
of, orbital space platforms or natural bodies in
space.
' Systems supporting this task will contribute to a capability to destroy or significantly
degrade the enemy's ABM or interceptor aircraft, the early warning and control systems for
these defensive measures, as well as the command, control and force management capability
for the enemy's offensive weaponry. Systems or programs specifically directed towards key
personnel in the chain of command are included under this task also.
'Note that these Specific Objectives do not include the detection, identification and tracking
aspects of the ASW or surface naval problem. Only the "kill" capability is considered here.
 Node 12
SPECIFIC OBJECTIVES DEFINITIONS ELEMENT #
The destruction of enemy cities and population 121
centers.
The destruction or disruption, either temporarily 122
or for an extended period of time, of enemy trans-
portation systems which serve primarily civilian
purposes and needs.
The destruction or disruption, either temporarily 123
or for an extended period, of enemy communica-
tion systems which serve the civilian sector.
The destruction of selected military and industrial 124
value targets.
The destruction, incapacitation, or neutralization 131
of enemy regional or national defensive forces
intended to counter a strategic attack.4
The destruction, incapacitation, or neutralization 132
of enemy command, control, and communication
systems specifically intended for strategic military
tasks.
The destruction, incapacitation, or neutralization 133
of enemy strategic warning and national intelli-
gence systems.'
The destruction or incapacitation of selected spe- 134
cial targets or personnel critically involved in
enemy strategic planning, intelligence, and/or
command and control.
Systems responsive to this Specific Objective are those whose primary mission is the attack
on the defensive systems. Those systems or subsystems which are a part of an offensive weapon
system and play a secondary or supportive role to that system are not considered here.
Systems responsive to this Specific Objective are those whose primary mission is the destruc.
tion of such targets. Subsystems of offensive systems which serve to support that system and
to neutralize or degrade enemy warning capability are not included hero.
 TASKS DEFINrrsONs ELEMENT #
The transportation of forces, weapons, or materiel 21
to any point or area on the earth in order to
provide support for offensive and defensive
measures on a worldwide basis.?
The establishment of bases for weapons or per- 22
sonnel in remote locations. (These bases are
nominally stationary or relatively slowly ' moving
or are not characterized by their mobility per se.
These bases are characterized by the relative
harshness of the environment and have strategic
value because of it.)7
Node 21
SPECIFIC OBJECTIVES DEFINITIONS ELEMENT #
The transportation of forces, weapons, or ma- 211
teriel by aircraft or lighter-than-air vehicles in
support of strategic offensive and defensive mis-
sions."
The transportation of forces, weapons, or materiel 212
in or on bodies of water, providing sealift support
capabilities for offensive and defensive strategic
missions.0
The transportation of forces, weapons, or materiel 213
over land in support of strategic offensive and
defensive missions.10
The transportation of forces, weapons, or materiel 214
via space vehicles or platforms.'i
? Systems supporting this task will provide an airlift and sealift capability for long-range air
and naval operations, the patrol of large areas, and the supply of remote bases. Mobile land
systems of a strategic nature are included under this task.
'Systems or basing facilities supporting this task are manned and unmanned space/orbital
platforms, surface and subsurface ocean platforms, and terrestrial bases and installations in
conditions of extreme environment such as the arctic or rugged high-altitude regions,
"Systems responsive to this Specific Objective include large transport aircraft, tankers, and
patrol and reconnaissance aircraft. Large aircraft which may serve as command and control
.centers and as support platforms for strategic weapons are also included.
' This Specific Objective covers submarines, surface naval vessels, and "interface" vehicles
such as GEMs and SESs. Such systems are intended to provide support for long-range naval
operations, patrol and reconnaissance, and resupply and logistic support.
'*This Specific Objective represents logistic support, mobile reconnaissance and patrol, and
mobile launch platforms for strategic weapons. Of particular interest are those systems capable
of maneuvering in hostile environments and those which support long-range penetration into
regions contiguous to the Soviet Union.
"Includes the development of low-cost reusable boosters and spacecraft and advanced shuttle
vehicles.
28 CCC
 Node 22
SPECIFIC OIIJECTIvEs DEFINITIONS ELEMENT #
The deployment and basing of weapons, personnel, 221
or materiel in space or in earth orbit or on natural
bodies in space.12
The deployment and basing of weapons, personnel, 222
or materiel beneath the surface of bodies of
water.ls
The deployment and basing of weapons, personnel, 223
or materiel on the surface of bodies of water.14
The deployment or basing of weapons, personnel, 224
or materiel on land, particularly in an extreme or
hostile environment.15
sary for the planning, conduct, and direction of
military operations.17
The collection, analysis, evaluation, and dissemi-
nation of information, not specifically designated
for military missions, for the purpose of estimating
and assessing extant and potential enemy capabili-
ties and intentions. Such information is intended
to provide support to policymakers and provide
guidance for long-range national planning.ls
The planning, direction, and control of the move- 31
ment and application of strategic forces and re-
sources in a conflict environment.16
The acquisition and analysis of information neces- 32
"Systems responsive to this Specific Objective would include reconnaissance platforms, deep
space command and/or bombardment platforms, etc.
"Systems responsive to this Specific Objective will include manned and unmanned submarine
platforms which are relatively fixed in location.
"Systems responsive to this Specific Objective are, for example, aircraft and missile carriers
or platforms which are slowly moving or fixed in location.
" Such bases would be located in arctic or polar regions, mountainous or high-altitude areas,
or deserts.
" Systems responsive to this task assist in infonnation processing, analysis, and dissemination;
decision-making; and force management and force allocation.
"Systems supporting this task include those for ASW and naval reconnaissance and surveil-
lance; satellite detection and tracking; acquisition of general reference materials and data for
planning pertaining to the capabilities, resources, and potential areas of operations.
"Systems or programs embraced by this task include those pertaining to the analysis and
estimation of foreign capabilities and intentions, future threats, verification of arms limitations,
early warning systems, and post-attack assessment.
 SPECwxc Onjscrxvzs DEFINITIONS EL mNT *
The planning, direction, and control of the move- 311
ment and application of Soviet strategic aero-
space offensive forces (Strategic Rocket Troops
and Long Range Aviation).10
The planning, direction, and control of the move- 312
ment and application of Soviet naval forces.
The planning, direction, and control of Soviet 313
space missions.
The planning, direction, and control of the Soviet 314
aerospace defensive forces.
The transmission of information by a combina- 315
tion of systems incorporated into an integrated and
standardized point-to-point network serving the
Soviet state, military, industrial, civil, and party
telecommunication needs.
The detection, identification, and tracking of 321
satellites and orbital platforms, preferably before
their first pass over the Soviet Union. (May in-
clude, for identification purposes, a close inspec-
tion and/or boarding or docking by Soviet ve-
hicles.)
The detection, identification, and tracking of naval 322
ships and vessels, including submarines and sub-
surface platforms, as well as military and civilian
surface traffic.
The acquisition of global geographic and geodetic 323
surveys and mappings, surveys of terrestrial mag-
netism, and oceanographic data pertinent to mili-
tary missions.
The acquisition, analysis, and utilization of meteor- 324
ological data, including global electromagnetic
activity (thunderstorms, solar flares, etc.), for
synoptic and tactical weather prediction.
"A launch-on-warning capability is included in this Specific Objective.
E1
30 SEGER
 Node 33
SPECIFIC OBJECTIVES DEFINITIONS ELEMENT #
The detection and early warning of strategic attack 331
on the Soviet Union.20
The determination of the nature and extent of 332
damage inflicted on the enemy and the damage
sustained by the Soviet Union in a strategic ex-
change and the generation of information pertinent
to subsequent actions, including recovery and the
termination of hostilities.
The detection and evaluation of long-term threats 333
and the estimation of future enemy intentions and
capabilities.
The collection and evaluation of information, not 334
designated specifically for military missions, on the
current and projected (near-term) strategic force
deployment, characteristics, and capabilities of
extant and potential enemies.21
attack from an enemy strategic aerospace strike.28
The protection of the population and the entire
national wealth from the effects of "weapons of
mass destruction" (nuclear, chemical, and bac-
teriological weapons ).24
ELEMENT #
Direct defensive action taken to destroy or re- 41
duce the effectiveness of attacks by enemy aero-
space vehicles or threats, including strategic bal-
listic missiles, aircraft and cruise missiles, and
orbital weapons.22
Defensive actions, other than active measures and 42
warning systems, taken to reduce the probability
of, and minimize the effects of damage by, an
"Systems responsive to this Specific Objective may include those for the detection of launch
of enemy offensive strategic weapons, a long-range high-capacity system for target acquisition
and tracking, and systems for the detection of weapon detonation.
" This Specific Objective includes the means for verification of a possible limitation on
strategic arms,
This task is concerned with the active point and area defense of civilian and military targets
by missiles, interceptor aircraft, artillery, electronic countermeasures, etc.
"This task is essentially the passive defense of military targets by deceptive techniques,
dispersion, mobility, and protective constriction (hardening).
" This task is essentially the passive defense of the civilian sector by evacuation, dispersal,
protective shelters, and those measures intended to provide rescue and emergency work and to
perpetuate the national economy in a wartime environment.
31
 32
Node 41
SPECIFIC OBJECTIVES DEFINITIONS ELEMENT .#
The active protection of selected point and/or 411
area targets, civilian or military, from an attack
by enemy strategic ballistic missiles (ICBMs and
SLBMs).
The active protection of selected point and/or area 412
targets, civilian or military, from an attack by
enemy strategic aircraft and cruise missiles.
The active protection of the Soviet Union from 413
an attack by orbital and space-based threats.
SPECIFIC OBJECTIVES DEFINITIONS ELENmNr #
The protection of selected military installations 421
and strategic forces by protective construction or
basing techniques (hardening).
The protection of selected military facilities, con- 422
trol centers, or weapons (offensive or defensive)
by mobility or the use of mobile basing concepts.
The protection of selected military installations 423
and facilities by deceptive techniques (cover,
camouflage, concealment, etc.).
SPECIFIC OnJECTIVEs DF.FINITUV. S ELEMENT #
The dispersal of workers and employees from 431
installations of the national economy and the
evacuation of the population from large cities and
the more important industrial centers.-''s
The sheltering of people collectively in protective 432
structures and the provision of individual means
of protection against blast, thermal radiation, and
radioactivity.
Measures directed toward increasing the ability 433
of important sectors of the economy, all types of
transportation, communications and utilities, and
industrial power supplies to survive an attack
and function in wartime.
" The evacuation and dispersal are "strategic," i.e., accomplished in advance of an attack and
dependent upon adequate warning of that attack, (Tactical evacuation is the relatively hasty
withdrawal immediately prior to or after the launch of an attack.) Dispersal is the organized
removal of the population from enterprises of industry, transport, and communication-which do
not cease functioning in wartime---from large cities to predetermined areas outside the cities.
Evacuation is the transportation of the nonworking population, primarily children, old people,
disabled and sick persons, from large cities to predetermined areas.
-SECRET-
 APPENDIX C
The Delphi elicitation of nodal priority numbers
covered the period Nov 70 - Apr 71. The exercise
began with a series of individual briefings to pro-
spective participants. At that time, an instruction
booklet 12 and some associated descriptive ma-
terials describing the background for the exercise
were provided to the participant. One nodal ques-
tionnaire and an Administrative Questionnaire were
also left with the participant. The Administrative
Questionnaire contained a self-rating section in
which the participant indicated on a simple 1-5
scale his opinion as to his relative expertness on the
16 nodes in the relevance tree. An opportunity was
presented for the respondent to indicate a simple
preference for a few nodes to be involved with and
the respondent was asked to indicate others that
he felt might contribute to the exercise. Finally,
the Administrative Questionnaire asked the re-
spondent for permission to be identified as a par-
ticipant in the exercise, with the understanding
that specific responses would not be attributed
by name and that opinions expressed are personal
and professional and need not reflect official Agency
views. All participants were assigned a code number
for the exercise.
Almost all participants in the exercise were
assigned five nodes in the relevance tree to con-
sider in the exercise. One of these nodes was
assigned initially by the Project Director (Nodes
0-4), while the remaining four were allocated on
the basis of the Administrative Questionnaires re-
turned. Every attempt was made to insure that par-
ticipants were contributing in areas in which they
felt competent, but at the same time, maintaining
a `mix' of backgrounds and Agency components
for all of the nodes.
Slightly over 100 people were approached for
participation. Of these, 99 indicated a willingness
to contribute. Agency personnel from all Direc-
torates were involved. There was, as expected, some
loss of participation over the course of the exercise,
amounting to about 24% at its termination. The
initial and final distributions of participants is
given in the table below.
Of the 75 participants who contributed through-
out the entire exercise, about 39% were analysts;
36% were supervisory personnel from Branch
Chiefs through Division Chiefs; 13% were from
staffs such as ONE and NIPE; 12% were Office
Directors and Deputy Directors.
A few comments might be made concerning the
self-rating and the resultant biasing of the sample
for expertness. On the scale 1-5, where 5 represents
a high assessment of expertness, the overall average
group self-rating for all nodes by all participants
was about 2.9. In contrast, the overall average self-
rating for the participants on the nodes they were
involved with was about 3.9, with a high of 4.5 on
Node 41 to a low of 3.1 on Node 2. On Node 0,
which was assigned without the benefit of the.
Administrative Questionnaire, the average group
self-rata,+g was 3.7.
Assignment of Priority Numbers-Scaling
Procedure
The participants in the exercise were called upon
to supply two types of information or expressions
 of opinion: numerical assignments of priority and
comments or statements supporting these assign-
ments. The Delphi Questionnaires consisted of two
basic parts. Part A was concerned with the nu-
merical assessment of priority. The Specific Ob-
jectives level in the relevance tree is to have a set
of weights (relevance numbers) associated with
it and these weights should in some sense reflect
the degrees to which upgrading the individual Spe-
cific Objectives would enhance higher-level Soviet
goals. To this end, priority was selected as the key
concept in the evaluation of the elements in the
relevance tree. Time was not handled explicitly in
the assessments, i.e., there was no attempt to break
the basic 5-15 year period into segments.
A typical Part A questionnaire was of the form
indicated below:
To meet national goals and objectives over the next 5 to
15 year period, the Soviet Union may seek to improve
significantly or upgrade their capabilities in (title of
node). What is the relative priority that you feel the
Soviets would place on obtaining significant improve-
ment or upgrading in each of the elements supporting
this objective? (Assign a priority number to each of
the elements below, using a scale 0-100, where 0=no
priority, 100=top priority.)
Element # ...... (Title) ..............
Element # ...... (Title) ............. .
Element # ...... (Title) ..............
A rather unusual 0-100 scale was used for the
numerical assignments of priority in this exercise.
A value of 100 was intended to denote the feeling
that a top priority is given by the Soviets for ob-
taining a truly significant improvement in capability.
A value of 0 was intended as a reflection of no
priority, however, it was not to be interpreted as
meaning that no effort would be expended in that
area. An clement rated 0 may well have improve-
ments, but the 0 indicates that the Soviets do not
attach a real priority to their realization. Inter-
mediate levels of priority are reflected by numbers
between 0 and 100. The scale and its associated
definitions are intended to reflect emphasis on
improvement and not simply a resource allocation.
A respondent could reflect relative priority between
alternatives at a node as well as an overall priority,
since he was not required to assign a 100. Alterna-
tive approaches which were rejected were the
normalized, or 'split-100', method in which the
sum of the assignments at a node is fixed, and the
method in which a fixed amount is assigned to the
highest alternative at the node (the so-called '100-
top' method). The normalized approach was felt
to be too restrictive and might tend to lead to
conservative assignments. More information is con-
veyed by the selected procedure than by either
the 'split-100' or '100-top' methods. The procedure
selected is interestingly similar to the '100-top'
method, for which recent work by Dalkey tends
to support as preferable over the 'split-100' in Delphi
examinations of relative value."
Non-Numerical Responses and Feedback Materials
Part B of the Delphi Questionnaires was involved
with the elicitation of comments concerning the
priorities, i.e., the respondent was asked to indicate
what he had in mind when he made the assign-
ments. These brief supporting statements provide
an outlet for qualification of the numerical assign-
ments and enabled respondents to indicate trends
that are not apparent from the simple numeric
values.
The most important application of the com-
ments made in Part B of the questionnaires
was in the feedback material provided to the
respondents after each round. In addition to some
statistical information which described the pre-
vious round's results (median and inter-quartile
range), the feedback material contained a com-
pilation of the comments made by the respondents.
This material amounted to some 2-9 pages of ver-
batim text or quotations arranged in appropriate
groups and identified for communication purposes
by the participant's ID number. The comments
were edited slightly and sometimes paraphrased,
but were, for the most part, complete comments
provided by the respondents. There was no attempt
made to synthesize groups of comments or to ab-
stract individual or collected comments. The feed-
back material was classified and restricted to the
Secret level,
An Investigation of Sensitivity to Arms-Control
Two complete rounds of a Delphi process were
carried out on all sixteen nodes in the relevance
tree. The results obtained from Round 2 (described
in detail elsewhere) indicated that it would
not be particularly advantageous to pursue addi-
tional rounds. A review of the results and com-
341:
 ments indicated, however, that it would be quite
desirable to carry out a third round with some
modification in the basic questionnaire on a few
selected nodes to determine their sensitivity to
arms-control. It was quite evident that some of the
dispersion in the assignments was due to uncer-
tainties about future arms-control environments.
In order to assess the extent of the dispersion due
to the mixture of assumptions concerning SALT
and future arms-control environments and to esti-
mate the sensitivity of the results to these con-
tingencies, five nodes (0, 1, 3, 4, 41) were selected
for further investigation by asking for priority
numbers for the elements at these nodes under
three given hypothetical arms-control environments
or scenarios. The basis for selection was partly
the frequency with which SALT or arms-control
assumptions were cited in Part B of the Delphi
Questionnaires for the first and second rounds.
The three hypothetical Arms-Control Environ-
ments (ACE's) considered are described below.
These definitions were derived from open literature
sources 10'21 and are purely hypothetical, with an
eye for extremes. They were not intended to be
entirely realistic or even achievable and it was
suggested to the participants that they attempt to
assign priority numbers assuming the existence of
a given ACE and to avoid arguing the viability or
likelihood of that ACE.
For the partial Round 3, the numerical por-
tion (Part A) of the Delphi Questionnaires was of
the following revised form:
To meet national goals and objectives over the next
5 to 15 year period, the Soviet Union may seek to
improve significantly or upgrade their capabilities in
(title of node). For each of the hypothetical Arms-
Control Environments, what is the relative priority that
you feel the Soviets would place on obtaining signi-
ficant improvement or upgrading in each of the ele-
ments supporting this objective. (Assign a priority
number to each on a 0-100 scale, where O= no priority,
100=top priority,)
Element # .... (Title) ......
Element # .... (Title) ......
Hypothetical Arms-Control Environments
ACE 0 Negotiations between the US and the SU are
terminated within the next year with little likeli-
hood of a resumption of SALT in the near
future, Assume, however, that treaties and agree-
ments already in effect (or nearly so) remain in
force (i.e., the 1963 Nuclear Test Ban; the
banning of bombs in orbit; the agreement to
maintain Latin America as a nuclear weapons-
free zone; the 1968 Nuclear Non-Proliferation
Treaty; and the treaty banning weapons on the
seabed).
ACE 1 A limited arms-control agreement is arrived at
within the next year. Assume an agreement
which restricts deployment of ABM's to the
National Command Authority (NCA) of the SU
and the US (i.e., Moscow and Washington).
Assume that talks continue for the purpose of
negotiating additional agreements.
ACE 2 A relatively full arms-control agreement is ar-
rived at within the next year or so. Assume an
agreement in which: ABM deployments are
restricted to the NCA; offensive strategic weap-
ons are frozen at then-current levels and types
(missiles and aircraft) with a ban on the con-
struction of new launchers and a restriction on
any changes in deployed systems to those that
do not change their external characteristics;
MIRV warhead flight testing, production, and
deployment is banned and the US removes such
warheads already deployed on their Minuteman
and Poseidon missiles. Assume that talks continue
toward possible additional agreements.
The revised questionnaires contained an array,
rather than a single column as in the first two
rounds, to be filled with priority numbers. As before,
normalization was not required. The respondents
were reminded of the importance of obtaining ap-
propriate relative assignments within a column but
that consistency across a row was desirable and
secondary in importance.
 APPENDIX D
TABULATION OF NODAL RESULTS
The following table contains a compilation, by node, of the priority numbers
elicited in the exercise. The table presents the results for the two complete rounds
for all nodes and the results of the third round for the five nodes examined
under the hypothetical Arms-Control Environments.
The number in parentheses at the left is the number of respondents for the
given node-round. Reading across from the left, the columns contain the following
quantities: the element ID number; the first or lower quartile (Q,); the median
(M) ; the third or upper quartile (Q j); the average (mean) of the responses (A);
the normalized average (NA) ; and the average of the set of normalized responses
(ANR).
93
M
Q.
A
NA
ANR
Rnd 1
(49)
1
50
75
90
69,3
.270
.270
2
25
50
65
47.8
.186
.180
3
50
65
80
66.2
.257
.260
4
50
80
95
73.8
.287
.290
Rnd 2
(44)
1
50
75
80
67.3
.266
.264
2
25
50
65
46.9
.186
,182
3
50
70
80
65.3
.258
.265
4
60
80
90
73.2
.290
.289
Rnd 3
(30)
1
60
85
100
79.2
.285
.288
ACE-0
2
30
50
65
49.2
.177
.173
3
60
70
80
69.8
.251
.249
4
70
80
90
80.0
.287
.290
Rnd 3
(30)
1
50
70
100
72.8
.282
.289
ACE-1
2
30
50
(30
48.2
.187
.182
3
65
75
90
75.8
.293
.294
4
50
60
75
61.5
.238
.235
Rnd 3
(30)
1
30
35
70
48.5
.216
.217
ACE-2
2
25
40
50
45.8
.204
.198
3
60
90
100
78.8
.351
.359
4
40
50
70
51.3
.229
.226
NODE 1
Rnd 1
(33)
11
55
80
90
71.9
.422
.440
12
40
50
75
45.2
.266
.280
13
30
50
80
53.1
.312
.280
Rnd 2
(30)
11
60
75
90
70.7
,409
.427
12
25
50
60
43.6
.252
.247
13
35
60
75
58.5
.339
,326
Rnd 3
(19)
11
75
85
100
84.5
.435
.445
ACE-0
12
40
50
70
53.4
.275
.265
13
50
50
70
56.3
.290
.290
Rnd 3
(19)
11
60
75
90
73.4
.441
.462
ACE-1
12
30
40
50
42.1
.253
.238
13
30
50
70
50.8
.306
.300
End 3
(19)
11
40
50
50
48.4
.387
.422
ACE-2
12
20
30
50
34.5
.276
.259
13
25
40
50
42.1
.337
.319
36
 ECRET
Ql
M
Q:
A
NA
ANR
Rnd 1
(21)
21
40
50
60
51.2
.536
.584
22
20
45
80
44.3
.464
.416
Rnd 2
(16)
21
35
50
55
49.4
.527
.570
22
20
45
70
44.4
.473
.430
Rnd 1
(28)
31
50
80
90
70.2
.337
.330
32
50
70
80
66.4
.319
.320
33
50
75
90
71.8
.344
.350
Rnd 2
(23)
31
60
75
75
72.4
.345
.353
32
50
70
75
69.3
.330
.316
33
50
70
80
68.0
.325
.331
Rnd 3
(17)
311...
60
80
80
69.7
.329
.326
ACE-0
32
60
70
80
67.6
.319
.320
33
65
70
90
74.4
.352
.354
Rnd 3
(17)
31
60
70
80
69.1
.322
.318
ACE-1
32
60
70
80
67.6
.315
.316
33
70
80
90
77.9
.363
.366
Rnd 3
(17)
31
45
70
85
65.3
.307
.300
ACE-2
32
45
75
80
65.3
.307
.312
33
80
90
95
81.8
.386
.388
Rnd 1
(29)
41
80
100
100
85.9
.560
.580
42
30
50
65
44.7
.291
.280
43
10
20
30
22.9
.149
.140
Rnd 2
(^7)
41
80
100
100
85.9
.550
.562
42
40
50
60
49.3
.316
.310
43
10
20
25
20.9
.134
.128
Rnd 3
(14)
41
85
90
100
85.7
.551
.570
ACE-0
42
25
50
65
48.6
.312
.292
13
10
20
25
21.4
.137
.138
Rnd 3
(14)
41
50
75
90
71.1
.518
.542
ACE-1
42
25
45
60
45.7
.333
.305
43
10
20
25
20.4
.149
.153
Rnd 3
(14)
41
40
50
75
58.9
.499
.526
ACE-2
42
20
45
60
42.1
.356
.327
43
5
15
25
17.1
.145
.147
Rnd 1
(25)
111
55
80
90
71.6
.235
.240
112
70
85
100
81.0
.266
.270
113
30
50
80
51.8
.170
.170
114
40
70
80
59.6
.195
.190
115
20
45
60
40,9
.134
.130
Rnd 2
(23)
111
80
90
90
77.8
.247
.251
112
75
90
95
83.0
.264
.268
113
40
50
70
52.4
.167
.164
114
50
70
75
61,1
.194
.193
115
20
40
60
40.4
.128
.124
 Rnd 1 (23)
121
03
50
470
38.7
.232
.230
122
10
20
50
26.1
.157
.140
123
10
25
50
34.6
.208
.220
124
50
75
90
67.2
.403
.410
Rnd 2 (21)
121
10
35
50
36.1
.223
.216
122
10
15
35
23.7
.146
.124
123
10
25
50
33,2
.204
.204
124
50
75
85
69.3
.427
.456
Rnd 1 (19)
131
30
60
100
64.5
.298
.300
132
40
70
90
64.5
.298
.320
133
25
70
80
54.7
.253
.240
134
10
20
50
32.9
.152
.140
Rnd 2 (18)
131
40
75
100
68.9
.303
.305
132
50
65
80
63.6
.280
.297
133
25
65
80
59.2
.260
.252
134
10
40
50
35.8
.157
.146
Rnd 1 (14)
211
40
55
70
53.6
.276
.270
212
50
60
75
62.5
.322
.340
213
25
55
60
5
1.1
.263
.260
214
5
15
50
27.1
.139
.130
Rnd 2 (13)
211
50
60
70
56.5
.282
.282
212
50
60
80
60.5
.333
.338
213
35
60
65
53.1
.266
.264
214
10
10
40
23.8
.119
.116
Rnd 1 (13)
221
10
25
75
44.6
.236
.220
222
45
50
70
51.5
.273
.290
223
25
30
75
41.9
.222
.220
224
40
60
75
50.8
.269
.270
Rnd 2 (12)
221
10
45
75
43.8
.234
.209
222
40
55
70
55.8
.300
.308
223
20
30
50
37.9
.203
.214
224
35
50
70
48.8
.263
.269
Rnd 1 (14)
311
40
80
85
67.9
.206
.200
312
70
90
100
78.9
.239
.240
313
30
35
70
46.4
.141
.140
314
70
75
90
72.1
.219
.220
315
50
70
80
64.3
.195
.200
End 2 (11)
311
50
80
85
65.0
.206
.205
312
70
90
95
81.8
.259
.263
313
30
40
65
46.4
.147
.150
314
60
80
80
65.9
.209
.207
315
45
60
75
56.4
.179
.175
 Rnd 1 (16)
321
Ql
60
M
80
Q,
85
A
72.2
NA
.311
ANR
.320
322
75
75
100
76.8
.331
.330
323
25
55
65
50.0
.215
.210
324
20
30
40
33.1
.143
.140
Rnd 2 (13)
321
70
80
80
74.6
.333
.339
322
70
80
90
77.7
.348
.345
323
30
50
60
42.7
.191
.189
324
20
30
30
28.5
.128
.127
Rnd 1 (21)
331
50
90
100
74.5
.328
.330
332
10
30
55
36.9
.162
.160
333
20
50
80
51.7
.228
.230
334
50
75
80
64.0
.282
.280
Rnd 2 (16)
331
75
90
100
84.1
.369
.373
332
15
25
45
29.4
.129
.129
333
20
45
80
51.6
.226
.226
334
50
80
80
62.8
.276
.272
Rnd 1 (26)
411
60
90
100
82.2
.440
.470
412
30
70
80
58.3
.312
.310
413
20
40
70
46.3
.248
.220
Rnd 2 (22)
411
70
90
95
83.0
.444
.461
412
40
70
80
61.8
.330
.323
413
20
40
65
42.3
.226
.216
Rnd 3 (13)
411
80
90
100
89.6
.447
.454
ACE-0
412
50
75
80
62.7
.313
.315
413
25
45
70
48.1
.240
.231
Rnd 3 (13)
411
55
75
90
70.4
.388
.388
ACE-1
412
50
65
80
63.8
.352
.355
413
30
45
70
47.3
.260
.257
Rnd 3 (13)
411
45
60
85
62.7
.390
.390
ACE-2
412
35
55
80
55.4
.345
.340
413
20
50
60
42.7
.265
.270
Rnd 1 (21)
421
50
60
80
65.0
.398
.400
422
35
80
90
67.6
.415
.420
423
10
30
45
30.5
.187
.180
Rnd 2 (17)
421
50
60
80
61.2
.374
.366
422
70
80
90
77.3
.472
.481
423
15
30
30
25.3
.154
.153
NODE 43
Rnd 1 (18)
431
10
20
40
25.0
.283
.300
432
15
20
30
27.5
.311
.290
433
10
30
60
35.8
.406
.410
Rnd 2
(14)
431
10
20
25
20.0
.223
.256
432
10
20
30
24.6
.274
.279
433
10
35
80
45.0
.503
.465
SGT 39
 REFERENCES
3. Beller, William S. "Technique Ranks Space Objectives,"
Missiles and Rockets, 7 Feb 66, p 22-24, U
4. Jantsch, E. Technological Forecasting in Perspective,
Paris, OECD, 1967, p 219-233, U
5. Fisher, M. "Toward a Mathematical Theory of Rele-
vance Trees," Technological Forecasting, v 1, 1970, p 381-
389, U
6. Beckerhoff, D. "Goal Systems for Governmental R&D
Planning," Technological Forecasting, v 1, 1970, p 363-369,
U
7. Nozicka, G., et al. "Technological Forecasting and
Simulation for Program Selection (TEFORS)," Volume II,
User's Guide, Control Data Corporation, AFATL-TR-70-20,
Feb 70, p 40-44, U
8. Dalkey, N. C. The Delphi Method: An Experimental
Study of Group Opinion, The Rand Corp., RM-5888-PR,
Jun 69, U
9. Dalkey, N. C., et al. The Delphi Method III. Use of
Self-Ratings to Improve Group Estimates, The Rand Corp.,
RM-6115-PR, Nov 69, U
10. Dalkey, N. C., and Rourke, D. L. Experimental Assess-
ment of Delphi Procedures with Group Value Judgments,
The Rand Corp., R-612-ARPA, Feb 71, U
13. The Rand Corp. A Bibliography of Selected Rand
Publications: Delphi, SB-1019, Jan 71, U
14. Dulles, Allen. The Craft of Intelligence, New York,
The New American Library, 1965, p 150-151, U
15. Wegner, L. H., et al. "The Trade-Off Study Re-
visited," Chapter 21, Systems Analysis and Policy Planning:
Applications in Defense, E. S. Quade and W. I. Boucher
(eds.), New York, American Elsevier Publishing Co., 1968,
16. Miller, George A. "The Magical Number Seven, Plus
or Minus Two," Psychological Review, v 63, 1956, p 81, U
17. Moore, John R., et al. "An Analytical Approach to
Scoring Model Design-Application to Research and De-
velopment Project Selection," IEEE Transactions on Engi-
neering Management, v EM-16, Aug 69, p 95, U
18. Tversky, A. "On the Optimal Number of Alternatives
at a Choice Point," J. Math. Psychology, v 1, 1964, p 386-
391, U
19. Rathjens, C. W. and Kistiakowsky, C. B. "The Limita-
tion of Strategic Arms," Scientific American, v 222, Jan 70,
p 19-29, U
20. Scoville, H. "The Limitation of Offensive Weapons,"
Scientific American, v 224, Jan 71, p 15-25, U
21. "Science and the Citizen," Scientific American, v 224,
Mar 71, p 44, U
41