Some Limitations in Systems Analysis in Intelligence Activities
APPROVED FOR RELEASE 1994
CIA HISTORICAL REVIEW PROGRAM
2 JULY 96
Suggestions on how to think about systems analysis
SOME LIMITATIONS IN SYSTEMS ANALYSIS IN INTELLIGENCE ACTIVITIES
R. C. Shreckengost
The procedure to be followed in using operations research or systems analysis techniques to identify optimal actions in large, complex systems is somewhat akin to the recipe for tiger soup, i.e., take one tiger. ... If consistent and well-behaved values and objectives are at hand the analyst may then proceed to apply the art to achieve an optimal concoction. Unfortunately, the multitudinous values generally required to explore fully the optimum allocation of resources among diverse intelligence tasks and responsibilities are as critical to the process as the tiger to tiger soup—but far more difficult to bag, assuming that a suitable set of values even exists.
The extension of operations research and systems analysis (OR/SA) techniques from the initial areas of their use—business and military operations—to other types of activities has been difficult and slow. A question persists about the feasibility of orderly analysis and quantification to identify desirable decisions affecting large-scale social systems. However, the demonstrated effectiveness of these techniques in business, military, and similar areas has stimulated effort to extend the methodology to these other areas where the problems of analysis are admittedly extremely complex and difficult.
Philosophers have always been concerned with values and objectives in human affairs, so that the complexities of social and political choice are well-recognized even though few "systems" or paradigms for choosing among alternatives have evolved. In contrast, OR/SA techniques have only recently been developed and put to use, in part because a computer is required for any major analysis. It is not yet possible to describe definitively the degree and manner in which the new methods may be adapted to the analysis of age-old problems. This paper sketches some of the factors which are critical to two of the basic notions of analysis—value and optimality. The notions of value and optimality are reflected in the quantitative expressions used in analytical calculations. The large number and variety of considerations bearing on value and optimality which must be treated in the analysis of intelligence systems will indicate some areas where better techniques and practices may be needed to facilitate the application of OR/SA techniques and enhance their acceptance.
Business and Military Applications
The chapter headings of many texts on operations research or systems analysis suggest the characteristic problems for which these techniques have been developed. Generally they are concerned with business operations such as inventory control, the movement of goods from factory to warehouse, replacement problems, queuing, and, of particular interest here, resource allocation among needs. In business operations the objectives are often relatively easy to describe—maximize profit, minimize loss, obtain a certain share of the market, etc. Further, the problem of quantification is usually relatively straightforward using dollars as units of measure. Although relatively simple compared to intelligence problems the relations of several competing objectives may be complex, and not so easy to resolve. Peter Drucker suggests eight business areas in which objectives are important: market standing, innovation, productivity, physical and financial resources, profitability, manager performance and development, worker performance and attitude, and public responsibility.1 The relative emphasis given to any of these areas reflects executive judgments, and OR/SA techniques for coupling across these areas do not exist in any useful form. These judgments must reflect temporal factors and the external forces of the markets over which the executive may have little or no control. Further, some of these areas are difficult in terms of value quantification: the measurement of manager performance, for example, in any universally satisfactory way has yet to be achieved. So, even in some relatively straightforward areas where OR/SA has been used extensively, a more comprehensive approach is needed.
In the determination of product mix and in similar tasks where OR/SA has had substantial success, the assumptions have been generally well understood—that demand for a product may be statistical in nature and that the assumed statistics may be incorrect, or that certain assumptions with respect to linearity may not exactly describe real life—but the assumptions required to achieve mathematical tractability have not been so severe as to vitiate the usefulness of the analysis.
In military operations as well as business OR/SA techniques have been employed with greater success in the analysis of problems of restricted scope. In analyzing radar operations, for example, the number of enemy aircraft detected, or similar units of value measurement, have served as readily accepted scales. In contrast, analyses of the allocation of British bomber aircraft to protect shipping rather than attack German industrial sites during World War II did not enjoy any convenient or widely accepted scale of measurement, and the decisions which were made were largely political rather than analytical.[Top of page]
Ordering Objectives and Values
In considering the problems of budget allocation among diverse activities, it should be appreciated that the problems of value quantification and the identification of objectives, which are basic to the determination of optimal resource allocation, are still extant in many of the activities in which OR/SA techniques have been most widely employed. In one way or another the problems to be analyzed must be modeled or structured in some orderly way—and clear objectives and the use of reasonably well-behaved values which permit a useful degree of precision in ranking alternative actions to assess optimality are basic to the development of an acceptable model or structure.
A fundamental impediment to broad acceptance of quantification and ordering of values and objectives may be the implication of right and wrong, or that an optimum decision does exist. This is slippery ground at best, and some of the ancient concerns of logic are paralleled in the problems of value and objective selection. Aristotle's famous law of contradiction, "Nothing can be both A and non-A," for example, must be accompanied by some ground rules. Many things change color with time, or have spatial distributions of color, e.g., the sky may be blue at noontime and black at night, or, if there are clouds some patches of sky may be blue arid some not blue, etc. And, of course, this matter of color may depend on whether the observer is on the ground or in an airplane. Then it appears that nothing can be A and non-A at the same time, in the same place, and under the same circumstances. In a somewhat analogous fashion it might be stated that the acquisition of data for intelligence purposes has a particular value with respect to alternative allocations pertinent to some intelligence objective. But the value of acquisition in the form of an option rather than an outright purchase may be quite different, and the additional effects of place, circumstances, etc., as well as time, are readily perceived. The determination of value with respect to even a seemingly simple item, such as a radar's frequency, may be considerably more complex than what is normally encountered in classical DR/SA problems, such as inventory.[Top of page]
Problems in Combining Priorities
The concept of rationality imposes another specific problem in arriving at values. In OR/SA problems rationality is frequently explained in terms of decision-maker's preferences: if A is preferred to B, and B is preferred to C, then the decision maker is said to be rational if A is preferred to C.2 This creates no problem in classical DR/SA applications, but may be a barrier when a consensus or majority of opinion is used to rank preferences or values. For example, suppose Individual I prefers A to B and B to C, Individual II prefers B to C and C to A, and Individual III prefers C to A and A to B. A majority prefers A to B and B to C, but a majority also prefers C to A. Although the individual decision makers may be rational, their collective preferences may not be. Starting with this well-known paradox 3f voting it may be shown that it is generally impossible to construct a social welfare function indicating preferences for alternatives when more than two alternatives and more than one person are involved except through imposition or a dictatorial process.3
The development of values and the structuring of problems for analysis of collection effectiveness are more complicated undertakings than those in areas in which OR/SA techniques have been developed. Means are lacking for obtaining an ordering of values by combining individual orderings. These considerations detract from the acceptability and credibility of using OR/SA techniques in this area.[Top of page]
Perils in Problem Partitioning
Important problems on a still broader scale than value assignment and ordering also exist. One often tacit but important premise is that the optimal solutions to sub-problems comprise an optimal solution to a total problem. OR/SA problems of great scope are frequently divided into parts that are of more convenient proportions for analysis. The results of these several sub-analyses may not add up to an overall optimal solution—neglecting that the selection of objectives, hence a determination of optimality, may be difficult or impossible because of the possible non-rational situation described for values.
Efforts by individual players to score as often as possible do not add up to the optimal strategy for a basketball team. Although the sport team represents a rather trite example, the equivalent may be recognized in large scale social systems. Jay Forrester's urban studies indicate that large social systems may be counter-intuitive, and that piecemeal programs intended to ameliorate some selected urban problems may in fact do more harm than good .4 A renowned authority in OR/SA, C. West Churchman in discussing this partitioning of problems and the resulting suboptimization states:
... it is clear that no person or group of persons—scientist, politicians, or whatever—can honestly say that. he understands enough to guarantee by his decisions and recommendations an improvement of even a small sector of society. We are all suboptimizers, perhaps prone to the most dangerous kinds of suboptimization.5
Churchman further develops the need for a comprehensive understanding of a system in order to satisfactorily determine how the problems can be partitioned, analyzed, and reassembled. After noting that Plato, Spinoza, and others since have seemed to believe it possible to expand the use of models—and that this philosophy is often used today to sell systems science and operations research—so that ultimately nothing might escape the eventual embrace of rational models, he strongly states, "The trouble with this philosophy is that it is wrong, dangerously wrong, pigheadedly wrong, philosophically inexcusable."6 The paradox lies in his belief that the end product—the complete model—is needed in order to obtain the information with which to build the model.[Top of page]
Although the problems are formidable and the prospects for achieving a fully satisfactory procedure now appear nil, the importance of improving decisionmaking is so great that extensive effort to this end is justified. In terms of values and objectives particular attention might be given to certain characteristics which are of special concern when OR/SA techniques are employed in analyses of intelligence activities. The matter of overall benefit is perhaps most difficult because it is so pervasive and appears in so many difficult forms. An example such as the acquisition of a new reconnaissance system may suggest many legitimate benefit concerns. How will the acquisition be made? What segment of the intelligence community will benefit from the acquisition? Will the procurement hinder or help other efforts? Will the interests of the intelligence community members be equally affected? Any procedure which purports to embrace and rationalize ,diverse and somewhat independent interests must at least provide visibility of all important facets—or risk rejection. Welfare economics, ethics, and other formal approaches hold little promise for any technique for integrating individual values—this has been the subject of debate for centuries. Can these interests and views be satisfactorily created without resorting to an integrated form? May not a system of costs and benefits be devised which more completely and honestly effects various points of view?
The benefit analysis characteristics, difficult as they are, must also .include temporal effects. To what degree should the present (or future) e sacrificed in order to provide greater benefits in the future (or resent)? In view of Jay Forrester's finding that large systems may be counter-intuitive, any technique which does not provide for a look at he future is less than adequate. Although technological forecasting in neatly organized scientific and technical fields is difficult and uncertain, sociological and political forecasting is far more difficult and certain. Coupling temporal considerations seriously exacerbates value assessment.
A second factor typifying characteristics which should be considered in assessing values and optimality is the degree of reversibility associated with any action. An increasing_ public awareness of the unforeseen consequences of some irreversible act attests to the importance of this characteristic in determining values and preferences. Ultimately such concerns emphasize maintaining the status quo.[Top of page]
Possible Approaches for Improvement
Several approaches illustrate lines along which some improvements might be made in working toward values, objectives, and optimality. or example, a listing of what is, or is not, implied in any set of preferences or values would at least suggest the bounds within which an R/SA analysis has been conducted. Knowing if the problem of radar data acquisition has been analyzed using the values of an analyst at the national level—or the values of a technical expert in radar characteristics—or the values of a tactical operations officer—provides sight to the results of the analysis. In some cases it might be useful to restrict the scope of the analysis severely, speculate on the value perturbations that might result from a set of different political condemns and their relation to optimality. Some appreciation of the sensitivity of the analysis or the OR/SA approach to political state ,particulars might be surfaced.
Some reconciliation of different value preferences might be achieved rough the use of the Delphi technique.
Deriving implied values from current resource allocations appears to be especially intriguing. For example, appropriations for the intelligence community imply some current preferences, values, and objectives as well as past investments and commitments. The dollar amounts might be regarded as representing de facto values. Can a form for analysis be constructed on these values "from the bottom up" which would provide clearer visibility and, at the same time, be suitable for more refined examination and analysis using OR/SA techniques?
The frustration and concern which is so evident in C. West Churchman's views quoted earlier is also shared, however, in his view that, "It is a mistake to use man's failure to develop an adequate measure of utility of the social structure as evidence that such attempts are futile."7[Top of page]
Aekoff, Russell L., Shiv K. Gupta, and .1. Sayer Minas. Scientific Method: Optimizing Applied Research Decisions. New York: John Wiley Inc., 1962. 464 pp.
The last chapter, the Ideals of Science and Society: An Epilogue, deals with the problems inherent in extending analysis techniques to large-scale social and political problems. Objectives and ideals in terms of operations research techniques are treated and their implications recognized. Concern for the unwarranted influence of pseudoscience is expressed since, for example, on pg. 442, "Such developments as operations research, management science, and systems analysis and engineering have brought science into the domain of important social and economic problem solving. Those who use these services are not always aware of the limitations of current technology."
Arrow, Kenneth J. "A Difficulty in the Concept of Social Welfare." Journal of Political Economy. Vol. 58, No. 4 (August 1950). pp. 328 346.
It is demonstrated that, given certain natural conditions, there can exist no method of deriving social choices by aggregating individual preferences. This finding is fundamental in limiting the meaning of optimality and the usefulness of objectives in the analysis of highly complex systems. For example, in the sense that individual choices can not he aggregated the concept of the greatest good for the greatest number is a logical impossibility.
Churchman, C. West. Challenge to Reason. New York: McGraw-Hill Book Co., 1968. 223 pp.
Directed to the proposition that the understanding, or analysis, of a part of a system is dependent upon having an understanding of the total system, severe criticism is developed regarding some simplifications which are not uncommon in formulating and "solving" OR/SA problems. Recommendations for more ethical and better practices are found throughout.
__________ and A. G. Schainblatt. "PPB: How Can It Be Implemented?" Public Administration Review. Vol. 29 No. 2 (March/April 1969). pp. 178-189.
A systems approach to the problem of state resource allocation among the facets of a particular problem, alcoholism, is used to illustrate the potential as well as the limitations and shortcomings of the approach.
Easton, David, Systems Analysis of Political Life. New York: Wiley and Sons, 1965. 507 pp.
Easton's third book in a long range program to develop a general theory in political science presents a generalized model in systems analysis terms and abundant detail on the subtle coupling of many practical factors which must be considered in airy analysis. (See Forrester for a companion reading.)
Forrester, Jay W. Urban Dynamics. Cambridge: The MIT Press, 1969. 28.5 pp.
A description of large scale simulation and modeling applied to urban problems employing about 20 equations to relate significant urban parameters indicating population distribution, industry, etc. Forrester estimates something like 100 equations would be required to provide a sufficiently representative simulation of a largescale social system at the national level which could be useful for the examination of alternatives for political leaders. (See Easton for a companion reading.)
Hadley, G. Probability and Statistical Decision Theory. San Francisco: Holden-Day, Inc., 1967. 580 pp.
The second chapter is devoted to the Theory of Utility and discusses the notion of rational behavior in some detail,
Miller, David W. and Martin K. Starr. Executive Decisions and Operations Research. Englewood Cliffs: Prentice-Hall, Inc., 1960. 446 pp.
A realistic treatment of the problems inherent in establishing values and objectives, and determining optimality when working with sub-parts of a total system. In Chapters 2 and 3 there are sections on Social Scientists and Decisions, Goals, Purposes, and Rational Behavior; Conflict Between Goals; Bounded Rationality; Multiple Objectives, etc., and there are further sections throughout the book which deal with value-objective problems, and their effect on the acceptability of the solutions obtained to OR/SA problems.
National Academy of Sciences. Technology: Processes of Assessment and Choice. A report to the Committee on Science and Astronautics, U.S. House of Representatives. Washington: Government Printing Office, 1969. 163 pp.
A review of existing procedures of assessment, the development of the numerous problems entailed, and a description of the multiple objectives that such a process should accommodate leads to a recommendation. It is recognized that procedures might be developed which could be used as a basis for resource allocations in the public sector, including, but not limited to allocations for research and development. however, no hope is given for the development of an algebra which could lead to a net index of social desirability, and there is some fear that abuses might arise in the assessment process similar to those in which systems analysis on occasion has been used to provide a misleading mantle of objectivity for essentially predetermined value preferences.
Schlesinger, James R. Planning-Programming-Budgeting: Uses and Abuses of Analysis. A memo prepared at the request of the Sub-Committee on National Security and International Operations of the Committee on Government Operations, U.S. Senate, 90th Cong. 2nd Sess. Washington: Government Printing Office, 1968.
Tucker, Samuel A., ed. A Modern Design For Defense Decision. Washington: Industrial College of the Armed Forces, 1966. 259 pp.
[Top of page]
A wide-ranging McNamara-Hitch-Enthoven anthology on the employment of operations research, systems analysis, and the introduction and growth of the PPBS in the Department of Defense. The very serious problems in applying these techniques in large systems under time-dependent. conditions are abundantly illustrated.