Military-Economic Estimating: A Positive View
18 SEPT 95
Defense of the intelligence input to systematic planning for defense.
MILITARY-ECONOMIC ESTIMATING: A POSITIVE VIEW
Julie O. Kerlin
In an article entitled "Economic Intelligence in Defense Planning" in a recent issue of this journal,1 Colonel Clyde C. Wooten examines aspects of the economic-military research done on the Soviet Union as input to systems analysis of U.S. defense needs. He concludes with some exasperation that the data are unusable, the methodology questionable, and the product spurious.2 In this essay we shall examine the character of the product and show that in the process an intelligent methodology provides a logical ordering for data which are indeed sparse but which can be used to advantage in place of an otherwise unknown, intuitive input into military judgments. Military plans must reflect, among other factors, judgments concerning the potential enemy; exposing the basis for these judgments and putting the data into logical array improves the plans and provides flexibility through control of changes in the data.
Aggregate Resources and Specific Allocation
The product of military-economic intelligence on the USSR divides naturally into two categories. The first covers general or aggregate aspects of the Soviet economy and its military establishment. An illustration of this type is a judgment made in 1962 on the size of the Soviet gold reserves. By estimating the reserves—through techniques analogous to those used for military costing3—at a level much lower than had been accepted on the basis of informed intuitive estimates, economic analysts exposed an important aspect of the Soviets' gunsor-butter problem. Another illustration is the estimates CIA made public of significantly reduced growth rates for Soviet GNP in the 1960's.4 Western economists and many Soviet economists have since endorsed the general order of that judgment and the predictions made at that time.5 It is because of studies such as these that Colonel Wooten is able to write, "We know that the Soviets have important resource allocation problems."
The relationship between Soviet aggregate military expenditures and GNP suggested by the terms "stress," "strain," and "economic feasibility" can serve, in conjunction with assumptions about the economic and political environment, as a broad guide to the U.S. military decision maker. In a gross sense this relationship expresses the marginal cost to the Soviets of various possible defense postures in terms of their economic and political goals, say living standards or industrial growth. For this purpose changes in the relationship are more important than absolute levels; consequently the need for certainty in the measurements gives way to a less stringent one that they be taken in a consistent and reasonable way.
No more detailed exposition of analytic procedures in this macroeconomic field will be attempted in the limited space available here. The second category, to which this paper is primarily addressed, concerns allocations of military expenditure to particular weapon systems. Because, as Colonel Wooten points out, it is difficult to isolate military expenditures in the published Soviet budget, the total is estimated by adding up cost estimates of Soviet forces. Costing and production estimates are also the way to get at the allocation of these military resources among weapon systems, an important ingredient in the U.S. military planner's decision on how to allocate his own resources. The kernel of Colonel Wooten's question is, "Can present data and methods quantify Soviet military forces and potential with sufficient accuracy to assist the U.S. military planner in this decision?" We believe they can and do, with due regard to the statements of uncertainty in which the producers of the estimates, especially when prognosticating, seek to communicate the nature of their product anc the care needed in using it.
The following explanations of the costing methodology, the meaning of the results, and the way the process is handled in the community should help answer some of the questions raised by Colonel Wooten It does not deal, however, with his doubts about the validity of using ruble costs to get a measure of opportunity cost or economic scarcity in the Soviet Union. The academic detail involved would take up a disproportionate amount of space, and the question has been discussed elsewhere by experts on the Soviet economy.6
Methodology: Ruble-Dollar Ratios
First, dollar prices of Soviet equipment are estimated: these are what it would cost the U.S. DoD to procure the Soviet weapons from U.S. manufacturers. The technical specifications of the Soviet weapon are given to the manufacturers in the same way they would be for a proposed U.S. weapon. Some adjustments to the U.S. technical environment may be allowed, for example in standard dimensions of materials or in standard processing techniques the alteration of which to meet Soviet specifications would be expensive without significantly affecting performance. The purpose of the dollar figures is stated, with the necessary caveat, in a recent paper: The dollar valuations of Soviet military programs ... can provide US planners with an appreciation of the physical magnitude of given Soviet programs and also provide a useful basis for comparison with US programs. Because of significant differences in the price structures of the two countries, however, the dollar valuations of Soviet programs do not necessarily provide the most accurate indication of the relative costs of given programs from the Soviet point o£ view.7
Then it is attempted to cost the Soviet military expenditures in rubles, that is in terms of the Soviet economic environment. Ideally, this will show what portion of Soviet resources are being directed to the military effort and the distribution of this effort among different military programs as seen by the Soviet planners. How close reality comes to the ideal depends not only on how closely the ruble reflects relative economic scarcities-the question on which the reader has just been referred to expert treatises-but also on how good the estimated ruble prices are. The Soviets' ruble prices for some military items like aircraft and tanks have become available from time to time. Others must be constructed from information concerning the weapon, inputs to the weapon, and the technical circumstances of its production.8
When the ruble prices must be constructed, the U.S. manufacturer is asked to apply his experience to cost the Soviet weapon using Soviet inputs, Soviet plant, and ruble values. The ruble values of the inputs are drawn from information available on the same or manifestly similar inputs outside the military field-construction costs, wages, electronic components, inputs into merchant ships, etc. Division of the thus estimated ruble price of a weapon by the estimated dollar price produces a ruble-dollar ratio for that weapon. This ratio remains valid until there is significant new information on or changes in the proportions of different inputs or in their ruble or dollar prices.
Because of the very different scarcity relationships in the U.S. and Soviet economies, the more specific the ruble prices are to each weapon the better they will express their relative value. A single ruble-dollar ratio for all military hardware would ignore the different scarcity proportions in different weapons which should be approximately reflected in their ruble prices. At the other extreme one might try to cost in rubles every one of the thousands of pieces of military hardware or even all the inputs to them. As a practical compromise, a ruble-dollar ratio is constructed for each broad category of weapon and used to convert to rubles the dollar cost of all weapons in that category.
The use of production experience to estimate costs of proposed production, the basis for this method of estimating Soviet weapon costs, has been found sufficiently reliable by U.S. manufacturers in prognostications about their own new products except with respect to R&D and to the possibility of subsequent changes in the product or conditions. The uncertainties of R&D, as explained below, are segregated from the costing of individual Soviet weapons. The uncertainty of possible subsequent changes remains, but no more than in any future estimating, whether of U.S. or foreign, civilian or military production. Estimates are always subject to change in the light of subsequent information. Their reliability depends on the quality of the data base, the soundness of the producers' judgment, and equally important, the context in which they are used.
Production techniques improve with experience, and consequently costs decline with continued output. This phenomenon has been studied quantitatively, the rate of decline being represented by a "learning curve." A learning curve for each type of production is constructed on the basis of U.S. and Soviet experience and of known production conditions. From it can be read the cost per unit at any cumulative production level.9
It is useful to know the independently estimated order of battle as a check on production estimates when choosing the appropriate cost along the learning curve. Often it is even necessary to use order-of-battle estimates in order to arrive at production estimates. But this use of corroborative or combined sources to arrive at quantities and costs involves no such circular reasoning with respect to judging economic feasibility as Colonel Wooten seems to think. Gross economic feasibility could be judged without reference to production estimates simply by costing the order-of-battle estimate.
As Colonel Wooten emphasizes, R&D-or more accurately Research, Development, Test, and Evaluation-is probably the most difficult area not only to cost but even to define. Definitional and statistical efforts on U.S. RDT&E activities have a very short history, and such activities are not attributable with precision to particular weapon systems. Consequently no attempt is now being made to cost Soviet RDT&E by weapon system for use in aggregate figures. Test vehicle estimates are dropped from production estimates, and RDT&E figures are separately derived from published Soviet data-the budgetary "science" figure and "funds for science from other sources." A ruble-dollar ratio is independently estimated for this line. The difficulties involved in judging costs of research and development in the Soviet Union, as in the United States, do not, therefore, permeate the military expenditures estimate but are confined to the one item, RDT&E. Those who worry about analogy costing because of differences between U.S. and Soviet RDT&E practices should find solace in this segregation.10
The High-Low Range
There is a misunderstanding about a most important attempt to define the cost range of possible Soviet choices. Colonel Wooten s distressed by the 45% gap between some of the low and high assumptions made for the level of Soviet forces in the study "Soviet Defense Expenditures" (CIA/ RR MP 65-1, 2 June 1965) because he interprets these as defining the range of confidence for a single estimate A the most probable Soviet outlays. But the high and low projection lines do not bound such a confidence range, a range within which any single-estimate line is as likely as any other. The inference that a similar range of uncertainty must be applied to any estimate on Soviet defense expenditures and the conclusion that no meaningful. measure can be based on such uncertain estimates are therefore completely wrong.
The range is comparable rather to that in a scatter diagram where the high and low occurrences have been traced to band the scatter. Within the full band one can expect all occurrences to fall.; there is 100% confidence that any event will be in this range. The statistician would then draw narrower bands according to the data, defining the range within which, say, 99%, 95%, 90%, and 85% of the events occur. These bands would not necessarily parallel the 100% boundaries. If the events were systematically related to the variables against which they were plotted one would expect a high proportion of them to fall within a narrow band covering perhaps 10% to 20% of the full range.
The high and low assumptions of the analysts might similarly be described as covering a comprehensive range of Soviet military force structures or as banding the spread of practical possibilities. Within this range, considering technical, military, economic, and political circumstances, a single most probable force structure is then chosen. While there is of course less confidence in the single estimate than in the whole banded area, there is greater confidence in it than in any other single structure possible. And the degree of confidence is not at all determined by the width of the banded area.
The high-low estimates thus provide a frame within which judgments applied to the problem of U.S. forces can be accommodated to the widest range of circumstances. The dual estimates are a quantified means to reduce the appearance of incontrovertibility in a single estimate and a basis for introducing contingency insurance into decisions based on it.11 The defined spread of alternatives makes it possible for policy decisions to allow not only for uncertainties in the data but for the possibility of unforeseen environmental changes that could lead Soviet policy makers to shift their decisions. It permits a variety of sensitivity analysis on the policy level, making for greater flexibility in the decisions.
Community and Confidence
An expansion of interagency cooperation has accompanied the community's efforts to meet DoD requirements on Soviet military expenditures. The inclusion of more cost analysis in the NIEs has resulted in broader participation in the evaluation of cost estimates, the published figures and discussion being an agreed summary of contributions from the community. Much benefit is derived from the distribution of the detailed contributions to participating agencies. Community review in the representatives' Meetings helps clarify methodology and communicate degrees of confidence.
In the NIE, confidence levels are indicated in general terms by words like "possible" and "probable." Explanations of methodology, documentation, and specific discussions of confidence are available in the individual contributions and, often in greater detail, in published research and analysis on which the contributions are based. To require that each NIE contain all these materials would make it so compendious that there would no doubt be a request for the separate publication and distribution of a summary-i.e. the NIE.
The Intelligence Assumptions for Planning and National Intelligence Projects for Planning papers, also subjected to the USIB review process, do contain discussions of the limitations of their expenditure estimates. Footnotes, in addition, provide an alert to areas of disagreement. Machine runs for these estimates are available to readers having a need for more detailed input figures. Another interagency cooperative effort to estimate military expenditures, the CIA/DIA Joint Analysis Group's alternative projections for Soviet forces, use generally the same procedures and pricing factors in costing. The first five pages of the "Foreword" of their costing study is devoted to a discussion of cost estimates and their limitations. The Foreword also explicitly invites users "to consult with JAG regarding the degree of confidence associated with cost items of particular concern to them and ... inform JAG of any projections which appear to be in error."
Consumers are everywhere encouraged to consult the military-economic analysts. Users of the NIEs and NIPP have ready access to the producers. In particular, a close working relationship has been established between personnel of the office of the Assistant Secretary of Defense for Systems Analysis, a major user, and cost analysts in CIA, so that they are in continual consultation without any formalities. Other agencies have similar relationships when needed.
The increased formal and informal cooperation within the intelligence community provides an opportunity for joint improvement of cost estimates as additional input data become available and as methodology is improved. The concomitant increase in interaction between the producers and consumers of expenditures estimates enables the producers to keep aware of and respond rapidly to the needs of the policy maker, including his need to know the limitations of the product for a particular use.
The State of the Art
Do the acknowledged gaps in information and the large role played by human judgment in arriving at estimates make it impossible for the intelligence community to produce figures which will meet the needs of the systems analyst? His needs are less exacting than may be realized. Analysis for Military Decisions provides a thorough review of the systems analysis approach to organizing the components of complex questions of choice in the face of uncertainty, questions such as occur in the field of national defense. It emphasizes repeatedly that the systems analyst does not pretend to provide certain or absolute answers as input to military policy. He himself must accept input data derived from human judgments to form the basis for other judgments. In a recent DoD-sponsored symposium on cost analysis, Alan Enthoven offered his general impression that the art of systems analysis is now at about the same state as medicine was during the latter half of the nineteenth century; that is, it has reached the point where it can on the average do more good than harm. He added that it would be just as unwise to stop systems analysis now as it would have been to stop medical research then.
Under these circumstances producers and consumers should both be continuously alert for errors and unjustified conclusions. One kind of error is illustrated in that correctly cited by Colonel Wooten, the 1964 extrapolation of a declining trend in Soviet economic assistance to "non-aligned" nations. The error was not in the economic quantification but in the implied judgment that the Soviets would not reexamine the political value of expenditures consuming such a small portion of the national product. This particular prognostication should have been recognized as one especially vulnerable to political considerations.
Errors resulting from gaps in data also occur. Such was the absence of information on production at Airframe Plant No. 30 and its consequent omission in 1960-65 estimates. This is the reason for continuous scrutiny of all source data; data gaps must be expected to occur, especially in the more dynamic sectors of military procurement. The error introduced in this case, cumulated for the five-year period, was approximately 0.1 billion rubles, by itself too small to have appreciable influence on aggregate figures given in billion rubles "correct" to one decimal. It does however represent a limitation, one of the uncertainties of which intelligence customers are warned. It should always be assumed that there are gaps in production information, many of them filled by technical judgments but some completely unknown and unmeasurable except by interpolation from order-of-battle estimates.
We conclude, therefore, that the data base, though not solid, is usable, the methodology logical, and the product rational. Reasonable intelligence estimates, as an element in making military decisions, must be quantified; and no alternative method of quantification has been proposed. There is no question but that many improvements can be made-and indeed this is an important aspect of the work of military intelligence analysts throughout the community-but within the framework of the present methodology and organization.
The real lesson to be drawn from the discomfort Colonel Wooten feels with the current product may be a need for quantification of uncertainty through sensitivity analysis, measuring the effects of variations in input. Such analysis, done at the technical input level, would not be suitable for incorporation into the published NIE, but it could be communicated to users in other ways. Although sensitivity analysis is practiced to a limited but growing extent in the current methodology, it is legitimate to suggest that it and other methods describing confidence levels be made an integral part o£ all basic work on estimates of Soviet military expenditures.
It would be wrong, however, to expect statistical techniques and descriptive language to eliminate uncertainty as a continuing element in Soviet military expenditures estimates. Acceptance of uncertainty in military policy formation is a necessary ingredient in preparation for reality.12
3 The methodology is summarized in Paul R. Storm's "Estimating the Soviet Gold Position" in Studies VII 4, p. 1 ff. For a contemporary appraisal see "Soviet Gold Production & Reserves Reconsidered" by Keith Bush in Soviet Studies (University of Glasgow) Vol XV No. 4 (April 1966), p. 490.
6 For example: Value and Plan by Gregory Grossman, University of California Press, 1960; "The Soviet Price System" by Morris Bornsteen in The American Economic Review, 1962; Price Determination in the USSR by Lazar Xloeseyevieh Kantor, JPRS translation no. 34,385, 3 March 1966.
12 For an excellent discussion which includes references to alternatives for quantitative estimates of Soviet military posture as well as the present and future needs of systems analysts see Dealing with Uncertainty About the Enemy: Some Rand Experience in Systems Analysis for Military Planning, by F. I. Mossman (RM4416-PR, Jan 1966). This study gives a clear picture of the need for and use of intelligence estimates on enemy posture. It shows the method of handling the expected uncertainties and at the same time indicates areas of hoped-for improvement in quantity and quality.