Library

 

Notes on Estimating

APPROVED FOR RELEASE 1994
CIA HISTORICAL REVIEW PROGRAM
18 SEPT 95

CONFIDENTIAL
No Foreign Dissem

The evolution of some techniques in the national estimating system.

NOTES ON ESTIMATING

Keith Clark

Since NIE 1 appeared in 1950, more than a thousand National Estimates have been considered and approved by the United States Intelligence Board or its predecessor, the Intelligence Advisory Committee. This large number of very solemn documents, the collective progeny of the intelligence community at large, have been delivered through the midwifery of the Board of National Estimates and its Staff. Both the process and the product have undergone certain changes in the course of seventeen years, and there ought to be some lessons in a review of this evolution, not only for the midwives in ONE but for all who participate in the process of conception, gestation and delivery.

My purpose is to identify, primarily from the ONE viewpoint, some recurrent dilemmas and common pitfalls in producing estimates, to note different ways of coping with these, and to suggest some main sources of strength or weakness, as well as some avoidable wastes of time and effort. No two estimators would identify all the same problems as being important or perennial enough to rank as matters of continuing professional concern, but I offer my observations under two headings: (1) Style and Scope: the treatise versus the short answer. (2) Methods and Discipline: predictive estimating and prophecy.

Having drafted, chaired, or otherwise participated in many of the National Estimates, I disqualify myself from engaging in much praise or condemnation, but some subjective judgments seep through. I hasten to add that the judgments which follow, the arguments which support them, and the idiosyncrasies which pervade them are my own; they do not necessarily reflect the opinions of any colleagues on the Board or on the Staff, though I am indebted to members of both and to other professionals for some of the ideas.

[Top of page]

 

Studies and Short Answers

These tags denote two sets of values, or schools of thought, each valid by its own lights, which often collide when estimates are written, debated, and coordinated. It is not a question of mere prose style. Everyone agrees that for our purposes good writing calls for economy in words. It is a question of scope and approach. Some look on estimates as vehicles for educating the reader in all he ought to know about the problem posed. They reason justly that an informed policymaker, like an informed electorate, is a good thing, and the more informed the better. Acknowledging that NIE's are not encyclopedias and do have severe limits on length of discussion and depth of detail, adherents of this approach nonetheless strive to incorporate as much information as possible into the document, and don't like it when something they consider information or insight of cardinal importance is defined by someone else as superfluous detail.

At the other extreme are the short-answer men. They are imbued with a perfectly correct conviction that most high-level policy-makers have too much to read as it is, and that if the intelligence community sins in its publications, it is in the direction of too much rather than too little. In common mercy, as well as in the interests of getting the essential message across, they conclude that estimates should be sheared of all that is not strictly necessary to making the main judgments, and that the latter should be supplied as crisply and quickly as possible.

It is a rare estimate that does not give rise to some clash of opinion along these lines, and since it is a very subjective matter, prevailing doctrine or fashion shifts from time to time and from person to person, in fact, individuals feel differently on different occasions, depending on whom they are writing for, their own depth of knowledge and interest in the subject, their patience or lack of it, and many other variables. Speaking only of ONE, I once thought it generally correct to say that the Board favored short answers and the Staff liked informative detail. This is probably more often true than not, but there are so many instances of the Board's demanding the addition of information and detail to staff drafts that the generalization is not very valid.

The National Estimates show fluctuating trends in this respect over the years. Insofar as general patterns can be discerned and briefly described, we leaned in the earlier years toward spareness. This reflected the strong military influence on early estimative methods, an influence which made for short answers to short and crisp questions. It also reflected the kind of problems which preoccupied estimators in those days almost exclusively—direct Communist threats to the United States and its allies and interests. The problems were relatively clear-cut or were made to appear so, and could be sharply defined. Thus NIE 1, of 3 November 1950, was on "Prospects for Communist Armed Action in the Philippines During November."

We then entered a kind of baroque period (mid-fifties to early sixties) in which estimates became more informative, full of subtleties, refinements, and detail, aimed at describing and assessing foreign societies and governments in a more complex way. This evolution was helped along by the participation of more civilians in the process, with their academic skills and habits of work. It was also partly due to a growth in the amount of intelligence available (e.g., photography of the USSR). And it was probably most of all the result of requirements for estimates on more complex subjects. For example, the nationalist revolution in the undeveloped world then in full flower gave rise to important policy problems for the United States and consequently to the need for estimates on a subject that was new and complex. It required conceptualization and even some new vocabulary; short answers to short questions would not do.

[Top of page]

 

Choosing Between Them

In recent years, we have followed an eclectic approach—using both methods and often mixing them, with the choice being made by the predilections of those involved after more or less considered judgment about the requirements and preferences of the consumer. I shall not argue for one approach over the other. In the present state of the art, and in light of varying consumer needs, we probably do best to be eclectic. But I offer a few observations about some pitfalls in the choice.

One observation chiefly concerns "country" or "area" estimates. These are not done as frequently as they once were, but the art form is far from dead. What is dead—or ought to be, I think—is the classic 60 or 70 paragraphs that methodically discussed almost every subject under the sun relating to a country or region in a kind of mechanical way, under the headings Introduction, Political, Foreign Policy, Economic, and Military. Experience has persuaded most of us that this approach involves much waste motion, and that country or area estimates can most usefully emphasize a few main points, sometimes a single main theme with variations. It seldom requires more than 20 paragraphs or so to render these judgments for any country, with all the supporting detail necessary.

And to do it in shorter scope increases the chances of attaining several desirable ends: one is that the estimate will be read and remembered by officials at high levels; a second is that the truly important judgments will shine forth clearly, and not be hidden or dulled by clouds of detail; a third is that the estimate will not become obsolete or obsolescent quite so fast when day-to-day developments put one detail or another out of date; and a fourth, rather bureaucratic one, is that short papers take less time to do, at least in the stages of coordination and consideration by the USIB. (It is a true, if lamentable, fact that time spent in discussing and coordinating papers often varies more nearly with the quantity of words to be gone through than it does with the importance and complexity of the problem at hand; we sometimes devote so much effort to not being wrong about secondary and even trivial matters, or to group discussions of literary idiosyncrasies, that we lack the energy and perspective to make sure that we are right about the big questions.)

The foregoing amounts to a rather more dogmatic argument for short papers than I really want to make. Let me note two or three exceptions to the main proposition. One is the kind of estimate occasionally requested (or in some cases annually expected) by high-level consumers who are already broadly familiar with the problems about which they ask. Certain levels of brevity and simplification which might be just right for many kinds of estimates would tell these particular consumers nothing they don't already know. In these cases a considerable degree of informative detail becomes mandatory if the estimate is to have any value. Certain annual Soviet and Chinese papers fall into this category, since generalized assessments of the Russian and Chinese military threats are of negligible use to anyone. Another exception is formed by some special estimates on, say, reactions to given U.S. courses of action. No one needs to be told that Communist and neutralist reactions to some forward military move by the United States would be adverse; they need to know how adverse, and in what ways—particularly the difference between verbal responses and retaliatory actions on the part of the governments in question. Sometimes we cannot make these distinctions clearly, but we ought to try.

Another occasional exception is the "how to think about" estimate—most often addressed to some fairly new and unfamiliar foreign policy problem, or some particular aspect of an area or country which intelligence feels it would be useful to conceptualize in a nonconventional way. The purpose may be more to structure the problem than to forecast the outcome. In such papers, it is probably best to get more leisurely, to give more information, detail, flavor, and atmospherics than are otherwise called for. In sum, there are problems which cannot be treated shortly if the estimate is to do the job it should. But we can at least try when we start these jobs to be clear in our own minds what the job is.

[Top of page]

 

Prediction and Prophecy

One of the most persistent half-truths held in the intelligence profession and among our customers is that estimates are predictions of things to come, prophecies of the future. This is dogma and it is also largely true, but when couched in these terms it frequently leads us down some unfortunate paths and stultifies our thinking. Prediction is indeed the heart of the matter, but there is a world of difference between predictive estimating and mere prophecy. Lest I appear to make a case by pejoratives, let me define my terms.

I use the term predictive estimating to suggest a process which takes due account of its own limitations and uncertainties. It begins with awareness of present unknowns, the slippery ground we start on because of the things we don't know, or can't be sure we know, about the past or present. It goes on to the future to predict what can be predicted—by induction from some kinds of evidence, deduction from other kinds, testing hypotheses against all evidence available, and the rest of the familiar intellectual disciplines hopefully instilled in us all. But as it moves along these tried and true paths, predictive estimating differs from mere prophecy in its continuing awareness of its limitations in the face of the extraordinarily complex array of matters which will in fact determine future developments.

More specifically, it distinguishes between constants and variables, and shows awareness of interaction between them; it defines critical points—crossroads or crunches—and suggests alternative lines of development leading from these; it admits ignorance and uncertainty when it reaches the outer limits of evidence, analysis, and logical speculation; without yielding to the crudities of "worst case" estimating, it also avoids the pretentious and useless fallacy of the "single best guess"; it distinguishes—sometimes explicitly, always implicitly—the model of a fairly tidy and rational world delineated for purposes of analysis and comprehensible exposition versus the messier world of flesh and blood and emotion; it keeps in mind the fact that foreign governments—even apparently monolithic dictatorships—are as often as not inwardly subject to conflicting pressures, ambivalences, and contradictory impulses, even though usage often compels us to talk as though "the Soviets," "Peiping," or "Israel" were each of one mind—whole, coherent, and consistent.1

Prophecy, as I use the term, implies that the future is already there, deep within the crystal ball, to be discerned by those who are wise and lucky enough to do so. It invites a great leap from A to Z, aided by intuition and hope. Predictive estimating does not reject these aids altogether, but it is based essentially on a concept of the future as too complicated and chancy to permit easy leaps from where you are to where you want to be. It is, in short, both more responsible and humbler than prophecy. It is also typically less dramatic, more cautious and tentative in its conclusions, and perhaps less exciting to read. Sometimes it is possible to startle or intrigue by statements of boldly impressive foresight, but this is legitimate only if a laborious and disciplined intellectual process has been gone through first.

All this may sound like pretentious counsels of perfection, and in any case inconsistent with earlier remarks on the desirability of short papers. Certainly a published estimate which self-consciously spelled out its own scrupulous observance of all the rules suggested above I would be an infinitely elaborate and tedious document, too much like a Ph.D. thesis in one of the fields of social science where concentration on methodology crowds out content. But I am talking here as much about an intellectual process as about the visible product delivered to the printer. We all use various forms of verbal shorthand in getting our message across; without them, analysis and estimating could not be communicated. But there is a difference between short cuts in getting the message across and short cuts in thinking about what the message should be. The latter can be indulged in only at the risk of sacrificing quality and, eventually, credibility. Like icebergs, estimates must have a lot of substance below the visible surface if they are to hold together and stand up.

[Top of page]

 

Guessing Games

The record of National Estimates over the years in these respects is a mixed one. One practice occurs often enough in various guises to warrant some criticism. It is the temptation or compulsion to estimate with apparent confidence about any question that anyone in authority wants to know about. The potent old blandishment, that if the estimators don't supply answers someone less qualified will, can sometimes be resisted only by appearing mutinous.

But the plain fact is that estimates on some questions are of negligible worth, no matter how sophisticated the thinking behind them, and we ought honestly to say so. We may be paid to estimate, but we are not paid to do the impossible, and certainly not to pretend to do the impossible when we can't. A confession of ignorance or uncertainty may annoy someone who wants practical answers to practical problems, but in the long run it is better to annoy than to con him. This is not an argument for refusing to do difficult tasks, or even to try what may look like impossible ones; it is an argument for being clear, to ourselves and to our readers, just how safe it is to skate on the ice in certain areas and just where the ice, for all we know or might wish otherwise, may be water.

One case in point is the amount of time devoted to predicting the survivability of governments. Using again the "country paper" as a whipping horse, these are too often conceived of as vehicles for quoting odds on whether an incumbent regime will be in place when "the period of this estimate" draws to a close. The trouble is that when it is possible to say yes or no with a really high degree of assurance, the answer is usually so obvious that no literate policy-maker really needs to be told it; and in cases where the forecast is much more uncertain—often, for example, in unstable and volatile countries of the underdeveloped world—no prudent policy-maker is going to place many chips on that particular prognostication.

I am not arguing for total abolition of this kind of estimate. It probably has to be made, the odds have to be quoted, the conclusion may even be informative and helpful at the time it is published. But as a continuing guide to planning and action in the real world it has severe limitations, and we ought to avoid exaggerating its importance. Among other defects, it becomes obsolescent quickly, since in these matters one wants the latest information, whether it changes a conclusion reached earlier or not; even the best estimate as of a given date cannot allow for all the accidents, whimsicalities, and other variables likely to affect the outcome in close questions of this sort; very often what the United States does or does not do will help determine the results (we normally leave this factor aside) ; and many of these situations are quite literally tossups, touch-and-go matters, in which rational planning must be kept flexible and contingent, with shadings rather than sharp choices in between alternatives.

We have too often focused on this kind of question as though a "probably yes" or "probably no" were the single most important answer we could give, one on which our reputations as estimators will stand or fall. I suspect that this particular kind of forecast is often read by our policy-making friend with a healthier skepticism about its real value than we ourselves show; and then the whole thing is forgotten unless and until something happens in the benighted country, in which case the estimate is dragged from the files and the prediction is either pointed to with pride or viewed with chagrin by those who made it. This review of the record, though interesting to professional estimators, is not very important in a broader sense, and certainly should not be made the touchstone of estimative reputations or a very serious criterion of quality. Success or failure in this kind of spot forecasting is too much a matter of luck and chance. It often comes closer to what I have defined above as prophecy than to predictive estimating, and is consequently not very useful as a responsible help to planning and action. We may have to indulge in it, but we should not confuse ourselves about its usefulness.

[Top of page]

 

Cards on the Table

One way in which estimates have grown more sophisticated deserves special mention, strong endorsement, and even more attention in the future: that is the laudable practice of leveling more with the reader on questions of methodology and our own confidence in certain estimates. I am not talking about the words we use for expressing degrees of probability, whether we conclude that something is "probable," "unlikely," or "almost certain." These terms are essential tools of the trade, available to all in a well-defined glossary, accepted and used by most writers and readers, and already the subject of several scholarly articles in this journal. To gain common agreement on the meaning of these terms has been no easy achievement, but it has now largely been done.

What I am applauding here is rather the practice of saying more about sources and methods, what can be expected of the evidence, and—more importantly—what cannot. To do so is to tread on delicate ground. There are many who feel that intelligence loses potency if it hints at the mysteries behind its findings, and the subject is apt to be particularly touchy in National Estimates, since any comment on the strengths and limitations of sources or methods tends to be translated into favorable or adverse reflections on some particular contributing agency's present and potential importance. Anyone who has participated in an estimate on strategic warning or concealment and deception will recognize the symptoms, but they are not confined to these subjects. Obviously there are distinct limits on how far one ought to go in telling all. Security and the "need to know" principle obviously impose distinct limitations. In many cases the whole story about sources and methods would also be tedious to the reader, and it is often unnecessary to an honest and useful paper. But it is also often quite relevant to giving the reader a sophisticated understanding of what he can rightly expect and what he would be foolish to count on.

We were probably pushed or pulled into being more forthcoming on this score than we might have volunteered on our own. Ten or fifteen years ago intelligence did go about its business—including estimating—with a propensity for the mysteries of the priesthood which has since diminished. The collective "we believe," as it appeared in the earlier estimates, had an aloof and oracular tone which has undergone subtle changes in recent years. I have a feeling that the propositions which it introduced were put forward in the fifties with less fear of contradiction or challenge than in the period since.

Perhaps the chief reason for the change was the new style of foreign and defense policy-making introduced by the Kennedy administration and still carried on. Broadly speaking, two things happened simultaneously: intelligence was taken more seriously than ever before as a continuing and responsible contributor to decision making; and it had to come down from the mountain and engage more vigorously in asserting and defending its judgments in strenuous debates before some very tough-minded audiences. The process was marked by much closer communication between intelligence producers and users, each became more familiar with the other's needs and assets, and estimates were geared more closely to practical problems in their scheduling and subject matter. All very fine, flattering, and generally beneficial—but it cost something.

The price was that intelligence lost something of its former mystery, autonomy, and immunity. Oracular assertions were out, argumentation which marshalled data was in. More and more technical experts lined the walls at meetings on increasingly complicated questions—and we would have been lost without them. Formal, published NIE's were preceded, accompanied, and followed up by a great deal of less formal paper and a lot of informal talk. Judgments could no longer be made, published, and filed away until next year; they came under constant scrutiny and had constantly to be defended or modified in the light of an increasing flow of intelligence. Information about U.S. policy plans was made available to intelligence to a degree previously unheard of, and estimates took cognizance of this in various ways. In the prevailing atmosphere, a few extreme heretics were heard to challenge the first premise of all—that policy-making and intelligence were, or should be, separate and distinguishable functions. The translation of some former intelligence officers into high policy positions seemed to add force to the radical new winds of opinion.

I suspect that some of the more drastic efforts to remodel the whole system in the early sixties will, in time, be seen as excessive reaction to some previous rigidities and excessive compartmentation. Intelligence and policy-making are likely to remain distinctly separate functions—with accompanying differences in perspective and a certain amount of intellectual and bureaucratic tension between them, some of it wasteful, some of it creative. But our particular professional world will never be quite the same as it was before. Having experienced the joys and sorrows of a more direct and responsible role, of seeing the product sold to sophisticated customers in a competitive market, few members of the profession would willingly return to the mysteries and immunities of an overcompartmentalized Olympus, even if they had the option. And they don't.

[Top of page]

 


Bibliography

1 To illustrate an effect of this approach: a number of National Estimates in recent years have employed the device of presenting the most likely judgment on the central question, and then, in immediately following paragraphs introduced by the sensible admission that this reasoning might be in error, of going on to suggest the implications of alternative hypotheses—even if the odds don't appear to favor them. I cannot escape the belief that on close questions of particularly crucial importance this practice adds enormously to the usefulness of the document.

[Top of page]

 

CONFIDENTIAL
No Foreign Dissem


Historical Document
Posted: May 08, 2007 08:12 AM
Last Updated: Aug 04, 2011 02:43 PM