A Modest Proposal for a Revolution in Intelligence Doctrine
After some dozen years’ immersion in intelligence, I still find myself reacting uncomfortably to its rather cavalier disregard for the footnote. In that strange way each profession has of altering accepted words to its own meanings, “footnote” in the jargon of the intelligence community designates primarily the notation of a major disagreement on the part of a member with an otherwise agreed estimate. Here, however, I am referring to the footnote in its academic, scholarly, or scientific sense, as a device for identifying and in some cases even evaluating the source material used for a particular textual statement. Such a footnote is deeply scorned by practitioners of intelligence and makes only a rare appearance in most intelligence products.
During my years of intelligence apprenticeship I of course noted the omission, but I assumed that the master craftsmen knew best and there were very good reasons for it. I assumed that the suppression of footnotes was part of one’s overall conversion from scholarship to intelligence: the paramount need of intelligence was a timely answer to a current problem. Intelligence could not afford the luxury of extended research, the comforting security of having explored all possible sources, the devotion of a lifetime of effort to the isolation and exact determination of one particular item of knowledge-culminating in a painstaking and exhaustive documentation of the entire research process.
And now, I
suppose, after these several years I am something of a master craftsman myself.
I have my brood of apprentices—and I teach them the same doctrine and they
practice it. But throughout the whole process I continue to be troubled. I
wonder if the abandonment, for the most part, by the intelligence community of
the somewhat elaborate and carefully developed apparatus of scholarship has
been altogether to the good. I wonder if we have not in fact been paying for it
by an undesired but real degradation of the intelligence effort.
[Top of page]
As one trained in the rigorous academic disciplines, I find abandonment of the reassuring apparatus of scholarship disturbing in itself. But it is more than this general loss that disturbs me. There are certain specific practices that also provoke a sense of uneasiness. For example, and I find this quite ironic, the higher the level of the intelligence product, the less complete is its visible documentation. In other words, the more serious its import and the closer it is to the influential official who will act upon it, the slighter is its overt back-up.
At the lowest level, of course, is the raw intelligence report. This report is generally extraordinarily well evaluated and supported. No scholar could really, within the normal limits of national security, ask much more. The source, particularly in CIA-originated reports, is carefully and intelligently described as to his professional knowledge and competence, his outlook, his opportunity to gather the information, and his previous reliability. Not only the date of acquisition of this information but place as well is given. In some reports the rapporteur also provides a field evaluation of the substantive information elicited from the source. The user of this kind of report can easily and effectively apply the canons of evidence in evaluating and testing the information.
But as we move up the ladder of intelligence reports the documentation gets sparser. The NIS (National Intelligence Summary), to use a well-known example, is in effect a scholarly monograph, digesting a great multitude of raw reports. Its total documentation usually consists of a single, very brief paragraph commenting on the general adequacy of the source material. No individual item within the NIS section can be tracked down to a particular source or specific group of sources. As one moves in the NIS from the individual chapter sections to the overall brief, the documentation becomes even more general and less meaningful.
At the more exalted level of the NIE (National Intelligence Estimate), documentation even in the generalized form of comments on sources has usually disappeared altogether. One is forced to rely on the shadings given to “possibly,” “probably,” and “likely” and on other verbal devices for clues as to the quantity and quality of the basic source data. These examples from the NIS and NIE are paralleled in a great many other publications of similar refinement. One may admire the exquisite nuances and marvel at what a burden of knowledge and implicit validation the compressed language of a finished “appreciation” can be forced to carry, but one cannot help being concerned about the conclusions. Upon what foundations do those clever statements rest?
If the final products were at least based upon documented intermediate inputs, the uneasiness might be somewhat less. But in my own experience the “contributions” or inputs, with the exception of certain economic papers, are normally devoid of any specific identification of the kinds and types of reports or other evidence upon which they are based. And in my experience those inputs are often based on other inputs prepared at a lower echelon until at last we reach the analyst with access to the raw data. At the upper level of joint or national discussion and negotiation and compromise, which eventuates in the exquisite nuance, the carefully hedged phrase, or sometimes a dissenting footnote, the remove from the original evidence can be, and often is, considerable.
is not, of course, quite as dire as I have portrayed it. The intermediaries, in
the process of review and consolidation of inputs, do query the preparers of
these concerning items of unusual importance or of a critical nature, and in
some cases they join the basic analyst in an examination of the raw data itself
in order to get a firmer grasp of a particular issue. Furthermore, the final
product, before being accepted and promulgated, is often returned to the
analyst who prepared the initial input, and he has an opportunity to note any
deviations from what he believes the situation to be. These processes do
provide a measure of control and cross-check, some assurance that the available
material has been thoroughly exploited and properly interpreted. But such
processes seem partial and makeshift at best. They do not always occur. And
they do not, of course, provide external participants in the final product with
any real insight into the quality and quantity of material utilized by their
[Top of page]
Another situation that troubles me—and this is a related problem—is the vast array of editors and reviewers under various guises and the several levels of examination to which an intelligence product is subjected before it is finally approved for publication. What troubles me is not the review, but the basis upon which it is accomplished. I recognize that many of these reviewers are highly talented, experienced individuals. Many are extremely devoted and conscientious and do their best to do a thoroughgoing job. But what basis do they have for their exalted “substantive” review?
In my experience, these reviewers have not generally—the notable exception would be members of the Board of National Estimates—been systematically exposed to the current take of raw data. Their knowledge of current intelligence events is based on hurried reading of generalized intelligence reports or on sporadic attendance at selected briefings. They are not aware in any particular instance—nor should they be—in any real detail of the material actually available on a particular subject. How do they know that this study in their hands for review has indeed explored the appropriate material? What variety of data has been utilized? Has the most recent material been examined? How can they do a spot-check on a particular item? Was a certain report seen, read, evaluated, and then discarded as erroneous, or was omission of the data in it inadvertent?
Lacking the apparatus of documentation, the reviewer generally has available only two methods by which to analyze the draft before him. One is to discover an internal inconsistency that calls into question the paper's overall accuracy or logic. The other is to find a statement that seems to contradict something he may have seen recently in his generalized reading and, on a hunch, to question its validity. The great bulk of any study, despite the reviewer’s best intentions, is beyond his capability to question, analyze, evaluate, or critically review. What a haphazard and random method this is for high-level substantive critique!
As a result much high-level review, in my experience, has consisted of the discovery of occasional typographical errors, small inconsistencies in numbers cited in different paragraphs or on different pages, minor inconsistencies in nomenclature, say between a figure or chart and a textual reference, unpreferred usage in spelling or hyphenating certain words, and other venial errors which a diligent proofreader should have caught. Any commentary on substantive validity, depth of research, or adequacy of analysis has been rare and exceptional. The minor changes are dutifully made, assurances given that more care will be exhibited next time, and the study is accepted and published as the agency’s or the community’s considered view.
I know that
this is the system we live with, and I know that it often works surprisingly
well. I know also that at times there are many vigorous discussions involving
substance, and that in this oral exchange there is often a rigorous testing of
propositions by an examination of the pertinent evidence. But much reviewing is
done without this stimulating personal dialogue, without considering the
evidence, and it is of this that I seriously wonder, is it worth the time and
effort? Are we in fact getting our money’s worth? Or are we not deluding
ourselves? Is the review structure we have erected to assure ourselves that we
are getting a high quality product not for the most part really a mere facade?
Does the Emperor have any clothes?
[Top of page]
If reviewing is sometimes a pious, well-intentioned fraud (one that I myself have had to commit), analysis at the basic journeyman level also at times leaves much to be desired. Not all analyses, of course, are based directly on the raw data, with its usable annotations and evaluations. Much analysis incorporates so-called finished intelligence, some of which is poorly dated, and the exact sources of which are not at all identified. Even the good and conscientious analyst does not know, nor does he have any means of learning, upon how solid a foundation that finished intelligence is based. It has an official imprimatur; so, not having supporting raw data in his files or time to procure and re-examine it—and, more important, following the traditional procedure of analysts—he uses it in his own study. His product eventually becomes a new piece of finished intelligence, which he or his successor will use in yet another study. And so the fragile structure can continue to be built of fragile materials. The weaknesses continually compound.
Another danger is the overconfident, glib, and persuasive analyst who writes his studies “off the top of his head.” He can prepare a report rapidly and defend it with great self assurance, relying on his memory and general knowledge of the subject matter. Sometimes this assurance is justified. But how do we know when? Then there is the intermediate intelligence officer who sometimes, for whatever reason, ignores his analytical staff and prepares a report on his own, again off the top of his head. It gets into the chain, and how is the next reviewer, or even consumer, to know that it has no substantial basis of research?
The hazards of
insufficient documentation are evident enough to need no further elaboration.
The value of proper documentation, moreover, and the system for it are not
unknown to intelligence officers of the community. Most, whether in uniform or
out, have at some time in their formal training been exposed to documentation
and its virtues, if only in the preparation of a term paper. Many continue to
evaluate externally prepared reports and monographs in part by reference to their
bibliographies and footnotes. The scholarly habits persist—except in the
intelligence field itself.
[Top of page]
Part of the reason for this condition is an item of cardinal intelligence doctrine: do not betray the source. Concern for protection of sources is of course legitimate, but it can be carried to extremes. As illustrated above, there appears to be a contradiction in the respective application of this doctrine to raw reports and to finished intelligence. Meticulous definition of the source in an individual raw report is accepted (and correctly) as necessary to the proper appreciation of the report’s content. It would appear equally necessary in finished studies derived therefrom.
The argument can be made that finished intelligence has a wider circulation than the raw reports and that there is therefore a greater risk of jeopardizing sources by identifying them in the finished product. In some cases this concern may indeed be valid—and could certainly be met by producing undocumented versions for the bulk of the circulation. But for internal consumption by operating officials who want to know (or should want to know!) the actual amount, validity, and reliability of the basic information, a documented form should be available. And it should certainly be available during the process of shaping up the final report-to the intermediate analysts, reviewers, and negotiators.
I am not
persuaded, however, that fear of source compromise is a wholly valid argument.
Footnotes will reveal report numbers, subjects, place of origin, and
rapporteurs, but would not necessarily identify sensitive sources. Many sources
are open or obvious and could be cited without danger. If a source is
particularly sensitive, even its nature need not be revealed, but a neutral documentary
reference should make it possible for a properly cleared user to run it down.
(In exceptional cases of extremely sensitive sources it might of course be
necessary to prepare versions at that level of sensitivity.) With effort and
imagination, I believe that the source-compromise problem can be successfully
met. One practical suggestion is included in the procedure recommended below.
[Top of page]
Another argument that can be and often is advanced is that documentation is time-consuming and time is a luxury that intelligence cannot afford. Admittedly it is time-consuming to prepare documentation; it would increase analytical, typing, and perhaps reproduction time. It could even be argued that it would increase editing, review, and final processing time. This is a plausible argument-but anyone familiar with the realities of much intelligence production will, I’m afraid, be unimpressed. Anyone who has been personally involved with the time lags in production of NIS sections, say, with the prolonged back-and-forth traffic of editing and “nit-picking” at most routine papers, will not believe that in much intelligence production time is quite so greatly of the essence. I strongly feel that the additional burden would be more than compensated by the improved substantive quality of the final product and that, as a matter of fact, much time would be saved. There would, for example, be no frustrating searches for the uncited sources of questioned statements.
It can also be argued that footnoting is a cumbersome, awkward, and excessively time-consuming method of documentation—and here I would agree. I would not, for intelligence purposes, advocate the adoption of the formal, extended-entry, bottom-of-the-page footnote system, requiring exasperatingly frequent repetition of document source and title and producing further complications in proper textual alignment and pagination. I would propose a very simple system based upon that used in scientific journals. In this system sources are listed in a single bibliography and numbered serially. Textual references to sources are made in parentheses following the relevant statement by use of two groups of numbers separated by a comma, the first identifying the source by the number it has in the bibliography and the second giving the page reference.
Extended discussions of particular source problems can appear as a series of appended numbered notes, referenced in the text by the appropriate note number in parentheses. This system is easy to employ and should present no difficulties to the analyst; it should cause only minor inconvenience to the consumer. And if a particular report needs to be sanitized quickly of specific source references the bibliography and appended notes can simply be detached.
documentation has languished so long and amiably in desuetude in the
intelligence community I do not know. Inertia and the relief from old academic
requirements may be part of the answer. But however it came about, the present
non-documentation system is well established and flourishing. The habit is
almost an addiction. Efforts to upset it fly in the face of human laziness,
tradition, even vested interest. In a sense, it is job protection for the
mediocre analyst: it does not expose his work to careful examination. Years of
living with undocumented intelligence has blunted our perception of its dangers
and inadequacies. The voice of protest—or is it conscience?—that is sometimes
heard is exceedingly small. Yet I think it is challenging.
[Top of page]
Import an Old Revolution
It seems to me that we need a major revolution in intelligence doctrine. What we need is the intelligence equivalent of the Academic Revolution that occurred in our schools of higher learning some hundred years ago when modern research methods were first introduced, primarily from Germany. This Academic Revolution, as all students of intellectual history know, brought to graduate academic disciplines (both scientific and humanistic) the tools, concepts, and apparatus of modern scholarship. Along with concepts of free inquiry, thorough exploitation of original sources, and objectivity it brought the requirement for precise documentation. A common methodology and certain common standards were developed; and the field of scholarship, originally the domain of the self-trained amateur, gradually became professionalized.
Intelligence is undergoing this kind of evolution. Its operations are becoming professionalized; a professional esprit and a common methodology are gradually developing. This journal has been an important step in that direction, following the classic pattern: it provides a necessary forum for the discussion of professional problems and helps create a common background of classic cases, basic concepts, general principles, and key problems in intelligence. It is in this forum that I should like to see argued out the advantages and disadvantages of a proper documentation of intelligence conclusions and findings. I have stated—perhaps overstated?—the case in its favor as a real necessity. Is there a valid defense for the status quo?
In addition to a serious, probing, and hopefully rewarding discussion of the problem, I would also recommend experimental application of the proposed doctrine to some specific areas of intelligence production. As a beginning, I would suggest it be tried on selected NIEs and NISs, with careful evaluation of the results after reasonable trial periods. Do they seem worth the additional encumbrances? What is the response of consumer officials to the improved documentation? Has there indeed been a qualitative improvement in the product? Or is it clear that formal, detailed documentation has no real part to play in intelligence, that it is and has been properly excluded from intelligence methodology?
In addition to this formal trial on standard products, it seems to me that policy officials requesting ad hoc intelligence studies or reports could very well consider including among their proposed terms of reference a requirement for thorough documentation. Since such a requirement may not occur to them (assuming they are unlikely to have read this particular plea), the intelligence officials discussing the proposed terms of reference might suggest it be included. Let us make the offer and see if it is opted.
The end result
of this discussion and selective application should be the development of an
agreed working methodology for intelligence documentation. The methodology must
be realistic. I should not like to see (and shudder at the possibilities!) the
establishment of inflexible requirements for its application. The apparatus of
documentation should be applied only where it helps, not where it hinders.
Certainly daily field operational intelligence is an area where it might prove
to be an impediment and costly luxury. But through intelligent trial and error
a practical doctrine should evolve.
A system that has proved its worth in every other professional field surely deserves careful examination and consideration by members of this one. It does not seem too soon to consider applying here the concepts of a revolution now some hundred years old.
[Top of page]
A “master craftsman” from State’s intelligence bureau takes up the challenge and presents the case
Allan EvansThe eloquent lead article in the last issuechallenges anyone to come forth with a valid defense of the status quo that prevails in our community with respect to footnotes. Age predisposes me to defend status quos; my frequent statements in talking to intelligence officer groups put me on the spot to repeat my arguments against the use of footnotes. It may be that these views are conditioned by circumstances in the Department of State and that these circumstances differ materially from those in the Department of Defense —if so, it will be all the more useful to unearth variations in the taste and requirements of major groups of consumers at whom our community is aiming. Let us see what can be said.
Customer is King
The first and most important arguments are that our customers won’t read fat papers and “almost certainly” in overwhelming majority don’t want to be bothered with documentation. I think no truth in our business is more thoroughly substantiated by experience (either footnoted or not) than that the impact of a paper varies in close inverse relation to its size. We have, of course, the NIS, which is indifferent to bigness, but it is an intelligence document of a very special kind, designed for universal reference. The Department of State issues stout papers, but for policy more often than intelligence purposes. There are technical areas of the government which revel in extensive analyses. So far, however, as the general run of day-to-day operation in this Department goes, our Bureau is prepared to stand by the idea that, other things being equal, the shortest paper has the most impact.
In closely related vein, our consumers are not going to spend their time summoning up the documents they see referred to in footnotes. They think of our intelligence papers as the product either of particular analysts whom they know by name and whom they have learned to trust, or of a particular organization which they trust to employ analysts who are reliable. They expect Intelligence to speak as authority, to present its conclusions with confidence, and they don’t want it to transfer to them the responsibility of reviewing the evidence all over again.
Indeed, many consumers couldn’t review the evidence. Many readers—those overseas, for example—simply don’t have the files of material that we use here at headquarters. Why tantalize them with alluring footnote references to luscious sources that are inaccessible to them?
the excellent suggestion that footnotes be organized in the modern manner at
the back of the paper and be therefore removable. When for special reasons
footnotes are actually used, the device would be valuable. In the usual case,
however, it would leave unjustified superscript figures throughout the text, to
annoy people and intrude a real if small barrier to smooth absorption of the
message. There might well be physical problems about tearing out and
restapling. These are minutiae, but in the bulk they might grow important. I
doubt that the real answer to the problem with consumers lies along this line.
[Top of page]
Quality and Control
These then are two positive arguments against introducing an apparatus of footnotes into intelligence papers. Let us now look at some of the arguments put forward in favor of this procedure. As an historian, I can only applaud the appeal to the past in evocation of the great scholarly revolution brought about by German methods well over a century ago. But aren’t a number of people becoming a little sceptical about some parts of this revolution? Are there not even sporadic attempts to escape from the yoke of that ultimate German invention, the Ph.D.? Only the other day I heard a notable authority on American scholarship draw a distinction between the research associated with our Germanic discipline and what might rightly be called thinking. Perhaps we should patronize the scholarly revolution of our own age rather than that of the past, and stress the production of ideas.
There is worry that without footnotes mediocre analysts will float texts which are unreliable. What about the danger that mediocre analysts, under cover of footnotes, will float texts in which they are able to avoid the challenge of decisive thinking? I don't say that only one of these two dangers exists. I think that they both exist, and I suspect that they rather cancel out as arguments one way or the other.
The article suggests that without the footnote the operation of review and upper-level control is a hollow pretense. The answer here would be in brief that without good supervision and control no amount of footnotes will guarantee quality, but that if the supervision and higher control are good the footnotes will not be necessary.I think the article is a little unfair to the reviewer. According to the terms set forth, every reviewer would have to be an expert in the subject of the paper he was reviewing, or would have to make himself an expert by reading all the material in the footnotes. Teachers, I think, will realize that this concept is too categorical. With good but not infinite knowledge of the subject, and with sound intuitions about how style, logic, and marshalling of ideas relate to accuracy and integrity of thinking, teachers and scholars do very well at reviewing the works of students and colleagues. These are the qualities required in the leaders of intelligence operations; without these qualities no apparatus will make intelligence products worth the money.
It is true that the judgments of an NIE float in the empyrean and impress
with their apparently unrooted boldness. It is also true, however, that the
writers of those sentences approach them with prayer and fasting, and work them
out in fiery give and take, often over long periods of time, in working groups
which can test to their heart’s content the background of information and fact
that underlies each agency's opinion. If sometimes our NIEs approach being a
little too empyrean, so do the problems that our superiors and world affairs
force us to examine.
[Top of page]
There are many lesser points. Certainly for intercommunication within the intelligence community indications of source might be useful; it would be a question of time and effort. As for the awful thought that many analysts may take advantage of the status quo to scamp their scholarly attention to detail in intelligence work, I should argue both that most of them are thoroughly dedicated and that the few who do try to get away with it are quickly found out.As a matter of fact, the working drafts of analysts often do have annotations, and are carefully filed for reference.
There is one small suggestion in the article on which comment requires a reference to the inner workings of a friendly agency; let me nevertheless rush in and remark that some part of the difficulty about documentation may be peculiar to the Defense Department because of its habit of sending estimators rather than the basic analysts to working groups. Is it possible that this mode of operating through layers accounts for some of the feeling that we lack full exchange of working data? I venture to suggest that the advantages and disadvantages of this procedure well merit discussion.
In the end,
there is one final and to my mind clinching argument. As I have told many
audiences, the essence of an NIE is what it says about things to come-indeed,
the culminating feat of the whole intelligence process is to project the
customer’s view near or far into the coming weeks or years. And, who will
footnote the future? Here internally, within the intelligence game itself,
resides the chief positive argument against footnotes—that a reliance on them
will blunt our willingness, if not our ability, to push along trails that
cannot be blazed with documents or references, and to explore what may lie
[Top of page]