The Collector's Role in Evaluation

18 SEPT 95


Imprecision and real hazards in an arbitrarily prescribed division of functions.


Bruce L. Pechan

Ever since the establishment of a defined and ordered central intelligence program, the community has performed one of its fundamental functions on the basis of a fiction. This fiction has by now come to be accepted as fact in some circles, and there is a dangerous chance that ultimately it could be universally accepted. I refer to the notion that the collector of information is not qualified or authorized, much less obligated, to participate in the evaluation of the reports he transmits. If this idea in its full implication is ever accepted by the collector, it will do great harm not only to our evaluative and estimative performance but to our performance in clandestine collection as well.

The official fiction makes Evaluation a ritual which only analysts are ordained the high priests to perform. Clandestine collectors, with their often impressive qualifications, may subject an item to a thorough process which bears all the earmarks of evaluation, but this is not officially accepted as Evaluation and may not be designated by that term. Recipients of clandestine reports are protected against any such misconception by the solemn warning, "This is unevaluated information."

This pre-emption of the word "evaluation" to denote a particular step in what is a composite process has left us floundering for terms to apply to other steps. For the field collector's judgment as to the probability that a report is true we must use the synonym "appraisal" in order to preserve the analytic monopoly on Evaluation. The collector's judgment as to the significance of an item of information must be designated vaguely "comments" to avoid the implication that he has some evaluative responsibility.

The tortured circumlocutions that must thus be employed in referring to the collector's role in evaluation are unbecoming in a profession in which search for objective truth and precision in the use of language are cardinal principles; but the official fiction has more serious consequences than these semantic ones. The best of our collectors, ignoring the codified absurdities, have for years been offering their own evaluations as appropriate, unconcerned by what name they are called. But some collectors have been honestly confused by the hazy language describing their evaluative functions, and some have accepted literally the dictum that the collector has no responsibility in the evaluation of an item's significance. When this happens, the quality of collection and reporting inevitably suffers and valuable judgments are lost.

There is a tendency in some quarters to regard collection as a technical process and collectors as mere technicians. But a technique that employs human agents rather than black boxes requires considerably more than technical skill. It is the purpose of this paper to examine the duties of the clandestine collector in connection with evaluation in order to clarify his natural and proper role in making the judgments of which evaluation consists. The discussion will be confined to evaluation done for the benefit of estimators and policy makers, not touching the slightly different characteristics of the same process undertaken for the purpose of guiding collectors in the pursuit of further information. Although addressed specifically to the role of the covert collector, much that follows will apply equally to that of the overt collector.


Determination of Probability

Evaluation, as the term is used in the intelligence world, consists of determinations on two matters-the truth or probability of facts reported, and their significance if true. Evidence about probability is of two kinds. One kind of evidence lies in the origin and acquisition of the report, i.e., the reliability, capability, access to information, etc., of the source, and the circumstances surrounding his acquisition of the information. The second kind has to do with the information itself-the amount of confirmatory or contradictory information already in hand, or in the absence of direct confirmation or contradiction, the internal logic of the new information in its relation to what is already known.

The collector is held responsible for providing the first, external kind of evidence-an evaluation (officially labeled such) of his source's reliability, and an account of the circumstances surrounding the acquisition of the information. These two elements of external evidence are to be used by the analyst as factors in determining the degree of probability that the information is true. In providing them the collector has discharged his major assigned responsibility in the evaluative process, and he is not regarded as having thereby engaged in evaluation of the information itself. To see whether this is a realistic view let us look more closely at the nature of these two elements.

Source. The one accepted evaluative judgment of the collector, his source evaluation, is generally considered to be independent of the particular information reported. Source reliability is regarded as a relatively stable factor, and a C source has to prove his reliability over a considerable period to be advanced to a B rating. Once he achieves this, he is not deprived of it unless his reliability shows a decrease over another considerable period. This is a convenient practice and serves reasonably well for most of our clandestine sources, but in any given instance it may be invalid. A source who is reliable in one field may be less reliable in another, whether for lack of competence, lack of access, or lack of will to be reliable. The collector who smugly rests on his source's B rating is flirting with disaster; there is always the possibility that the B rating may not apply in the instance at hand.

This means that the collector must be constantly alert to possible shifts in his source's reliability, brought about by variations in the source's access to information or in his motivation or by lack of competence to report intelligently on a new subject. These can be detected only by regular analysis and evaluation of the significance of the information being received in terms of the source's capability and access and especially of his motivation. This analysis may be only a matter of form in the case of a staunch anti-Communist Japanese reporting the movements of Communist organizers, but it can be an endlessly complex job when a Japanese socialist is reporting on Japanese political undercurrents.

Circumstances. In the simplest of situations, that in which the source is reporting on something of which he has direct participating knowledge (such as a plot in which he is one of the leading conspirators), the source evaluation bears directly on the probable truth of the information. But when the source is reporting information he acquired second hand, the source evaluation can bear only on the credibility of his account of how he got the information; and the circumstances of its acquisition here gain importance as the only valid external evidence bearing directly on the probability of its being true.

An account of the circumstances of acquisition is expected of clandestine collectors for each individual report. Many agent operations have a standard pattern in which agents of fairly stable reliability report regularly in one or two sectors and their mode of acquisition remains unchanged in any important particular over a long period of time. But if a situation being reported on grows tense or if security conditions for the source become more stringent, this comfortable stability diminishes; the circumstances surrounding acquisition become of increasing importance and tend to vary appreciably from report to report. This is particularly true of political reporting on an area in crisis where the source may have to acquire his information in a variety of ways. In such situations the factors of acquisition pertinent to a determination of truth are often closely related to the significance of the information. To get at these factors the collector must analyze the information itself and work back from that point to draw out his source to best advantage.

In sum, the collector must for his own purposes evaluate the information he receives if he is to perform with discrimination his tasks of evaluating the source and reporting acquisition data. He thus has a head start toward fulfilling his third assigned function in the evaluative process, that of "appraising" factual probability. For this purpose, however, before he can make a judgment of validity, he needs additionally a respectable store of knowledge on the subject to which the new information pertains. We shall return to this point shortly.


Determination of Significance

In evaluating the significance of a report there are at least four elements to be determined. The first is whether it has relevance to an established requirement. The second is its meaning in terms of the requirement, in other words its place and contribution in the fulfillment of the requirement. The third is its relative weight, impact, or importance. And the fourth is the timeliness of the information, with particular reference to the timing of events predictable therefrom.

A study of the extended implications of these several elements shows some significant characteristics of the evaluation function. First, evaluation is an exercise of human judgment, under the best of circumstances subject to human limitations and human error. Any single evaluation may be wholly accurate, but the sum total of all our evaluations will fall short of perfection. Second, it is a transitory judgment, rarely if ever a fixed and stable truth. It is subject to change with changes in the facts themselves or with the acquisition of new information. Third, it is an organic process comprising a number of successive steps, and though each of these may be complete in its own terms no one of them is the whole of evaluation. Some of these steps occur very early in the official life of an item of information. Let us then look at them more closely, and determine who it is that logically takes each of them.

In denying responsibility to the collector for evaluation we have overlooked entirely that he is de facto authorized to exercise evaluative judgment regarding relevance, importance, and timeliness, and that these judgments of his are often final and irrevocable. His right to kill a report for lack of relevance is uncontested. To cut down marginal reporting he is given lively encouragement even to kill information of limited relevance or importance. And because of his frontline location in the collection process, he must judge both timeliness and importance in deciding whether to send a report by pouch or by one of several orders of cable precedence. Whether his judgments are good or bad, the actions based on them are definitive: once a report is destroyed in. the field it is beyond our reach to recall and reconsider. Or once pouched as having limited timeliness or importance, the delay in its transmission is an unchangeable fact.

These judgments, however, are not Evaluation with a capital E, which is apparently thought to require a study in depth of relevance, importance, and factual probability for which the collector is not qualified. Let us then examine the prerequisites for such a study.


Capabilities of the Collector

The qualifications needed to be capable of evaluation in depth may be summed up in two categories: ability as a thinking man to reach sensible conclusions, and command of a pertinent body of knowledge on which to base them. With purely personal capabilities there is no reason to suppose that the collector is less generously endowed than the analyst; there is nothing in the analytic function that increases nor in the collecting that decreases the native ability of the naked man to think. It is held in some quarters, however, that the practice of the different functions enhances the ability of the one and detracts from that of the other to apply his logical capacity to the function of evaluation in depth, that analysis contributes to and collection detracts from the conditioning of the mental equipment for evaluation.

This effect is said to be produced in two ways. First, since the analyst must evaluate regularly as part of his job, his equipment for evaluation becomes more and more highly trained, whereas the collector, who by definition does not evaluate as a necessary part of his job, lets his equipment, however naturally great and highly trained when he starts out, become rusty from disuse. I have already shown that evaluation of the significance of information is a necessary adjunct to a collector's proper performance of his assigned evaluative responsibilities. I shall deal further with this point in a moment.

The second argument is that the influence of contact with his sources and his personal interest in the success of his operations render the collector undependable as a maker of objective judgments. I will concede that bias may be introduced into his thinking by his identification with sources and operations, but I believe the danger is often overstated. First of all, the possibility of such bias is accepted by collectors them selves, and the seasoned collector builds up a healthy skepticism to minimize it. Second, since his personal interests are bound up in the success of his operations, the experienced collector realizes that overselling his product may reflect badly on himself and so cultivates a counteracting tendency to undersell.

Admitting that bias, while not inevitable, remains a possibility in the collector's evaluative judgments, we should note that it is not peculiar to the collector. The analyst too may, and sometimes does, exhibit bias toward a point of view and interpret new facts in such a way as to make them support a preconceived conclusion. Pearl Harbor and a number of more recent strategic surprises bear memorable witness to the possibility that analysts may reject or downgrade evidence not in accord with their preconceptions. And since bias is a human failing that afflicts all of us to some degree, the best way to insure objectivity in evaluation is to take into account the judgments of all those in a position to render valid opinions.

Is the collector in such a position? Theoretically, only a person with access to all related knowledge can render the definitive evaluation of a new fact. The analyst most nearly meets this condition: in addition to his own knowledge of the area or subject matter he can call on a vast organized store of related facts from all available sources. But he does not approach an allness in this store, and on some areas and subjects it is pitifully thin. So if it were true that evaluation is not Evaluation unless it is based on the entirety of data we should have no Evaluations whatever. Actually, we seldom need all the facts to make a valid evaluative judgment. Of the hundred thousand facts about a country we may have stored up in machine records, ranging from the makeup of the party of the opposition in 1897 to the number of aluminum teeth worn by the current labor minister, we may find that only sixteen are of any use in determining the significance of a new item. The work-a-day analyst quickly learns to conform his consideration of data on hand to those of substantial pertinence. Otherwise he would turn out precious few evaluations.

Given that the analyst must evaluate on the basis of incomplete data, what kind of data is he most likely to lack? On many countries and subjects his store of organized and usable information grows sparse as we approach the immediate now, because of the inescapable lags in acquisition, transmission, organization, and assimilation of up-to-the-minute facts. In a rapidly changing situation he usually lacks the facts most essential for a valid evaluation of new information. That this is recognized by analysts themselves is shown in a recent review of clandestine reporting on a certain area: "... During a critical situation ... field interpretations of the significance and probability of the information reported are needed by the customers to a greatly increased extent." This plea lacks any official standing as a directive for collectors to evaluate, but it remains a direct and realistic expression of need by one set of analysts aware of the limitations placed on their own judgments by circumstances.

The field collector, and I speak here of both the overt and the covert collector, is best situated to acquire what the analyst most keenly lacks: current information on the area in which he works. He it is who can immerse himself wholly in the life of the area, have daily contact with broad segments of its people and its thought, and develop a capacity for judgment which under some circumstances is beyond the capability of the distant analyst. The 1956 uprising in Hungary was unforeseen in our national estimates not because we lacked excellent analysts working on the area or knowledge of the history of Hungary or information on the economic situation, but primarily because we had no qualified body of observers present on the scene to report the things that could be experienced and interpreted only by being there.

It is argued that the collector can fill this gap by furnishing the analyst that evidence on which he himself would base an evaluative judgment were he called on to make one. But this is impractical in many cases. There are often too many small details, some too elusive to capture in a written report, too closely bound up with the physical presence of the collector in the area. Many of the indicators simply do not speak to the analyst in his remote office with the same ring they have for the collector experiencing them in the field.

The collector is not only thus uniquely qualified under certain circumstances to evaluate new information, he must evaluate if he is to produce good reports. We have already discussed the importance of evaluating the. significance of in formation as an adjunct to the task of evaluating the source and providing useful acquisition data. Evaluative judgments are required also in getting the maximum of information from agents and informants. The skilled collector in the field, overt or covert, far from being a mere technician, is a whole intelligence community in miniature. When he debriefs an agent he runs through the entire intelligence cycle, sometimes several times over. As an agent makes a report, the collector evaluates it as to relevance. If it is relevant, he hastily evaluates its significance as nearly as he can and uses it to formulate new requirements. These he immediately puts to the agent in the form he judges most likely to draw out additional facts and related information which the agent may have overlooked, not realized he knew, or intended not to reveal. By this means the collector often greatly increases the substantive value of his report.

The important thing here is that the collector is limited by his capacity to judge significance. When he reaches the limit of the body of particular facts and broad general knowledge at his command and therefore of his ability to analyze the new facts in relation to them, he must content himself with accepting whatever the agent offers and sending it on for others to work on. It is axiomatic that, other things being equal, a collector who is well grounded in a subject is a better collector in that subject than one who is not. He can instantly analyze, evaluate, and extemporize his own requirements, short-circuiting by days or weeks the process of getting further guidance from the analyst. The mental operations of analysis and evaluation, by whatever terms they are designated, are inextricably involved in this short-circuiting, and it is clear that the skilled collector must have an evaluative technique well polished by use.


Limitations on the Collector

Limitations there certainly are on the qualifications of the individual field collector to evaluate. Being only one person, he cannot become expert in all subjects on which he may collect information, and his judgment about the significance of many items he reports is therefore of little or no value. On geographic areas other than the one in which he is working he can have at best only a limited up-to-date knowledge; hence, while he may be able to evaluate political news meaningfully in terms of its local impact, he may have little to offer with respect to its impact abroad. For these reasons, each collector should take stock of the limitations of his knowledge and not attempt to go beyond their bounds.

Although in some situations the collector may have all the data available to the analyst and more, he can never be sure that this is so. The analyst usually has, and in every instance may have, pertinent other-source information not available to the collector. Further, by virtue of the nature of his job, the collector is usually inhibited from indulging in the depth and thoroughness of deliberation expected of the analyst. Hence no evaluation by the collector, however accurate and thorough it may prove to be, can properly be regarded as more than tentative until it has been reviewed and confirmed by the analyst.

Finally, full cognizance should be taken of the fact that the primary job of the collector is to collect, and in this job analysis and evaluation are means to the end. It would be foolish for the collector to waste his time writing evaluations of every item he collects. He should make evaluative comments only when he believes he has something to offer that the analyst can probably not supply.

Recognition of these limitations should keep the collector's evaluations of the significance of information within workable limits. At the same time, to insure to the community that his judgments, which at times are irreplaceable, are not lost, as well as to enable him to enhance his own collection technique, it should be clearly acknowledged to the collector that he has a responsibility to make evaluations. It should be a part of his indoctrination to accept that responsibility and understand its limitations, of his training to learn how to carry it out, and of his performance to act his logical part in the evaluative process with skill and discrimination. Improvement in reporting will not be the least of the resulting benefits.

By the same token the analyst should be aware of the worth and of the limitations of field evaluation, as well as of his own. He should not act arbitrarily in discounting collector opinion, but give it the weight it deserves; he should never simply discard it because it does not agree with his own beliefs. Differences of opinion should be carefully examined, documented, and in important cases referred back to the collector for further consideration. Such a procedure may be cumbersome, but reliable evaluations cannot be arrived at by denying the collector's responsibility to express his opinion or by ignoring it once expressed.


Posted: May 08, 2007 07:37 AM
Last Updated: Aug 04, 2011 02:34 PM