HANDBOOK OF BAYESIAN ANALYSIS FOR INTELLIGENCE

Document Type: 
Collection: 
Document Number (FOIA) /ESDN (CREST): 
CIA-RDP86B00269R001100080001-0
Release Decision: 
RIPPUB
Original Classification: 
K
Document Page Count: 
30
Document Creation Date: 
December 12, 2016
Document Release Date: 
August 21, 2002
Sequence Number: 
1
Case Number: 
Publication Date: 
June 1, 1975
Content Type: 
STUDY
File: 
AttachmentSize
PDF icon CIA-RDP86B00269R001100080001-0.pdf1.65 MB
Body: 
Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 For Official Use Only , Handbook of Bayesian Analysis for Intelligence For Official Use Only OPR-506 June 1975 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Releas7,3R03/19yRAlt5W08M90269R001100080001-0 CENTRAL INTELLIGENCE AGENCY DIRECTORATE OF INTELLIGENCE OFFICE OF POLITICAL RESEARCH June 1 975 HANDBOOK OF BAYESIAN ANALYSIS FOR INTELLIGENCE In the preparation of this study, the Office of Political Re- search consulted other offices of the Central Intelligence Agency. Their comments and suggestions were appreciated and used, but no formal attempt at coordination was undertaken. Comment, would be welcomed by the author, FOR OFFICIAL USE ONLY Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For ReleaspaoomipitTFAT0269R001100080001-0 Summary. CONTENT Page Bayes 1. What are its Capabilities and Benefits? 2. When is it Useful? 2 How to Initiate a Bayesian Exercise 1. Flowchart of Setup Procedure 4 2. Defining the Scenarios. 4 3. Participants. 5 4. Time Span and Frequency. 6 5. Designing the Format 7 6. An Initial Meeting 7 7. Practice Runs 7 The OPR Bayesian Analyses 1. More than One Analyst. 8 2. Printing the Evidence 9 3. Graphs and Visibility 9 4. Identifying the Participants 10 How to Manage an On-Going Bayes 1. Flowchart of Periodic Procedure 13 2. Routine Procedure 13 3. The Calculations A. The Basic Method 14 B. The Simplified Method 16 4. Problems in Mid-course 17 5. Terminating and Evaluating a Bayes. 18 Inherent Weaknesses and Problems of the Bayesian Technique 1. Limited Applicability 19 2. Data Problems 19 3. Problems over Time. 20 4. Problems with Numbers 20 5. Manipulation 21 III FOR OFFICIAL USE ONLY Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 FOR OFFICIAL USE ONLY Page Appendix 1. The Statistical/Mathematical Basis for the Calculations I. Rationale 23 2. Derivation 23 3. Example 25 Appendix 2. Interactive Programs in APL and BASIC 1. Interactive APL Program 27 2. Interactive BASIC Program 28 Figures: 1. Bayesian Analysis on the Likelihood of a Major North Viet- namese Military Offensive. 11 2. Bayesian Analysis on the Likelihood of Major Hostilities in the Middle East in the Next 30 Days 11 3a. A Bayesian Analysis of the Likelihood of Sino-Soviet Hostilities 11 3b. Individual Assessments of Sino-Soviet Tensiori 11 iv FOR OFFICIAL USE ONLY Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Releas?,aoup/A3L4Filkl-Tposippo269Rooi 100080001-0 SUMMARY Within the Intelligence Community, the term "Bayes" has been bandied about from time to time over the past decade, as one of a number of techniques to supplement traditional analysis. There have been frequent assessments of the Bayesian technique and infrequent applications of it to the analysis of an intelligence question, usually on an experimental basis. During 1974 and 1975, the Office of Political Research used Bayesian analy- sis in three separate studies dropping the qualifier "experimental" after the first?and the office intends to continue its application in the future, when such an approach is relevant. The technique produced a style of report which conveyed much information in a very concise format. It is our view and the frequently-voiced opinion of others that this represented an advance in communications over traditional methods of reporting. This handbook will attempt to explain the use of the Bayesian approach as a forecasting tool for intelligence. It is written for potential users of the technique, both to make OPR's experience available to others, and to en- courage and facilitate its proper use. It is meant as a self-help guide on how to conduct such an exercise. Our experiences are intended to be instructive in a general way; the techniques described represent a particular solution to problems of political intelligence, but could be adapted with minor or major changes to other intelligence and bureaucratic situations. The presentation explains the rationale and format of the three OPR analyses, the procedures developed to administer them, and the benefits and problems of the Bayesian method. Neither the analyses nor this handbook wallows in methodological soul-searching; in both cases, the purpose is utility rather than perfection. There will be, of necessity, some examination of the mathematical formulae involved. The bulk of this handbook, however, is a catalog of the practical administrative details of such a project. To avoid "perfection of means and confusion of goals [which] seem to characterize our age" (Einstein) the need for careful definition of the problem and careful project design is repeatedly stressed. FOR OFFICIAL USE ONLY Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For ReleaseFg0p2iMr3/:RAt7INP(8269,0269R001100080001-0 BAYES 1. What are its Capabilities and Benefits? The task of intelligence is to answer questions about the capabilities and intentions of various international actors; information may be abundant or meager, but it is usually incomplete or incoherent. The main feature of Bayesian analysis is that it provides a rational, consistent, and objective process whereby many apparently unrelated facts can be combined to produce an overall assessment. If an intelligence problem can be formulated into a set of questions which can be answered in probabilistic terms how likely one of a set of outcomes is?it may lend itself to Bayesian analysis. There is no magic and no inherent wisdom in Bayes. In simplest terms, the Bayesian technique consists of a statistical formula and a procedure for its use. (Statistics is a discipline which allows one to deal with uncertainty in an organized fashion without being vague or imprecise; Bayesian statistics differ from classical statistics in that they allow an analyst to use his own expert understanding of a situation along with probabilistic judgments based on evidence.) The Bayesian technique is an organizing device which allows an analyst to assign probabilities to the likelihood of various carefully drawn scenarios about an intelligence problem, and thereafter to evaluate fragments of evidence in terms of those hypothetical scenarios. The Bayesian formula then aggregates these numbers mathematically, rather than by the inductive logic of an analyst, into an overall set of probabilities. This has several advantages: ?More information can be extracted from the available data because the technique allows each piece of evidence, central or marginal, to add its weight to the final assessment in a systematic way; thus, a number of small items can outweigh a large one, and the probabilities are not at the mercy of the most recent or most visible item. ?The procedure provides a reproducible sequence of steps for arriving at the final figures; a disagreement among analysts can thus often be seen to be a disagreement over the meaning of certain items of evidence rather than an unresolvable difference of opinion. ?The formulation of the questions forces the analyst to consider alternative explanations of the facts he sees, thus loosening the bonds of 1 FOR OFFICIAL USE ONLY Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 established opinions. In other words, he is asked to look at how well the evidence explains possible scenarios other than thi one he has already decided is most likely. The use of quantified judgments allows the results of the analysis to be displayed on a numerical scale, rather than through the use of terms like "probable," "likely," "unlikely," or "possible." Also, the work of a number of analysts can be arrayed in graphic form, with ranges and averages. The formal procedure has been shown to be less conservative than analysts' informal opinions, and to drive probabilities away from 50-50 faster and farther than the analysts' overall subjective judgments do. This is often initially unsettling for the analysts, but mot have admitted that they later agreed with the assessment. The mere fact that a team of experts is asked to assess periodically the evidence on an important intelligence question provides managers of intelligence production with a degree of assurance that the question is indeed being monitored. Before embarking on their initial Bayesian projt-ct, the OPR analysts assigned to coordinate the exercise reviewed the Agency's previous experi- ments with Bayesian analysis, read the available literature on the theory and application of Bayes, and consulted with individuals familiar with the technique both in and outside of the Agency. The earlier work of Jack Zlotnick and Charles Fisk, and the advice of Gordon Sliahin, in various offices of the Agency, provided especially useful guidelines for the design of these projects. The rule of Bayes is an established statistical tethnique with a variety of applications in the social sciences and industry, but its application to intelligence questions is both more complex and less precise. The political, economic, strategic, and social events of the world are imperfectly under- stood and difficult to measure. Hence, to use a technique like Bayes it is necessary to turn to expert judgments expressed quantitatively. The values assigned by expert analysts are, of course, approximate, but they provide a rough basis for comparison and analysis. OPR's experience suggests that it is relatively easy to induce analysts accustomed to quAlitative expressions of probability to shift to numerical assessments. An ever-present danger, however, is the tendency to attribute more precision to the numbers than is warranted, and it should be stressed that the numbers are always only Approximations. 2. When is it Useful? The starting point for any investigation must always be the careful formulation of a relevant question. Before discussing the rule of Bayes further or the procedures developed to work with it, it is important to remember several overriding substantive considerations. 2 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 What is the nature of the intelligence problem; what is the question and what data are available? In the intelligence process the nature of the problem must dictate which methods and approaches will be most successful. There are many techniques for enhancing traditional analysis, and it is possible that something other than Bayes would be more appropriate to the level and complexity of the issue involved; trend analysis, an econometric model, or a Delphi exercise might, for example, be more relevant. Many of the more rigorous techniques require good data, and to the extent that a problem becomes more data-rich, it may actually cease to be an intelligence problem. There is also a certain delay involved in the genesis of a project, which might preclude its use in fast-developing situations. To be the type of question which is susceptible to Bayesian analysis; ?It must lend itself to formulation in categories which overlap very little, such as war versus no war, or development of a complete nuclear capability versus development of a peaceful nuclear capability versus no nuclear development. Bayes is useful only when the question is expressed as a specific set of outcomes; thus, the Bayesian approach would be useless as a predictor of something as amorphous as future Middle East relations. The question would have to be re-cast in terms of specific alternatives, that is, a set of more or less mutually exclusive scenarios of Middle East develop- ments. In this process, however, there is a danger that the question will be so simplified that the answer would be neither relevant nor interesting. ?There must be a fairly rich flow of data at least peripherally related to the question. For example, in the nuclear example above, data on all related materials and processes would be relevant. If data are sparse, the technique is very sensitive to each piece and is less reliable although it may still be useful. ?It must be the type of activity which produces preliminary signs and is not a chance or random event. For example, it would be fruitless to attempt to predict whether Giscard would be killed in an automobile accident. Bayesian analysis can measure only preparations for and indications of the hypothesized outcomes. For whom and at what level is the question important? This will have a direct bearing on the level of resources which can be commanded for an analytical effort. The consumer receives a product which displays results concisely and graphically, and unless he or she reacts negatively to such displays, this can be an important consideration. Each production office would have to weigh the benefits and costs of the method?on one side the rigor, reproducibility, and sex appeal of the method, and on the other the need for education, monitoring, and coordina- tion of a number of analysts who may not find the method immediately congenial. Having considered the above questions, one should ask whether Bayes is applicable. 3 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 HOW TO INITIATE A BAYESIAN EXERCISE 1. Flowchart of Setup Procedure Customers Ask a Question Coordinating Other Office Offices Define the Question Arrange for Participants Set Time Span and Frequency Design the Format Contribute Analysts Hold Initial Meeting Arrange Practice Runs Advise on Graphics and Printing 2. Defining the Scenarios Once a question has been selected or assigned as a candidate for a Bayesian analysis, the first step is to formulate it into categories which are more or less mutually exclusive, that is, which cover the relevant possibilities and overlap as little as possible. It is imperative not to let the question run away with the formulation; that is, it must not be chosen simply because it can be answered using the Bayesian technique. If so,-it may not necessarily be the question that the customer really wants answered. This formulation is easiest when there is a single event which one is interested in predicting. For example, the actual formulation of the first OPR study, in what are termed "scenarios" or "hypotheses," was: A. The North Vietnamese will launch a major military offensive (defined as a countrywide offensive on the scale of the Tet 1968 or March Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 1972 fighting, or attacks on a similar level generally confined to one or two military regions) before the end of June, 1974. B. The North Vietnamese will not launch such an offensive. When this formulation as a dichotomy can be done, it allows a simplified assignment and computation technique, as explained later. Often, however, the problem is not so simple, and to formulate an intelli- gence problem as an either-or question would be to strip it of its complexity and interest. For example, the second project was designed to monitor the Sino-Soviet conflict, and the following possible scenarios were identified: A. The Soviets will undertake a nuclear strike against Chinese strategic or nuclear targets within six months. B. The Soviets will launch a large-scale conventional attack against China within six months. C. The Soviets will launch a localized cross-border attack, with limited objectives, on a scale larger than the 1969 incidents within six months. D. The Chinese will launch a localized cross-border attack, with limited objectives, on a scale larger than the 1969 incidents within six months. E. One or more minority groups on either side of the border will revolt, following instigation by the opposite side within six months. F. Neither side will undertake any of the above types of hostilities within six months. 3. Participants The second step is to decide the number and variety of participants that are needed to carry out the Bayesian analysis. OPR has been able to enlist participants in its various Bayesian projects from a wide variety of offices within CIA, analysts who concentrate on economic, strategic, politi- cal, and geographic affairs, or who analyze propaganda or imagery, or who follow the broad spectrum of current events. For Intelligence Community projects, participants have been enlisted from the Army, Navy, Air Force, DIA, NSA, INR, the Intelligence Community Staff, and the National Intelligence Offices. As stated before, the importance of the question will affect the number of persons willing or even available to give time to this type of effort. The number of participants in our reports has varied from six to thirteen, and although a larger number would clog the logistical mechanisms and clutter certain types of graphs, there is no theoretical limit. Nor is there any reason the analysis could not be done by one person. Depending on the purpose of the exercise, it is possible for an individual, an office, an agency, or an interagency group to carry out a Bayesian analy- 5 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 sis. OPR, in its projects thus far, has had the benefit of working on topics of high-level interest, with a specific consumer demand that this technique be applied. On a topic of lesser moment, or where there is little visible en- couragement from above, it would still be practical and quite useful for a small group or even an individual to undertake a I3ayesian analysis. The scale on which OPR has been able to carry out its projects is not essential to a useful application of the technique. Monitoring an intelligence situation is most frequently done by an office or an individual within an office, and the Bayesian technique can be applied just as profitably on that level, without the overlay of logistics and cartography. In a scaled-down procedure, the logistics of the exercise would diminish to the point where everything but the selection and evaluation of evidence could be recorded, stored, processed, and printed via a computer terminal. A member of OPR's Analytical Techniques Group could provide assistance in setting up the exercise and training the participants, after which another individual could administer the project. A small in-office Bayes could serve well to support or supplement conventional analysis, and would be better suited to following a fast-moving situation than the elaborate productions OPR has designed. It might even serve as the basis for a later full-scale project. After the exercise has been initiated, a single analyst or a small number of analysts in a single office could draw up a list of individual items of evidence; probabilities could then be assigned to each item and consolidated by computer or by hand. This could be done on any schedule: weekly, daily, or whenever new evidence appears. Readouts of revised probabilities would be immediately available at any time, and the results could be circulated via narratives, hand-drawn graphs, or computer-drawn graphs. 4. Time Span and Frequency The third step is to determine how long the exercise should run and how often reports should be issued. This depends on the volume of data and the expected speed of development. For a data-rich question of great urgency and some probability, it may be necessary to run the exercise once or twice a week over a period of many months. Lacking any of these attributes, however, the problem may lend itself to a more leisurely schedule. OPR's reports have been issued at intervals of one week, two weeks, four weeks, and six weeks, and have lasted for periods of six months to one year. The reporting period can be changed in the course of an exercise, as the question becomes more or less urgent, without upsetting continuity. Another consideration which has great practical impact upon frequency is the cost, in man-hours of analysis and coordination, of running a large project. As developed by OPR, these studies have kept to a minimum the time required of analysts, though this was achieved by a large expenditure Approved For Release 2002/08/3 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 of time on the part of the coordinators. Whereas each coordinator spent approximately two full days for each periodic assessment, each analyst spent only one-quarter to one hour. Even this large investment of coordina- tion time might be decreased by a greater use of computer terminals and an interactive data-gathering program, already developed but untested. 5. Designing the Format Finally, the format for presenting the results should be designed. If the exercise is of sufficient import to be published, that is, if there is a known or perceived consumer demand for it, then the design of the graphic presenta- tion is a very important step in communicating the analysis. Any carto- graphic aid, either for initial design or for continuing support, should be enlisted early in this process. If there is no perceived high-level demand for such an analysis, meaning that it would be done for the benefit of working-level analysts, a less formal and expensive presentation would probably be more appropriate and just as effective. 6. An Initial Meeting It will be necessary to have an initial meeting of participants at which the rationale, format, and procedures are explained. It is also convenient at this time to collect the following information, some of which will be used for the list of participants in the publication: ?Name as it is to appear in print; ?Title and office name; ?Number of years in office; Number of years on subject; ?Office telephone number(s); ?Tube station or LDX address; ?An alternate analyst to stand in during vacations; ?Starting probabilities for each hypothesis. The latter numbers should represent the participants' current overall feeling for the probability that each event will occur as stated. (See the section on calculations for a further description of the starting probabilities.) 7. Practice Runs It will be useful to arrange one and perhaps two practice runs, to familiarize the analysts with the routine, and also to work out the logistics, the cartography, and the printing of the report. The wording of the hypoth- eses should be reviewed at this point, to assure that the proper question is being addressed. Although the basic design of the exercise and the formula- tion of the questions will remain fairly constant once the formal publication begins, the work of familiarizing analysts with the routine will never 7 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 completely end, as occasionally someone will forget- to cite sources or to return contributions on time. All of the foregoing constitutes the design and definition of the project. Careful attention at this stage is extremely important to the subsequent conduct of the exercise. It is also the point at which _basic questions about goals and intentions must be resolved. Much of it can be handled by one person--preferably the one who will later coordinate the exercise--in consultation with both consumers and participants. THE OPR BAYESIAN ANALYSES The three Bayesian analyses run by OPR in 1974-75 relied heavily for their success on supplementary techniques relating to data-gathering, presentational format, and administration. The Ba3esian aspects of the reports formed the core of the analysis, around which a number of other carefully-tailored aspects were draped. 1. More than One Analyst One of the central features of the studies was the use of a group of analysts rather than a single expert. This more than anything else influenced the data-gathering process, the format of the publication, and the actual production procedure. The reasons for this decision were: --To bring to the exercise a range of expertise beyond the skills and experience of any single analyst; --To supply a richer mix of evidence on the questions by asking each analyst to contribute anything he or she considered important. Most political, strategic, or military intelligence problems involve such varied inputs as propaganda analysis, photographic interpretation, and logistic matters. (The evidence was then shown to all :4Articipants without identifying the contributors); and --To provide a balance of expertise in which the effects of organizational and individual biases are minimized. It is well-known that different analysts will tend to place greater reliance on different types and sources of intelligence. The consolidated list provided an opportunity for each analyst to call an item of evidence to the attention of his colleagues, and assured that the participants would consider each item explicitly before judging its relevance. To avoid the unpredictable and oft-decried effects of group dynamics, however, each analyst worked on the probabilistic assessments individually, and relayed them to the OPR coordinator. This also avoided the time wasted in a group meeting and the problem of assembling the group on a convenient schedule. 8 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 2. Printing the Evidence Each periodic report contained the pieces of evidence identified and used by the participants, along with only a paragraph or two on the principal trends during the period. No attempt was made to formulate or coordinate a lengthy textual analysis of the situation. This allowed the reader - To see the basic evidence rather than just a summary and hence to better understand the analysts' assessments; To make his own direct assessments if he so desires, or just to keep up with the topic by viewing the evidence regularly; and ----To keep a concise record of the situation. 3. Graphs and Visibility The ability to portray the results of the analysis graphically was one of the strongest arguments for using a quantitative method like Bayes, and the graphs in the publications have been well-received. In each study, the probability of the hypothesized event, usually "how likely are major hostilities within a certain time period?" was immediately visible on a broken line or bar chart. This conveyed much information at a glance, especially with the broken-line charts, which illustrate trends far more concisely and vividly than do words. On both the broken-line and bar charts the range of estimates around the central measure showed clearly and concisely how much disagreement there was. Different formats were tried in the various reports, each demonstrating certain advantages and disadvantages. Arrayed on the foldout page are examples of the graphs and combinations of graphs (with representative values) used in the various reports. As a rule, the aim was to make the graphs as prominent as possible, while trading off the amount of information against the degree of clutter, the use of color against the cartographic and printing constraints, and the size and placement of the graphs against the requirements of the overall publication. In figure I, from the Vietnam study, each analyst was represented by a separate line showing his or her position in successive weeks. This extensive detail was possible because there was only one positive question to be an- swered and there were only six analysts participating. (The negative scenario could of course be omitted because it was the mathematical complement of the positive one.) The assessments were represented by six lines on a single graph, plus one for the average (arithmetic mean). For the Middle East study with four scenarios and thirteen analysts, this format was reduced to four charts?one for each scenario each of which displayed the median, the interquartile range, and the range, because the full representation would have been confusing, time-consuming, and 9 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 expensive in printing. One of those four charts is represented in figure 2. This format does have the disadvantage, however, that individuals are not identified; consequently a full-page supplementary graph was supplied for that purpose, similar to figure 3b, on which each analyst gave a hunch judgment of hostilities in the next ninety days. This supplementary graphic was totally non-Bayesian. The provision of individual lines allowed each analyst to be visibly identified, as in the Vietnam reports, and thus to advocate a certain position if desired. This visibility was pleasing to the analysts and interesting for the readers, with the result that there was little if any pressure on the analysts to try to manipulate their positions on the other graphs (see the section on manipulation). In figure 3a, the assessments of thirteen analysts for six scenarios of Sino-Soviet hostilities were condensed into a bar graph which showed, for each scenario, the average of the thirteen probabilities along with the lowest and the highest values assigned. To provide continuity from period to period and to illustrate trends, a single broken-line chart was added which showed the progression of the average figures over time. In addition, in- dividual lines were provided on a single supplementary chart, figure 3b, which was an attempt to quantify and measure a related problem, the "level of tension" between the two countries; this was also a non-Bayesian assessment. Each analyst was asked to gauge his overall feeling of the level of tension between the two countries on a pre-arranged historical scale, independently of the chances for hostilities. Gossamer though this concept is, the graph performed well as an indicator of changes in the atmosphere between the two countries; once each individual chose his or her position on the chart?as a hawk, a dove, or a middle-of-the-roader ?the ups and downs of the graph showed clear trends from month to month, and there was no' need for the analysts to agree on a single number. 4. Identifying the Participants The final presentational technique was the listing of all participants by name, and their identification on at least one graph with a trend line of assessment. This visibility, as mentioned, helped to motivate participants, and in the latter two studies provided them a direct medium for expressing themselves in accord with their opinions on the issue. 10 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 c Probability 'Ye Figure 2 10 December 1973-30 June 1974 Current status of the Assessment 100 90 80 70 60 50 ????? mma ?0`4%.* 000 rtrft, 40 30 20 10 0 ....... ......... ? ? ? ? ? 41) ? ? ......... .. .... ............ . ...... 10 18 27 Dec '73 .. Participants: 10 17 24 Jan '74 ? Central Tendency of the Group's Estimates Bayesian Analysis on the Likelihood of Major Hostilities in the Middle East in the Next 30 Days Assessment as of 9 April Likelihood of any Major Hostilities, Regardless of the Initiator: % Probability 100 12 26 12 26 Feb Feb Mar Mar ? ? ? Lowest estimate by any Apr 9 Apr Average of all analysts , Highest estimate by any Figure 3a Apprae8 4ittrAitIlb Imrcisualitbe tilekentiseedadallesiamktectgoe0Mildes Before 1September 1974 Current Status of the Assessment -- as of 1 March STAT @TAT A The Soviets will undertake a nuclear strike against Chinese strategic or nuclear targets. B The Soviets will launch a large- scale conventional attack against China. C The Soviets will launch a localized cross-border attack, with limited objectives, on a scale larger than the 1969 incidents. D The Chinese will launch a localized cross-border attack, with limited objectives, on a scale larger than the 1969 incidents. ? One or more minority groups on either side of the border will revolt, following instigation by the opposite side. F Neither side will undertake any of the above types of major hostilities. ? Lowest estimate by any of the participants Probability % 10 20 30 40 50 60 70 80 90 100 a Average of 1111 Highest estimate by any of all estimates the participants A Time Chart Showing the Movement of the Group's Averages for Probability % Hypotheses A Through E (as described above) 10 9 DEC JAN FEB MAR APR MAY JUN Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 566534 Figure 3b Individual Assessments Of Sino-Soviet Tensions Level of Tension 100 I 90 ; 80 70 is defined as the level of tension 70 , in August 1969 10 is netted as through 1958 566535 60 50 .. ? ? .... ? .... ..................... ?00.1 ???? 40 or) 20 DEC JAN FEB MAR APR MAY JUN The points on this chart were ccosee by each participant on an intuitive basis usino the rouoh ouidphoos show, on the chart ? Average Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 HOW TO MANAGE AN ON-GOING BAYES 1. Flowchart of Periodic Procedure: Analysts Collect and Submit Evidence Evaluate Items 2. Routine Procedure Coordinator Services Consolidate Evidence Compute Revised Probabilities Revise Graphics Compose Report Cartography Printing Once an exercise is started, the procedure is periodic and recurring. On the first day of each period, each analyst submits the items of evidence he has seen since last round relating to the question. The submission is in the form of one or two sentences summarizing the item, along with the date, source, and classification, e.g., Statement in the Soviet journal, Problems of the Far East by Yumjaagiyn Tsedenbal, First Secretary of the Central Committee of the Mongolian People's Revolutionary Party, alleging that groups of Chinese soldiers violate the Mongolian border, fell trees, start forest fires, and herd diseased cattle into his country. (Foreign Broadcast Information Service Daily Report, 10 January, UNCLASSIFIED) 13 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 This can best be done on preprinted forms. The choice of data is left entirely to the analyst, who should include anything relevant, and exclude what can be judged to be irrelevant. There will be some overlap and considerable diversity in what is submitted. Later the same day, the coordinator to whom the data is sent consoli- dates the items, resolving differences of wording, emphasis, and meaning, and distributes the complete list of items to all participants. At the coordina- tor's discretion, one standard item may be added which asks the participants to evaluate the volume of data itself. This is explained later in the section on weaknesses of the method. By the following day, the analysts working individually should evaluate the items, as explained below, and return their numerical assessments to the coordinator. 3. The Calculations There are two ways in which the Bayesian assignments and subsequent calculations can be made, a basic and a simplified way. They are both based on the same mathematical relationships, but one works with ratios of prob- abilities and the other with probabilities directly. The simplified is restricted to use with questions formulated in an either-or fashion, such as war or no war. The. basic method, however, can be used in all cases, for either-or questions and for multiple scenario questions. Both methods require a set of starting probabilities which represent each analyst's best judgment of the situation at the start of the exercise, and the subsequent evaluations build on these. A. The Basic Method For example, using the basic method and the following scenarios, A: Large-scale aggression by country X within six months B: Small-scale provocations by country X within six months C: No planned hostilities within six months an analyst might assign a set of starting probabilities to A, B, and C of 25%, 25%, and 50% likely at the start of the exercise. (The assignment of these probabilities is done entirely subjectively, although techniques exist to refine what a person really means by these numbers.) Note that the sum of these probabilities equals 100, because the scenarios supposedly cover all possible events. This is the only time the analyst will have to assign numbers which total 100. In the course of the exercise, the analysts will evaluate each item of evidence as it relates to the scenarios, formulating the question as "how likely (0-100%) is it that this item would occur if A is planned?", "how likely is it that this item would occur if B is planned?", and "how likely is it that this item would occur if C is planned?" The peculiar formulation of the 14 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 questions is dictated by the nature of Bayes' theorem, which depends upon conditional probabilities as explained in the first appendix. The set of probabilities assigned for each item need not add to 100. In this case, the wording of a very bellicose speech by the leader of country X might be 90% likely if that country were contemplating outright aggression against its neighbor, 90% likely if small-scale border provocations were planned, and 80% likely even if nothing were planned. This procedure is followed for each item independently, without refer- ence to the other items. The only exception is the discretionary final item, which refers explicitly to the other items: "How likely is it during this period that this volume of information (no more and no less) would occur if A is planned?", et cetera. The mathematics of revising the estim'ates, once the coordinator receives them, is straightforward though tedious. It is recommended that an inter- active computer program be used to save time and reduce arithmetic errors. The calculations are performed for each analyst separately; in other words, analyst A's set of probabilities from the previous round are updated by the conditional probabilities for each item in turn, resulting in a final set of revised probabilities for this round, which are used for the publication and as starting probabilities for the next round. The same procedure is then followed for each other analyst. The general formula for determining the revised probability for each new item is P(S111) ? P(Si) x P(I/Si) I(P(Si) x P(1/S,)) where P(S)?is the starting probability for scenario P(I/Si) is the conditional probability of an item P(Si/I) is the revised probability of scenario i m?is the number of scenarios For a full elaboration of the mathematics, see Appendix 1; Appendix 2 contains the code for interactive programs in APL and BASIC which will perform these calculations. The following is an example of the interaction using the starting probabilities assigned above, the probabilities assigned to the hypothetical speech, and two other sets of probabilities chosen for illustrative purposes. (The user's entries start in column 5, the computer's responses in column 1): BAYES THIS PROGRAM CALCULATES EVENT PROBABILITIES BY THE BAYESIAN FORMULA. YOU WILL BE ASKED TO ENTER A SET OF PRIOR PROBABILITIES, THE NUMBER OF ITEMS OF EVI- DENCE, AND THE ITEM PROBABILITIES. 15 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 PRIOR PROBABILITIES FOR THIS ANALYST, IN ORDER. 25 25 50 HOW MANY PIECES OF EVIDENCE? 3 PROBABILITIES FOR ITEM 1, IN ORDER. 90 90 80 INTERMEDIATE RESULTS: 26.47 26.47 47.06 PROBABILITIES FOR ITEM 2, IN ORDER. 90 90 90 INTERMEDIATE RESULTS: 26.47 26.47 47.06 PROBABILITIES FOR ITEM 3, IN ORDER. 60 20 10 INTERMEDIATE RESULTS: 61.36 20.45 18.18 P(Si/I) = 61.36 20.45 18.18 (a very small change) (no change) (a significant change) It should be noted from perusing the above, as one of the lessons to be learned early about the diagnosticity of evidence, that the items which have the greatest impact upon the final values are the items with the greatest spread of numbers. Item 2 had no effect on the probability of the scenario ,because, although it had large probabilities associated with it, the item gave no indication of what scenario was more likely; it was not diagnostic. The assignment of three nineties implied that the item was very likely to occur no matter what scenario was true. Item 3, however, was a sensitive indicator, because it was very unlikely if the second and third scenarios were true; the effect of item 3 was to depress significantly the revised probability for those two scenarios. This concept of diagnosticity can usually be conveyed to the participants by showing them the results of the calculations for a round or two. B. The Simplified Method Using the simplified method when the question is a yes-no either-or formulation reduces the effort for both the analyst and the coordinator. It is even feasible to perform all the calculations by hand. In this procedure, each item is evaluated as before and assigned two numbers?one for each 16 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 hypothesis?but they are reported as a ratio, of one over the other. Identical results are obtained whichever hypothesis is in the numerator, but the practice must of course be consistent. For example, if an item is only 10% likely if A is true, and 90% likely if B is true, and the ratio is to be reported as A over B, the evaluation for the item would be 10/90. The assignment procedure is followed for each item independently, and the mathematics of revision is then considerably simplified. For each analyst the starting probabilities are Formed into a ratio (A-25% and B-75% would become 25/75) and simply multiplied against all the item ratios consecutively. Since multiplication is commutative, this can be done in any order, and much cancellation can usually be done if the whole set of ratios is considered. For example, 75 96 $6 50 3750 15 ?x?x?x ?x- 25 Or 4-6 96 10 250 1 The resulting figure is the revised ratio, and should be kept as the starting ratio for the next round. The ratio must be converted to probabilities (percentages) by the follow- ing simple method for reporting purposes: add the numerator and denomina- tor together and divide the total into 100. In the case above this produces 100/16, or 6.25. The result of this division can be used to multiply both the numerator and the denominator, producing the percentages for A and B, respectively. Again in this case, A = 15 x 6.25 = 93.75, and B = 1 x 6.25 = 6.25. Remember that for the next round the starting number is the ratio, not the percentages. This procedure can be further simplified by reducing the numerical scale. If probabilities are only allowed to be multiples of 10, such as 10, 20, 50, and 90, the calculations become much easier. This level of precision is thought by some to be as much as an untrained mind can estimate, al- though many analysts have demonstrated their ability to use the full 100-point scale. Also, because 10/90 is the same as 1/9, the ratios can be assigned using just the numbers from 1 through 9. Whichever method is used, the basic or the simplified, each analyst will end up with a set of probabilities, which can be reported in a number of ways, as described in the section on graphics. 4. Problems in Mid-course The most frequent problem in conducting a Bayesian exercise is one common to any group project: the management of a group of individuals, some of whom will make unusual demands and require special attention. There will also be occasional logistics problems, and delays in cartography and printing, for which there is no ready solution. 17 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Holidays, vacations, and dropouts are another problem which should be anticipated. Ideally each participant should designate an alternate who would be familiar with the exercise and could take over when necessary. When that is impractical, the exercise can be run with a diminished number of participants; in that case, the individual's line can either be left out or extended without change, and any averages can of course be calculated on the basis of a smaller number of analysts. Finally, there is always the possibility that the basic design of the exercise or the formulation of the question will be found lacking, but that is the reason a practice run or two is recommended. 5. Terminating and Evaluating a Bayes In some exercises the scenarios specify a fixed deadline at which time the event either has or has not taken place, such as the OPR report on the likelihood of a North Vietnamese offensive before July, 1974 (the end of the dry season). It is clear that such a project would be terminated at that time, if not before. Certain other questions will be formulated to look at the probabilities of various scenarios within a continuously-moving time frame, such as the report on the likelihood of Sino-Soviet hostilities within six months. A project such as this may be terminated at any time, or extended indefinitely. The Bayesian method upon completion results it) an archive of evidence, evaluations, and predictions which lend themselves to various forms of evaluation; OPR has undertaken only the grossest measures of performance, however. A wealth of material lies untouched for rating collection systems, analyst bias, the use of intelligence in the community, or?that most dubious measurement of all analyst performance. The main criterion for evaluation is the accuracy of prediction, by individuals and by group, although this may not be as straightforward as it seems. Because of the myriad variables in the prediction equation, an event may occur which was only 10% probable the day before, or an event which was scheduled to take place may fail to materialize Thus there have been times of great uncertainty during our reporting periods when the probability of a scenario rose, only to fall back again. Did this mean that the high prob- ability of the event occurring was somehow in error? In retrospect, it seems to mean that the event could very well have occurred at that time if other factors had coincided; the evaluation really cannot be considered "wrong." Generally, OPR's studies in 1974 successfully predicted non-events, that is, they showed that the evidence did not support any of the positive scenarios, and none of them occurred. In this case, the only point to be noted is how early in the exercise the evaluations moved away from an indeterminate figure toward a strong probability of no change. It has been our experience that the Bayesian calculations show this movement earlier than the analyst's own judgment would. 18 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Until one of the positive scenarios actually occurs during the course of a Bayesian exercise, there is no way of knowing the predictive value of the technique. When the positive event does take place, it will be possible to conduct a much more searching evaluation of the Bayes procedure. What were the earliest indicators? What evidence was missing, overlooked, or misperceived? When did the trend lines signal a significant alteration in the situation? How did the Bayesian assessment compare with other intelligence assessments? INHERENT WEAKNESSES AND PROBLEMS OF THE BAYESIAN TECHNIQUE 1. Limited Applicability The first and foremost reservation in the use of the technique, as noted earlier, is that it is applicable only to certain types of questions. They must be capable of definition as a set of fairly distinct outcomes or hypotheses. Also, the procedure as developed is cumbersome enough to discourage its use for questions on a crisis schedule, although it is conceivable that the technique could be so adapted, using computer terminals and greatly simplified presentational techniques. 2. Data Problems There is the problem of identifying which evidence is relevant? whether certain peripheral items should be included. And if included, whether they should carry less weight than other items. The OPR exercises have delegated that decision to the analysts. After all. they are experts, and their frequent disagreement over items shows that objective measures of relevance would be virtually impossible to devise. Little editorial judgment is imposed by the coordinator in the process of consolidating evidence, and any item which appears to be even peripherally relevant is included for evaluation. Nevertheless, each analyst is then allowed to ignore any item he or she considers irrelevant. This gives the participants great leeway over what they rate, but insures that they at least see the evidence and make an explicit decision whether it is relevant. Furthermore, if two or more items are seen as overlapping, the participant is asked to rate only one of them. Related to this is the problem of source reliability. Although some methodologists have suggested that each analyst assign a numerical rating of source reliability along with each item, to be incorporated into the calculations as a weight, the OPR studies have avoided placing this extra burden on the analyst by requesting that the probabilities reflect how much 19 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23: CIA-RDP86600269R001100080001-0 faith the analyst places in each item. As mentioned earlier, the effect of an item will depend on the spread of probabilities assigned, and items of greater salience and reliability will automatically be assigned a greater range of probabilities, if the analyst understands the process and rates items thought- fully. The problem of discerning items of disinformation or fabrication is left to the judgment of the participants; indeed, one of the benefits of the Bayesian process as a group exercise may be the opportunity to identify and sift out such items. The problem of negative evidence is another point which bothers strict methodologists. This refers to the fact that the absence of any positive evidence may in itself be highly indicative, and the journalistic bias toward reporting events rather than non-events compounds the situation. That is, we tend to get news only of events or changes, whereas the fact that the status quo is being maintained can be quite significant. This problem has been recognized?as discussed in previous sections? and is partially satisfied by including an item such as "how likely is it that exactly this volume of evidence would occur (and be seen) if hypothesis A is true?", et cetera. 3. Problems Over Time There are indisputable difficulties in the use of the method in a project continuing over many months. First of all, the questions probably require some reference to a time period (explicit or implicit), that is, an event to occur within the proximate month or year. And as the project continues, the timeframe must either contract or move forward. Contraction is illu- strated by OPR's first reports, which investigated the probability of an event before a certain fixed date. In this case, the passage of time and the reduction of the period remaining may itself be of significance. The administrator may wish to include an item to that effect for evaluation. Moving the timeframe forward is illustrated by the second and third OPR studies, which looked at the probability of events "within the next X months." This "sliding window" approach brings up the problem of retaining or discarding data which was evaluated months earlier with regard to an earlier frame of possibilities. One solution, adopted by the second study, which covered a year, is to drop earlier evaluations, maintaining only the most current three months of evidence multiplied against the original probabilities. The starting probabilities were also updated at intervals. 4. Problems with Numbers There are also two numerical problems. The first is that multiplication by zero must not occur; once a probability becomes zero it can never recover. Thus any evaluation of zero probability must be replaced by a very small number. The second is more profound, and is the problem of how individual 20 Approved For Release 2002/08/23: CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 participants handle numbers, especially probabilities. It has been OPR's experience that some analysts think easily in probabilities, others have to work at it each time, and a few need constant attention and retraining. It is essential that the process of assigning probabilities or ratios be reviewed on an individual basis when necessary. We have found that the administrator can recognize when a participant is uncertain of the procedure, and one of the most effective teaching devices for sharpening the ability to assign probabilities is to show him or her the effects of each of the evaluations on the revised probabilities. 5. Manipulation Finally, there is the problem of conscious manipulation. An analyst may assign his probabilities in a manner which reflects a pre-determined goal rather than unbiased judgment This is especially true where the Bayesian probabilities affect the individual trend lines, as in the first study. Solutions to the problem could include disciplinary action or expulsion from the exercise, but the later OPR reports to some extent circumvented this problem by providing another outlet--the supplementary graphs?for the expression of individual opinions. 21 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 APPENDIX 1 THE STATISTICAL/MATHEMATICAL BASIS FOR THE CALCULATIONS The statistical formula which underlies this method first appeared in an essay by the Reverend Thomas Bayes in 1763. It is a tool of statistical inference, used to find the probability that an observed event was caused by one source rather than another. 1. Rationale The purpose of Bayesian analysis is to determine the probability that each scenario (hypothesis) is true, given that certain items of evidence have been seen. The Bayesian formula allows this to be calculated in an indirect fashion, from the probabilities that the items would be seen if each scenario were true. In our application of the method, analysts assign subjective values to the latter probabilities (that items would be seen), and the OPR coordinators calculate the former (that the scenarios are true) by applying the Bayesian formula. Only one other thing is necessary: a beginning set of probabilities for the scenarios, which the analysts supply for the initial round only, based solely on intuition. (This initial assignment, and the subsequent assessments, could be refined by various academic techniques, but we have not risen to that level of perfection or analyst rapport.) 2. Derivation The formula itself is derived from the following basic statistical identities and transformations: S?represents a scenario, or hypothesis, such as "The USSR is planning to launch a nuclear attack on China." I----represents an item of evidence, such as "TASS reports that the Chinese are deploying nuclear missiles, thereby threatening world peace." P(S)--is the probability of a scenario being true. P(I)--is the probability of an item occurring and being seen. 23 Approved For Release 2002/08/23 : CIA-RDP86B00269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 S n I?is the intersection of S and I, that is, both S and I. P(I/S)?is the conditional probability of I given S, that is, that an item I .would occur if a particular scenario S were true. P(S/I)?is the conditional probability of S given I, that is, that a scenario would be true if an item were seen. 1. P(S n I) = P(I n S) commutativity of intersection P(I n S) 2. P(I/S) ? P(S) definition of conditional probability P(S n I) P(I n S) 3. P(I/S) ? from 1 and 2 P(S) P(S) 4. P(I n S) = P(S) x P(I/S) transposition of 2 At this point, we define m mutually exclusive scenarios, Si, of which one must occur. That is, EP(S1) =1 - 1 5. P(I) =P(I n SI) +P(I n S2) + . . . +P(I n Sm) 6. P(I) = (P(Si) x P(I/S1)) +. . .+ (P(Sm) x P(I/Sm)) 7. P(I) = x P(I/Si)) by definitions from 4 and 5 restatement of 6 The desired probability is the probability that each Si is true given that a piece of evidence has been seen, or, using this notation, P(S/I). 8. P(Si/I) ? 9. P(Si/I) ? P(I n Si) P(I) P(Si) x P(I/S;) P(I) P(S1) x P(I/Si) 10. P(Si/I) ? m E(P(S) x P(I/Si)) from 3 substitution of 4 in 8 substitution of 7 in 9 24 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 The last formula, which is the rule of Bayes, says that, given an analyst's starting probabilities P(Si) For all scenarios, and his assessments of how likely an item would be if S were true, P(I/Si), then the new probability of Si, now that I has occurred, can be calculated using formula 10. 3. Example Assume an analyst has assigned the following probabilities, intuitively, to scenarios one and two, that the USSR is planning to launch a nuclear attack on China within six months, and that she is not: P(S ) = 10% or .1 P(S2)= 9()% or .9 Also assume that the following item comes in and that the analyst assigns probabilities that it would occur, first assuming that Russia is planning a nuclear attack, and second, assuming that she is not planning one: "TASS reports that the Chinese are deploying nuclear missiles, thereby threatening world peace." P(I/S1) =99% or .99 P(I/S2) = 80% or .8 This information can be used to revise the probabilities that each scenario is true by using the Bayesian formula: Similarly, P(Si II) = P(Si) x P(I/Si) .1 x .99 .099 = --= .12 .099 + .72 .819 E(P(Si) x P(I/Si)) i P(S2/I)= P (S 2 ) x P (VS 2 ) .72 =?=.88 .099 + .72 .819 Notice that the two new probabilities add to 1 (100%) even though the item probabilities did not. As this is a recursive process, in which a succession of items of evidence are assessed, the new probabilities are used as P(Si) in calculating the effect of the next item. 25 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 Approved For Release 2002/08/23 : CIA-RDP86600269R001100080001-0 APPENDIX 2 INTERACTIVE PROGRAMS IN APL AND BASIC 1. Interactive APL Program V B;INSTR;INSTR1;INTERMEDIATE;COUNT;PRIOR;REVISED;TIMES;PROB;TOTAL 113 12] 133 'INSTRUCTIONS? TYPE YES OR NO.' [4] INSTR1-4-3-(p(INSTR-441)) [5] +16x1INSTR1 [6] 'THIS PROGRAM CALCULATES EVENT PROBABILITIES' [7] BY THE BAYESIAN FORMULA' [8] [9] 'YOU WILL BE ASKED TO ENTER A SET OF PRIOR PROBABILITIES,' [10] 'THE NUMBER OF ITEMS OF EVIDENCE, AND THE ITEM PROBABILITIES.' 111] 1 [12] 'TO CORRECT A MISTAKE: ON A DELTA, MOVE THE CURSOR BACK AND HIT [13] ON A 2741, BACKSPACE AND HIT ATTN.' 1143 'IF YOU WANT TO STOP, TYPE THF KEY.' [15] [16] 'PRINT INTERMEDIATE RESULTS? TYPE YES OR NO.' [17] INTERMEDIATE-4-3-(p(INTERMEDIATP4-1])) 118] [19] COUNT-4-0 [20] 'PRIOR PROBABILITIES FOR THIS ANALYST, IN ORDER' [21] PRIOR-410 [22] REVISED+PRIOR [23] 'HOW MANY PIECES OF EVIDENCE' [24] TIMES-4110 [25] COUNT+COUPT+1 [26] 'PROBABILITIES FOR ITEM ';COUNT;', IN ORDER' [27] PROB-4-0 [28] +31x1((pPROB)=(PPRIOR)) [29] 'WRONG NUMBER OF ENTRIES' [30] 426 [31] TOTAL?+/(REVISEDxPROB) [32] REVISED+(REVISEDxPROB)-1TOTAL [33] +35xt(INTERMEDIATE) [34] 'INTERMEDIATE RESULTS: ';REVISEDx100 [35] +25xt(CO(JNT