Document Type: 
Document Number (FOIA) /ESDN (CREST): 
Release Decision: 
Original Classification: 
Document Page Count: 
Document Creation Date: 
November 4, 2016
Document Release Date: 
May 17, 2000
Sequence Number: 
Case Number: 
Publication Date: 
December 13, 1990
Content Type: 
PDF icon CIA-RDP96-00789R002100030001-5.pdf660.75 KB
Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 SECRET DEFENSE INTELLIGENCE AGENCY DT-9-1010-9 EVALUATIOA IIIETHOD5 (U) INITIHL REPORT 13 December, 1990 SECRET Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 CIA-RDP96-00789R002100030001-5 ~ LA EVALUATION METHODS Date of Publication 13 December 1990 This is a Department of Defense Intelligence Document prepared by the Technology Assessment and Support Office, Directorate for Scientific and Technical Intelligence Defense Intelligence Agency. PREPARED BY Technology Assessment and Support Office (DT-S) NOT RELEASABLE TO FOREIGN NATIONALS Classified by: DIA/DT Declassify: OADR Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 TABLE OF CONTENTS I. PURPOSE II. BACKGROUND III. SCOPE IV. OPERATIONAL PROJECT EVALUATION METHODOLOGIES A. SCALE APPROACH B. FUZZY SET APPROACH C. CONCEPT ANALYSIS D. CONTROL GROUPS E. IN-GROUP CONTROLS V. UTILITY EVALUATION CONSIDERATIONS VI. EVALUATION/DATA PROBLEMS VII. PREREQUISITES FOR EVALUATION A. RESEARCH REQUIREMENTS B. PROJECT TYPE C. ROLE OF RECORDS/PROTOCOLS/PROCEDURES ]BIBLIOGRAPHY FIGURES 1. EVALUATION SCALES 2. DATA CATEGORIES 3. REMOTE VIEWING DATA/TARGET EXAMPLE 4. DATA DEVELOPMENT, EVALUATION AND ASSESSMENT PROCESS Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 EVALUATION METHODS I. (U) PURPOSE: (S) The purpose of this initial report is to provide a summary of methods that have been used for evaluating psychoenergetics He., remote viewing) data in both a research and applications-oriented environment. II. (U) BACKGROUND: (U) When modern research into psychoenergetics He. Extrasensory Perception) began in the early 1930's at Duke University, NC., the experiments were designed to accommodate easy to use statistical methods. Consequently, a small number of forced choice targets (e.g., a set of five cards with different symbols) were developed as targets. Results from experiments using such targets could be readily compared with those expected from chance guessing. If results exceeded a preset value (usually one out of twenty) a case for phenomenon existence could be made, especially if the experimental trials were large in number (ie., several thousand). (U) Although these early statistical methods were convenient, they could not be applied to evaluate results from the remote viewing experiments that began in the early 1970's. Targets in most remote viewing experiments are not limited to a small set of possibilities; most early remote viewing experiments Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 used natural sites or National Geographic pictures that could be almost anything. In addition, data output was radically different from that required for early ESP research. Instead of "guesses" as to what card was the target (usually, a hunch or intuition prompted the participant), a remote viewing subject actually developed pictorial and written material. Data from remote viewing sessions required a "free response" style that by its very nature did not fit with any clear-cut statistical approach. Consequently data evaluation, as well as phenomenon existence assessments, became much more complex. (U) Several statistical-based methods were subsequently developed to help in remote viewing data assessment. Over-all results, even if statistically significant, could not make a strong case for phenomenon existence due to the small number of trials involved in a typical remote viewing series. It was inherently more time consuming to perform a single remote viewing experiment than a card guessing series of hundreds, if not thousands, of trials for any single participant. (S) Consequently, free response evaluations from the research environment were aimed at assessing data uniqueness on a trial-to trial basis. These required establishment of large (at least 100 or more) homogeneous targets in a fixed target pool that needed complex statistics for assessing results. Improved evaluation methods followed based on artificial intelligence approaches. Unfortunately, most if not all of these statistical 2 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 CIA-RDP96-00789R002100030001-5 MMMMMM~ approaches were difficult to apply in an operational environment where targets were not homogenous and where importance of various target elements varied from one trial (or project) to another. Worse, not all relevant information required for data assessment would be known, and some "ground truth" data may even have been wrong. (S) Nonetheless an evaluation procedure, even if subjective in nature, had to be developed initially to at least provide some basis for estimating or evaluating the results from operational projects. Later, improved methods that examined both data accuracy and data reliability were developed based on methods used for applications research projects. These improved methods have reduced, but not eliminated, the subjective nature of operational project evaluations. The accumulation of a large track record for given individuals over time, and performing meta-analysis of this accumulated data base, would be needed to further improve over-all assessment of such remote viewing data. (S) Even though workable methods for assessing data accuracy and reliability have been developed, there is yet another consideration for operational projects: How useful was the data? This requires another set of evaluation (ie., utility assessment) that is customer dependent. Results from such assessments, unfortunately, ranged widely due to differences in evaluation criteria that were used. Utility assessment criteria need to be defined in advance of any operational project in order to minimize 3 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 subjective aspects. This has not always been accomplished in the past and needs to be part of any future application project procedure, whenever possible. (C) However, specific evaluation methods are only meaningful when the over-all remote viewing activity is based on sound methodological procedures. Procedural aspects can easily be developed to ensure that the activity is in fact consistent with sound scientific methodology. Appropriate procedures are discussed in companion reports, and are only briefly addressed in this report. III. (U) SCOPE: (C) The following sections provide brief summaries of the various approaches that have been used by this unit for evaluating results from operational or applications-oriented projects. A follow up report is planned that will review in more detail the evaluation methodologies used for research and for applications- oriented projects. Other relevant issues are also discussed. IV. (U) OPERATIONAL PROJECT EVALUATION METHODOLOGIES: (S) There are two main issues in evaluating remote viewing data; (1) What is the definition of the target; (2) What is the definition of the remote viewing response. Various methods examined are simply different ways of comparing and evaluating the target and the response. Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 A. (U) SCALE APPROACH: (S/NF) A subjective based value scale of 0 through 5 was used in the past; a value of 0 indicated no correlation to ground truth; a value of 5 indicated a perfect match. Recently, scale values of 0 through 3 (with +, -, variations) have been used. By whatever range of scale values used, the viewers' raw or summarized data is compared to known information about a target. The best possible judgement is made concerning approximate degree of correlation to "ground truth". An example of a specific scale evaluation approach is shown on figure 1. This 0 through 3 evaluation scale illustrates numerical ratings, percentages, and descriptions for degree of correlation with regard to essential elements of information (EEI) desired for each project. (S/NF) Figure 2 lists the major target catagories that are usually of interest in any remote viewing project. Not all of these are of concern for any given task. Complex targets such as S&T facilities for example, are generally more difficult to evaluate than straight forward projects which have only 1 or 2 elements to consider. Where possible, major target catagories of interest (e.g., facility function) would be specified as part of the desired information in advance of the session. However, all the raw data is-'exa'mined no matter what its relative importance or category. This provides a gauge of individual strengths and weaknesses useful for future target/person matching. Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 .J z a 0 w .0 J -z LE - Wa z 0 0 o.. .o.. 'o . o CO LO F 0) 0 0 0 0 r M Li) 0) Approved For Release 2000/08/08 : CIA RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Cl) z CO z _j 0 Cf) z m z F- - U) 1- z - rn UJI Z a. lir- V yCl) .. 0 (E E. a . J' W Q -! z V) z w o w .LL a P:~ =z X cn.. cn t- > CC . ~ LL- J p m 2 0c ? a ~- .Q ? w . f, iii Cl) cc j ^ - ua r W I U Q U) a }yam ~... a...W:.... .W 8) W. 4/. 9GO W ,rr d N L1. a 4me cn O O z 1 0. w O C) U) ^ --, z z tic w w CC 7 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 B. (U) FUZZY SET APPROACH: (C) Fuzzy Set Theory is an objective mathematical framework for both verbal and visual analysis which is utilized for evaluating imprecise data. Imprecision results from sketches that illustrate general shapes or approximate spatial relationships. Verbal data generally includes more content (ie., analysis) than visual information. The fuzzy set theory allows for numerical values to be assigned to target elements that represent their degree of importance. These numerical values would be assigned by consumers (or other expert personnel) in advance of any project. Numerical estimates are also made of the raw data after the session that represents its degree of correlation to and importance with the intended target. Thus, the remote viewing data can then be quantified by appropriate calculations to determine data accuracy and reliability. Accuracy is defined as the percentage of target material that is described correctly by a viewer; reliability is defined to be the percentage of the over-all response that correlates to the target. (S/NF) Figure 3 shows an example of some of the data provided in a recent remote viewing experiment conducted by Stanford Research Institute to illustrate this procedure. In this example, the target used for the experiment was a microwave generator, support equipment, and testing equipment. A viewer described over seventy functions, objects and relationships. Over-all accuracy, Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 r r. r r using the fuzzy set approach, was calculated to be about 80%. However, over-all reliability was only 65%; this indicated that 35% of the raw data had no correlation to the intended target. The product of accuracy and reliability yields a "figure of merit" what is also useful for over-all data assessment and for examining viewer performance over time. Additional details with regard to this experiment, are in reference no. 11 of the bibliography. In practice, a simplified version of this approach can be used to minimize analysis time. C. (U) CONCEPT ANALYSIS: (S/NF) Concept analysis is based on analyzing data according to the over-all concept rather than on smaller bits of information usually found in a remote viewer's response. For example, in figure 3, one of the responses to a target was "a fairly long narrow channel". In concept analysis, the concept of tube, or possibly gun, would be emphasized rather than breaking apart original words such as long, narrow, or channel. Although this can be a useful approach, some meaningful data may be over looked. This method had not been widely used, although it was useful for initiating the fuzzy set approach. D. (U) CONTROL GROUPS: (U) In Research & Development activities, control groups are often necessary to establish a data baseline to which the results of other experiments can be compared. Generally, a control group is a randomly selected group of people run according Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 CA m n M m SG1 B REMOTE VIEWING DATA/TARGET EXAMPLE ""~ CO44*)EA 'DL Ut t (4.,S Co SQSp63DED $~ f~ ~ u-~'s . GETT VE,Qj # m n m Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 ? SOmC- KWD OF adO L8` 7T Approved For Release 2000/08/08 CIA-RDP96-00789R002100030001-5 to one set of protocols, while another group is run according to a different set of protocols. Both groups seek to achieve the same goal. After a designated testing period, the results are compared. The results indicate the effectiveness of each protocol independently of the other. Control groups can also indicate the extent to which remote viewing data is different from data generated by knowledgeable experts given the same background information. This can permit estimates to be made regarding validity of the remote viewing data based on standard statistical procedures. E. (U) IN-GROUP CONTROLS: (U) A simple method often used in the research community involves the use of a comparative' approach. In this method, the raw data from a session is compared to one of several possible targets, one of which is the correct one. Judges blind to the actual target attempt to make the best match. If they succeed, a case can be made for remote viewing success. Statistics are straight forward. However, due to low numbers of targets generally used for this comparison (usually 4 to 6), statistical strength is quite low. This method is useful, however, and can provide insight into the remote viewing process. It does, however, minimize the significance of highly unique data elements and is not a good indication of data usefulness. V. (U) UTILITY EVALUATION CONSIDERATIONS: (S/NF) Utility refers to how useful remote viewing data 11 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 proved to be to the consumer. Usually some form of scale value or statements ranging from "very useful" to "of little use" have been provided as feedback. These utility evaluations, when performed, have usually suffered from lack of consistent evaluation criteria. In many cases, the criteria were not ever agreed upon in advance of project initiation. This is an important issue that will be addressed in future applications-oriented research. VI. (U) EVALUATION/DATA PROBLEMS: (S/NF) Sometimes it is difficult to complete evaluations since ground truth may not be totally known, or possibly the raw data may contain predictive information of an unspecified future time period. In such cases, only partial evaluations are possible, and final assessments may require months or years. Such potential delays in data evaluation pose serious problems for the reviewer (ie., feedback not possible), as well as for the consumer who may require timely information. (S/NF) Another problem is who should do the evaluating? If only customers evaluate the operational data, they may not be capable of observing trends or patterns that could be useful. If evaluations are solely determined by individuals not involved in operations, they may emphasize aspects that are not operationally important. A combination of both views must be considered when ;possible and implemented in the evaluation process. It is also necessary to minimize or eliminate the role of,the data procedures in final data evaluation since this would present a potential for Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 assessment biasing. (S/NF) Other problems can include viewers being exposed to inferences and deductions by open press sources or television for certain current event projects. A monitor or observer present during a remote viewing session could also pose a problem since he or she could unknowingly bias the viewer. Biasing the source can be due to subliminal cueing. Consequently, thorough records must be kept regarding possible target related knowledge of those present in the remote viewing session. VII. (U) PREREQUISITES FOR EVALUATION: A. (U) RESEARCH REQUIREMENTS: (S) Research requirements are more stringent than operational,requirements since proof of principle or the search for difficult-to-detect variables are involved. Consequently, there is a strong need for well-defined targets and tightly controlled protocol so that appropriate statistics can be applied. B. (U) PROJECT TYPE: (S/NF) The various projects worked on by this office include foreign personalities, military related targets, event predictions, as well as search projects involving location of target personalities or moving equipment. Evaluation procedures with regard to search are very clear cut because either the location is accurate ("a hit") or it is not ("a miss"). Therefore, search information can be evaluated separately from broad categorical data. Training results are easier to evaluate Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 in terms of data accuracy since targets can be easily controlled or defined. C. (U) ROLE OF RECORDS/PROTOCOLS/PROCEDURES: (S/NF) The role of session records is an extremely important one. These records would include the people involved, information provided to the project personnel, project timing and other relevant data. Such project details are recorded and maintained in a permanent file or automated data bases. Specific protocols are also followed to insure proper records and other procedures are followed. A companion report, item 6 in the bibliography, contains protocol and methodology details. (C) To further assist and improve the over-all evaluation ,process, future projects will be evaluated and assessed according to the procedures illustrated on figure 4. This flow diagram contains all the essential steps necessary for insuring that appropriate actions occur and range from task initiation through final data assessment and feedback. Details will be developed to clarify the various roles of each major phase and to identify guidelines for establishing uniform evaluation criteria. Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 ApproveTc~~or SIP d~STe?2,d~ j6??:di9k* J bb ? 02100030001-5 REQUIREMENTS %v. Vva1~{.. l'117f1LIJ1J (TRADITIONAL ANALYSIS) (SPECIFIC ASSUMPTIONS) ALL-SOURCE ANALYSTS (OUTPUT) (COMPLETED EVALUATI,NS) (OVERALL ASSESSMENT) L(FINAL REVIEW BOARD ASSESSMENT (feedback) (feedback) (OUTPUT) DT/DT-3 (EVALUATION) ALL-SOURCE ANALYSTS OVERSIGHT PANEL 15 Approved For RleIease 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 CIA-RDP96-00789R002100030001-5 1. "Sun Streak: Grill Flame Project Report (U)," Defense Intelligence Agency, Washington DC, (October 19, 1983), SECRET-NOFORN-WNINTEL-SAP. 2. "Special Report: Sun Streak Evaluation (U)," Defense Intelligence Agency, Washington DC, (November 2 1989), SECRET-NOFORN-WNINTEL-SAP. 3. Daniel Druckman and John A. Swets, Editors, "Enhancing Human Performance Issues, Theories, and Techniques" National Academy Press, Washington DC (1988), UNCLASSIFIED. 4. John A. Palmer, Charles Honorton and Jessica Utts," Reply to the National Research Council Study on Parapsychology", Parapsychology Association, Inc., Research Triangle Park, NC (1988), UNCLASSIFIED. 5. "Report of a Workshop on Experimental Parapsychology", International Security and Commerce Program, Office of Technology and Assessment, United States Congress (February 22, 1989), UNCLASSIFIED. Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 CIA-RDP96-00789R002100030001-5 6. "Operational Project Protocols (U)," DIA Document No. DT-S-1006-SS, Defense Intelligence Agency, Washington, DC (November 27, 1990), SECRET-NOFORN-WNINTEL. 7. "SRI Studies in Remote Viewing (U)," Science Panel Report (March 1, 1984), SECRET-WNINTEL. 8. "Data Evaluation Issues," Paper, (Pending publication by .DT-S), UNCLASSIFIED. 9. Beverly S. Humphrey et al., "Fuzzy Set Applications in Remote Viewing Analysis", Final Report--Objective A, Task 3 Covering the period 1 October 1986 to 30 September 1987, Project 1291, SRI :International, Menlo Park, CA (December 1987), UNCLASSIFIED. :10. Edwin C. May, et al., "Applications of Fuzzy Sets to Rote Viewing Analysis (U)", Final Report--Objective F, Task 1 Covering the Period 1 October 1987 to 30 September 1988, Project 1291, SRI International, Menlo Park, CA (December 1988), SECRET-NOFORN-WNINTEL. 11. Edwin C. May, "An Application Oriented Remote Viewing Experiment (U)", Final Report Covering the Period 1 May 1988 to April 1989, Project 2740, SRI International, Menlo Park, CA 17 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 (April 1989), SECRET-NOFORN. 12. Wanda L. W. Luke et al., "A Prototype Analysis System For Special Remote Viewing Tasks (U)", Final Report--Task 6.0.3 Covering the Period 1 October 1988 to 30 September 1989, Project 1291, SRI International, Menlo Park, CA (October 1989), SECRET-NOFORN-WNINTEL. 13. Beverly S. Humphrey et al., "Remote Viewing Evaluation Techniques (U),", Final Report--Objective A, Task 4, Project 1291, SRI International, Menlo Park, CA (December 1986), SECRET-NOFORN-WNINTEL. 14. K. Ramakrishna Rao "The Basic Experiments in Parapsychology," McFarland & Company, Inc., Jefferson, NC, and London (1984), UNCLASSIFIED. Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5