REPLICATION AND META-ANALYSIS IN PARAPSYCHOLOGY

Document Type: 
Collection: 
Document Number (FOIA) /ESDN (CREST): 
CIA-RDP96-00792R000100130004-9
Release Decision: 
RIFPUB
Original Classification: 
K
Document Page Count: 
41
Document Creation Date: 
November 4, 2016
Document Release Date: 
September 5, 2003
Sequence Number: 
4
Case Number: 
Content Type: 
MAGAZINE
File: 
AttachmentSize
PDF icon CIA-RDP96-00792R000100130004-9.pdf4.42 MB
Body: 
Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Statistical Science 1991, Vol. 6, :No. 4, 363-403 Replication and Meta-Analysis in Parapsychology Abstract. Parapsychology, the laboratory study of psychic phenomena, has had its history interwoven with that of statistics. Many of the controversies in parapsychology have focused on statistical issues, and statistical models have played an' integral role in the experimental work. Recently, parapsychologists -have been using meta-analysis as a tool for synthesizing large bodies of work. This paper presents an overview of the use of statistics in parapsychology and offers a summary of the meta-analyses that have been conducted. It begins with some anecdotal information about the involvement of statistics and statisti- cians with the early history of parapsychology. Next, it is argued that most nonstatisticians do not appreciate the connection between power and "successful" replication of experimental effects. Returning to para- psychology, a particular experimental regime is examined by summariz- ing an extended debate over the interpretation of the results. A new set of experiments designed to resolve the debate is then reviewed. Finally, meta-analyses from several areas of parapsychology are summarized. It is concluded that the overall evidence indicates that there is an anoma- lous effect in need of an explanation. Key words and phrases: Effect size, psychic research, statistical contro- versies, randomness, vote-counting. In a June 1990 Gallup Poll, 49% of the 1236 respondents claimed to believe in extrasensory per- ception (ESP), and one in four claimed to have had a personal experience involving telepathy (Gallup and Newport, 1991). Other surveys have shown even higher percentages; the University of Chicago's National Opinion Research Center re- cently surveyed 1473 adults, of which 67% claimed that they had experienced ESP (Greeley, 1987). Public opinion is a poor arbiter of science, how- ever, and experience is a poor substitute for the scientific method. For more than a century, small numbers of scientists have been conducting labora- tory experiments to study phenomena such as telepathy, clairvoyance and precognition, collec- tively known as "psi" abilities. This paper will examine some of that work, as well as some of the statistical controversies it has generated. Jessica Utts is Associate Professor, Division of Statistics, University of California at Davis, 469 Kerr Hall, Davis, California 95616. Parapsychology, as this field is called, has been a source of controversy throughout its history. Strong beliefs tend to be resistant to change even in the face of data, and many people, scientists included, seem to have made up their minds on the question without examining any empirical data at all. A critic of parapsychology recently acknowledged that "The level of the debate during the past 130 years has been an embarrassment for anyone who would like to believe that scholars and scientists adhere to standards of rationality and fair play" (Hyman, 1985a, page 89). While much of the controversy has focused on poor experimental design and potential fraud, there have been attacks and defenses of the statistical methods as well, sometimes calling into question the very foundations of probability and statistical inference. Most of the criticisms have been leveled by psy- chologists. For example, a 1988 report of the U.S. National Academy of Sciences concluded that "The committee finds no scientific justification from research conducted over a period of 130 years for the existence of parapsychological phenomena"- (Druckman and Swets, 1988, page 22). The chapter on parapsychology was written by a subcommittee Approved For Release 2003/09/10 : CIAsRDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 364 chaired by a psychologist who had published a similar conclusion prior to his appointment to the committee (Hyman, 1985a, page 7). There were no parapsychologists involved with the writing of the report. Resulting accusations of bias (Palmer, Hon- orton and Utts, 1989) led U.S. Senator Claiborne Pell to request that the Congressional Office of Technology Assessment (OTA) conduct an investi- gation with a more balanced group. A one-day workshop was held on September 30, 1988, bring- ing together parapsychologists, critics and experts in some related fields (including the author of this paper). The report concluded that parapsychology needs "a fairer hearing across a broader spectrum of the scientific community, so that emotionality does not impede objective assessment of experimen- tal results" (Office of Technology Assessment, 1989). It is in the spirit of the OTA report that this article is written. After Section 2, which offers an anecdotal account of the role of statisticians and statistics in parapsychology, the discussion turns to the more general question of replication of experi- mental results. Section 3 illustrates how replica- 4: L l ~~r (,,-, \ rf vrvPtP~ by s! ientists in many fields. Returning to parapsychology in Section 4, a particular experimental regime called the "ganz feld" is described, and an extended debate about the 'interpretation of the experimental results is discussed. Section 5 examines a meta-analysis of recent ganzfeld experiments designed to resolve the debate. Finally, Section 6 contains a brief account of meta-analyses that have been conducted in other areas of parapsychology, and conclusions are given in Section 7. 2. STATISTICS AND PARAPSYCHOLOGY Parapsychology had its beginnings in the investi- gation of purported mediums and other anecdotal claims in the late 19th century. The Society for Psychical Research was founded in Britain in 1882, and its American counterpart was founded in Boston in 1884. While these organizations and their members were primarily involved with investigat- ing anecdotal material, a_ few of the early re- searchers were already conducting "forced-choice" experiments such as card-guessing. (Forced-choice experiments are like multiple choice tests; on each trial the subject must guess from a small, known set of possibilities.) Notable among these was Nobel Laureate Charles Richet, who is generally credited with being the first to recognize that prob- ability theory could be applied to card-guessing experiments (Rhine, 1977, page 26; Richet, 1884). F. Y. Edgeworth, partly in response to what he considered to be incorrect analyses of these experi- ments, offered one of the earliest treatises on the statistical evaluation of forced-choice experiments in two articles published in the Proceedings of the Society for Psychical Research (Edgeworth, 1885, 1886). Unfortunately, as noted by Mauskopf and McVaugh (1979) in their historical account of the period, Edgeworth's papers were "perhaps too diffi- cult for their immediate audience" (page 105). Edgeworth began his analysis by using Bayes' theorem to derive the formula for the posterior probability that chance was operating, given the data. He then continued with an argument "savouring more of Bernoulli than Bayes" in which "it is consonant, I submit, to experience, to put 1/2 both for a and 0," that is, for both the prior proba- bility that chance alone was operating, and the prior probability that "there should have been some additional agency." He then reasoned (using a Taylor series expansion of the posterior prob- ability formula) that if there were a large prob- ability of observing the data given that some additional agency was at work, and a small objec- tive probability of the data under chance, then the latter (binomial) probability "may be taken as a rough measure of the sought a posteriori probabil- ity in tavour of mere cnanee 1pa5C a~ worth concluded his article by applying his method to some data published previously in the same journal. He found the probability against chance to be 0.99996, which he said "may fairly be regarded as physical certainty" (page 199). He concluded: Such is the evidence which the calculus of probabilities affords as to the existence of an agency other than mere chance. The calculus is silent as to the nature of that agency-whether it is more likely to be vulgar illusion or ex- traordinary law. That is a question to be decided, not by formulae and figures, but by general philosophy and common sense [page 1991. Both the statistical arguments and the experi- mental controls in these early experiments were somewhat loose. For example, Edgeworth treated as binomial an experiment in which one person chose a string of eight letters and another at- tempted to guess the string. Since it has long been understood that people are poor random number (or letter) generators, there is no statistical basis for analyzing such an experiment. Nonetheless, Edge- worth and his contemporaries set the stage for the use of controlled experiments with statistical evalu- ation in laboratory parapsychology. An interesting historical account of Edgeworth's involvement and the role telepathy experiments played in the early history of randomization and experimental design is provided by Hacking (1988). Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 and Greenwood, 1937). Stuart, who had been an 'undergraduate in mathematics at Duke, was one of Rhine's early subjects and continued to work with him as a researcher until Stuart's death in 1947. Greenwood was a Duke mathematician, who appar- ently converted to a statistician at the urging of Rhine. Another prominent figure who was distressed with Kellogg's attack was E. V. Huntington, a mathematician at Harvard. After corresponding with Rhine, Huntington decided that, rather than further confuse the public with a technical reply to Kellogg's arguments, a simple statement should be made to the effect that the mathematical issues in Rhine's work had been resolved. Huntington must have successfully convinced his former student, Burton Camp of Wesleyan, that this was a wise approach. Camp was the 1937 President of IMS. When the annual meetings were held in December of 1937 (jointly with AMS and AAAS), Camp released a statement to the press that read: Dr. Rhine's investigations have two aspects: experimental and statistical. On the exper- imental side mathematicians, of course, have nothing to say. On the statistical side, however, recent mathematical work has established the fact that, assuming that the experiments have been properly performed, the statistical analysis is essentially valid. If the Rhine investigation is to be fairly attacked, it must be on other than mathematical grounds [Camp, 19371. One statistician who did emerge as a critic was William Feller. In a talk at the Duke Mathemati- cal Seminar on April 24, 1940, Feller raised three criticisms to Rhine's work (Feller, 1940). They had been raised before by others (and continue to be raised even today). The first was that inadequate shuffling of the cards resulted in additional infor- mation from one series to the next. The second was what is now known as the "file-drawer effect," namely, that if one combines the results of pub- lished studies only, there is sure to be a bias in favor of successful studies. The third was that the results were enhanced by the use of optional stop- ping, that is, by not specifying the number of trials in advance. All three of these criticisms were ad- dressed in a rejoinder by Greenwood and Stuart (1940), but Feller was never convinced. Even in its third edition published in 1968, his book An Intro- duction to Probability Theory and Its Applications still contains his conclusion about Greenwood and Stuart: "Both their arithmetic and their experi- ments have a distinct tinge of the supernatural" (Feller, 1968, page 407). In his discussion of Feller's position, Diaconis (1978) remarked, "I believe Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 One of the first American researchers to use statistical methods in parapsychology was John Edgar Coover, who was the Thomas Welton Stanford Psychical Research Fellow in the Psychol- ogy Department at Stanford University from 1912 to 1937 (Dommeyer, 1975). In 1917, Coover pub- lished a large volume summarizing his work (Coover, 1917). Coover believed that his results were consistent with chance, but others have ar- gued that Coover's definition of significance was too strict (Dommeyer, 1975). For example, in one evaluation of his telepathy experiments, Coover found a two-tailed p-value of 0.0062. He concluded, "Since this value, then, lies within the field of chance deviation, although the probability of its occurrence by chance is fairly low, it cannot be accepted as a decisive indication of some cause beyond chance which operated in favor of success in guessing" (Coover, 1917, page 82). On the next page, he made it explicit that he would require a p-value of 0.0000221 to declare that something other than chance was operating. It was during the summer of 1930, with the card-guessing experiments of J. B. Rhine at Duke University, that parapsychology began to take hold as a laboratory science. Rhine's laboratory still exists under the name of the Foundation for Re- search on the Nature of Man, housed at the edge of the Duke University campus. It wasn't long after Rhine published his first book, Extrasensory Perception in 1934, that the attacks on his methodology began. Since his claims were wholly based on statistical analyses of his experiments, the statistical methods were closely scrutinized by critics anxious to find a conventional explanation for Rhine's positive results. The most persistent critic was a psychologist from McGill_ University named Chester Kellogg (Mauskopf and McVaugh, 1979). Kellogg's main argument was that Rhine was using the binomial distribution (and normal approximation) on a se- ries of trials that were not independent. The experi- ments in question consisted of having a subject guess the order of a deck of 25 cards, with five each of five symbols, so technically Kellogg was correct. By ].937, several mathematicians and statis- ticians had come to Rhine's aid. Mauskopf and McVaugh (1979) speculated that since statistics was itself a young discipline, "a number of statisticians were equally outraged by Kellogg, whose argu- ments they saw as discrediting their profession" (page 258). The major technical work, which ac- knowledged that Kellogg's criticisms were accurate but did little to change the significance of the results, was conducted by Charles Stuart and Joseph A. Greenwood and published in the first volume of the Journal of Parapsychology (Stuart Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 366 J. UTIS Feller was confused.. . he seemed to have decided the opposition was wrong and that was that." Several statisticians have contributed to the literature in parapsychology to greater or lesser degrees. T. N. E. Greville developed applicable statistical methods for many of the experiments in parapsychology and was Statistical Editor of the Journal of Parapsychology (with J. A. Greenwood) from its start in 1937 through Volume 31 in 1967; Fisher (1924, 1929) addressed some specific prob- lems in card-guessing experiments; Wilks (1965a, b) described various statistical methods for parapsy- chology; Lindley (1957) presented a Bayesian anal- ysis of some parapsychology data; and Diaconis (1978) pointed out some problems with certain ex- periments and presented a method for analyzing experiments when feedback is given. Occasionally, attacks on parapsychology have taken the form of attacks on statistical inference in general, at least as it is applied to real data. Spencer-Brown (1957) attempted to show that true randomness is impossible, at least in finite se- quences, and that this could be the explanation for the results in parapsychology. That argument re- emerged in a recent debate on the role of random- ness in parapsychology, initiated by psychologist J. Barnard Gilmore (Gilmore, 1989, 1990; Utts, 1989; Palmer, 1989, 1990). Gilmore stated that "The ag- nostic statistician, advising on research in psi, should take account of the possible inappropriate- ness of classical inferential statistics" (1989, page 338). In his second paper, Gilmore reviewed several non-psi studies showing purportedly random sys- tems that do not behave as they should under randomness (e.g., Iversen, Longcor, Mosteller, Gilbert and Youtz, 1971; Spencer-Brown, 1957). Gilmore concluded that "Anomalous data ... should not be found nearly so often if classical statistics offers a valid model of reality" (1990, page 54), thus rejecting the use of classical statisti- cal inference for real-world applications in general. Implicit and explicit in the literature on parapsy- chology is the assumption that, in order to truly establish itself, the field needs to find a repeat- able experiment. For example, Diaconis (1978) started the summary of his article in Science with the words "In search of repeatable ESP experi- ments, modern investigators ... " (page 131). On October 28-29, 1983, the 32nd International Con- ference of the Parapsychology Foundation was held in San Antonio, Texas, to address "The Repeatabil- ity Problem in Parapsychology." The Conference Proceedings (Shapin and Coly, 1985) reflect the diverse views among parapsychologists on the na- ture of the problem. Honorton (1985a) and Rao (1985), for example, both argued that strict replica- tion is uncommon in most branches of science and that parapsychology should not be singled out as unique in this regard. Other authors expressed disappointment in the lack of a single repeatable experiment in parapsychology, with titles such as "Unrepeatability: Parapsychology's Only Find- ing" (Blackmore, 1985), and "Research Strategies for Dealing with Unstable Phenomena" (Beloff, 1985). It has never been clear, however, just exactly what would constitute acceptable evidence of a re- peatable experiment. In the early days of investiga- tion, the major critics "insisted that it would be sufficient for Rhine and Soal to convince them of ESP if a parapsychologist could perform success- fully a single 'fraud-proof' xperiment" (Hyman, 1985a, page 71). However, as soon as well-designed experiments showing statistical significance emerged, the critics realized that a single experi- ment could be statistically significant just by chance. British psychologist C. E. M. Hansel quan- tified the new expectation, that the experiment should be repeated a few times, as follows: If a result is significant at the .01 level and this result is not due to chance but to informa- tion reaching the subject, it may be expected that by making two further sets of trials the antichance odds of one hundred to one will be increased to around a million to one, thus en- abling the effects of ESP-or whatever is re- sponsible for the original result-to manifest itself to such an extent that there will be little doubt that the result is not due to chance [Hansel, 1980, page 2981. In other words, three consecutive experiments at p:5 0.01 would convince Hansel that something other than chance was at work. This argument implies that if a particular experi- ment produces a statistically significant result, but subsequent replications fail to attain significance, then the original result was probably due to chance, or at least remains unconvincing. The problem with this line of reasoning is that there is no consid- eration given to sample size or power. Only an experiment with extremely high power should be expected to be "successful" three times in succession. It is perhaps a failure of the way statistics is taught that many scientists do not understand the importance of power in defining successful replica- tion. To illustrate this point, psychologists Tversky and Kahnemann (1982) distributed a questionnaire Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 REPLICATION IN PARAPSYCHOLOGY 867 to their colleagues at a professional meeting, with the question: An investigator has reported a result that you consider implausible. He ran 15 subjects, and reported a significant value, t = 2.46. Another investigator has attempted to duplicate his pro- cedure, and he obtained a nonsignificant value of t with the same number of subjects. The direction was the same in both sets of data. You are reviewing the literature. What is the highest value of t in the second set of data that you would describe as a failure to replicate? [1982, page 281. In reporting their results, Tversky and Kahne- mann stated: The majority of our respondents regarded t = 1.70 as a failure to replicate. If the data of two such studies (t = 2.46 and t = 1.70) are pooled, the value of t for the combined data is about 3.00 (assuming equal variances). Thus, we are faced with a paradoxical state of affairs, in which the same data that would increase our confidence in the finding when viewed as part of the original study, shake our confidence when viewed as an independent study [1982, page 281. At a recent presentation to the History and Phi- losophy of Science Seminar at the University of California at Davis, I asked the following question. Two scientists, Professors A and B, each have a theory they would like to demonstrate. Each plans to run a fixed number of Bernoulli trials and then test Ho: p 0.25 versus Ha: p > 0.25. Professor A has access to large numbers of students each semester to use as subjects. In his first experiment, he runs 100 subjects, and there are 33 successes (p = 0.04, one-tailed). Knowing the importance of replication, Professor A runs an additional 100 sub- jects as a second experiment. He finds 36 successes (p = 0.009, one-tailed). Professor B only teaches small classes. Each quarter, she runs an experiment on her students to test her theory. She carries out ten studies this way, with the results in Table 1. I asked the audience by a show of hands to indicate whether or not they felt the scientists had successfully demonstrated their theories. Professor A's theory received overwhelming support, with approximately 20 votes, while Professor B's theory received only one vote. If you aggregate the results of the experiments for each professor, you will notice that each con- ducted 200 trials, and Professor B actually demon- strated a higher level of success than Professor A, with 71 as opposed to 69 successful trials. The one-tailed p-values for the combined trials are 0.0017 for Professor A and 0.0006 for Professor B. To address the question of replication more ex- plicitly, I also posed the following scenario. In December of 1987, it was decided to prematurely terminate a study on the effects of aspirin in reduc- ing heart attacks because the data were so convinc- ing (see, e.g., Greenhouse and Greenhouse, 1988; Rosenthal, 1990a). The physician-subjects had been randomly assigned to take aspirin or a placebo. There were 104 heart attacks among the 11,037 subjects in the aspirin group, and 189 heart attacks among the 11,034 subjects in the placebo group (chi-square = 25.01, p < 0.00001). After showing the results of that study, I pre- sented the audience with two hypothetical experi- ments conducted to try to replicate the original result, with outcomes in Table 2. I asked the audience to indicate which one they thought was a more successful replication. The au- dience chose the second one, as would most journal editors, because of the "significant p-value." In fact, the first replication has almost exactly the same proportion of heart attacks in the two groups as the original study and is thus a very close repli- cation of that result. The second replication has TABLE 1 Attempted replccations for professor B 10 4 0.22 15 6 0.15 17 6 0.23 25 8 0.17 30 10 0.20 40 13 0.18 18 7 0.14 10 5 0.08 15 5 0.31 20 7 0.21 TABLE 2 Hypothetical replications of the aspirin / heart attack study Replication #1 Heart attack Replication #2 Heart attack Aspirin 11 1156 20 2314 Placebo 19 1090 48 2170 Chi-square 2.596, p = 0.11 13.206, p = 0.0003 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 368 J. UTI'S very different proportions, and in fact the relative risk from the second study is not even contained in a 95% confidence interval for relative risk from the original study. The magnitude of the effect has been much more closely matched by the "nonsig- nificant" replication. Fortunately, psychologists are beginning to no- tice that replication is not as straightforward as they were originally led to believe. A special issue of the Journal of Social Behavior and Personality was entirely devoted to the question of replication (Neuliep, 1990). In one of the articles, Rosenthal cautioned his colleagues: "Given the levels of sta- tistical power at which we normally operate, we have no right to expect the proportion of significant results that we typically do expect, even if in na- ture there is a very real and very important effect" (Rosenthal, 1990b, page 16). Jacob Cohen, in his insightful article titled "Things I Have Learned (So Far)," identified an- other misconception common among social scien- tists: "Despite widespread misconceptions to the contrary, the rejection of a given null hypothesis gives us no basis for estimating the probability that a replication of the research will again result in rejecting that null hypothesis" (Cohen, 1990, page 1307). Cohen and Rosenthal both advocate the use of effect sizes as opposed to significance levels when defining the strength of an experimental effect. In general, effect sizes measure the amount by which the data deviate from the null hypothesis in terms of standardized units. For instance, the effect size for a two-sample t-test is usually defined to be the difference in the two means, divided by the stan- dard deviation for the control group. This measure can be compared across studies without the depen- dence on sample size inherent in significance lev- els. (Of course there will still be variability in the sample effect sizes, decreasing as a function of sam- ple size.) Comparison of effect sizes across studies is one of the major components of meta-analysis. Similar arguments have recently been made in the medical literature. For example, Gardner and Altman (1986) stated that the use of p-values "to define two alternative outcomes-significant and not significant-is not helpful and encourages lazy thinking" (page 746). They advocated the use of confidence intervals instead. As discussed in the next section, the arguments used to conclude that parapsychology has failed to demonstrate a replicable effect hinge on these mis- conceptions of replication and failure to examine power. A more appropriate analysis would compare the effect sizes for similar experiments across ex- perimenters and across time to see if there have been consistent effects of the same magnitude. Rosenthal also advocates this view of replication: The traditional view of replication focuses on significance level as the relevant summary statistic of a study and evaluates the success of a replication in a dichotomous fashion. The newer, more useful view of replication focuses on effect size as the more important summary statistic of a study and evaluates the success of a replication not in a dichotomous but in a continuous fashion [Rosenthal, 1990b, page 281. The dichotomous view of replication has been used throughout the history of parapsychology,- by both parapsychologists and critics (Utts, 1988). For example, the National Academy of Sciences report critically evaluated "significant" experiments, but entirely ignored "nonsignificant" experiments. In the next three sections, we will examine some of the results in parapsychology using the broader, more appropriate definition of replication. In doing so, we will show that the results are far more interesting than the critics would have us believe. 4. THE GANZFELD DEBATE IN PARAPSYCHOLOGY An extensive debate took place in the mid-1980s between a parapsychologist and critic, questioning whether or not a particular body of parapsychologi- cal data had demonstrated psi abilities. The experi- ments in question were all conducted using the ganzfeld setting (described below). Several authors were invited to write commentaries on the debate. As a result, this data base has been more thor- oughly analyzed by both critics and proponents than any other and provides a good source for studying replication in parapsychology. The debate concluded with a detailed series of recommendations for further experiments, and left open the question of whether or not psi abilities had been demonstrated. A new series of experi- ments that followed the recommendations were conducted over the next few years. The results of the new experiments will be presented in Section 5. 4.1 Free-Response Experiments Recent experiments in parapsychology tend to use more complex target material than the cards and dice used in the early investigations, partially to alleviate boredom on the part of the subjects and partially because they are thought to "more nearly resemble the conditions of spontaneous psi occur- rences" (Burdick and Kelly, 1977, page 109). These experiments fall under the general heading of "free-response" experiments, because the subject is asked to give a verbal or written description of the Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 REPLICATION IN PARAPSYCHOLOGY 369 target, rather than being forced to make a choice from a small discrete set of possibilities. Various types of target material have been used, including pictures, short segments of movies on video tapes, actual locations and small objects. Despite the more complex target material, the statistical methods used to analyze these experi- ments are similar to those for forced-choice experi- ments. A typical experiment proceeds as follows. Before conducting any trials, a large pool of poten- tial targets is assembled, usually in packets of four. Similarity of targets within a packet is kept to a minimum, for reasons made clear below. At the start of an experimental session, after the subject is sequestered in an isolated room, a target is selected at random from the pool. A sender is placed in another room with the target. The subject is asked to provide a verbal or written description of what he or she thinks is in the target, knowing only that it is a photograph, an object, etc. After the subject's description has been recorded and secured against the potential for later alter- ation, a judge (who may or may not be the subject) is given a copy of the subject's description and the four possible targets that were in the packet with the correct target. A properly conducted experi- ment either uses video tapes or has two identical sets of target material and uses the duplicate set for this part of the process, to ensure that clues such as fingerprints don't give away the answer. Based on the subject's description, and of course on a blind basis, the judge is asked to either rank the four choices from most to least likely to have been the target, or to select the one from the four that seems to best match the subject's description. If ranks are used, the statistical analysis proceeds by summing the ranks over a series of trials and .comparing the sum to what would be expected by chance. If the selection method is used, a "direct hit" occurs if the correct target is chosen, and the number of direct hits over a series of trials is compared to the number expected in a binomial experiment with p = 0.25. Note that the subjects' responses cannot be con- sidered to be "random" in any sense, so probability assessments are based on the random selection of the target; and decoys. In a correctly designed ex- periment, the probability of a direct hit by chance is 0.25 on each trial, regardless of the response, and the trials are independent. These and other issues related to analyzing free-response experiments are discussed by Utts (1991). 4.2 The Psi Ganzfeld Experiments The ganzfeld procedure is a particular kind of free-response experiment utilizing a perceptual isolation technique originally developed by Gestalt psychologists for other purposes. Evidence from spontaneous case studies and experimental work had led parapsychologists to a model proposing that psychic functioning may be masked by sensory in- put and by inattention to internal states (Honorton, 1977). The ganzfeld procedure was specifically de- signed to test whether or not reduction of external "noise" would enhance psi performance. In these experiments, the subject is placed in a comfortable reclining chair in an acoustically shielded room. To create a mild form of sensory deprivation, the subject wears headphones through which white noise is played, and stares into a constant field of red light. This is achieved by taping halved translucent ping-pong balls over the eyes and then illuminating the room with red light. In the psi ganzfeld experiments, the subject speaks into a microphone and attempts to describe the target material being observed by the sender in a distant room. At the 1982 Annual Meeting of the Parapsycho- logical Association, a debate took place over the degree to which the results of the psi ganzfeld experiments constituted evidence of psi abilities. Psychologist and critic Ray Hyman and parapsy- chologist Charles Honorton each analyzed the re- sults of all known psi ganzfeld experiments to date, and they reached strikingly different conclusions (Honorton, 1985b; Hyman, 1985b). The debate con- tinued with the publication of their arguments in separate articles in the March 1985 issue of the Journal of Parapsychology. Finally, in the Decem- ber 1986 issue of the Journal of Parapsychology, Hyman and Honorton (1986) wrote a joint article in which they highlighted their agreements and disagreements and outlined detailed criteria for future experiments. That same issue contained commentaries on the debate by 10 other authors. The data base analyzed by Hyman and Honorton (1986) consisted of results taken from 34 reports written by a total of 47 authors. Honorton counted 42 separate experiments described in the reports, of which 28 reported enough information to determine the number of direct hits achieved. Twenty three of the studies (55%) were classified by Honorton as having achieved statistical significance at 0.05. 4.3 The Vote-Counting Debate Vote-counting is the term commonly used for the technique of drawing inferences about an experi- mental effect by counting the number of significant versus nonsignificant studies of the effect. Hedges and Olkin (1985) give a detailed analysis of the inadequacy of this method, showing that it is more and more likely to make the wrong decision as the Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 number of studies increases. While Hyman ac- knowledged that "vote-counting raises many prob- lems" (Hyman, 1985b, page 8), he nonetheless spent if of his critique of the ganzfeld studies showing why Honorton's count of 55% was wrong. Hyman's first complaint was that several of the studies contained multiple conditions, each of which should be considered as a separate study. Using this definition he counted 80 studies (thus further reducing the sample sizes of the individual studies), of which 25 (31%) were "successful." Honorton's response to this was to invite readers to examine the studies and decide for themselves if the varying conditions constituted separate experiments. Hyman next postulated that there was selection bias, so that significant studies were more likely to be reported. He raised some important issues about how pilot studies may be terminated and not re- ported if they don't show significant results, or may at least be subject to optional stopping, allowing the experimenter to determine the number of tri- als. He also presented a chi-square analysis that "suggests a tendency to report studies with a small sample only if they have significant results" (Hyman, 1985b, page 14), but I have questioned his analysis elsewhere (Utts, 1986, page 397). Honorton refuted Hyman's argument with four rejoinders (Honorton, 1985b, page 66). In addition to reinterpreting Hyman's chi-square analysis, Honorton pointed out that the Parapsychological Association has an official policy encouraging the publication of nonsignificant results in its journals and proceedings, that a large number of reported ganzfeld studies did not achieve statistical signifi- cance and that there would have to be 15 studies in the "file-drawer" for every one reported to cancel out the observed significant results. The remainder of Hyman's vote-counting analy- sis consisted of showing that the effective error rate for each study was actually much higher than the nominal 5%. For example, each study could have been analyzed using the direct hit measure, the sum of ranks measure or one of two other measures used for free-response analyses. Hyman carried out a simulation study that showed the true error rate would be 0.22 if "significance" was defined by re- quiring at least one of these four measures to achieve the 0.05 level. He suggested several other ways in which multiple testing could occur and concluded that the effective error rate in each ex- periment was not the nominal 0.05, but rather was probably close to the 31% he had determined to be the actual success rate in his vote-count. Honorton acknowledged that there was a multi- ple testing problem, but he had a two-fold response. First, he applied a Bonferroni correction and found that the number of significant studies (using his definition of a study) only dropped from 55% to 45%. Next, he proposed that a uniform index of success be applied to all studies. He used the num- ber of direct hits, since it was by far the most commonly reported measure and was the measure used in the first published psi ganzfeld study. He then conducted a detailed analysis of the 28 studies reporting direct hits and found that 43% were sig- nificant at 0.05 on that measure alone. Further, he showed that significant effects were reported by six of the 10 independent investigators and thus were not due to just one or two investigators or laborato- ries. He also noted that success rates were very similar for reports published in refereed journals and those published in unrefereed monographs and abstracts. While Hyman's arguments identified issues such as selective reporting and optional stopping that should be considered in any meta-analysis, the de- pendence of significance levels on sample size makes the vote-counting technique almost useless for as- sessing the magnitude of the effect. Consider, for example, the 24 studies where the direct hit meas- ure was reported and the chance probability of a direct hit was 0.25, the most common type of study in the data base. (There were four direct hit studies with other chance probabilities and 14 that did not report direct hits.) Of the 24 studies, 13 (54%) were "nonsignificant" at a = 0.05, one-tailed. But if the 367 trials in these "failed replications" are com- bined, there are 106 direct hits, z = 1.66, and p = 0.0485, one tailed. This is reminiscent of the dilemma of Professor B in Section 3. Power is typically very low for these studies. The median sample size for the studies reporting direct hits was 28. If there is a real effect and it increases the success probability from the chance 0.25 to an actual 0.33 (a value whose rationale will be made clear below), the power for a study with 28 trials is only 0.181 (Utts, 1986). It should be no surprise that there is a "repeatability" problem in parapsychology. 4.4 Flaw Analysis and Future Recommendations The second half of Hyman's paper consisted of a "Meta-Analysis of Flaws and Successful Outcomes" (1985b, page 30), designed to explore whether or not various measures of success were related to specific flaws in the experiments. While many crit- ics have argued that the results in parapsychology can be explained by experimental flaws, Hyman's analysis was the first to attempt to quantify the relationship between flaws and significant results. Hyman identified 12 potential flaws in the ganzfeld experiments, such as inadequate random- Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 ization, multiple tests used without adjusting the significance level (thus inflating the significance level from the nominal 5%) and failure to use a duplicate set of targets for the judging process (thus allowing possible clues such as fingerprints). Using cluster and factor analyses, the 12 binary flaw variables were combined into three new variables, which Hyman named General Security, Statistics and Controls. Several. analyses were then conducted. The one reported with the most detail is a factor analysis utilizing 17 variables for each of 36 studies. Four factors emerged from the analysis. From these, Hyman concluded that security had increased over the years, that the significance level tended to be inflated the most for the most complex studies and that both effect size and level of significance were correlated with the existence of flaws. Following his factor analysis, Hyman picked the three flaws that seemed to be most highly corre- lated with success, which were inadequate atten- tion to both randomization and documentation and the potential for ordinary communication between the sender and receiver. A regression equation was then computed using each of the three flaws as dummy variables, and the effect size for the experi- ment as the dependent variable. From this equa- tion, Hyman concluded that a study without these three flaws would be predicted to have a hit rate of 27%. He concluded that this is "well within the statistical neighborhood of the 25% chance rate" (1985b, page 37), and thus "the ganzfeld psi data base, despite initial impressions, is inadequate ei- ther to support the contention of a repeatable study or to demonstrate the reality of psi" (page 38). Honorton discounted both Hyman's flaw classifi- cation and his analysis. He did not deny that flaws existed, but he objected that Hyman's analysis was faulty and impossible to interpret. Honorton asked psychometrician David Saunders to write an Ap- pendix to his article, evaluating Hyman's analysis. Saunders first criticized Hyman's use of a factor analysis with 17 variables (many of which were dichotomous) and only 36 cases and concluded that "the entire analysis is meaningless" (Saunders, 1985, page 87). He then noted that Hyman's choice of the three flaws to include in his regression anal- ysis constituted a clear case of multiple analysis, since there were 84 possible sets of three that could have been selected (out of nine potential flaws), and Hyman chose the set most highly correlated with effect size. Again, Saunders concluded that "any interpretation drawn from [the regression analysis] must be regarded as meaningless" (1985, page 88). Hyman's results were also contradicted by Harris and Rosenthal (1988b) in an analysis requested by Hyman in his capacity as Chair of the National Academy of Sciences' Subcommittee on Parapsy- chology. Using Hyman's flaw classifications and a multivariate analysis, Harris and Rosenthal con- cluded that "Our analysis of the effects of flaws on study outcome lends no support to the hypothesis that ganzfeld research results are a significant function of the set of flaw variables" (1988b, page 3). Hyman and Honorton were in the process of preparing papers for a second round of debate when they were invited to lunch together at the 1986 Meeting of the Parapsychological Association. They discovered that they were in general agreement on several major issues, and they decided to coauthor a "Joint Communique" (Hyman and Honorton, 1986). It is clear from their paper that they both thought it was more important to set the stage for future experimentation than to continue the techni- cal arguments over the current data base. In the abstract to their paper, they wrote: We agree that there is an overall significant effect in this data base that cannot reasonably be explained by selective reporting or multiple analysis. We continue to differ over the degree to which the effect constitutes evidence for psi, but we agree that the final verdict awaits the outcome of future experiments conducted by a broader range of investigators and according to more stringent standards [page 351]. The paper then outlined what these standards should be. They included controls against any kind of sensory leakage, thorough testing and documen- tation of randomization methods used, better re- porting of judging and feedback protocols, control for multiple analyses and advance specification of number of trials and type of experiment. Indeed, any area of research could benefit from such a careful list of procedural recommendations. 4.5 Rosenthal's Meta-Analysis The same issue of the Journal of Parapsychology in which the Joint Communique appeared also car- ried commentaries on the debate by 10 separate authors. In his commentary, psychologist Robert Rosenthal, one of the pioneers of meta-analysis in psychology, summarized the aspects of Hyman's and Honorton's work that would typically be in- cluded in a meta-analysis (Rosenthal, 1986). It is worth reviewing Rosenthal's results so that they can be used as a basis of comparison for the more recent psi ganzfeld studies reported in Section 5. Rosenthal, like Hyman and Honorton, focused only on the 28 studies for which direct hits were known. He chose to use an effect size measure Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 called Cohen's h, which is the difference between the aresin transformed proportions of direct hits that were observed and expected: h 2 (aresin - arcsin VI-p). One advantage of this measure over the difference in raw proportions is that it can be used to compare experiments with different chance hit rates. If the observed and expected numbers of hits were identical, the effect size would be zero. Of the 28 studies, 23 (82%) had effect sizes greater than zero, with a median effect size of 0.32 and a mean of 0.28. These correspond to direct hit rates of 0.40 and 0.38 respectively, when 0.25 is expected by chance. A 95% confidence interval for the true effect size is from 0.11 to 0.45, corresponding to direct hit rates of from 0.30 to 0.46 when chance is 0.25. A common technique in meta-analysis is to calcu- late a "combined z," found by summing the indi- vidual z scores and dividing by the square root of the number of studies. The result should have a standard normal distribution if each z score has a standard normal distribution. For the ganzfeld studies, Rosenthal reported a combined z of 6.60 with a p-value of 3.37 x 10-11. He also reiterated Honorton's file-drawer assessment by calculating that there would have to be 423 studies unreported to negate the significant effect in the 28 direct hit studies. Finally, Rosenthal acknowledged that, because of the flaws in the data base and the potential for at least a small file-drawer effect, the true average effect size was probably closer to 0.18 than 0.28. He concluded, "Thus, when the accuracy rate expected under the null is 1/4, we might estimate the ob- tained accuracy rate to be about 1/3" (1986, page 333). This is the value used for the earlier power calculation. It is worth mentioning that Rosenthal was com- missioned by the National Academy of Sciences to prepare a background paper to accompany its 1988 report on parapsychology. That paper (Harris and Rosenthal, 1988a) contained much of the same analysis as his commentary summarized above. Ironically, the discussion of the ganzfeld work in the National Academy Report focused on Hyman's 1985 analysis, but never mentioned the work it had commissioned Rosenthal to perform, which contra- dicted the final conclusion in the report. 5. A META.-ANALYSIS OF RECENT GANZFELD EXPERIMENTS After the initial exchange with Hyman at the 1982 Parapsychological Association Meeting, Honorton and his colleagues developed an auto- mated ganzfeld experiment that was designed to eliminate the methodological flaws identified by Hyman. The execution and reporting of the experi- ments followed the detailed guidelines agreed upon by Hyman and Honorton. Using this "autoganzfeld" experiment, 11 experi- mental series were conducted by eight experi- menters between February 1983 and September 1989, when the equipment had to be dismantled due to lack of funding. In this section, the results of these experiments are summarized and com- pared to the earlier ganzfeld studies. Much of the information is derived from Honorton et al. (1990). Like earlier ganzfeld studies, the "autoganzfeld" experiments require four participants. The first is the Receiver (R), who attempts to identify the tar- get material being observed by the Sender (S). The Experimenter (E) prepares R for the task, elicits the response from R and supervises R's judging of the response against the four potential targets. (Judging is double blind; E does not know which is the correct target.) The fourth participant is the lab assistant (LA) whose only task is to instruct the computer to randomly select the target. No one involved in the experiment knows the identity of the target. Both R and S are sequestered in sound-isolated, electrically shielded rooms. R is prepared as in earlier ganzfeld studies, with white noise and a field of red light. In a nonadjacent room, S watches the target material on a television and can hear R's target description ("mentation") as it is being given. The mentation is also tape recorded. The judging process takes place immediately af- ter the 30-minute sending period. On a TV monitor in the isolated room, R views the four choices from the target pack that contains the actual target. R is asked to rate each one according to how closely it matches the ganzfeld mentation. The ratings are converted to ranks and, if the correct target is ranked first, a direct hit is scored. The entire proc- ess is automatically recorded by the computer. The computer then displays the correct choice to R as feedback. There were 160 preselected targets, used with replacement, in 10 of the 11 series. They were arranged in packets of four, and the decoys for a given target were always the remaining three in the same set. Thus, even if a particular target in a set were consistently favored by Rs, the probability of a direct hit under the null hypothesis would remain at 1/4. Popular targets should be no more Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 likely to be selected by the computer's random number generator than any of the others in the set. The selection of the target by the computer is the only source of randomness in these experiments. This is an important point, and one that is often misunderstood. (See Utts, 1991, for elucidation.) Eighty of the targets were "dynamic," consisting of scenes from movies, documentaries and cartoons; 80 were "static," consisting of photographs, art prints and advertisements. The four targets within each set were all of the same type. Earlier studies indicated that dynamic targets were more likely to produce successful results, and one of the goals of the new experiments was to test that theory. The randomization procedure used to select the target and the order of presentation for judging was thoroughly tested before and during the experi- ments. A detailed description is given by Honorton et al. (1990, pages 118-120). Three of the 11 series were pilot series, five were formal series with novice receivers, and three were formal series with experienced receivers. The last series with experienced receivers was the only one that did not use the 160 targets. Instead, it used only one set of four dynamic targets in which one target had previously received several first place ranks and. one had never received a first place rank. The receivers, none of whom had had prior exposure to that target pack, were not aware that only one target pack was being used. They each contributed one session only to the series. This will be called the "special series" in what follows. Except for two of the pilot series, numbers of trials were planned in advance for each series. Unfortunately, three of the formal series were not yet completed when the funding ran out, including the special series, and one pilot study with advance planning was terminated early when the experi- menter relocated. There were no unreported trials during the 6-year period under review, so there was no "file drawer." Overall, there were 183 Rs who contributed only one trial and 58 who contributed more than one, for a total of 241 participants and 355 trials. Only 23 Rs had previously participated in ganzfeld experi- ments, and 194 Rs (81%) had never participated in any parapsychological research. While acknowledging that no probabilistic con- clusions can be drawn from qualitative data, Hon- orton et al. (1990) included several examples of session excerpts that Rs identified as providing the basis for their target rating. To give a flavor for the dream-like quality of the mentation and the amount of information that can be lost by only assigning a rank, the first example is reproduced here. The target was a painting by Salvador Dali called "Christ Crucified." The correct target received a first place rank. The part of the mentation R used to make this assessment read: .,.. I think of guides, like spirit guides, leading me and I come into a court with a king. It's quiet .... It's like heaven. The king is some- thing like Jesus. Woman. Now I'm just sort of summersaulting through heaven ... . Brooding .... Aztecs, the Sun God .... High priest ....Fear . . . . Graves. Woman. Prayer . . . . Funeral . . . . Dark. Death ... Souls .... Ten Commandments. Moses .... [Honorton et al., 1990]. Over all 11 series, there were 122 direct hits in the 355 trials, for a hit rate of 34.4% (exact bino- mial p-value = 0.00005) when 25% were expected by chance. Cohen's h is 0.20, and a 95% confidence interval for the overall hit rate is from 0.30 to 0.39. This calculation assumes, of course, that the proba- bility of a direct hit is constant and independent across trials, an assumption that may be question- able except under the null hypothesis of no psi abilities. Honorton et al. (1990) also calculated effect sizes for each of the 11 series and each of the eight experimenters. All but one of the series (the first novice series) had positive effect sizes, as did all of the experimenters. The special series with experienced Rs had an exceptionally high effect size with h = 0.81, corre- sponding to 16 direct hits out of 25 trials (64%), but the remaining series and the experimenters had relatively homogeneous effect sizes given the amount of variability expected by chance. If the special series is removed, the overall hit rate is 32.1%, h = 0.16. Thus, the positive effects are not due to just one series or one experimenter. Of the 218 trials contributed by novices, 71 were direct hits 7(32.5%, h = 0.17), compared with 51 hits in the 137 trials by those with prior ganzfeld experience (37%, h = 0.26). The hit rates and effect sizes were 31% (h = 0.14) for the combined pilot series, 32.5% (h = 0.17) for the combined formal novice series, and 41.5% (h = 0.35) for the com- bined experienced series. The last figure drops to 31.6% if the outlier series is removed. Finally, without the outlier series the hit rate for the com- bined series where all of the planned trials were completed was 31.2% (h = 0.14), while it was 35% (h = 0.22) for the combined series that were termi- nated early. Thus, optional stopping cannot account for the positive effect. Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 There were two interesting comparisons that had been suggested by earlier work and were pre- planned in these experiments. The first was to compare results for trials with dynamic targets with those for static targets. In the 190 dynamic target sessions there were 77 direct hits (40%, h = 0.32) and for the static targets there were 45 hits in 165 trials (27%, h = 0.05), thus indicating that dynamic targets produced far more successful results. The second comparison of interest was whether or not the sender was a friend of the receiver. This was a choice the receiver could make. If he or she did not bring a friend, a lab member acted as sender. There were 211 trials with friends as senders (some of whom were also lab staff), result- ing in 76 direct hits (36%, h 0.24). Four trials used no sender. The remaining 140 trials used nonfriend lab staff as senders and resulted in 46 direct hits (33%, h = 0.18). Thus, trials with friends as senders were slightly more successful than those without. Consonant with the definition of replication based on consistent effect sizes, it is informative to com- pare the autoganzfeld experiments with the direct hit studies in the previous data base. The overall success rates are extremely similar. The overall direct hit rate was 34.4% for the autoganzfeld stud- ies and was 38% for the comparable direct hit studies in the earlier meta-analysis. Rosenthal's (1986) adjustment for flaws had placed a more con- servative estimate at 33%, very close to the observed 34.4% in the new studies. One limitation of this work is that the auto- ganzfeld studies, while conducted by eight experi- menters, all used the same equipment in the same laboratory. Unfortunately, the level of fund- ing available in parapsychology and the cost in time and equipment to conduct proper experiments make it difficult to amass large amounts of data across laboratories. Another autoganzfeld labora- tory is currently being constructed at the Univer- sity of Edinburgh in Scotland, so interlaboratory comparisons may be possible in the near future. Based on the effect size observed to date, large samples are :needed to achieve reasonable power. If there is a constant effect across all trials, resulting in 33% direct hits when 25% are expected by chance, to achieve a one-tailed significance level of 0.05 with 95% probability would require 345 sessions. We end this section by returning to the aspirin and heart attack example in Section 3 and expand- ing a comparison noted by Atkinson, Atkinson, Smith and Item (1990, page 237). Computing the equivalent of Cohen's h for comparing obser- ved heart attack rates in the aspirin and placebo groups results in h = 0.068. Thus, the effect size observed in the ganzfeld data base is triple the much publicized effect of aspirin on heart attacks. 6. OTHER META-ANALYSES IN PARAPSYCHOLOGY Four additional meta-analyses have been con- ducted in various areas of parapsychology since the original ganzfeld meta-analyses were reported. Three of the four analyses focused on evidence of psi abilities, while the fourth examined the rela- tionship between extroversion and psychic func- tioning. In this section, each of the four analyses will be briefly summarized. There are only a handful of English-language journals and proceedings in parapsychology, so retrieval of the relevant studies in each of the four cases was simple to accomplish by searching those sources in detail and by searching other bibliographic data bases for keywords. Each analysis included an overall summary, an analysis of the quality of the studies versus the size of the effect and a "file-drawer" analysis to deter- mine the possible number of unreported studies. Three of the four also contained comparisons across various conditions. 6.1 Forced-Choice Precognition Experiments Honorton and Ferrari (1989) analyzed forced- choice experiments conducted from 1935 to 1987, in which the target material was randomly selected after the subject had attempted to predict what it would be. The time delay in selecting the target ranged from under a second to one year. Target material included items as diverse as ESP cards and automated random number generators. Two investigators, S. G. Soal and Walter J. Levy, were not included because some of their work has been suspected to be fraudulent. Overall Results. There were 309 studies re- ported by 62 senior authors, including more than 50,000 subjects and nearly two million individual trials. Honorton and Ferrari used z// as the measure of effect size (ES) for each study, where n was the number of Bernoulli trials in the study. They reported a mean ES of 0.020, and a mean z-score of 0.65 over all studies. They also reported a combined z of 11.41, p = 6.3 x 10-25. Some 30% (92) of the studies were statistically significant at a = 0.05. The mean ES per investigator was 0.033, and the significant results were not due to just a few investigators. Quality. Eight dichotomous quality measures were assigned to each study, resulting in possible Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 scores from zero for the lowest quality, to eight for the highest. They included features such as ade- quate randomization, preplanned analysis and au- tomated recording of the results. The correlation between ;study quality and effect size was 0.081, indicating a slight tendency for higher quality studies to be more successful, contrary to claims by critics that the opposite would be true. There was a clear relationship between quality and year of publication, presumably because over the years experimenters in parapsychology have responded to suggestions from critics for improving their methodology. File Drawer. Following Rosenthal (1984), the authors calculated the "fail-safe N" indicating the number of unreported studies that would have to be sitting in file drawers in order to negate the signifi- cant effect. They found N = 14,268, or a ratio of 46 unreported studies for each one reported. They also followed a suggestion by Dawes, Landman and Williams (1984) and computed the mean z for all studies with z > 1.65. If such studies were a ran- dom sample from the upper 5% tail of a N(0,1) distribution, the mean z would be 2.06. In this case it was 3.61. They concluded that selective reporting could not explain these results. Comparisons. Four variables were identified that appeared to have a systematic relationship to study outcome. The first was that the 25 studies using subjects selected on the basis of good past performance were more successful than the 223 using unselected subjects, with mean effect sizes of 0.051 and. 0.008, respectively. Second, the 97 stud- ies testing subjects individually were more success- ful than the 105 studies that used group testing; mean effect sizes were 0.021 and 0.004, respec- tively. Timing of feedback was the third moderat- ing variable, but information was only available for 104 studies. The 15 studies that never told the subjects what the targets were had a mean effect size of -0.001. Feedback after each trial produced the best results, the mean ES for the 47 studies was 0.03;5. Feedback after each set of trials re- sulted in mean ES of 0.023 (21 studies), while delayed feedback (also 21 studies) yielded a mean ES of only 0.009. There is a clear ordering; as the gap between time of feedback and time of the actual guesses decreased, effect sizes increased. The fourth variable was the time interval be- tween the subject's guess and the actual target selection, available for 144 studies. The best results were for the 31 studies that generated targets less than a second after the guess (mean ES = 0.045), while the worst were for the seven studies that delayed target selection by at least a month (mean ES = 0.001). The mean effect sizes showed a clear trend, decreasing in order as the time interval increased from minutes to hours to days to weeks to months. 6.2 Attempts to Influence Random Physical Systems Radin and Nelson (1989) examined studies de- signed to test the hypothesis that "The statistical output of an electronic RNG [random number gen- erator] is correlated with observer intention in ac- cordance with prespecified instructions" (page 1502). These experiments typically involve RNGs based on radioactive decay, electronic noise or pseu- dorandom number sequences seeded with true ran- dom sources. Usually the subject is instructed to try to influence the results of a string of binary trials by mental intention alone. A typical protocol would ask a subject to press a button (thus starting the collection of a fixed-length sequence of bits), and then try to influence the random source to produce more zeroes or more ones. A run might consist of three successive button presses, one each in which the desired result was more zeroes or more ones, and one as a control with no conscious intention. A z score would then be computed for each button press. The 832 studies in the analysis were conducted from 1959 to 1987 and included 235 "control" stud- ies, in which the output of the RNGs were recorded but there was no conscious intention involved. These were usually conducted before and during the experimental series, as tests of the RNGs. Results. The effect size measure used was again z / v , where z was positive if more bits of the specified type were achieved. The mean effect size for control studies was not significantly different from zero (-1.0 x 10-s). The mean effect size for the experimental studies was also very small, 3.2 x 10-4, but it was significantly higher than the mean ES for the control studies (z = 4.1). Quality. Sixteen quality measures were defined and assigned to each study, under the four general categories of procedures, statistics, data and the RNG device. A score of 16 reflected the highest quality. The authors regressed mean effect size on mean quality for each investigator and found a slope of 2.5 x 10-5 with standard error of 3.2 x 10-5, indicating little relationship between quality and outcome. They also calculated a weighted mean effect size, using quality scores as weights, and found that it was.very similar to the unweighted mean ES. They concluded that "differences in methodological quality are not significant predictors of effect size" (page 1507). File Drawer. Radin and Nelson used several methods for estimating the number of unreported Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 studies (pages 1508-1510). Their estimates ranged from 200 to 1000 based on models assuming that all significant studies were reported. They calculated the fail-safe N to be 54,000. 6.3 Attempts; to Influence Dice Radin and Ferrari (1991) examined 148 studies, published from 1935 to 1987, designed to test whether or not consciousness can influence the results of tossing dice. They also found 31 "con- trol" studies in which no conscious intention was involved. Results. The effect size measure used was z / v , where z was based on the number of throws in which the die landed with the desired face (or faces) up, in n throws. The weighted mean ES for the experimental studies was 0.0122 with a stan- dard error of 0.00062; for the control studies the mean and standard error were 0.00093 and 0.00255, respectively. Weights for each study were de- termined by quality, giving more weight to high- quality studies. Combined z scores for the exper- imental and control studies were reported by Radin and Ferrari to be 18.2 and 0.18, respectively. Quality. Eleven dichotomous quality measures were assigned, ranging from automated recording to whether or not control studies were interspersed with the experimental studies. The final quality score for each study combined these with informa- tion on method of tossing the dice, and with source of subject (defined below). A regression of quality score versus effect size resulted in a slope of - 0.002, with a standard error of 0.0011. However, when effect sizes were weighted by sample size, there was a significant relationship between quality and ef- fect size, leading Radin and Ferrari to conclude that higher-quality studies produced lower weighted effect sizes. File Drawer: Radin and Ferrari calculated Rosenthal's fail-safe. N for this analysis to be 17,974. Using the assumption that all significant studies were reported, they estimated the number of unreported studies to be 1152. As a final assess- ment, they compared studies published before and after 1975, when the Journal of Parapsychology adopted an official policy of publishing nonsigni- ficant results. They concluded, based on that an- alysis, that more nonsignificant studies were published after 1975, and thus "We must consi- der the overall (1935-1987) data base as suspect with respect to the filedrawer problem." Comparisons. Radin and Ferrari noted that there was bias in both the experimental and control studies across die face. Six was the face most likely to come up, consistent with the observation that it has the least mass. Therefore, they examined re- sults for the subset of 69 studies in which targets were evenly balanced among the six faces. They still found a significant effect, with mean and stan- dard error for effect size of 8.6 x 10-3 and 1.1 x 10-3, respectively. The combined z was 7.617 for these studies. They also compared effect sizes across types of subjects used in the studies, categorizing them as unselected, experimenter and other subjects, exper- imenter as sole subject, and specially selected sub- jects. Like Honorton and Ferrari (1989), they found the highest mean ES for studies with selected subjects; it was approximately 0.02, more than twice that for unselected subjects. 6.4 Extroversion and ESP Performance Honorton, Ferrari and Bem (1991) conducted a meta-analysis to examine the relationship between scores on tests of extroversion and scores on psi-related tasks. They found 60 studies by 17 investigators, conducted from 1945 to 1983. Results. The effect size measure used for this analysis was the correlation between each subject's extroversion score and ESP score. A variety of measures had been used for both scores across stud- ies, so various correlation coefficients were used. Nonetheless, a stem and leaf diagram of the corre- lations showed an approximate bell shape with mean and standard deviation of 0.19 and 0.26, respectively, and with an additional outlier at r = 0.91. Honorton et al. reported that when weighted by degrees of freedom, the weighted mean r was 0.14, with a 95% confidence interval covering 0.10 to 0.19. Forced-Choice versus Free-Response Re- sults. Because forced-choice and free-response tests differ qualitatively, Honorton et al. chose to exam- ine their relationship to extroversion separately. They found that for free-response studies there was a significant correlation between extroversion and ESP scores, with mean r = 0.20 and z = 4.46. Fur- ther, this effect was homogeneous across both investigators and extroversion scales. For forced-choice studies, there was a significant correlation between ESP and extroversion, but only for those studies that reported the ESP results to the subjects before measuring extroversion. Honorton et al. speculated that the relationship was an artifact, in which extroversion scores were temporarily inflated as a result of positive feedback on ESP performance. Confirmation with New Data Following the extroversion /ESP meta-analysis, Honorton et al. attempted to confirm the relationship using the autoganzfeld data base. Extroversion scores based on the Myers-Briggs Type Indicator were available for 221 of the '241 subjects who had participated in autoganzfeld studies. Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 REPLICATION IN PARAPSYCHOLOGY The correlation between extroversion scores and ganzfeld rating scores was r = 0.18, with a 95% confidence interval from 0.05 to 0.30. This is con- sistent with the mean correlation of r = 0.20 for free-response experiments, determined from the meta-analysis. These correlations indicate that ex- troverted subjects can produce higher scores in free-response ESP tests. 7. CONCLUSIONS Parapsychologists often make a distinction be- tween "proof-oriented research" and "process- oriented research." The former is typically con- ducted to test the hypothesis that psi abilities exist, while the latter is designed to answer questions about how psychic functioning works. Proof- oriented research has dominated the literature in parapsychology. Unfortunately, many of the studies used small samples and would thus be nonsignificant even if a moderate-sized effect exists. The recent focus on meta-analysis in parapsy- chology has revealed that there are small but consistently nonzero effects across studies, experi- menters and laboratories. The sizes of the effects in forced-choice studies appear to be comparable to those reported in some medical studies that had been heralded as breakthroughs. (See Section 5; also Honorton and Ferrari, 1989, page 301.) Free- response studies show effect sizes of far greater magnitude. A promising direction for future process-oriented research is to examine the causes of individual differences in psychic functioning. The ESP/ex- troversion meta-analysis is a step in that direction. In keeping with the idea of individual differ- ences, Bayes and empirical Bayes methods would appear to make more sense than the classical infer- ence methods commonly used, since they would allow individual abilities and beliefs to be modeled. Jeffreys (1990) reported a Bayesian analysis of some of the RNG experiments and showed that conclu- sions were closely tied to prior beliefs even though hundreds of thousands of trials were available. It may be that the nonzero effects observed in the meta-analyses can be explained by something other than ESP, such as shortcomings in our understand- ing of randomness and independence. Nonetheless, there is an anomaly that needs an explanation. As I have argued elsewhere (Utts, 1987), research in parapsychology should receive more support from the scientific community. If ESP does not exist, there is little to be lost by erring in the direction of further research, which may in fact uncover other anomalies. If ESP does exist, there is much to be lost by not doing process-oriented research, and much to be gained by discovering how to enhance and apply these abilities to important world problems. ACKNOWLEDGMENTS I would like to thank Deborah Delanoy, Charles Honorton, Wesley Johnson, Scott Plous and an anonymous reviewer for their helpful comments on an earlier draft of this paper, and Robert Rosenthal and Charles Honorton for discussions that helped clarify details. REFERENCES ATKINSON, R. L., ATKINSON, R. C., SMITH, E. E. and BEM, D. J. (1990). Introduction to Psychology, 10th ed. Harcourt Brace Jovanovich, San Diego. BELOFF, J. (1985). Research strategies for dealing with unstable phenomena. In The Repeatability Problem in Parapsychol- ogy (B. Shapin and L. Coly, eds.) 1-21. Parapsychology Foundation, New York. BLACKMORE, S. J. (1985). Unrepeatability: Parapsychology's only finding. In The Repeatability Problem in Parapsychology (B. Shapin and L. Coly, eds.) 183-206. Parapsychology Foundation, New York. BURDICK, D. S. and KELLY, E. F. (1977). Statistical methods in parapsychological research. In Handbook of Parapsychology (B. B. Wolman, ed.) 81-130. Van Nostrand Reinhold, New York. CAMP, B. H. (1937). (Statement in Notes Section.) Journal of Parapsychology 1 305. COHEN, J. (1990). Things I have learned (so far). American Psychologist 45 1304-1312. COOVER, J. E. (1917). Experiments in Psychical Research at Leland Stanford Junior University. Stanford Univ. DAWES, R. M., LANDMAN, J. and WILLIAMS, J. (1984). Reply to Kurosawa. American Psychologist 39 74-75. DIACONIS, P. (1978). Statistical problems in ESP research. Sci- ence 201 131-136. DOMMEYER, F. C. (1975). Psychical research at Stanford Univer- sity. Journal of Parapsychology 39 173-205. DRUCKMAN, D. and SWETS, J. A., eds. (1988) Enhancing Human Performance: Issues, Theories, and Techniques. National Academy Press, Washington, D.C. EDGEWORTH, F. Y. (1885). The calculus of probabilities applied to psychical research. In Proceedings of the Society for Psychical Research 3 190-199. EDGEWORTH, F. Y. (1886). The calculus of probabilities applied to psychical research. II. In Proceedings of the Society for Psychical Research 4 189-208. FELLER, W. K. (1940). Statistical aspects of ESP. Journal of Parapsychology 4 271-297. FELLER, W. K. (1968). An Introduction to Probability Theory and Its Applications 1, 3rd ed. Wiley, New York. FISHER, R. A. (1924). A method of scoring coincidences in tests with playing cards. In Proceedings of the Society for Psychi- cal Research 34 181-185. FISHER, R. A. (1929). The statistical method in psychical re- search. In Proceedings of the Society for Psychical Research 39 189-192. GALLUP, G. H., JR., and NEWPORT, F. (1991). Belief in paranor- mal phenomena among adult Americans. Skeptical Inquirer 15 137-146. GARDNER, M. J. and ALTMAN, D. G. (1986). Confidence intervals rather than p-values: Estimation rather than hypothesis testing. British Medical Journal 292 746-750. Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 GILMORE, J. B. (1989). Randomness and the search for psi. Journal of Parapsychology 53 309-340. GILMORE, J. B. (1990). Anomalous significance in pararandom and psi-free domains. Journal of Parapsychology 54 53-58. GREELEY, A. (1987). Mysticism goes mainstream. American Health 7 47 -49. GREENHOUSE, J. B. and GREENHOUSE, S. W. (1988). An aspirin a day ... ? Chance 124-31. GREENWOOD, J. A. and STUART, C. E. (1940). A review of Dr. Feller's critique. Journal of Parapsychology 4 299-319. HACKING, I. (1988). Telepathy: Origins of randomization in ex- perimental design. Isis 79 427-451. HANSEL, C. E. M. (1980). ESP and Parapsychology: A Critical Re-evaluation. Prometheus Books, Buffalo, N.Y. HARRIS, M. J. and ROSENTHAL, R. (1988a). Interpersonal Ex- pectancy Effects and Human Performance Research. Na- tional Academy Press, Washington, D.C. HARRIS, M. J. and ROSENTHAL, R. (1988b). Postscript to Interper- sonal Expectancy Effects and Human Performance Research. National Academy Press, Washington, D.C. HEDGES, L. V. and OLKIN, I. (1985). Statistical Methods for Meta-Analysis. Academic, Orlando, Fla. HONORTON, C. (1977). Psi and internal attention states. In Handbook of Parapsychology (B. B. Wolman, ed.) 435-472. Van Nostrand Reinhold, New York. HONORTON, C. (1985a). How to evaluate and improve the repli- cability of parapsychological effects. In The Repeatability Problem in Parapsychology (B. Shapin and L. Coly, eds.) 238-255. Parapsychology Foundation, New York. HONORTON, C. (:1985b). Meta-analysis of psi ganzfeld research: A response to Hyman. Journal of Parapsychology 49 51-91. HONORTON, C., BERGER, R. E., VARVOGLIS, M. P., QUANT, M., DERR, P., SCHECHTER, E. I. and FERRARI, D. C. (1990). Psi communication in the ganzfeld: Experiments with an automated testing system and a comparison with a meta- analysis of earlier studies. Journal of Parapsychology 54 99-139. HONORTON, C. and FERRARI, D. C. (1989). "Future telling": A meta-analysis of forced-choice precognition experiments, 1935-1987.. Journal of Parapsychology 53 281-308. HONORTON, C., FERRARI, D. C. and BEM, D. J. (1991). Extraver- sion and ESP performance: A meta-analysis and a new confirmation. Research in Parapsychology 1990. The Scare- crow Press, Metuchen, N.J. To appear. HYMAN, R. (1985a). A critical overview of parapsychology. In A Skeptic's Handbook of Parapsychology (P. Kurtz, ed.) 1-96. Prometheus Books, Buffalo, N.Y. HYMAN, R. (1985b). The ganzfeld psi experiment: A critical appraisal. Journal of Parapsychology 49 3-49. HYMAN, R. and HONORTON, C. (1986). Joint communiqud: The psi ganzfeld controversy. Journal of Parapsychology 50 351-364. IVERSEN, G. R.,, LONGCOR, W. H., MOSTELLER, F., GILBERT, J. P. and YouTz, C. (1971). Bias and runs in dice throwing and recording: A few million throws. Psychometrika 36 1-19. JEFFREYS, W. H. (1990). Bayesian analysis of random event generator data. Journal of Scientific Exploration 4 153-169. LINDLEY, D. V. (1957). A statistical paradox. Biometrika 44 187-192. MAUSKOPF, S. H. and MCVAUGH, M. (1979). The Elusive Science: Origins of Experimental Psychical Research. Johns Hopkins Univ. Press. MCVAUGH, M. R. and MAUSKOPF, S. H. (1976). J. B. Rhine's Extrasensory Perception and its background in psychical research. Isis 67 161-189. NEULIEP, J. W., ed. (1990). Handbook of replication research in the behavioral and social sciences. Journal of Social Behav- ior and Personality 5 (4) 1-510. OFFICE OF TECHNOLOGY ASSESSMENT (1989). Report of a work- shop on experimental parapsychology. Journal of the Amer- ican Society for Psychical Research 83 317-339. PALMER, J. (1989). A reply to Gilmore. Journal of Parapsychol- ogy 53 341-344. PALMER, J. (1990). Reply to Gilmore: Round two. Journal of Parapsychology 54 59-61. PALMER, J. A., HONORTON, C. and Urrs, J. (1989). Reply to the National Research Council study on parapsychology. Jour- nal of the American Society for Psychical Research 83 31-49. RADIN, D. I. and FERRARI, D. C. (1991). Effects of consciousness on the fall of dice: A meta-analysis. Journal of Scientific Exploration 5 61-83. RADIN, D. I. and NELSON, R. D. (1989). Evidence for conscious- ness-related anomalies in random physical systems. Foun- dations of Physics 19 1499-1514. RAO, K. R. (1985). Replication in conventional and controversial sciences. In The Repeatability Problem in Parapsychology (B. Shapin and L. Coly, eds.) 22-41. Parapsychology Foun- dation, New York. RHINE, J. B. (1934). Extrasensory Perception. Boston Society for Psychical Research, Boston. (Reprinted by Branden Press, 1964.) RHINE, J. B. (1977). History of experimental studies. In Hand- book of Parapsychology (B. B. Wolman, ed.) 25-47. Van Nostrand Reinhold, New York. RICHET, C. (1884). La suggestion mentale et le calcul des proba- bilites. Revue Philosophique 18 608-674. ROSENTHAL, R. (1984). Meta-Analytic Procedures for Social Re- search. Sage, Beverly Hills. ROSENTHAL, R. (1986). Meta-analytic procedures and the nature of replication: The ganzfeld debate. Journal of Parapsychol- ogy 50 315-336. ROSENTHAL, R. (1990a). How are we doing in soft psychology? American Psychologist 45 775-777. ROSENTHAL, R. (1990b). Replication in behavioral research. Journal of Social Behavior and Personality 5 1-30. SAUNDERS, D. R. (1985). On Hyman's factor analysis. Journal of Parapsychology 49 86-88. SHAPIN, B. and COLY, L., eds. (1985). The Repeatability Problem in Parapsychology. Parapsychology Foundation, New York. SPENCER-BROWN, G. (1957). Probability and Scientific Inference. Longmans Green, London and New York. STUART, C. E. and GREENWOOD, J. A. (1937). A review of criti- cisms of the mathematical evaluation of ESP data. Journal of Parapsychology 1 295-304. TVERSKY, A. and KAHNEMAN, D. (1982). Belief in the law of small numbers. In Judgment Under Uncertainty: Heuristics and Biases (D. Kahneman, P. Slovic and A. Tversky, eds.) 23-31. Cambridge Univ. Press. Urrs, J. (1986). The ganzfeld debate: A statistician's perspec- tive. Journal of Parapsychology 50 395-402. UTrs, J. (1987). Psi, statistics, and society. Behavioral and Brain Sciences 10 615-616. Urrs, J. (1988). Successful replication versus statistical signifi- cance. Journal of Parapsychology 52 305-320. Urrs, J. (1989). Randomness and randomization tests: A reply to Gilmore. Journal of Parapsychology 53 345-351. Urrs, J. (1991). Analyzing free-response data: A progress report. In Psi Research Methodology: A Re-examination (L. Coly, ed.). Parapsychology Foundation, New York. To appear. WILKS, S. S. (1965a). Statistical aspects of expeirments in telepath. N.Y. Statistician 16 (6) 1-3. WILKS, S. S. (1965b). Statistical aspects of experiments in telepathy. N. Y. Statistician 16 (7) 4-6. Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 J. UITS 379 Comment M. J. Bayard and James Berger 1. INTRODUCTION There are many fascinating issues discussed in this paper. Several concern parapsychology itself and the interpretation of statistical methodology therein. We are not experts in parapsychology, and so have only one comment concerning such mat- ters: In Section 3 we briefly discuss the need to switch from P-values to Bayes factors in discussing evidence concerning parapsychology. A more general issue raised in the paper is that of replication. It is quite illuminating to consider the issue of replication from a Bayesian perspec- tive, and this is done in Section 2 of our discussion. Many insightful observations concerning replica- tion are given in the article, and these spurred us to determine if they could be quantified within Bayesian reasoning. Quantification requires clear delineation. of the possible purposes of replication, and at least two are obvious. The first is simple reduction of random error, achieved by obtaining more observations from the replication. The second purpose is to search for possible bias in the original experiment. We use "bias" in a loose sense here, to refer to any of the huge number of ways in which the effects being measured by the experiment can differ from. the actual effects of interest. Thus a clinical trial without a placebo can suffer a placebo "bias"; a Survey can suffer a "bias" due to the sampling frame being unrepresentative of the actual population; and possible sources of bias in parapsychological experiments have been extensively discussed. Replication to_Reduce Random Error If the sole goal of replication of an experiment is to reduce random error, matters are very straight- forward. Reviewing the Bayesian way of studying this issue is, however, useful and will be done through the following simple example. M. J. Bayarri is Titular Professor, Department of Statistics and Operations Research, University of Valencia, Avenida Dr. Moliner 50, 46100 Burjassot, Valencia, Spain. James Berger is the Richard M. Brumfield Distinguished Professor of Statistics, Purdue University, West Lafayette, Indiana 47907. EXAMPLE 1. Consider the example from Tversky and Kahnemann (1982), in which an experiment results in a standardized test statistic of z1 = 2.46. (We will assume normality to keep computations trivial.) The question is: What is the highest value of z2 in a second set of data that would be consid- ered a failure to replicate? Two possible precise versions of this question are: Question 1: What is the probability of observing z2 for which the null hypothesis would be rejected in the replicated ex- periment? Question 2: What value of z2 would leave one's overall opinion about the null hypothe- sis unchanged? Consider the simple case where Z1 -- N(z1 0, 1) and (independently) Z2 -- N(z2 101 1), where 0 is the mean and 1 is the standard deviation of the normal distribution. Note that we are considering the case in which no experimental bias is suspected and so the means for each experiment are assumed to be the same. Suppose that it is desired to test Ho: 0 s 0 versus Hl: 0 > 0, and suppose that initial prior opinion about 0 can be described by the noninformative prior r(0) = 1. We consider the one-sided testing problem with a constant prior in this section, be- cause it is known that then the posterior probabil- ity of H0, to be denoted by P(H0 data), equals the P-value, allowing us to avoid complications arising from differences between Bayesian and classical answers. After observing z1 = 2.46, the posterior distribu- tion of 0 is ir(0Iz1) =N(012.46,1). Question 1 then has the answer (using predictive Bayesian reasoning) P(rejecting at level a z1) 0000 1 C. 727 d8 dz2 c< - 2.46 =1-~ ), where 4 is the standard normal cdf and c,, is the (one-sided) critical value corresponding to the level, a, of the test. For instance, if a = 0.05, then this probability equals 0.7178, demonstrating that there is a quite substantial probability that the second experiment will fail to reject. If a is chosen to be the observed significance level from the first exper- iment, so that c, = z1, then the probability that the Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 Approved For Release 2003/09/10 : CIA-RDP96-00792R000100130004-9 second experiment will reject is just 1/2. This is nothing but a statement of the well-known martin- gale property of Bayesianism, that what you "ex- pect" to see in the future is just what you know today. In a sense, therefore, question 1 is exposed as being uninteresting. Question 2 more properly focuses on the fact that the stated goal of replication here is simply to reduce uncertainty in stated conclusions. The an- swer to the question follows immediately from not- ing that the posterior from the combined data (z1, z2) is 'r (B 1 Z1, Z2) = N(9 I (z1 + z2)/2, 1/vG ), P(Ho ldata) = t(- (z1 + z2)/s). Setting this equal to P(Ho I z1) and solving for z2 yields z2 = (V2-- - 1)z1 = 1.02. Any value of z2 greater than this will increase the total evidence against H0, while any value smaller than 1.02 will decrease the evidence. Replication to Detect Bias The aspirin example dramatically raises the is- sue of bias detection as a motive for replication. Professor Utts observes that replication 1 gives results that are fully compatible with those of the original study, which could be interpreted as sug- gesting that there is no bias in the original study, while replication 2 would raise serious concerns of bias. We became very interested in the implicit suggestion that replication 2 would thus lead to less overall evidence against the null hypothesis than would replication 1, even though in isolation replication 2 was much more "significant" than was replication 1. In attempting to see if this is so, we considered the Bayesian approach to study of bias within the framework of the aspirin example. EXAMPLE 2. For simplicity in the aspiring exam- ple, we reduce consideration to 0 = true difference in heart attack rates between aspirin and placebo populations multiplied by 1000; Y = difference in observed heart attack rates be- tween aspirin and placebo groups in original study multiplied by 1000; Xi = difference in observed heart attack rates be- tween aspirin and placebo groups in Replica- tion i multiplied by 1000. We assume that the replication studies are ex- tremely well designed and implemented, sQ that one is very confident that the Xi have mean 0. Using normal approximations for convenience, the data can be summarized as X1 -- N(x110, 4.82), X2 - N(x210, 3.63) with actual observations xl = 7.704 and x2 = 13.07. Consider now the bias issue. We assume that the original experiment is somewhat suspect in this regard, and we will model bias by defining the mean of Y to be ,i=0+0, where 0 is the unknown bias. Then the data in the original experiment can be summarized by Y--N(yIv,1.54), with the actual observation being y = 7.707. Bayesian analysis requires specification of a prior distribution, a(f3), for the suspected amount of bias. Of particular interest then are the posterior distri- bution of 0, assuming replication i has been performed, given by ir(al y, xi)