An Experiment in Decision Analysis in Israel in 1975
Speaking to Policymakers
“Minister of Foreign Affairs Alon wanted his own analysis of the outlook for the region in early 1975”
Knowing when and how to apply structured analytical techniques has been a continual challenge for US Intelligence Community analysts and the subject of IC training courses for years. In this article Nobel Laureate Daniel Kahneman and his colleague Zvi Lanir, then with the Israeli government, attempted to apply a technique to a pressing, real world issue for Jerusalem in 1975. At the time, Lanir was serving as leader of the just-established Center of Research and Political Planning in the Ministry of Foreign Affairs of the State of Israel, an entity similar in function to the Policy Planning Staff of the US State Department. Kahneman was professor of psychology at the Hebrew University in Jerusalem. There, he was involved in research in judgment and decisionmaking.
The two became involved in a research and analysis project requested by the then-minister of foreign affairs, Ygael Alon. Publication was not the objective of the project, which at the time was classified. More immediate policy concerns were at play. Lanir and Kahneman report on it more than 30 years later because, as they point out, some elements of their approach to the task are still relevant today in a variety of domains, including business and international relations. These elements include considerations in choosing structured techniques, communicating the results to senior policy makers, and coping with the mixture of success and failure the two experienced in the effort.
* * *
Historical and Intellectual Context
Israel in 1975 was still living the trauma of the catastrophic failure of intelligence that had preceded the costly 1973 war in which more than 2,300 soldiers lost their lives in three weeks— an equivalent loss of life for US forces today would be over 100,000 dead. Confidence in the Israeli intelligence community was badly shaken, and there was interest in new approaches. There was also a belief that decentralization of intelligence appraisal might prevent a recurrence of what was considered— with some benefit of hindsight—to have been blind allegiance to an incorrect conception of the strategic situation and of enemy intentions.
In the tense early months of 1975, Minister of Foreign Affairs Ygael Alon, in particular, wanted to have his own analysis of the political outlook in the region as US Secretary of State Henry Kissinger engaged in his famous exercise in shuttle diplomacy, attempting to achieve an agreement between Egypt and Israel. One message Kissinger delivered to the Israeli government was the CIA's judgment that failure to negotiate peace would have dire consequences for the region. The Intelligence Branch of the Israel Defense Forces generally concurred with this pessimistic view. The foreign minister was skeptical.
Meanwhile, on the academic scene, the 1970s was probably the heyday of decision research and analysis. Demonstrations of biases in human judgment under conditions of uncertainty were seen as supporting the prescriptive approach of decision analysis, by undermining, if only slightly, faith in the ability of decision makers to reach optimal decisions without aid. Danny was strongly influenced by the Stanford branch of decision analysts, led by the charismatic Ron Howard, who preached that options in complex decisions should be represented by "equivalent gambles," denominated in a common currency of utility or money. That form of decision analysis required an explicit formulation of a decision maker's values, as well as his or her sense of the probabilities of various outcomes—which were normally assumed to agree with the probabilities generated by experts in the organization to which the decision maker belonged.
At the time, Danny accepted the ideals of decision analysis, a field that provided, in his view, a gold standard for rational decisionmaking. However, he was aware of many obstacles to the application of decision analysis to significant choices. In particular, it was obvious that eliciting formal utilities and value trade-offs from decision makers was completely unrealistic in many situations, certainly including those involving strategic political and military decisions.
However, it still seemed to us that decision makers could benefit from systematic consideration of the probabilities of significant outcomes and of the effects of their choices on these probabilities. We also believed that properly trained analysts could make fairly sensible quantitative judgments of probability and conditional probability—or at least make judgments that convey useful information.
When Zvi approached Danny about this project, both were naively enthusiastic about what they saw as a chance to improve the rationality of decisionmaking on truly important issues. As a practitioner of intelligence analysis, Zvi's views had been formed in the field, but he was sympathetic to the academic approach and to decision analysis as a tool for improving rational decisionmaking. We worked as a team in designing and implementing the exercise.
The ideas that guided our approach to the problem were borrowed from the field of human engineering—the field devoted to making instruments "user-friendly" (a term that became popular much later). The foreign minister had asked for a report, and it was immediately obvious to us that the report should represent the opinions of a wide range of experts. In the spirit of human engineering, our task was to structure a report that would convey information to the decision maker as efficiently as possible and to structure the experts' task accordingly, while keeping their task manageable.
What should replace the "equivalent gamble" as a representation of the decision maker's uncertainty? The proposed solution was to start from the national leaders' concerns, the possibilities that sometimes kept them awake at night (the ultimate decision maker in Israel at the time was Golda Meir, who famously suffered from both worries and insomnia).
We construed "concerns" as events that are feared or hoped for and proposed that a list of these events, each associated with a probability would provide a representation of the "national gamble," the risk profile of the state of Israel. Borrowing an image from a field in which human engineering has been especially useful, the list of concerns is analogous to the display of dials in an airplane's cockpit. In this analogy, every dial the decision maker monitors shows the probability associated with a particular concern. The display in the cockpit is designed to make it easy for the pilot to notice changes in critical parameters.
Critical information must appear reliably and obviously, but the number of dials must be as small as possible because attention is a limited resource. Most of the time, of course, the displays are stable and their changes are predictable, but the early detection of anomalies is crucial for safe flying.
Our fantasy was to design an efficient cockpit for strategic decisionmaking. The key idea was to make it easy for decision makers to focus on new information. Most of the content of the reports decision makers hear or read is already known to them. The cluttered messages make it difficult to identify significant news.
Our plan was to help the consumers of information detect potentially instructive surprises: these are concerns in which the judgments of experts violate the decision maker's expectations. It is worth noting that, unlike standard decision analysis, the numerical values of probabilities do not matter very much in such an application because the focus is on change (from one time to the next) and on differences (between the decision maker's own assessments and those of the experts).
The dials in the cockpit display show current values of the critical variables. Continuous monitoring of these critical probabilities could serve a purpose in providing information to leaders about an evolving set of threats and opportunities. When choices must be made, decision makers require an assessment of conditional probabilities. They need to know how the probabilities of critical events may be affected by the selection of different options—in the present image, anticipated settings of the various dials contingent on the choice.
We knew that the assessment of conditional probabilities is difficult, and we therefore provided the expert judges with elaborate instructions on how to make these assessments. As we shall see later, however, we had underestimated the difficulties.
Minister Alon initially defined the question to be answered: What were the possible consequences of alternative outcomes of the current negotiations?
Zvi and the minister further refined the question in consultations in attempts to make it precise enough for the proposed method of study.
The consultation with Alon yielded five contingencies we were to examine:
- A) The negotiations succeed.
- B1) The negotiations fail, in the view of the US mediators, because of unreasonable demands by Egypt on bilateral issues.
B2) The negotiations fail, in the US view, because of unreasonable
demands by Egypt on issues involving other Arab states.
- B3) The negotiations fail, in the US view, because of Israeli rigidity.
- B4) The United States abandons the negotiations at an early stage, without assigning blame to either side.
These contingencies would not be chosen unilaterally by Israel (unlike the options considered in most decision analyses), but their probability of occurrence was partly controlled by its conduct of the negotiations. The point of the exercise was to provide an independent assessment of the strategic costs and benefits of various contingencies—presumably to make it easier for the decision makers to assess the concessions they should be willing to make in order to prevent the worst contingencies from occurring.
The research team defined its task as preparing a summary document for the foreign minister that would present expert judgments on two types of questions:
- The First Major Event. What would be the first major event that could be expected to occur if the negotiations failed for each of the reasons posited in the four failure contingencies?
- Critical Events and Concerns. What would be the effects of each of the five contingencies on the strategic risks and opportunities facing Israel during that period, defined as the realization of one of a number of critical events or concerns (e.g. a cut in oil supplies, an outbreak of hostilities)?
All of these events were to be identified by the research team.
The goal of the first set of questions was to provide some guidance for the construction of scenarios that would focus decision makers’ attention for the period immediately following the negotiations. The second set of questions was intended to facilitate the assessment of the relative values of the contingencies by allowing an easy comparison of their anticipated effects on national concerns.
The judgments that were summarized in the report were elicited from 19 individuals selected from three groups: (1) Intelligence analysts in the Ministry of Foreign Affairs; (2) Academic experts in relevant domains (Middle Eastern studies, Soviet Union studies, American studies); (3) Mid-level personnel in the Ministry of Foreign Affairs who were not otherwise involved in the issue at hand.
The judgments were obtained in structured interviews conducted by graduate students sophisticated in probability concepts and trained by Danny. The role of the interviewers was limited to obtaining answers to questions about the concepts of probability and conditional probability. The average duration of the interviews was three hours; some lasted considerably longer. All interviews were conducted over two working days, and the final report was ready three or four days later. The main features of the procedure we devised were to produce (1) a report that represented the views of a large number of knowledgeable individuals and (2) a report that would be much easier to prepare than the fully documented analytical essays (e.g. estimates) that are often used to support decisionmaking, even though they are not always studied in detail by busy leaders.
Each interview began with general instructions explaining the two goals of the project: introducing an experimental procedure for the structured elicitation and transmission of expert opinion to decision makers and provision of an analysis of a current and significant issue. The interview subjects were told that they were expected to summarize the reasoning behind the answer to each of the questions. Each was told: “The listing of arguments is an essential element of the procedure. The analysis of the arguments will make it possible to identify the source of disagreements among participants in the study and will influence the search for additional data that could help resolve these disagreements.”
The instructions provided highly specific descriptions of the four failure contingencies (B1–B4) listed above. In evaluating the contingent probabilities, the respondents were instructed to assume that a failure would occur before April 1975, which was the renewal date for a previous agreement between Egypt and Israel. They were also instructed to assume that no events serious enough to change the political picture would have occurred in the meantime.
The “First Event” Task. The list of “first key events” contained seven items. The instruction given to the respondents specified a key event as one which “dominates the attention of all parties for a significant period and compels them to take new decisions immediately.” The respondents’ task was to rank the seven events by their probability of being the first to occur, conditional on each of the four contingencies for failure of the negotiations.
The respondents were given elaborate instructions about the evaluation of conditional probabilities. In particular, they were told that if they considered one of the contingencies highly unlikely, the occurrence of that contingency would indicate that their current model was probably wrong and that it should be revised.
The seven possible events were:
American-Russian initiative launched to convene a peace conference in Geneva.
- Joint Russian-Arab initiative launched to convene a peace conference in Geneva.
- War of attrition on the northern front (Syria) ensues.
- Egyptians commit serious violation of the current military agreement.
- Egypt, Syria, or both together refuse to renew the mandate of the UN presence.
- Crisis in
the Israel-US relationship occurs.
- A new Egyptian-Russian agreement, including military cooperation, is reached.
“Critical Event” Task.
The critical events used to characterize the contingencies were selected to represent Israel’s major strategic concerns. In the first stage of the procedure, members of Zvi’s office independently nominated events to be included on the critical list.
The list was then reduced in several steps. First, all events had to satisfy the “clairvoyance test”: the event should be sufficiently well-defined to enable a clairvoyant to determine unequivocally whether the event would or would not occur. Second, all the events had to be major concerns—the kind of events that might keep a decision maker awake at night. Third, dependencies were eliminated, so that if event A entailed event B, only A was included in the list. Finally, events were eliminated if all members of the research team agreed that its probability would not be affected by the outcome of the negotiations. The final list consisted of 22 critical events/concerns.
deterioration of American-Soviet relations
- Joint American-Soviet initiative to resolve the conflict without coordination with Israel
- Expulsion of Israel from the UN
- Cancellation of oil supply agreements between Israel and Iran
- Regime change in Egypt leading to a new policy
- Regime change in Saudi Arabia leading to a new policy
- Regime change in Jordan leading to a new policy
- Declaration of the Palestine Liberation Organization (PLO) as a government in exile
- Signing of a formal agreement with Syria
- Signing of a formal agreement with Jordan
- US recognition of the PLO
- Renewal of massive Soviet military aid to Egypt
- Deployment of operational Soviet military units in Egypt or Syria
- War of attrition on the Northern Front (Syria)
- War of attrition on both fronts (Egypt and Syria)
- All-out war with Syria
- All-out war with both Syria and Egypt
- In case of war, deployment of Soviet troops in the region
- In case of war, intensive American effort to resupply the Israel Defense Forces
- In case of war, extreme economic steps by the Arab countries, including oil boycott and disruption of international financial markets
- In case of war, American military intervention to take control of the oil fields
- In case of war, Soviet threat to use nuclear weapons
The instructions to the respondents during the critical events part of the interview were: “We will consider a list of possible events that would have a significant impact on the security of Israel. You will be asked to evaluate the likelihood of these events occurring before the end of 1975, under various assumptions. For each event, we shall ask the following questions:
- What is the probability of this event if the current negotiations succeed?
- Are there substantial differences (more than 10 percent) in the probability of this event that are dependent on how the negotiations fail (B1–B4)?
- If the probability of the event is not sensitive to how the negotiations fail, what is its probability in the event negotiations fail for whatever reason?
- If the probability of the event is sensitive to how the negotiations fail, what are the probabilities for each of the four contingencies (B1–B4)?
The instructions for probability judgments made use of the then fashionable idea of a reference gamble. Respondents were asked to consider a wheel of fortune and to choose between gambling on the target event and gambling on the pointer falling in the winning region of the wheel.
When all interviews were completed, simple statistical analyses were carried out. We were particularly interested in possible differences between the three groups of respondents: the professional analysts—who had access to secret intelligence data—the academic experts, and the well-informed nonprofessionals. Rather to our surprise, we found no systematic differences. This observation on a small sample is compatible with the conclusion that Philip Tetlock reached in a very large study of political forecasting. In these complex situations, the returns to extra knowledge and expertise appear to be rapidly diminishing. A more encouraging observation was that the political leanings and strategic preferences of the judges (hawks or doves) were not easily discernible from their responses to the questionnaire.
The report we prepared for the Ministry of Foreign Affairs was brief: a total of less than 2,000 words.The main section, labeled “Conclusions,” consisted of three lists and a table.
list of events that were judged not sensitive to the five
including the success or failure of the negotiations. This list included 12 events. (See table on the right.) The average rating of likelihood assigned to each of these events was reported, as well as a few sentences summarizing arguments supporting a high or a low judgment of likelihood.
- A list of the events that were judged to be sensitive to the success or failure of the negotiations but not to the reason for the failure. Seven events were listed in this section. The average judgments of likelihood under the two contingencies appeared next to each other, providing a clear representation of sensitivity. A summary of the explanations was again offered. The most significant cost of a failure of the negotiations was an increase in the probability of war, a conclusion on which there was general consensus, although the probability of war was not considered high. Whether the negotiations failed or succeeded was seen as essentially irrelevant to the probabilities of the events that might occur if war did break out. And the success or failure of the negotiations was not expected to influence other events.
- Only three events were judged to be sensitive to the mode of failure (and in particular to whether the failure was attributed to Israeli rigidity).
The next section presented, in tabular format, the answers to the “first event” question concerning the reaction to a failure of the negotiations, contingent on its perceived cause. The only distinction that the judges considered relevant was between the case in which the Israeli side was blamed for the failure and all other contingencies. In the former case, for example, the event considered most likely to occur first was a joint American-Russian initiative to convene a Geneva conference. If the Israeli side was not specifically blamed by the United States, the most likely reaction was a joint Arab-Russian initiative to convene the Geneva conference. The ranks of all seven events were shown, but no summaries of the reasoning were provided.
Presentation of the Report
The report was presented to the director-general of the Ministry of Foreign Affairs, as well as to the minister. Their reactions were instructive. Both were generally indifferent to the specific probabilities, and the director-general memorably commented on the judged effect of failure of the negotiations on the probability of war with Syria “10 percent increase? That is a small difference.” This statement is shocking for a decision analyst, because 1/10 of the disutility of all-out war is hardly a small matter.
The minister remarked politely that the probabilities were “interesting.” He then went on to say that he had found the report unusually helpful because of a few instances in which particular judgments or arguments in the report had surprised him. These surprises, he said, caused him to think more deeply about the issues in a way he found enlightening.
For various reasons, including the sudden death of Ygael Alon and changes in our personal and professional lives, we did not do much to follow up on this exercise. When we discussed the experience recently, we found that we had learned different morals from it and had seen different aspects of it as potentially useful to intelligence communities.
Danny had been particularly impressed by the conspicuous lack of interest in numerical judgments among the readers of the report. This greatly reduced his faith in the applicability of decision analysis. He remained quite pleased with some aspects of the procedure, including the representation of strategic uncertainty by a vector of conditional probabilities of critical events and the highly economical format of the report, which he still considers superior to the essay format in which intelligence assessments are often presented.
Zvi was shaken by the difficulty the respondents encountered in making conditional probability judgments, which also reduced his faith in the usefulness of numerical statements of probability. On the other hand, he was encouraged by the effect of the interviews on the respondents: by their own testimony, the requirement to answer specific questions that did not fit naturally with their prior conceptions had compelled them to rethink and deepen their views. Zvi also remained satisfied with the compact format of the report, which made it easy for the decision maker to identify disagreements with the judgments of experts, and therefore to learn from them.
When we recently revisited the details of the original report, we reached a conclusion that shocked us: the basic judgments of conditional probability that were supposed to have been the core of the report were profoundly flawed. Zvi’s skepticism about these judgments was fully justified, with consequences that were worse than he had imagined.
We had always known that the numerical values assigned to the probabilities were implausible: far too close to .50 to be taken literally. But the flaw we uncovered after several decades was deeper. The pattern of sensitivity judgments shown above strongly suggests that the respondents were not in fact evaluating the probabilities of critical outcomes conditional on success or failure of the negotiations.
- Instead, they were evaluating the direct causal impact of success or failure on these events.
Thus, the causal connection was immediately obvious to the judges when the probability of all-out war was considered as conditional on the outcome of the negotiations. However, the judges did not indicate that a failure of the negotiations would alter the probability of changes of regime in Jordan or Saudi Arabia. This was certainly a mistake. The respondents would surely have agreed that an all-out war was bound to increase political instability in the region, and thereby the probability that fragile regimes would fall, but their judgments showed that this obvious inference had not been made.
The pattern of results we saw in 1975 is precisely what the new interpretation of judgment heuristics would have predicted. In this interpretation, intuitive judgment is explained by a process of attribute substitution. When a person is asked a difficult question, the answer to a different but related question sometimes comes spontaneously to mind. If this occurs, the answer to the easier question is often mapped onto a corresponding answer to the question that was asked, without the respondent being aware that this substitution had occurred.
In the 1975 experiment, conditional probability was a very difficult attribute to judge, whereas judgments of causal influence came easily to mind. As students of human judgment, we are of course not surprised to find assessments of uncertainty that are susceptible to biases and do not conform to the logic of mathematical probability. We were still taken aback by the conclusion that our expert respondents largely failed to deal with the task of assessing conditional probabilities and answered a question they had not been asked.
In spite of this significant blemish, we believe that some elements of the procedure we have described may be useful in other contexts in which a decision maker requires expert help in assessing the uncertainties of a complex situation.
- First, we expect that even where judgments of probability are seriously flawed, changes and differences in these judgments are likely to contain useful information. Returning to the image of the decision cockpit that we introduced earlier, periodic assessments of the probabilities of critical events by a diverse set of experts will convey information to the decision maker when some probabilities change unexpectedly from one occasion to the next, or when a change in the significance of a concern (or an element of reasoning) otherwise violates the decision maker’s expectations.
- We still believe that a report that deals with the likelihood of discrete critical events and provides crisp arguments for judgments is an efficient way to convey new information to decision makers and to evoke new thinking from them. This procedure is likely to work best if the list of critical events—the set of dials in the cockpit—accurately represents the leader’s concerns and is periodically updated to reflect changes in these concerns.
Philip E. Tetlock, Expert Political Judgement: How Good Is It? How Can We Know? (Princeton, NJ: Princeton University Press, 2005).
Zvi would go on to write on this subject in Fundamental Surprise: The National Intelligence Crisis? (Tel Aviv, Israel: The Center for Strategic Studies, Tel Aviv University, 1983 (Hebrew))
Daniel Kahneman, “Maps of Bounded Rationality: Psychology for Behavioral Economics,” American Economic Review 93 5 (2003): 1449–75.
All statements of fact, opinion, or analysis expressed in this article are those of the author. Nothing in the article should be construed as asserting or implying US government endorsement of an article’s factual statements and interpretations.