Speaking to Policymakers
Zvi Lanir and Daniel Kahneman
“Minister of Foreign Affairs Alon wanted his own analysis of the outlook for the region in early 1975”
when and how to apply structured analytical techniques has been a
continual challenge for US Intelligence Community analysts and the
subject of IC training courses for years. In this article Nobel
Laureate Daniel Kahneman and his colleague Zvi Lanir, then with the
Israeli government, attempted to apply a technique to a pressing, real
world issue for Jerusalem in 1975. At the time, Lanir was serving as
leader of the just-established Center of Research and Political
Planning in the Ministry of Foreign Affairs of the State of Israel, an
entity similar in function to the Policy Planning Staff of the US State
Department. Kahneman was professor of psychology at the Hebrew
University in Jerusalem. There, he was involved in research in judgment
The two became involved in a research and
analysis project requested by the then-minister of foreign affairs,
Ygael Alon. Publication was not the objective of the project, which at
the time was classified. More immediate policy concerns were at play.
Lanir and Kahneman report on it more than 30 years later because, as
they point out, some elements of their approach to the task are still
relevant today in a variety of domains, including business and
international relations. These elements include considerations in
choosing structured techniques, communicating the results to senior
policy makers, and coping with the mixture of success and failure the
two experienced in the effort.
* * *
Historical and Intellectual Context
Israel in 1975 was still living the trauma of the catastrophic failure
of intelligence that had preceded the costly 1973 war in which more
than 2,300 soldiers lost their lives in three weeks— an equivalent loss
of life for US forces today would be over 100,000 dead. Confidence in
the Israeli intelligence community was badly shaken, and there was
interest in new approaches. There was also a belief that
decentralization of intelligence appraisal might prevent a recurrence
of what was considered— with some benefit of hindsight—to have been
blind allegiance to an incorrect conception of the strategic situation
and of enemy intentions.
In the tense early months of
1975, Minister of Foreign Affairs Ygael Alon, in particular, wanted to
have his own analysis of the political outlook in the region as US
Secretary of State Henry Kissinger engaged in his famous exercise in
shuttle diplomacy, attempting to achieve an agreement between Egypt and
Israel. One message Kissinger delivered to the Israeli government was
the CIA's judgment that failure to negotiate peace would have dire
consequences for the region. The Intelligence Branch of the Israel
Defense Forces generally concurred with this pessimistic view. The
foreign minister was skeptical.
Meanwhile, on the
academic scene, the 1970s was probably the heyday of decision research
and analysis. Demonstrations of biases in human judgment under
conditions of uncertainty were seen as supporting the prescriptive
approach of decision analysis, by undermining, if only slightly, faith
in the ability of decision makers to reach optimal decisions without
aid. Danny was strongly influenced by the Stanford branch of decision
analysts, led by the charismatic Ron Howard, who preached that options
in complex decisions should be represented by "equivalent gambles,"
denominated in a common currency of utility or money. That form of
decision analysis required an explicit formulation of a decision
maker's values, as well as his or her sense of the probabilities of
various outcomes—which were normally assumed to agree with the
probabilities generated by experts in the organization to which the
decision maker belonged.
At the time, Danny accepted the ideals
of decision analysis, a field that provided, in his view, a gold
standard for rational decisionmaking. However, he was aware of many
obstacles to the application of decision analysis to significant
choices. In particular, it was obvious that eliciting formal utilities and value trade-offs
from decision makers was completely unrealistic in many situations,
certainly including those involving strategic political and military
However, it still seemed to us that decision makers could benefit from systematic consideration of the probabilities
of significant outcomes and of the effects of their choices on these
probabilities. We also believed that properly trained analysts could
make fairly sensible quantitative judgments of probability and
conditional probability—or at least make judgments that convey useful
When Zvi approached Danny about this
project, both were naively enthusiastic about what they saw as a chance
to improve the rationality of decisionmaking on truly important issues.
As a practitioner of intelligence analysis, Zvi's views had been formed
in the field, but he was sympathetic to the academic approach and to
decision analysis as a tool for improving rational decisionmaking. We
worked as a team in designing and implementing the exercise.
[Top of page]
The ideas that guided our approach to the problem were borrowed from
the field of human engineering—the field devoted to making instruments
"user-friendly" (a term that became popular much later). The foreign
minister had asked for a report, and it was immediately obvious to us
that the report should represent the opinions of a wide range of
experts. In the spirit of human engineering, our task was to structure
a report that would convey information to the decision maker as
efficiently as possible and to structure the experts' task accordingly,
while keeping their task manageable.
replace the "equivalent gamble" as a representation of the decision
maker's uncertainty? The proposed solution was to start from the
national leaders' concerns,
the possibilities that sometimes kept them awake at night (the ultimate
decision maker in Israel at the time was Golda Meir, who famously
suffered from both worries and insomnia).
"concerns" as events that are feared or hoped for and proposed that a
list of these events, each associated with a probability would provide
a representation of the "national gamble," the risk profile of the
state of Israel. Borrowing an image from a field in which human
engineering has been especially useful, the list of concerns is
analogous to the display of dials in an airplane's cockpit. In this
analogy, every dial the decision maker monitors shows the probability
associated with a particular concern. The display in the cockpit is
designed to make it easy for the pilot to notice changes in critical
information must appear reliably and obviously, but the number of dials
must be as small as possible because attention is a limited resource.
Most of the time, of course, the displays are stable and their changes
are predictable, but the early detection of anomalies is crucial for
Our fantasy was to design an efficient
cockpit for strategic decisionmaking. The key idea was to make it easy
for decision makers to focus on new information. Most of the content of
the reports decision makers hear or read is already known to them. The
cluttered messages make it difficult to identify significant news.
Our plan was to help the consumers of information detect potentially
instructive surprises: these are concerns in which the judgments of
experts violate the decision maker's expectations. It is worth noting
that, unlike standard decision analysis, the numerical values of
probabilities do not matter very much in such an application because
the focus is on change (from one time to the next) and on differences (between the decision maker's own assessments and those of the experts).
dials in the cockpit display show current values of the critical
variables. Continuous monitoring of these critical probabilities could
serve a purpose in providing information to leaders about an evolving
set of threats and opportunities. When choices must be made, decision
makers require an assessment of conditional probabilities.
They need to know how the probabilities of critical events may be
affected by the selection of different options—in the present image,
anticipated settings of the various dials contingent on the choice.
knew that the assessment of conditional probabilities is difficult, and
we therefore provided the expert judges with elaborate instructions on
how to make these assessments. As we shall see later, however, we had
underestimated the difficulties.
[Top of page]
Minister Alon initially defined the question to be answered:
What were the possible consequences of alternative outcomes of the current negotiations?
and the minister further refined the question in consultations in
attempts to make it precise enough for the proposed method of study.
The consultation with Alon yielded five contingencies we were to examine:
- A) The negotiations succeed.
- B1) The negotiations fail, in the view of the US mediators, because of unreasonable demands by Egypt on bilateral issues.
B2) The negotiations fail, in the US view, because of unreasonable
demands by Egypt on issues involving other Arab states.
- B3) The negotiations fail, in the US view, because of Israeli rigidity.
- B4) The United States abandons the negotiations at an early stage, without assigning blame to either side.
contingencies would not be chosen unilaterally by Israel (unlike the options
considered in most decision analyses), but their probability of occurrence was
partly controlled by its conduct of the negotiations. The point of the exercise
was to provide an independent assessment of the strategic costs and benefits
of various contingencies—presumably to make it easier for the decision makers
to assess the concessions they should be willing to make in order to prevent
the worst contingencies from occurring.
research team defined its task as preparing a summary document for the foreign
minister that would present expert judgments on two types of questions:
First Major Event. What would be the first major
event that could be expected to occur if the negotiations failed for each of
the reasons posited in the four failure contingencies?
Events and Concerns. What would be the effects of
each of the five contingencies on the strategic risks and opportunities facing
Israel during that period, defined as the realization of one of a number of
critical events or concerns (e.g. a cut in oil supplies, an outbreak of
of these events were to be identified by the research team.
goal of the first set of questions was to provide some guidance for the
construction of scenarios that would focus decision makers’ attention for the
period immediately following the negotiations. The second set of questions was
intended to facilitate the assessment of the relative values of the
contingencies by allowing an easy comparison of their anticipated effects on
judgments that were summarized in the report were elicited from 19
individuals selected from three groups: (1) Intelligence analysts in the
Ministry of Foreign Affairs; (2) Academic experts in relevant domains (Middle
Eastern studies, Soviet Union studies, American studies); (3) Mid-level
personnel in the Ministry of Foreign Affairs who were not otherwise involved in
the issue at hand.
judgments were obtained in structured interviews conducted by graduate students
sophisticated in probability concepts and trained by Danny. The role of the
interviewers was limited to obtaining answers to questions about the concepts
of probability and conditional probability. The average duration of the interviews
was three hours; some lasted considerably longer. All interviews were conducted
over two working days, and the final report was ready three or four days later.
The main features of the procedure we devised were to produce (1) a report that
represented the views of a large number of knowledgeable individuals and (2)
a report that would be much easier to prepare than the fully documented
analytical essays (e.g. estimates) that are often used to support decisionmaking,
even though they are not always studied in detail by busy leaders.
[Top of page]
interview began with general instructions explaining the two goals of the
project: introducing an experimental procedure for the structured elicitation
and transmission of expert opinion to decision makers and provision of an
analysis of a current and significant issue. The interview subjects were told
that they were expected to summarize the reasoning behind the answer to each
of the questions. Each was told: “The listing of arguments is an essential
element of the procedure. The analysis of the arguments will make it possible
to identify the source of disagreements among participants in the study and
will influence the search for additional data that could help resolve these disagreements.”
instructions provided highly specific descriptions of the four failure
contingencies (B1–B4) listed above. In evaluating the contingent probabilities,
the respondents were instructed to assume that a failure would occur before
April 1975, which was the renewal date for a previous agreement between Egypt
and Israel. They were also instructed to assume that no events serious enough
to change the political picture would have occurred in the meantime.
“First Event” Task. The list of “first key events” contained seven
items. The instruction given to the respondents specified a key event as one
which “dominates the attention of all parties for a significant period and
compels them to take new decisions immediately.” The respondents’ task was to
rank the seven events by their probability of being the first to occur,
conditional on each of the four contingencies for failure of the
respondents were given elaborate instructions about the evaluation of
conditional probabilities. In particular, they were told that if they
considered one of the contingencies highly unlikely, the occurrence of that
contingency would indicate that their current model was probably wrong and
that it should be revised.
seven possible events were:
American-Russian initiative launched to convene a peace conference in Geneva.
Russian-Arab initiative launched to convene a peace conference in Geneva.
- War of
attrition on the northern front (Syria) ensues.
commit serious violation of the current military agreement.
Syria, or both together refuse to renew the mandate of the UN presence.
- Crisis in
the Israel-US relationship occurs.
- A new
Egyptian-Russian agreement, including military cooperation, is reached.
“Critical Event” Task.
The critical events used to
characterize the contingencies were selected to represent Israel’s major
strategic concerns. In the first stage of the procedure, members of Zvi’s
office independently nominated events to be included on the critical list.
list was then reduced in several steps. First, all events had to satisfy the
“clairvoyance test”: the event should be sufficiently well-defined to enable a
clairvoyant to determine unequivocally whether the event would or would not
occur. Second, all the events had to be major concerns—the kind of events that
might keep a decision maker awake at night. Third, dependencies were
eliminated, so that if event A entailed event B, only A was included in the
list. Finally, events were eliminated if all members of the research team
agreed that its probability would not be affected by the outcome of the
negotiations. The final list consisted of 22 critical events/concerns.
deterioration of American-Soviet relations
American-Soviet initiative to resolve the conflict without coordination with
of Israel from the UN
of oil supply agreements between Israel and Iran
change in Egypt leading to a new policy
change in Saudi Arabia leading to a new policy
change in Jordan leading to a new policy
of the Palestine Liberation Organization (PLO) as a government in exile
- Signing of
a formal agreement with Syria
- Signing of
a formal agreement with Jordan
recognition of the PLO
- Renewal of
massive Soviet military aid to Egypt
of operational Soviet military units in Egypt or Syria
- War of
attrition on the Northern Front (Syria)
- War of
attrition on both fronts (Egypt and Syria)
- All-out war
with both Syria and Egypt
- In case of war, deployment of Soviet troops in the region
- In case of war, intensive American effort to resupply the Israel Defense Forces
- In case of war, extreme economic steps by the Arab countries, including oil boycott and
disruption of international financial markets
- In case of war, American military intervention to take control of the oil fields
- In case of war, Soviet threat to use nuclear weapons
The instructions to the respondents during the critical events part of the
interview were: “We will consider a list of possible events that would have a
significant impact on the security of Israel. You will be asked to evaluate
the likelihood of these events occurring before the end of 1975, under various
assumptions. For each event, we shall ask the following questions:
- What is the probability of this event if the current negotiations succeed?
- Are there
substantial differences (more than 10 percent) in the probability of this
event that are dependent on how the negotiations fail (B1–B4)?
- If the
probability of the event is not sensitive to how the negotiations fail,
what is its probability in the event negotiations fail for whatever reason?
- If the
probability of the event is sensitive to how the negotiations fail,
what are the probabilities for each of the four contingencies (B1–B4)?
instructions for probability judgments made use of the then fashionable idea of
a reference gamble. Respondents were asked to consider a wheel of fortune and
to choose between gambling on the target event and gambling on the pointer
falling in the winning region of the wheel.
interviews were completed, simple statistical analyses were carried out. We
were particularly interested in possible differences between the three groups
of respondents: the professional analysts—who had access to secret
intelligence data—the academic experts, and the well-informed nonprofessionals.
Rather to our surprise, we found no systematic differences. This observation on
a small sample is compatible with the conclusion that Philip Tetlock reached
in a very large study of political forecasting. In these complex situations, the returns to extra knowledge and expertise
appear to be rapidly diminishing. A more encouraging observation was that the
political leanings and strategic preferences of the judges (hawks or doves)
were not easily discernible from their responses to the questionnaire.
[Top of page]
The report we prepared for the Ministry of Foreign Affairs
was brief: a total of less than 2,000 words.The main section, labeled
“Conclusions,” consisted of three lists and a table.
list of events that were judged not sensitive to the five
including the success or failure of the negotiations. This list
included 12 events. (See table on the right.) The average rating of likelihood
assigned to each of these events was reported, as well as a few sentences
summarizing arguments supporting a high or a low judgment of likelihood.
list of the events that were judged to be sensitive to the success or failure
of the negotiations but not to the reason for the failure. Seven events were
listed in this section. The average judgments of likelihood under the two
contingencies appeared next to each other, providing a clear representation of
sensitivity. A summary of the explanations was again offered. The most
significant cost of a failure of the negotiations was an increase in the
probability of war, a conclusion on which there was general consensus, although
the probability of war was not considered high. Whether the negotiations
failed or succeeded was seen as essentially irrelevant to the probabilities of
the events that might occur if war did break out. And the success or failure of
the negotiations was not expected to influence other events.
three events were judged to be sensitive to the mode of failure (and in
particular to whether the failure was attributed to Israeli rigidity).
The next section presented, in tabular format, the answers
to the “first event” question concerning the reaction to a failure of the
negotiations, contingent on its perceived cause. The only distinction that the
judges considered relevant was between the case in which the Israeli side was
blamed for the failure and all other contingencies. In the former case, for
example, the event considered most likely to occur first was a joint
American-Russian initiative to convene a Geneva conference. If the Israeli side
was not specifically blamed by the United States, the most likely reaction was
a joint Arab-Russian initiative to convene the Geneva conference. The ranks of
all seven events were shown, but no summaries of the reasoning were provided.
of the Report
The report was presented to the director-general of the
Ministry of Foreign Affairs, as well as to the minister. Their reactions were
instructive. Both were generally indifferent to the specific probabilities,
and the director-general memorably commented on the judged effect of failure of
the negotiations on the probability of war with Syria “10 percent increase?
That is a small difference.” This statement is shocking for a decision
analyst, because 1/10 of the disutility of all-out war is hardly a small
The minister remarked politely that the probabilities were
“interesting.” He then went on to say that he had found the report unusually
helpful because of a few instances in which particular judgments or arguments
in the report had surprised him. These surprises, he said, caused him to think
more deeply about the issues in a way he found enlightening.
[Top of page]
For various reasons, including the sudden death of Ygael
Alon and changes in our personal and professional lives, we did not do much to
follow up on this exercise. When we discussed the experience recently, we
found that we had learned different morals from it and had seen different
aspects of it as potentially useful to intelligence communities.
Danny had been particularly impressed by the conspicuous
lack of interest in numerical judgments among the readers of the report. This
greatly reduced his faith in the applicability of decision analysis. He
remained quite pleased with some aspects of the procedure, including the
representation of strategic uncertainty by a vector of conditional
probabilities of critical events and the highly economical format of the
report, which he still considers superior to the essay format in which
intelligence assessments are often presented.
Zvi was shaken by the difficulty the respondents encountered
in making conditional probability judgments, which also reduced his faith in
the usefulness of numerical statements of probability. On the other hand, he
was encouraged by the effect of the interviews on the respondents: by their own
testimony, the requirement to answer specific questions that did not fit
naturally with their prior conceptions had compelled them to rethink and deepen
their views. Zvi also remained satisfied with the compact format of the report, which made
it easy for the decision maker to identify disagreements with the judgments of
experts, and therefore to learn from them.
When we recently revisited the details of the original
report, we reached a conclusion that shocked us: the basic judgments of conditional
probability that were supposed to have been the core of the report were
profoundly flawed. Zvi’s skepticism about these judgments was fully justified,
with consequences that were worse than he had imagined.
We had always known that the numerical values assigned to
the probabilities were implausible: far too close to .50 to be taken literally.
But the flaw we uncovered after several decades was deeper. The pattern of
sensitivity judgments shown above strongly suggests that the respondents were
not in fact evaluating the probabilities of critical outcomes conditional on
success or failure of the negotiations.
- Instead, they were evaluating the direct causal impact of success or failure on these
Thus, the causal connection was immediately obvious to the
judges when the probability of all-out war was considered as conditional on the
outcome of the negotiations. However, the judges did not indicate that a
failure of the negotiations would alter the probability of changes of regime in
Jordan or Saudi Arabia. This was certainly a mistake. The respondents would
surely have agreed that an all-out war was bound to increase political instability
in the region, and thereby the probability that fragile regimes would fall, but
their judgments showed that this obvious inference had not been made.
The pattern of results we saw in 1975 is precisely what the
new interpretation of judgment heuristics would have predicted. In this
interpretation, intuitive judgment is explained by a process of attribute
substitution. When a person is asked a difficult question, the answer to a different but
related question sometimes comes spontaneously to mind. If this occurs, the
answer to the easier question is often mapped onto a corresponding answer to
the question that was asked, without the respondent being aware that this substitution
In the 1975 experiment, conditional probability was a very
difficult attribute to judge, whereas judgments of causal influence came
easily to mind. As students of human judgment, we are of course not surprised
to find assessments of uncertainty that are susceptible to biases and do not
conform to the logic of mathematical probability. We were still taken aback by
the conclusion that our expert respondents largely failed to deal with the task
of assessing conditional probabilities and answered a question they had not
In spite of this significant blemish, we believe that some
elements of the procedure we have described may be useful in other contexts in
which a decision maker requires expert help in assessing the uncertainties of a
- First, we expect that even where judgments of probability are seriously flawed,
changes and differences in these judgments are likely to contain useful information.
Returning to the image of the decision cockpit that we introduced earlier,
periodic assessments of the probabilities of critical events by a diverse set
of experts will convey information to the decision maker when some
probabilities change unexpectedly from one occasion to the next, or when a
change in the significance of a concern (or an element of reasoning) otherwise
violates the decision maker’s expectations.
- We still believe that a report that deals with the likelihood of discrete critical
events and provides crisp arguments for judgments is an efficient way to
convey new information to decision makers and to evoke new thinking from them.
This procedure is likely to work best if the list of critical events—the set
of dials in the cockpit—accurately represents the leader’s concerns and is
periodically updated to reflect changes in these concerns.
[Top of page]
E. Tetlock, Expert Political Judgement: How Good Is It? How Can We Know? (Princeton,
NJ: Princeton University Press, 2005).
would go on to write on this subject in Fundamental Surprise: The National
Intelligence Crisis? (Tel Aviv, Israel: The Center for Strategic
Studies, Tel Aviv University, 1983 (Hebrew))
Kahneman, “Maps of Bounded Rationality: Psychology for Behavioral Economics,” American
Economic Review 93 5 (2003): 1449–75.
All statements of fact, opinion, or analysis expressed in this
article are those of the author. Nothing in the article should be
construed as asserting or implying US government endorsement of an
article’s factual statements and interpretations.