To Improve Analytical Insight
Rachel K. Hanig and Mark E. Henshaw
The following essay was a winner in the 2007 DNI Galileo Competition, a program that awards authors of papers proposing innovative solutions to Intelligence Community challenges.
The authors argue that creation of a National Security Simulations Center would strengthen the accuracy and insight of intelligence analysis, improve IC collaboration, and create a testing ground for new analytic tools and methods.
The quality of IC analysis is inconsistent, and the challenges to sustaining a superior analytic track record look more formidable all the time.
Intelligence analysis too often is like investing in the stock market—past performance is not an indicator of future results. The quality of IC analysis is inconsistent, and the challenges to sustaining a superior analytic track record look more formidable all the time. The bar has always been set high and is moving higher as policymakers demand that analysts:
- be “timely”—at least on par with the public media;
- be analytically correct 100 percent of the time while offering broader strategic views that include longer lists of potential outcomes;
- be strategically relevant on increasingly complex topics as the volume of raw information to filter and analyze grows.
This pressure for increased speed, accuracy, and consistent strategic relevance is one of the primary factors pushing the analytic corps towards risk aversion and its analytical consequences. Under the best of circumstances, even the most experienced IC analysts, those with years of study and experience invested in single accounts, make mistakes by falling prey to mental biases and mindsets, intelligence gaps, or even “lack of imagination.”
Given uneven hiring cycles in the IC’s ranks over the past few decades, it won’t always be the most experienced analysts making the judgments upon which policymakers might rely.
IC Initiatives to Improve Analysis: Building Blocks for a Larger Solution
The IC has responded to these challenges with three major initiatives. The first came immediately after 11 September 2001 with a call for more diligent adherence to analytic tradecraft “best practices.” The problem was and remains that there really are few standard methods of analysis. Analysts are left largely to their own devices in developing systems for processing intelligence and depend on coordination with other analysts to catch the errors.
The second and most broad-ranging of these initiatives picked up steam after the 2003 Iraq WMD NIE fiasco. Several solutions, including a number of winning Galileo papers, focused on giving analysts better access to data before analysis occurs and promoting better coordination after the fact. Improving the IC’s data organization and inter-and intra-agency sharing is a necessary but ultimately insufficient first step.
Better information sharing and data access are always useful, but information sharing and data access are not analysis. Even perfect access to perfect information would be unhelpful if the analytical models used to process it were deficient, and even perfect coordination among analysts might not be enough to guarantee the models’ quality. So this begs the question: How can analysts stress-test the quality of their analytical models, theories, and theses without waiting for history to prove them right or wrong?
The third major initiative promotes the use of alternative analytic tools and techniques. Again, these are very useful. But the approach is potentially flawed because many structured analytical tools and techniques are employed as individual mental exercises. Their effectiveness can still be undermined by sloppy thinking. Ironically, the analysts who need to use them most desperately are most likely to use them ineffectively or incorrectly, or just not use them at all. Nor can we guarantee that the coordination process will catch sloppy application of alternative analytical tools in all cases since many senior analysts, though experienced in traditional analytical tradecraft, are no more experienced in the craft of alternative analysis than their junior counterparts. Many senior analysts, in fact, prove to be the most resistant to using such techniques.
All three of the above initiatives are critical elements of a larger solution; but even if all three were perfectly executed, analysts would still struggle to meet several of the policymakers’ requirements during crises. Quality analysis cannot be rushed. Strategic insights take time to develop, but when a crisis breaks, the time for analysts to engage in deep thinking is often past.
Proposed Solution: The National Security Simulations Center
solution that fuses all three initiatives together into a single whole and that
resolves the problem posed by the pressure for analytical timeliness would be
ideal. We propose that one solution is, ironically, both widely known and
little practiced by the IC, simulations.
Simulations can be very effective in stretching analysis and strengthening the methodological rigor that policy consumers value and expect. The use of simulations is not new. The US military has used them for years, primarily as training tools to help troops develop tactical and joint-service coordination skills. It is unfortunate that the IC has used simulations for the same reason only intermittently at best—there has never been a central, Intelligence Community, simulation hub equivalent to the National Strategic Gaming Center at the National Defense University in Washington, DC, or the Wargaming Center at the Naval War College in Newport, Rhode Island.
Such simulations as have been conducted were usually performed under the purview of individual agencies. However, the intelligence failures of recent years suggest that the IC should be staging simulations for another purpose: to develop strategic insights into potential geopolitical developments.
Simulations are not predictive, but they can allow analysts to explore key analytic questions and conclusions in far greater depth than is possible from behind a desk or in meetings with other analysts. A properly organized geopolitical simulation forces analysts into dynamic, social, stressful situations that simulate real-world conditions to expose the participants’ thinking, mindsets, biases, and assumptions to colleagues and observers positioned to identify analytic weaknesses.
simulations can also peel back the layers of intellectual cruft and weak
analysis to expose insights that might otherwise remain undiscovered—and do it
before real crisis hits, when there is almost no time for analytical
coordination and deep strategic thinking. In a sense, simulations give analysts
better ideas of what geopolitical changes might look like before having to
present their conclusions to policymakers.
Why a national center?
Experience shows that the preparation and execution of successful simulations are the product of both structured analytic work and art requiring a large number of expert people with a large variety of skills. The Intelligence Community would greatly benefit from a center with a dedicated staff versed in the arts and crafts of scenario development, construction of simulation tools and methodology, and subject-matter experts, not to mention the support personnel needed for such an endeavor.
The Director of National Intelligence already has the charter, provided by Congress in the Intelligence Reform and Terrorism Prevention Act of 2004, Section 1023, 119B, to create national interagency centers that focus on intelligence issues. The National Counterterrorism Center and National Counterproliferation Center are two current examples. However, a National Security Simulations Center (NSSC) would not focus on any single issue that threatens US interests. Not only could it address threats of all kinds, it could deal with other community priorities, as seen below.
Integration and Collaboration
The NSSC could regularly stage large-scale simulations that would bring together analysts and managers from multiple agencies. Such simulations would would give participants opportunities to share information, ideas, theories, and best practices in structured, realistic environments designed to push the participants toward common goals.
In this sense, the NSSC would function much like the NDU National Strategic Gaming Center or the Naval War College Wargaming Center. Such simulations would teach participants how to work together during crises, who to call, and the capabilities of their IC counterparts. The personal connections developed in such an environment would be highly useful during real crises, as participants would better know who to call and would have practiced real-time coordination with their counterparts.
the NSSC could stage simulations that go far beyond practicing tactical
responses to crisis scenarios. By having analysts participate in the scenario
development process, it would also become a strategic analysis
cross-pollination center. Previously proposed solutions to problems of
community coordination and integration could be field-tested in controlled
environments to determine their practicality and identify their strengths and
Engagement of outside experts
A simulation’s value rests directly on the quality of both the scenario and the participants. Backed by the DNI’s authority and resources, NSSC simulations could recruit high quality participants to lend expertise to scenario development and to participate in the simulations. It is not unreasonable to believe that former high-ranking government officials, corporate CEOs, leading academic thinkers, and other notable figures—including foreign participants—would be willing to participate in NSSC simulations. Their involvement would improve strategic analysis across the board and strengthen the outreach efforts of individual agencies, which now tend to be piecemeal and ad hoc. This would ensure that outside expertise finds broader audiences and becomes better aligned with the needs of individual agencies.
Staying ahead of geopolitical developments
The media’s rapid response to breaking events leaves the IC at a significant disadvantage in informing policymakers. The NSSC could help analysts remain both timely and strategically relevant by simulating as many events as possible before they happen, thereby buying analysts time that is irretrievably lost once an event actually occurs. In that sense, properly organized and managed, simulations could help analysts more quickly provide more informed perspective to policymakers.
In addition, as a simulation looked into potential developments, players would be in position to identify intelligence gaps and to begin developing targeting plans to fill those gaps.
Training new IC employees
The cyclical nature of hiring in the intelligence community is well documented and the number of analysts in the IC with less than five years experience has reached record highs. The NSSC could take the training of new people beyond the classroom by putting junior analysts into environments in which they could learn and practice tradecraft without having to worry about making embarrassing, or career-terminating, analytical errors or having their efforts dismissed or ridiculed by policymakers.
Alternative analysis techniques, which can be difficult to learn and properly apply, often lend themselves very well to being operationalized within simulations. Alternative futures analysis, Team A/Team B, and several others are particularly well suited for use in simulations.
The NSSC would also help new analysts learn how to better process the vast amounts of data available by teaching them how to determine what information would be most valuable to them and their policymaking customers. And, as they considered the relative importance of information, they could actually beginning mining the data they would need in a given circumstance.
Analytical tradecraft experimentation
Like any craft, intelligence analysis, and especially alternative analysis, must experiment continuously with new tools and techniques. The NSSC would be ideally suited to serve as a laboratory in which analysts could develop and field-test tradecraft innovations before deploying them to the IC at large. In fact, simulations might well point analysts towards new tools and techniques that might otherwise remain undiscovered, or suggest new uses previously unconsidered for existing tools. By increasing the frequency of interaction among analysts focused on specific problems, the NSSC would improve the odds that innovations could emerge from such social networking. The NSSC could be an idea factory for experimental tradecraft.
In sum, the NSSC could be an organization fit to play many roles in the community. Which role it would play at any given time would depend on the kind of simulation chosen for the particular exercise. Tradecraft training, strategic insight development, and testing of analytical tools and techniques all could be managed under the single roof of the highly flexible center.
Simulations are the one kind of exercise that can tie all other analytical tools and techniques together, both new and old, while enhancing inter-agency coordination at the same time. It’s difficult to think of any alternative concept that even promises a way to enhance IC-wide collaboration and allow analysts to develop strategic insights and perfect analytical tradecraft, all in single endeavor. Practice makes perfect, but opportunities to practice all three activities at once are, to say the least, rare.
Building the National Security Simulations Center
Having outlined justifications for creating such a center, the questions become: What should the National Security Simulations Center look like and how might it work?
The NSSC would require, at minimum, four key organizational components (see graphic on following page):
- A Research and Analysis Staff (R&A)
- Simulations Design Staff (SD)
- An Analytical Tools and Techniques Development Staff (AT&TD)
- Private Sector/Academia Outreach Staff (PS&AO)
Research and Analysis Staff (R&A)
The primary responsibility of the R&A would be to work with IC subject matter experts—CIA analysts, NCS officers, and other IC members engaged in analytical or targeting functions—to identify and craft intelligence questions suited for scenario testing. This would require R&A to mount in-depth research campaigns on underlying issue areas to identify three major requirements of each scenario:
Key variables, which must be observable and measurable in the real world by the IC; or if they aren’t observable (and therefore not measurable) could become so through the implementation of new technologies or collection programs.
Intelligence gaps, so the simulation designers could understand in advance where the holes in the simulation scenario would be and how they could best be addressed.
Environmental factors, including social, military, economic, diplomatic, and potential natural disasters beyond the control of key actors.
After a simulation is completed, R&A would be responsible for producing the analytic product documenting its key findings. Using appropriate analytical standards and tradecraft, the product would include key findings, warnings and indicators, and analytic conclusions. These might include strategic projections and key decision points and discussion of how things might have gone had different decisions been made. This analysis would all be directed toward extracting strategic insights that would give analysts and policymakers deeper understanding of the issues they face.
Simulation Design Staff (SD)
The primary responsibility of SD would be to take polished analytical concepts prepared by R&A and develop simulation scenarios to address them. SD would devise scenario story lines and geopolitical conditions that would best illuminate hidden assumptions, insights, and potential outcomes. SD would also create game mechanics to move players through scenarios. Broadly speaking, this would include identifying needed government, private sector, non-state and state roles and organizing players and teams. SD would also be responsible for creating supporting game materials— maps, manuals, and other accessories—and driving development of the computer network that would be used to deliver to players game injects and scenario information and that would provide the means by which players and teams would communicate with each other and with simulation controllers.
Once a simulation design phase is complete, SD would be responsible for conducting the live exercise. Those who create simulation scenarios are usually best prepared to adjudicate players’ actions within those artificial environments. The skills of scenario designers and adjudicators directly affect the validity of any simulation’s results. This is not an activity that can easily be taught. Constructing plausible and useful present and future conditions for a simulation and then managing the simulation is an art, not a science, and only time and experience teach it. SD would develop expertise as it created legitimate environments and judged players moves to ensure that simulation results would always be credible.
Analytical Tools & Techniques Development Staff (AT&TD)
To fulfill its mandate as an analytical research center, the NSSC would benefit greatly from having a separate team of methodologists who could observe simulations and explore new tools and techniques for addressing the problems players would confront. AT&TD could be an exceptional IC asset, as it could be a think-tank mandated to constantly drive analytical methodologies toward the cutting edge. It could develop and refine new approaches for tackling hard analytical problems until they were mature enough to be put to work in the IC.
Drawing from their respective charters and expertise, AT&TD, R&A and SD could cooperate to design simulation tools and techniques, with a particular focus on pioneering methods and software that could be used outside the center by analysts in small groups at their home facilities. Their work could be enhanced if the NSSC facility had a charter that, while allowing it to handle classified information, also allowed experimentation with new computer network technologies, and allowed for the simulation of 24-hour news media coverage.
The potential local and global influence of the media makes it an essential variable in the simulation environment. Accordingly, an NSSC facility would need distinct spaces, wired for Internet broadband communications and teleconferencing, where multiple teams of varying sizes—perhaps a dozen or more at a time-could play, with at least one dedicated auditorium capable of “hot wash” sessions, where all participants and observers could participate in pre-and after-action reviews.
Private Sector/Academia Outreach Staff (PS&AO)
The quality of any simulation, and therefore its analytical results, depends directly on the quality of its players. While the IC has more than its share of world-class experts on many subjects, its expertise is dwarfed by that found outside the IC in other government agencies, the private sector, and academia. The NSSC could not realize its full potential without taping into those reservoirs of talent outside the community.
PS&AO would be responsible for identifying outside experts willing and able to contribute their time and talents to working side-by-side with IC analysts to design simulations and to play them out to develop the conclusions. Backed by the name and prestige of the Office of the DNI, the NSSC almost certainly would attract leaders from every relevant field, including former and current heads of state and other high-ranking government officials, corporate CEOs, technology visionaries, and key academic figures. Their appearance in a centrally managed simulation would also ensure that their expertise was more widely shared among all the agencies than possible under present circumstances.
At our core, IC analysts are, first and foremost, investigators and scientists. As professional intelligence officers we aggressively search for meaning and strategic understanding of the world and the forces affecting it. We do this to make sense of the present and to give our nation’s leaders insight, context, and prescience about the future. However, we have been asked to increase the quality and relevance of our insight even as the volume of data increases and the time available to make sense of it decreases.
The National Security Simulations Center could be a 21st-century model for processing and analyzing potential geopolitical developments before they happen. The center would provide additional ways of exploring why things happen, why they break, and what geopolitical levers influence global changes. It would also be a training ground for IC officers to hone their craft. Uncovering hidden assumptions, identifying new indicators, illuminating alternative outcomes, and developing and testing new tools and techniques are tasks inherent in the process of designing and running simulations. As aptly stated by Peter Schwartz in The Art of the Long View, “The scenario process provides a context for thinking clearly about the impossibly complex array of factors that affect any decision.”
Doing what we, as analysts and intelligence collectors, do is going to get harder. The state of the world continues to become more complex. As a nation, how well we continue to influence that complexity is directly related to how well we first make sense of it. The DNI National Security Simulations Center, a seemingly natural step in the evolution of the intelligence profession, would go a long way toward helping us to better understand that world and to better serve our policymakers.