Library

 

Making the Analytic Review Process Work

Toward a Stronger Intelligence Product

Martin Petersen


The problem with the review process is not the layers of review but rather the quality of the review.”

If there is a first principle in producing written intelligence, it is that finished intelligence is a corporate product, not a personal one. Intelligence officers all have a stake in everything that leaves the building. A rigorous, focused review process is the best guarantee that the style, message, and tradecraft of every piece of finished intelligence meet the standards that the mission requires.

Like the tides, criticism of the analytic review process is predictable, relentless, eternal, and potentially destructive. Those who argue for more power to the drafter present a bill of particulars that alleges the process does little to improve the product, reduces judgments to the lowest common denominator, stifles creativity, and takes analysis out of the hands of the experts. Those who defend the review process counter that it sharpens focus, guarantees that the piece addresses policymaker concerns, taps all relevant expertise, and ensures a corporate product. Both sides agree on one thing—that there ought to be fewer layers of review—and both miss the key point.

The problem with the review process is not the layers of review but rather the quality of the review. In an imperfect business, this is the one thing that intelligence officers need to get right. My 30-plus years of experience leads me to conclude that there should be three levels of review and three broad areas of review for each piece of finished intelligence.

Editing is NOT review. Editing is a mechanical task that should be accomplished by the first-level reviewer or by a staff. Review is about thinking, about questioning evidence and judgments. It focuses on the soundness of the analytic points that are being made and the quality of the supporting evidence. Levels of review is NOT synonymous with layers of review. Layers of review speaks to how many cooks are involved with the broth; levels of review is about ascertaining the quality of the soup.

Each level of review has a different focus. The strength of the review process is directly related to the different perspective that each level brings, with succeeding levels focusing on ever broader issues that are hard for the author and firstline reviewer to see because they are so close to the substance.

 

Analytic Review Process at a Glance 

Level of Review

Checking for Style*

Clarifying the Message*

Monitoring Tradecraft*

First

Reviewer: Firstline supervisor, who is close to the subject matter.

Focus: What is in the piece.

  • Are grammar, spelling, and other technical aspects correct?
  • Is language crisp?
  • Is the flow logical?
  • Are key points clear and argued with evidence?
  • Can assertions be supported?
  • Is the what/so-what for the US evident?
  • Are facts correct?
  • Are sources described accurately?
  • Is the evidence correctly characterized?
  • Has all relevant evidence been considered?
  • Are alternate interpretations acknowledged?
  • Is the piece consistent with previous analysis?
  • Is it clear what is known and not known, and what the level of confidence is?
  • What assumptions underpin the analysis?
  • What are the key drivers and variables?

Second

Reviewer: Middle manager, who is somewhat expert but has a broader perspective with respect to substance and audience.

Focus: What underpins the piece.

  • Any confusing technical language or jargon?
  • Are key points clear and argued with evidence?
  • Can assertions be supported?
  • Is the what/so-what for the US evident?
  • Are alternate interpretations acknowledged?
  • Is it clear what is known and not known, and what the level of confidence is?
  • What assumptions underpin the analysis?
  • What are the key drivers and variables?
  • Is the piece consistent with previous analysis?
  • Have the right questions been asked? Any information gaps?
  • Are policymakers' concerns addressed?

Third

Reviewer: Office-level manger or senior officer of the organization, who is not expert but has a very broad context.

Focus: The audience.

  • Any confusing technical language or jargon?

  • Are key points clear and argued with evidence?
  • Can assertions be supported?
  • Is the what/so-what for the US evident?
  • Is it clear  what is known and not known, and what the level of confidence is?
  • Is the piece consistent with previous analysis?
  • What assumptions underpin the analysis?
  • What are the key variables and drivers?
  • Have the right questions been asked?
  • Is the piece clear to the non-expert?
  • Are policymakers' concerns addressed?

 

*Bold type indicates special focus

[Top of Page]

 

First-Level Review

The drafter's supervisor is almost always the first-level reviewer. Of all the reviewers, this individual is usually the most expert and closest to the substance, and therefore bears the greatest responsibility—after the author—for the substantive accuracy of the piece. The focus of the first-level review should be the content of the piece.

Style. Of the three broad areas of review—style, message, and tradecraft—style is the exclusive domain of the first-level reviewer. He or she does the “blue pencil” edit, assuring that the language is clear and crisp and that the piece flows logically. The second- and third-level reviewers should resist mightily the all too human temptation to “tweak the prose” or “polish the draft.”

Message. The first-level reviewer shares with the next two levels the responsibility for the clarity of the message. Review at this point is about thinking, not editing, and the first thing the reviewer should be thinking about is why anyone should take a minute out of his or her busy schedule to read something that took days to produce. Because it is “interesting” is not sufficient.

The what/so-what of the paper has to be crystal clear in the title or the first sentence. Without the ability to attract the reader, there is no point in producing the piece. The first level is the expert's eye view. The second level is the perspective of someone steeped in the subject matter but not genuinely expert, and the third level is from the point of view of the intelligent generalist. When they agree, the what/so-what is as clear as it is going to be. If the expert has a vulnerability, it is assuming too much knowledge on the part of the audience and its ability to see the connections.

Assertions are among the vehicles for conveying the message. Assertions are not facts. That Beijing is the capital of China is a fact. That Chinese military leaders take a harder line than civilian leaders on Taiwan issues is an assertion—a judgment based on earlier analytic work and evidence not cited. Assertions are necessary; intelligence is not geometry, and analysts have neither the time nor the space to prove every point. But, assertions are dangerous, too. They need to be questioned periodically, especially if long held and based on earlier analysis, for they have a tendency to become subconscious assumptions over time. Questioning assertions needs to begin at the first level, but the second- and third-level reviewers play an even more critical role here.

Tradecraft. The analyst and the first-level reviewer bear the primary responsibility for the quality of tradecraft. They have near total accountability for those things that higher levels of review are unlikely to catch, specifically:

Accuracy of information presented as facts. Silly things do slip through. I remember seeing a piece early in my career that made reference to the China-Thai border area. Thailand does not share a border with China.

Descriptions of the sources and what the reporting says precisely. It matters a great deal whether the source states that a country is attempting to acquire some technology or whether the country has actually acquired the technology, and the analysis must reflect this distinction, even if, on balance, the analyst believes that the country succeeded in acquiring the technology. Why? Because if intelligence officers are called to explain the analysis and to present the evidence, the evidence will only reflect the attempt; success in this instance is an analytic judgment. It also matters—at least to the audience—whether the source has direct knowledge or indirect knowledge. The rule is simple: the more politically sensitive the subject matter, the greater the requirement for absolute precision, because on a politically charged issue a small slip in one piece will be used to discredit all analytic work.

Characterizations of the body of evidence. A “majority of the reporting” means a majority, and “almost every” is not 60 percent.

These are right or wrong issues— either the piece correctly reports what the source says or it does not. The other six tradecraft issues that the first-level reviewer must consider are even more difficult, and therefore more important to pay attention to. In ascending order of difficulty:

Has all the relevant evidence been taken into consideration? Considered means just that: thought about.

Is there an alternate explanation or interpretation of the evidence and is it acknowledged? The drafter is not obligated to walk the reader through all of the various interpretations or permutations of the evidence. There ought to be a presumption that the analyst considered alternate interpretations before arriving at the interpretation presented. That said, an author owes it to the reader, especially on complex or controversial subjects, to inform him or her that another view exists, especially if that view has a measure of acceptance by other experts. “Suggests” and “indicates” may be the two favorite verbs of analysts. My experience is that what follows “suggests” or “indicates” is the analyst's pet theory, for which there is some but not compelling evidence. My first question as a reader—and it ought to be the first question of the reviewer—is what else might this development suggest or indicate and why is the theory put forth as the best possibility?

Is this piece consistent with previous analysis? If it is a departure, then that must be acknowledged explicitly and explained. Also, if it has been some time since the subject was treated, caution dictates a quick review of previous work—as an institution, we must never lose track of what precisely has been said—and probably some language in the piece by way of background for the reader. Over time, caveats and qualifiers are at risk of being dropped or forgotten, especially as different analysts or reviewers become involved, and this seems to have happened in the case of Iraq recent years.

Is it clear what is known and what is not known? And has the analyst conveyed the level of confidence in the judgments and conclusions put forth in the piece? There is a difference between “sharpening judgments” and “firming up judgments.” To sharpen a judgment is to be very clear about what the judgment is and how confident the analyst is in that judgment. To firm up a judgment begs the confidence level at best and in all likelihood conveys a greater degree of confidence than actually exists.

What assumptions underpin the analysis? There is no tougher issue to get at, nor one with graver consequences if left unexamined. Assumptions are different from assertions— assertions are explicit statements whereas assumptions tend to be implicit to the analysis. The CIA Directorate of Intelligence's work on the fall of the Shah of Iran in the 1970s and Iraq's weapons of mass destruction in the early 2000s share a common characteristic: each was premised on a strong, widely held assumption. In the case of Iran, it was assumed that the Shah was strong and the opposition weak and divided; in the case of Iraq, it was assumed that Saddam Hussein would not allow his stockpiles of chemical and biological weapons to erode. In both instances, the assumptions led analysts to interpret “could be” behavior as “is” behavior. If the analyst and the reviewer cannot articulate the assumptions on which the analysis rests, then they are flying blind. They have no idea how they can be wrong and no means to test if they are.

What are the key variables, changes in which would alter the assessment? The variables are the drivers and the causal links in the analysis, the “if A, then B” part of the analysis. They are closely tied to the assumptions but generally are more evident. Most of the key variables will not be mentioned in the piece, especially if US actions are one, as they almost always are. The easiest way to get at the variables is to ask “what if.” A change in one of these variables should alter the analysis, and indeed, should help the analyst think through how it might change. Spelling out, discussing, and periodically revisiting what the analyst believes to be the key drivers and shapers of the issue are also the best ways to identify the underlying assumptions. The 23 September 2004 transmittal message to the Duelfer Report on Iraq's weapons of mass destruction discusses hidden assumptions and Saddam's views and perceptions at length. The report argues that Saddam saw his situation, the United States, and Iran very differently than his adversaries did—a different set of causal links, variables, assumptions, and what-ifs. Sometimes the key variable is a strategic perspective, as it may have been in the Saddam case; at other times, it may be more mundane, like whether the railroad system can get the contraband to a port in time to make a sailing date.

[Top of Page]

 

Second-Level Review

The second-level reviewer is usually the issue manager, who is well steeped in the subject matter but not as expert as the analyst or the firstline supervisor. As a rule, this individual is closer to the policymaker. The comparative advantage in the review process is a broader perspective, with regard to both the substance and the audience. The second reviewer is better positioned to see how the piece at hand fits in with other work being done and how it relates to the audience's needs. The focus of review at the second level should be those things that underpin the piece.

Style. The second-level reviewer should not be editing the piece— the blue pencil function. Flow and presentation are checked at the first level, and the odds are good that there is a staff that will do the final technical edit. The one thing the second-level reviewer should keep an eye out for is technical language or jargon that would not be immediately clear to the audience.

Message. The real work of the second-level reviewer starts here. Whereas the firstline supervisor bears the principal responsibility for the substantive accuracy of the piece, subsequent reviewers, by virtue of their greater perspective, bear responsibility for the clarity of the message for the audience, ensuring that the points the piece is attempting to make are apparent and that the supporting evidence is compelling (at best) or supportive (at a minimum). If the analyst and the first-level reviewer have done their job, this should not be that demanding. The key question for the second-level reviewer is whether the what/so-what will engage the target audience and whether the key points will seem convincing. By virtue of expertise and perspective, the second-level reviewer is in the best position to judge whether the analyst has made the case for whatever is being argued.

With regard to assertions, the second-level reviewer needs to ask: Does the assertion make sense; can it stand on its own merit without further explanation or evidence needed to convince the audience? It is the audience that matters in making this judgment, not the analyst. On issues where there are sharp policy divisions or great uncertainty, it is especially important to pay attention to the assertions, because they are an easy way to question or discredit the overall message. An overdrawn or unexamined assertion is one of the easiest ways to go wrong quickly.

Tradecraft. This is where the second level reviewer does his or her heavy lifting. As the chart shows, four of seven tradecraft issues for consideration are exactly the same as they are for the first-level reviewer and should be looked at in the same way: Is there an alternate explanation or interpretation of the evidence, and is it acknowledged? Is it clear what is known and what is not known, and has the analyst conveyed the level of confidence in the judgments and conclusions put forth in the piece? And what assumptions underpin the analysis and what are the key variables, changes in which would alter our assessment?

Three other tradecraft issues are similar to ones at the first level of review, but have a different focus here largely because of the broader perspective of the second-level reviewer:

Is this piece consistent not only with previous work on this topic but also with other analysis being done in the issue group? Policymakers generally have a broader set of responsibilities than analysts and any differences, real and perceived, across issues will be readily apparent to them. It is not unusual for different observers (and thus different analysts) to have different perceptions of multiparty talks, for example, and this can create the impression that analytic products are all over the map on the issue under discussion. It is also important for reviewers at this level to be alert to linkages between issues—increased tension in the Taiwan Strait has implications for Japan as well, for example.

Does the piece ask the right questions and are there gaps in our knowledge that could have a major impact on our analysis? This is where the second-level reviewer has the greatest potential impact. More than at the first stage, the second-level reviewer ought to be the “what if” person—the one with enough expertise and enough perspective to get above the piece at hand. Specifically, it falls to the second-level reviewer more than anyone else to assure that the right questions are being asked, that blind spots in information are identified and factored in, and that alternate possibilities for key variables are considered. In the case of Iran in the 1970s, the points of views of the Shah and the opposition were well understood, but the view of wider Iranian society—the “swing vote”—was a key information gap. Some basic questions were not asked, including about the Shah's ability to follow a coherent course and the opposition's ability to work together. The willingness and ability of religious moderates to act as a third force was a key variable. Once a crisis becomes full blown, it is easy for analysts to get eaten up by the daily demand. This is precisely the moment when it is most necessary to step back and reconsider assumptions and variables, and ask the big “what if” questions.

Does the paper address known policymaker concerns and likely questions? Because the second-level reviewer is closer to the audience than the firstline supervisor, he or she is better positioned to ascertain whether the piece is likely to scratch the policymaker's itch. This reviewer is also better positioned to know whether the piece at hand is likely to generate additional questions, perhaps on related matters.

[Top of Page]

 

Third-Level Review

The third-level review should be done by the office director or the staff of a senior officer in the organization. On a particularly sensitive piece, both may weigh in. Their comparative advantage is that of the intelligent generalist who operates in a broad policy context. The third-level reviewer focuses on the piece almost exclusively from the perspective of the audience.

Style and Message. As at the previous stage, attention to style should be minimal, with an eye only for inappropriate or confusing jargon or technical language. One question should dominate the third-level review: will the intended audience find the piece convincing?

Tradecraft. The questions and concerns of the third-level reviewer are not significantly different from those of the second-level reviewer. The value added is in the broader perspective at this level, both in terms of the reviewer's contact with a wider range of policymakers and an improved ability to see the forest for the trees. Like earlier reviews, the third level needs to ponder core tradecraft questions: Is it clear what is known and not known and what the level of confidence is? What assumptions underpin the analysis? And does the piece address policymaker concerns?

The third-level reviewer should primarily be concerned with two pairs of tradecraft issues. The first set relates to clarity: Will the non-expert understand the piece, and is it consistent with other work being done in the organization? The second set is the more challenging, and it goes to the “what if” questions. As an intelligent general reader, the third-level reviewer is best positioned to ask the “dumb questions” that never would occur to the expert. The third-level reviewer should focus most on whether the right questions have been asked and what the key variables are.

[Top of Page]

 

Shared Responsibilities

In making sure that the all-important review process works, reviewers and analysts share two obligations: do all that they can to create an environment that facilitates an exchange of views and keep the discussion professional and not personalize the issues or get emotional. Beyond these common responsibilities, they each have their own set of responsibilities:

Reviewers must respect the views of analysts. A reviewer's experience and perspective are strengths, but so are an analyst's expertise and command of detail. Reviewers must be open to discussing substantive differences raised by analysts. Although the final say goes to the reviewers, the process should be a dialogue not a decree. Reviewers, moreover, have an obligation to put analysts at ease and to draw out their views. They should be specific about their concerns or issues. If a reviewer cannot explain what the problem is, the problem may be the reviewer. Reviewers should complete their work quickly. If the piece is a priority for the analyst, it has to be a priority for the reviewer. Finally, reviewers must be prepared to stand behind the analysts and their analysis at the conclusion of the review process.

Analysts, for their part, should submit the best draft they are capable of. They should respect the experience, perspective, and expertise of the reviewers, and accept that the final say belongs to them. An analyst can and should seek clarification if he or she does not understand what a reviewer is saying or wanting, raising any concerns about what the reviewer is suggesting by using data, history, alternate theories, or intelligence reporting. If analysts are unhappy with what reviewers have done, they should be ready to offer other language or suggest another approach to the issue at hand.

To be effective, the review process must remain collegial. There is no monopoly on either expertise or broad insight, and now more than ever the nation's security is linked to a fusion of the two.

 

 

Martin Petersen served in the CIA Directorate of Intelligence.

[Top of Page]

 


Historical Document
Posted: Apr 15, 2007 08:32 AM
Last Updated: Jun 26, 2008 02:22 PM