4.8 Design Seed 5: Conflict and Corroboration Support

One of the most important requirements of an analytic product is that it be
accurate, or at least that the analyst be well-calibrated as to its accuracy. This is
one of the main reasons that the same report written by an intelligence analyst
and an unknown author are viewed differently: by putting an intelligence
analysis organization's name on the report, there is an implicit "stamp of
approval" that increases its value.

Study participants clearly viewed the elimination of inaccuracies by finding
converging evidence across independent sources as a major component of the
value of an analytic product. The participants described and employed a variety
of strategies for tracking and resolving discrepant descriptions in order to reduce
their vulnerability to incorporating inaccurate information. Partly because this
cognitively difficult process of corroborating information and resolving
conflicting information was unsupported by the tools that they were provided,
nearly every participant experienced some breakdowns in this process.

During the study, the participants described strategies that they would use if
they had more time to conduct an analysis to ensure that information was
corroborated, such as printing out the documents and highlighting topics with
particular colors every time they appeared from independent sources. However,
in general, no study participants used these strategies during the simulated task
because of the short deadline, and described that under high workload
conditions they tended to shed this task in the workplace as well. In addition,
several noted that their strategies were not well-supported within the electronic
environment (e.g., it is difficult to see information from multiple documents in
parallel on the screen). Supporting these strategies would not be overly difficult
in an electronic environment, and this support forms the essence of this design
seed.

In summary, this design seed has characteristics of supporting the following
tasks to improve the accuracy of an analytic product:
• identifying data conflicts,
• highlighting uncertain information,
• remembering judgments about relationships between data,
• tracking "loose ends" that need to be resolved later,
• identifying when data comes from the same original source7, and
• identifying attempts of others to purposely misinform and deceive.

The primary design challenge with this design seed is how to minimize data
entry while still supporting the observed strategies. Although there are many
theoretically possible categories of relationships among data elements (Schum,
1994), in the strategies we observed, analysts only documented broad-brush
distinctions, e.g., same topic, uncertain, verified. Similarly, they did not
explicitly say what information was conflicting, just that they previously saw
something that they thought contradicted what they were reading then.
Therefore, design seeds that require overly specific data relationship categories
or require specifying what information conflicts with each other might be judged
to require too much work on the user's part to be useful.

As with the other design seeds, the level of machine intelligence could vary
greatly based on how much information is available to the machine intelligence
to process and how often the machine processing will be incorrect. At the lowest
level of machine intelligence, the software could simply display judgments made
by intelligence analysts, such as by displaying an underline mark under a word
that an analyst selected to be underlined for an unknown reason. At the highest
end, a user could ask the machine to critique the process that had been followed
in verifying the information was accurate. The machine could then:
• analyze the breadth of information sampling in time and in high profit
documents,
• identify potential data conflicts both in the information that was read as
well as potentially available in databases,
• check that information that was marked as corroborating came from
independent sources, and
• assess the quality of the documents that were most heavily used in the
analysis (most likely based on what information has been pulled or
marked from what documents).


7 Note that intelligence analysts refer to a potential vulnerability in analysis as "creeping validity." This
phrase is used to refer to situations where multiple reports appear to corroborate an event or other piece of
information that actually same from the same original source. In these cases, even though there are
multiple accounts, belief in the accuracy of the account should not be increased.



TABLE OF CONTENTS