4.12 Design Seed 9: Finding Updates

The final design seed is an attempt to address what is probably the most difficult
challenge in time-pressured computer-supported inferential analysis under data
overload conditions. When analyzing the data from the study participants, a
surprising finding was that the study participant who had the most prior
knowledge of the Ariane 501 scenario, the most technical knowledge about
rocket launcher technology, spent the most time during the analysis, and
generated a written briefing in addition to a verbal briefing made an inaccurate
statement in the written briefing. This inaccurate statement appears to be
explained by a particularly difficult challenge during data analysis: detecting
updates to once-believed-accurate information. Because the "findings" or data
set on which the analysis was based came in over time, there was always the
possibility of missing information that was released after the report that was
being read that could overturn or render previous information "stale" (see Figure
31 for examples in the Ariane 501 scenario). When these updates occurred on
themes that were not central enough to be included in report titles or
newsworthy enough to generate a flurry of reports, it was extremely difficult to
know if updates had occurred or where to look for them.

This design seed has the characteristic of:
• Helping analysts to locate updates that overturn or substantially change
an analytic conclusion
• Helping analysts to calibrate their assessment of analytic accuracy to the
likelihood that updates that render analytic conclusions inaccurate do not
exist.




It is interesting to note that most of the study participants never specifically
looked for updates during the analysis process or described strategies that would
do so. It is possible that training analysts about the need to search for updates
might be useful, although the reaction of one novice analyst to the critique that
he should look for updates was that it would be very hard to do with the tools
available to him. Updates could be reported hours, days, weeks, months, years,
or decades after an event. Many of the updates on more minor themes in the
Ariane 501 scenario did not cause a flurry of reports and were not reflected in the
date/title view of the reports.

It is possible that "agents" that suggest targeted query formulations and/or
"seed" representations with updates on a theme might be useful, particularly if
the agents have advanced natural language processing capabilities. We believe
that this design direction will require artifact-based investigations in order to
gain a better understanding of how to make the concept useful. Finding updates
over time is a difficult challenge for human intelligence with current tools, and
yet it is likely also to be so challenging for machine intelligence that conclusions
by the machine intelligence will likely be incorrect much of the time. How much
can the vulnerability to missing updates be reduced simply by having the
machine intelligence remind the human partner to look for updates? Are there
cues to informative areas where updates might be found, such as a flurry of
setpoint crossings in a short amount of time on interrelated systems? Should the
machine intelligence suggest possible candidate updates, either by "seeding" a
visualization or by requiring the human to explicitly consider recommended
items? What advancements in machine intelligence are required to make more
accurate seeding recommendations?
In addition to significant work required to ensure usefulness of this design seed,
there will be substantial usability problems to address. Visualizations will need
to be investigated to ensure that machine processing is observable and directable
by the user in order to make the human-machine teamwork effective and avoid
situations where the human agent is surprised by actions of the machine agent.



TABLE OF CONTENTS