|
Executive Summary This report details a complete, beginning-to-end Cognitive Systems Engineering (CSE) project tackling the challenges of conducting intelligence analysis under the condition of data overload. We first reviewed and synthesized the research base on data overload from multiple complex, high-consequence settings like nuclear power generation. The product of this activity was a diagnosis of the challenges of dealing with data overload in general. Then, leveraging this research base and previous experience in conducting Cognitive Task Analysis (CTA), we conducted a study to identify the aspects of this research base that applied to intelligence analysis as well as unique challenges. We observed expert intelligence analysts conducting an analysis on a selected unclassified scenario, the 1996 Ariane 501 rocket launch failure, with a baseline set of tools that supported keyword search, browsing, and word processing in an investigator-constructed database. From this study, we identified challenging tasks in intelligence analysis that leave analysts vulnerable to making inaccurate statements in briefings when they are working in a new topic area and are under short deadline constraints. In parallel, we identified limitations of the baseline tools in addressing these vulnerabilities that pointed to ideas for new design directions. In addition, the study findings were translated into objective criteria for evaluating the usefulness of any effort aimed at reducing data overload. In the final phase of the project, we shifted from an emphasis on problem definition to an emphasis on developing modular design concepts, or "design seeds," that could be incorporated into both ongoing and future design efforts. The design seeds were instantiated as animated fly-through mockups, or "Animocks," so that feasibility of certain usability considerations, such as the ability to display data in parallel in an easily interpretable form, could be explored without being forced into committing to a particular design. With Animocks, commitment to a particular hardware infrastructure, visualization instantiation, or combination of design seeds is lessened and there is greater flexibility to incorporate feedback about the usefulness of a design concept in addressing data overload. In addition, the design seeds are conceptually modular and based in challenging scenarios, which enables generalization of concepts across design projects and domains. With this strategy, we can better address one of the primary challenges of a research and development program, which is to develop research bases that can be translated into fieldable systems in multiple settings to prevent continuously engaging in individual, one-off design endeavors without learning how to improve systems over time. We believe that complementarity of research and design efforts and accompanying cross-stimulation is the main characteristic of effective Cognitive Systems Engineering R&D programs. |