Research Team

Emily S. Patterson

David D. Woods

Daniel J. Zelik

Research Support

Erik Aagard

Stoney Trent

Martin Voshell

Understanding Rigor in Information Analysis

Across information analysis domains, it is often difficult to recognize when analysis is inadequate for a given context. A better understanding of rigor is an analytic broadening check to be leveraged against this uncertainty. The purpose of this research is to refine the understanding of rigor, exploring the concept within the domain of intelligence analysis. Nine professional intelligence analysts participated in a study of how analytic rigor is judged. The results suggest a revised definition of rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation. Based on this insight, a model for assessing rigor was developed, identifying eight attributes of rigorous analysis. Finally, an alternative model of briefing interactions is proposed that integrates this framing of rigor into an applied context. This research, although specific in focus to intel analysis, shows the potential to generalize across forms of information analysis.

The topic landscape below provides a brief history and overview of our research into understanding rigor in information analysis. This landscape summarizes the developing understanding of rigor relative to the following five topic areas: (1) Exploring rigor to determine how it can be defined in the context of information analysis. (2) Studying professional analysts to identify the cues that they use to infer and evaluate rigor. (3) Assessing rigor by identifying metrics that differentiate between high and low rigor processes. (4) Applying the concept of rigor through innovations that address the challenges of analysis. (5) Looking Forward to discover how understanding rigor shapes the future of information analysis.

Understanding Rigor in Information Analysis Topic Landscape

The following paper, presented at the Eighth International Naturalistic Decision Making Conference, provides a more formal overview of our current research.

Modeling Rigor in Information Analysis: A Metric for Rigor Poster (PDF)

David Woods’ discussion of our analytical rigor research at a Google TechTalk provides a dynamic presentation of the material. Google TechTalks are designed to disseminate a wide spectrum of views on topics including Current Affairs, Science, Medicine, Engineering, Business, Humanities, Law, Entertainment, and the Arts. This talk was originally recorded on on April 10, 2007.

Reducing the Risk of Shallow Information Analysis Google TechTalk

Studying the Professional Analyst

Our exploration of analytical rigor is based, in part, on a research study involving professional intelligence analysts. This study examined how professional analysts judge the rigor behind an analysis. The study investigates the challenges that inhibit the understanding of rigor in intelligence analysis and explores cues used by analysts to identify analytic rigor—or lack of rigor. Nine professional intelligence analysts participated in a modified elicitation by critiquing method study, embedded in a scenario walkthrough. Findings from the study indicate that, while professional intelligence analysts can make perceptive assessments about the quality of an analysis process based on product quality, these perceptions are apt to change with insight into the analytic process.

The following paper, presented at the Human Factors and Ergonomics Society 51st Annual Meeting, provides an overview of the research study and findings.

The Impact of Process Insight on Assessment of Analytic Rigor Poster (PDF)

A complete write up of the study background, methodology, analysis, and results can be found in my Master’s Thesis (CSEL Report 2007-TR-02).

Zelik, D. J. (2007). When is analysis sufficient? A study of how professional intelligence analysts judge rigor. Unpublished M.S. thesis, The Ohio State University, Columbus, OH. (PDF) (ABSTRACT) (LIBRARY)

Analytical Rigor and Process Insight

Our recent research with professional information analysts has revealed promising directions for representing and sharing analyses to create process insight, suggesting two related directions for developing effective representations that support the assessment of rigor. First, our study suggests that effective representation incorporates a synthesis of critical process attributes, finding that analysts viewed analytic rigor as a multi-attribute assessment of sufficiency. Second, the research suggests that an effective representation for revealing analytic rigor facilitates the emergence of a participatory exchange among stakeholders. These two findings indicate that developing an effective representation for creating insight into an analytic process embeds indicators of critical process attributes into a participatory exchange interaction. Our approach for supporting rigor assessment builds a virtual map to the data space behind an analysis—a visualization that affords active shifting among organizing views of critical attributes of an analysis process.

The following paper, presented at the 2nd Annual Workshop on Meta-Information Portrayal, discusses in brief this approach for representing analyses processes.

Zelik, D., Patterson, E. S., & Woods, D. D. (2007). Supporting the assessment of rigor: Representing analysis to create process insight. Paper presented at the 2nd Annual Workshop on Meta-Information Portrayal, Washington, DC. (PDF) (VIDEO)

This research also reveals promising directions for applied explorations of the concept of rigor. We conceptualize the emergence of a tool that captures and represents the analysis process via an intuitive visual interface that not only allows people to better understand their own processes—and to better judge when analyses are sufficient for a given context—but that also allows them to meaningfully share their analysis processes with others. The proposal below outlines “A Strategy for Creating Insight into the Information Analysis Process.”

Research Proposal: Revealing Analytical Rigor (PDF)

Select Resources

See these other pages for additional resources that relate to the current work on analytical rigor in information analysis.

A Cognitive Systems Approach to Supporting Intel Analysis Topic Landscape

Institute for Collaborative Innovation II

Institute for Collaborative Innovation

Zelik, D., Patterson, E. S., & Woods, D. D. (2007, June). Understanding rigor in information analysis. Paper presented at the 8th International Conference on Naturalistic Decision Making, Pacific Grove, CA. (PDF) (VIDEO)

This recent poster provides a brief overview of our research on analytical rigor, contrasting a conventional perspective on rigor with a practice-driven perspective.

Zelik, D., Patterson, E. S., & Woods, D. D. (2007). Judging sufficiency: How professional intelligence analysts assess analytical rigor. In Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting [CD-ROM]. Santa Monica, CA: Human Factors and Ergonomics Society. (PDF)

This poster provides a summary of the findings in the HFES proceedings paper.

Analytical Rigor Poster (PDF)

This poster provides an overview of the rigor model, identifying the aspects of the attributes that contribute to low, moderate, and high rigor analysis processes. It also overviews the rigor metric as applied to the LNG Scenario study.