|
"How to Design for Human-Automation Coordination: Observability And Directability" More sophisticated automated systems or suites of automation represent an increase in autonomy and authority (Woods, 1996). Increasing the autonomy and authority of machine agents is not good or bad in itself. The research results indicate that increases in this capability create the demand for greater coordination. The kinds of interfaces and displays sufficient to support human performance for systems with lower levels of autonomy or authority are no longer sufficient to support effective coordination among people and more autonomous machine agents. When automated systems increase autonomy or authority without new tools for coordination, we find automation surprises contributing to incidents and accidents (for summaries see Woods, 1993; Woods, Sarter and Billings, 1997; Woods and Sarter, 2000). The field research results are clear -- the issue is not the level of autonomy or authority, but rather the degree of coordination. However, the design implications of this result are less clear. What do research results tell us about how to achieve high levels of coordination between people and machine agents? What is necessary for automated systems to function as cooperative partners rather than as mysterious and obstinate black boxes? The answer, in part, can be stated simply as -- Cooperating automation is both observable and directable. Observability: Opening up the Black Box One of the foundations of any type of cooperative work is a shared representation of the problem situation (e.g., Grosz, 1981; McCarthy et al, 1991). In human-human cooperative work, a common finding is that people continually work to build and maintain a “common ground” of understanding in order to support coordination of their problem solving efforts (e.g., Patterson et al., 1999). We can break the concept of a shared representation into two basic (although interdependent) parts: (1) a shared representation of the problem state, and (2) representations of the activities of other agents. The first part, shared representation of the problem situation, means that the agents need to maintain a common understanding of the nature of the problem to be solved. What type of problem is it? Is it a difficult problem or a routine problem? Is it high priority or low priority? What types of solution strategies are appropriate? How is the problem state evolving? The second part, shared representation of other agents’ activities, involves access to information about what other agents are working on, which solution strategies they are pursuing, why they chose a particular strategy, the status of their efforts (e.g., are they having difficulties? why? how long will they be occupied?), and their intentions about what to do next. Together with a set of stable expectations about the general strategies and behavior of other agents across contexts, mutual knowledge about the current situation supports efficient and effective coordination among problem solving agents (Patterson et al., 1999). Agents can anticipate and track the problem solving efforts of others in light of the problem status and thus coordinate their own actions accordingly. The communicative effort required to correctly interpret others’ actions can be greatly reduced (e.g., short updates can replace lengthy explanations). The ability to understand changes in the state of the monitored process is facilitated (e.g., discerning whether changes are due to a new problem or to the compensatory actions of others). An up to date awareness of the situation also prepares agents to assist one another if they require help. Notice how much of the knowledge discussed here is available at relatively low cost in “open” work environments involving multiple human agents. For example, in older, hardwired control centers, individual controllers can often infer what other controllers are working on just by observing which displays or control panels they are attending to. In the operating room, surgical team members can observe the activities of other team members and have relatively direct, common access to information about the problem (patient) state. The open nature of these environments allows agents to make intelligent judgments about what actions are necessary and when they should be taken, often without any explicit communication. However, when we consider automated team members, this information no longer comes for free – we have to actively design representations to generate the shared understandings which are needed to support cooperative work. Data Availability Does Not Equal Informativeness Creating observable machine agents requires more than just making data about their activities available. As machine agents increase in complexity and autonomy, simple presentations of low-level data become insufficient to support effective interaction with human operators. For example, many early expert systems “explained” their behavior by providing lists of the individual rules which had fired while working through a problem. While the data necessary to interpret the system’s behavior was, in a literal sense, available to operators, the amount of cognitive work required to extract a useful, integrated assessment from such a representation was often prohibitive. A more useful strategy was to provide access to the intermediate computations and partial conclusions that the machine agent generated as it worked on a problem. These were valuable because they summarized the machine agent's conception of the problem and the bases for its decisions at various points during the solution process. In general, increases in the complexity and autonomy of machine agents requires a proportionate increase in the feedback they provide to their human partners about their activities. Representations to support this feedback process must emphasize an integrated, dynamic picture of the current situation, agent activities, and how these may evolve in the future. Otherwise, mis-assessments and miscommunications may persist between the human and machine agents until they become apparent through resulting abnormal behavior in the process being controlled. For example, the relatively crude mode indicators in the current generation of airliner cockpits have been implicated in at least one major air disaster. It is clearly unacceptable if the first feedback pilots receive about a miscommunication with automation is the activation of the ground proximity alarm (or worse). Human agents need to be able to maintain an understanding of the problem from the machine agent’s perspective. For instance, it can be very valuable to provide a representation of how hard the machine agent is having to work to solve a problem. Is a problem proving especially difficult? Why? If the automated agent has a fixed repertoire of solution tactics, which have been tried? Why did they fail? What other options are being considered? How close is the automation to the limits of its competence? Having this sort of information at hand can be extremely important to allow a human agent to intervene appropriately in an escalating critical situation. Providing effective feedback to operators in complex, highly automated environments represents a significant challenge to which there are no ready-made solutions. Answering this challenge for the current and future generations of automation will require fundamentally new approaches to designing representations of automation activity (e.g., Sarter, 1999; Sklar and Sarter, 1999; Nikolic and Sarter, 2001). While the development of these approaches remains to be completed, we can at least sketch some of the characteristics of these representation strategies (Woods and Sarter, 2000). The new concepts will need to be: Event-based: representations will need to highlight changes and events in ways that the current generation of state-oriented display techniques do not. Future-oriented: in addition to historical information, new techniques will need to include explicit support for anticipatory reasoning, revealing information about what should/will happen next and when. Pattern-based: operators must be able to quickly scan displays and pick up possible abnormalities or unexpected conditions at a glance rather than having to read and mentally integrate many individual pieces of data. Directability: Who Owns the Problem? Giving human agents the ability to observe the automation’s reasoning processes is only one side of the coin in shaping machine agents into team players. Without also giving the users the ability to substantively influence the machine agent’s activities, their position is not significantly improved. One of the key issues which quickly emerges in trying to design a cooperative human-machine system is the question of control. Who is really in charge of how problems are solved? Billings (1996) has argued that as long as some humans remain responsible for the outcomes, they must also be granted effective authority and therefore ultimate control over how problems are solved. Giving humans control over how problems are solved entails that we, as designers, view the automation as a resource which exists to assist human agents in the process of their problem solving efforts. While automation and human activities may integrate smoothly during routine situations, unanticipated problems are a fact of life in complex work environments such as those where we typically find advanced automation. It is impossible in practice, if not in principle, to design automated systems which account for every situation they might encounter. While entirely novel problems may be quite rare, a more common and potentially more troublesome class of situations are those which present complicating factors on top of typical, “textbook” cases (cf. studies of brittleness of automated systems include Roth et al., 1987; Guerlain et al., 1996; Smith et al., 1997). These cases challenge the assumptions on which the pre-defined responses are based, calling for strategic and tactical choices which are, by definition, outside the scope of the automation’s repertoire. The relevant question is, when these sorts of problems or surprises arise, can the joint system adapt successfully? Traditionally, one response to this need has been to allow human operators to interrupt the automation and take over a problem manually. Conceiving of control in this way, an all-or-nothing fashion, means that the system is limited to operating in essentially one of two modes – fully manual or fully automatic. This forces people to buy control of the problem at the price of the considerable computational power and many potentially useful functions which the automation affords. What is required are intermediate, cooperative modes of interaction which allow human operators to focus the power of the automation on particular sub-problems, or to specify solution methods that account for unique aspects of the situation which the automated agent may be unaware of. In simple terms, automated agents need to be flexible and they need to be good at taking direction. Part of the reason that directability is so important is that the penalties for its absence tend to accrue during those critical, rapidly deteriorating situations where the consequences can be most severe. One of the patterns that we see in the dynamic behavior of complex human-machine systems during abnormal situations is an escalation in the cognitive and coordinative demands placed on human operators (Woods and Patterson, 2000). When a suspicious or anomalous state develops, monitoring and attentional demands increase; diagnostic activities may need to be initiated; actions to protect the integrity of the process may have to be undertaken and monitored for success; coordination demands increase as additional personnel/experts are called upon to assist with the problem; others may need to be informed about impacts to processes under their control; plans must be modified, contingencies considered; critical decisions need to be formulated and executed in synchronization with other activities. All of this can occur under time pressure (Klein et al., 2000). These results do not imply that automation work only as a passive adjunct to the human agent. This is to fall right back into the false dichotomy of people versus automation. Clearly, it would be a waste of both humans’ and automation’s potential to put the human in the role of micro-managing the machine agent. At the same time however, we need to preserve the ability of human agents to act in a strategic role, managing the activities of automation in ways that support the overall effectiveness of the joint system. As was found for the case of observability, one of the main challenges is to determine what levels and modes of interaction will be meaningful and useful to practitioners. In some cases human agents may want to take very detailed control of some portion of a problem, specifying exactly what decisions are made and in what sequence, while in others they may want only to make very general, high level corrections to the course of the solution in progress. Accommodating all of these possibilities is difficult and requires careful iterative analysis of the interactions between system goals, situational factors, and the nature of the machine agent. However, this process is crucial if the joint system is to perform effectively in the broadest possible range of scenarios (Roth et al., 1997; Guerlain et al., 1999; Smith et al., 2000; Smith, in press). In contrast to this, technology-driven designs tend to isolate the activities of humans and automation in the attempt to create neatly encapsulated, pseudo-independent machine agents. This philosophy assumes that the locus of expertise in the joint human-machine system lies with the machine agent, and that the human’s role is (or ought to be) largely peripheral. Such designs give de facto control over how problems are solved to the machine agent. However, experience has shown that when human agents are ultimately responsible for the performance of the system, they will actively devise means to influence it. For example, pilots in highly automated commercial aircraft have been known to simply switch off some automated systems in critical situations because they have either lost track of what the automation is doing, or cannot reconcile the automation’s activities with their own perception of the problem situation. Rather than trying to sort out the state of the automation, they revert to manual or direct control as a way to reclaim understanding of and control over the situation. The uncooperative nature of the automated systems forces the pilots to buy this awareness and control at the price of abandoning the potentially useful functions that the automation performs, thus leaving them to face the situation unaided. Summary When designing a joint system for a complex, dynamic, open environment, where the consequences of poor performance by the joint system are potentially grave, the need to shape the machine agents into team players is critical. Traditionally, the assumption has been that if a joint system fails to perform adequately, the cause can be traced to so-called "human error". However, if one digs a little deeper, they find that the only reason many of these joint systems perform adequately at all is because of the resourcefulness and adaptability that the human agents display in the face of surprises and uncooperative machine agents. The ability of a joint system to perform effectively in the face of difficult problems depends intimately on the ability of the human and machine agents to cooperate and capitalize upon the unique abilities and information to which each agent has access. For automated agents to become team players, there are two fundamental characteristics which need to be designed in from the beginning: observability and directability. In other words, users need to be able to see what the automated agents are doing and what they will do next relative to the state of the process, and users need to be able to re-direct machine activities fluently in instances where they recognize a need to intervene. These two basic capabilities are the keys to fostering a cooperative relationship between the human and machine agents in any joint system. |