The focus of the current study was on how the dismounted soldiers’ decision cycle is affected by the use of a display device for utilizing intelligence from an unmanned ground vehicle during a patrol mission. Via a handheld monocular display, participants received a route map and sensor imagery from the vehicle that was ~20–50 m ahead. Twenty-two male participants were divided into two groups, with or without the sensor imagery. Each participant navigated for 2 km in a military urban terrain training facility, while encountering civilians, moving and stationary suspects, and improvised explosive devices. The OODA loop (observe–orient–decide–act) framework was used to examine soldiers’ decisions. The experimental group was slower to respond to threats and to orient. They also reported higher workload, more difficulties in allocating their attention to the environment, and more frustration. These can be partially attributed to the novelty of the technological capability, but also to its implementation in the study. The breakdown of performance metrics into the OODA loop components enabled analysis of the major difficulties in the decision-making process. This evaluation highlights the need for new roles in combat-team setups and for additional training when unmanned vehicle sensor imagery is introduced.
The effectiveness of different instrument approach charts to deliver minimum visibility and altitude information during airport equipment outages was investigated. Eighteen pilots flew simulated instrument approaches in three conditions: (a) normal operations using a standard approach chart (standard-normal), (b) equipment outage conditions using a standard approach chart (standard-outage), and (c) equipment outage conditions using a prototype decluttered approach chart (prototype-outage). Errors and retrieval times in identifying minimum altitudes and visibilities were measured. The standard-outage condition produced significantly more errors and longer retrieval times versus the standard-normal condition. The prototype-outage condition had significantly fewer errors and shorter retrieval times than did the standard-outage condition. The prototype-outage condition produced significantly fewer errors but similar retrieval times when compared with the standard-normal condition. Thus, changing the presentation of minima may reduce risk and increase safety in instrument approaches, specifically with airport equipment outages.
The design and adoption of decision support systems within complex work domains is a challenge for cognitive systems engineering (CSE) practitioners, particularly at the onset of project development. This article presents an example of applying CSE techniques to derive design requirements compatible with traditional systems engineering to guide decision support system development. Specifically, it demonstrates the requirements derivation process based on cognitive work analysis for a subset of human spaceflight operations known as extravehicular activity. The results are presented in two phases. First, a work domain analysis revealed a comprehensive set of work functions and constraints that exist in the extravehicular activity work domain. Second, a control task analysis was performed on a subset of the work functions identified by the work domain analysis to articulate the translation of subject matter states of knowledge to high-level decision support system requirements. This work emphasizes an incremental requirements specification process as a critical component of CSE analyses to better situate CSE perspectives within the early phases of traditional systems engineering design.
Current usages of model-based systems engineering allow naïve substitutions of humans by machines. Human factors / ergonomics researchers have rejected such substitutions as the "substitution myth," for if work is reallocated from a human to a machine, then there is work incurred to ensure that the machine is working properly—it must be supervised. We construct a template for what automation should look like when the need for supervision is taken into account. The template can be applied to understand the arrangements for supervising automation in systems as they are and to explore the options for systems that are being designed. We consider examples from electronic warfare self-protection and the command and control of sensor-weapon systems in the land domain.
Policy capturing is a judgment analysis method that typically uses linear statistical modeling to estimate expert judgments. A variant to this technique is to capture decision policies using data-mining algorithms designed to handle nonlinear decision rules, missing attributes, and noisy data. In the current study, we tested the effectiveness of a decision-tree induction algorithm and an instance-based classification method for policy capturing in comparison to the standard linear approach. Decision trees are relevant in naturalistic decision-making contexts since they can be used to represent "fast-and-frugal" judgment heuristics, which are well suited to describe human cognition under time pressure. We examined human classification behavior using a simulated naval air defense task in order to empirically compare the C4.5 decision-tree algorithm, the k-nearest neighbors algorithm, and linear regression on their ability to capture individual decision policies. Results show that C4.5 outperformed the other methods in terms of goodness of fit and cross-validation accuracy. Decision-tree models of individuals’ judgment policies actually classified contacts more accurately than their human counterparts, resulting in a threefold reduction in error rates. We conclude that a decision-tree induction algorithm can yield useful models for training and decision support applications, and we discuss the application of judgmental bootstrapping in real time in dynamic environments.
Trust and trustworthiness judgments have been studied in the context of social, business, and romantic relationships as well as between humans and automation. This article extends the prior research to explore how programmers assess code for trustworthiness when asked to reuse existing computer code. We used cognitive task analysis methods to explore experienced programmers’ first-person perspectives on code reuse. We elicited specific cues and strategies used to assess trustworthiness in real-world scenarios. Using qualitative analysis techniques, we grouped cues into three trustworthiness factors: performance, transparency, and reputation. We also identified environmental factors that influence acceptable levels of trust, including customer needs and requirements, organizational resources and constraints, and consequences of failure. We propose a descriptive model based on these findings. These findings have important implications for organizations that intend to reuse, adapt, and extend code over time. Writing code with the factors such as reputation, transparency, and performance in mind will increase the likelihood that it will be trusted in the near term and be reusable in the future. Furthermore, this research provides an important foundation for future studies exploring trusting behaviors, individual differences, and the ability to detect malicious code.
To improve safety, work systems need to be designed that help humans successfully manage expected and unexpected situations. A resilience-based human factors method called strategies analysis for enhancing resilience (SAfER) has been developed to help practitioners identify ways to create systems that let humans more effectively control the range of different operating situations they may face. The SAfER method covers the identification of (a) critical system priorities that need to be preserved in order to sustain safe operations; (b) the range of decisions, actions, and strategies that humans might use to try to control different operating scenarios; and (c) design changes that help to promote actions that preserve safe operations and prevent or tolerate actions that might result in adverse outcomes. This paper describes the SAfER analysis conducted on an industrial crane-lift incident, and it compares the results from the SAfER analysis with those from a traditional incident investigation process. The findings from the comparison suggest that the SAfER method helps analysts identify and generate additional and potentially useful information on how the system design might be changed to improve crane-lift safety.
Presently, adaptive systems use various cognitive and cardiovascular measures to evaluate the functional state of the operator. One marker that has been largely ignored as an assessment tool is baroreflex sensitivity (BRS). This study examined the extent to which BRS changed in response to acute psychological and physical stressors. A total of 20 participants underwent 6-min exposures to a psychological stressor and a physical stressor. Baroreceptor sensitivity, blood pressure, heart rate, heart rate variability, stroke volume, cardiac output, mean blood pressure, total peripheral resistance, left ventricular ejection time, and pre-ejection period were continuously measured at rest and throughout the testing period. Compared to rest, BRS significantly decreased during both the psychological and physical stressors. BRS was reduced more with the psychological stressor than the physical stressor. Heart rate and systolic blood pressure significantly increased above rest during the psychological stressor but not during the physical stressor. There were no significant differences from rest or between stressors for the other physiological markers. BRS was more robustly responsive than other cardiovascular measures commonly used to assess the psychophysiological response to stress, suggesting BRS is a useful marker for evaluating operator functional state during psychological and physical tasks.
In a production context, safety-related rule violations are often associated with rule breaking and unsafe behavior even if violations are not necessarily malevolent. Nevertheless, our past experiments indicated a range of strategies between violation and rule compliance. Based on this finding, different types of rule-related behavior for managing organizations’ goal conflict between safety and productivity are assumed. To deepen the understanding of rule violations, the behavior of 152 participants in a business simulation was analyzed. Participants operated a plant as a production worker for 36 simulated weeks. In each week, they could choose to comply with safety rules or to violate them in order to maximize their salary. A cluster analysis of the 5,472 decisions made on how to operate the plant included the severity of the violations, the number of times participants changed their rule-related strategy, and the extent of failure/success of these strategies. Five clusters of rule-related behavior were extracted: the compliant but ineffective "executor" (15%), the production-optimizing and behavioral variable "optimizer" (13%), the successful and compliant "well behaved" (36%), the notoriously violating "inconvincible" (29%), and the "experimenter" (7%), who does not succeed with various violating strategies.
Systems that supported operators with higher levels of information-analysis and decision-selection automation had varying effects on human performance and situation awareness. We investigated whether information-processing and working-memory abilities moderate the effects of automation on human performance and situation awareness. To investigate such effects, we measured the information-processing ability and working-memory capacity of 60 participants. We also assessed their performance and situation awareness when they repeatedly controlled simulated air traffic with the support of different levels of information-analysis and decision-selection automation. Results indicated that performance increased, but situation awareness declined, when the levels of automation increased. The participants with better information-processing ability and working-memory capacity scored better in performance and situation awareness. The participants with higher information-processing ability and working-memory capacity profited from the higher levels of automation. In contrast, the participants with lower information-processing ability and working-memory capacity suffered under higher levels of automation. Authors of future research should thus consider individual differences when investigating the effects of automation and focus on identifying mechanisms that ensure that automation supports all operators.
It is a current trend in aviation to use categories of technical (e.g., knowledge) and nontechnical skills (e.g., situation awareness) to assess airline pilots’ performance. Several studies have revealed large disagreement between assessors when airline professionals use these categories to assess the performance of their peers. The aim of the present study is to investigate whether the categories themselves are at the source of disagreement. We explore the reasoning of flight examiners who assess an engine fire scenario in pairs. The results provide insight into the overlap of topics that constitute certain categories. Implications are drawn in regards to the use of assessment categories and their influence on pilot performance assessment.
Complex domains require cognitive work for which current approaches to training may be ill-suited. To improve training for cognitive work, Klein and Baxter have proposed Cognitive Transformation Theory (CTT), a learning theory that characterizes sensemaking processes as essential to the development of expertise. The objectives of this research were to compare CTT with the instructional strategies of two expert air traffic control instructors to evaluate the relevance of CTT’s four teaching practices, propose refinements to CTT, and identify potential instructional strategies to serve as guidance for the application of CTT. Data were collected using cognitive task analysis methods, including course observation, artifact examination, and knowledge elicitation with two instructors and seven of their students. Data were coded using categories derived from theory and patterns emergent within the data. Results suggest that many of the instructional strategies used were consistent with the teaching practices of CTT and that learning was aligned with the active sensemaking claims of CTT. An integrated set of instructional strategies and a few refinements to CTT are advanced to further its application to training in complex domains. Although this set of strategies may benefit current training practices, further research is needed to evaluate their effectiveness.
Experimentation was conducted comparing videotape self-modeling and videotape peer/other modeling to self-directed mental rehearsal (a covert modeling procedure) and a no-training (physical practice) control condition in training a mechanical device assembly task. Eighty male and female college students were introduced to the assembly task in a timed pretest trial and then videotaped performing the assembly task in a second trial. Over the next 4 days, subjects were randomly assigned to training conditions and repeatedly trained and tested in the mechanical device assembly task. The effects of training methods upon assembly times, self-efficacy expectations regarding assembly task performance, and subjective impressions of the nature and usefulness of training were examined. Superior assembly performance over initial training and at 4 months post-training follow-up was observed for the self-directed mental rehearsal training condition and discussed.
Where occupational performance outcomes are difficult to measure, there is a tendency to associate expertise with years of experience and/or previous occupational position. Although useful, these indicators represent composite constructs that embody a number of different variables, only some of which may be strongly associated with the transition to expertise. In identifying an alternative measure of expertise, it is necessary to consider the cognitive processes associated with expert performance, in particular, the role of cue utilization. The present study, conducted in the context of software engineering, was designed to test the relationship between cue utilization and self, peer, and error management indicators of expertise. The results indicated that participants who exhibited relatively higher levels of cue utilization were significantly more likely to self-report engaging in behaviors associated with expert decision making, to be nominated as an expert by their peers, and to demonstrate superior error management when developing solutions to development problems.
The implementation of automation relies on the assumption that automation will reduce the operator’s cognitive demand and improve performance. However, accepted models demonstrate the multidimensionality of cognitive resources, suggesting that automation must support an appropriate resource dimension to have an appreciable effect. To evaluate this theory, the present study examined the impact of various types of automation on an unmanned ground vehicle (UGV) operator’s performance, workload, and stress. The use of a visually demanding task allowed for comparison between an auditory alert (supporting the heavily burdened visual dimension) and a driving aid (supporting action execution, a relatively unburdened cognitive dimension). Static and adaptive (fluctuating based on task demand) levels were implemented for each automation type. Those receiving auditory alerts exhibited better performance and reduced Worry, but also increased Temporal Demand and Effort relative to those receiving driving automation. Adaptive automation reduced workload for those receiving the auditory alerts, and increased workload for those receiving the driving automation. The results from this research demonstrate the need to consider the multidimensionality of the operator’s cognitive resources when implementing automation into a system. System designers should consider the type of automation necessary to support the specific cognitive resources burdened by the task.
In this paper, we describe an integrative approach to understanding flight crew activity. Our approach combines contemporary innovations in cognitive science theory with a new suite of methods for measuring, analyzing, and visualizing the activities of commercial airline flight crews in interaction with the complex automated systems found on the modern flight deck. Our unit of analysis is the multiparty, multimodal activity system. We installed a variety of recording devices in high-fidelity flight simulators to produce rich, multistream time-series data sets. The complexity of such data sets and the need for manual coding of high-level events make large-scale analysis prohibitively expensive. We break through this analysis bottleneck by using our newly developed integrated software system called ChronoViz, which supports visualization and analysis of multiple sources of time-coded data, including multiple sources of high-definition video, simulation data, transcript data, paper notes, and eye gaze data. Four examples of flight crew activity serve to illustrate the methods, the theory, and the kinds of findings that are now possible in the study of flight crew interaction with flight deck automation.
We describe the all-engine-out landing of Air Transat Flight 236 in the Azores Islands (August 24, 2001) and use certain aspects of that accident to motivate a conceptual framework for the organization and display of information in complex human-interactive systems. Four hours into the flight, the aircraft experienced unusual oil indications. Two hours later, a fuel system failure led to a full-blown emergency that was not evident to the crew until it was too late. Although all relevant data to avoid the emergency were available to the aircraft computer systems, the design choices made about what to display and how to display it kept the pilots "in the dark." The framework proposed here consists of six levels, beginning from the extraction of data from physical signals, abstracting from raw data to form visual representations on the user interface, and finally integrating high-level elements and information structures. We illustrate how the framework can be used to analyze some of the shortcomings in current display design, and we discuss some principles of information organization and formal analysis of task logic that might help to improve design. Finally, we sketch a design for a helicopter engine display based on these principles.
The collective taskwork of a team spans the functions required to achieve work goals. Within this context, function allocation is the design decision in which taskwork functions are assigned to all agents in a team, both human and automated. In addition, the allocation of taskwork functions then creates the need for additional teamwork functions to coordinate between agents. In this paper, we identify important requirements for function allocation within teams of human and automated agents. Of note, many important attributes may be observed only within the detailed dynamics of simulation or actual operations, particularly when a function allocation requires tightly coupled interactions. Building on the preceding companion paper’s conceptual review of the requirements of effective function allocation, in this paper we develop a modeling framework that increases the number of aspects of function allocation that can be examined simultaneously through both static analysis and dynamic computational simulations. The taskwork and teamwork of a modern air transport flight deck with a range of function allocations is used as an example throughout, highlighting the range of phenomenon these models can describe. A follow-on companion paper discusses specific metrics of function allocation that can be derived both from such models and from observations in high-fidelity human-in-the-loop simulations or real operations.
Function allocation is the design decision in which work functions are assigned to all agents in a team, both human and automated. Building on the preceding companion papers’ review of the requirements of effective function allocation and discussion of a computational framework for modeling function allocation, in this paper, we develop specific metrics of function allocation that can be derived from such models as well as from observations in high-fidelity human-in-the-loop simulations or real operations. These metrics span eight issues with function allocation: (a) workload, (b) stability of the work environment, (c) mismatches between responsibility and authority, (d) incoherency in function allocations, (e) interruptive automation, (f) automation’s boundary conditions, (g) function allocations limiting human adaptation to context, (h) and mission performance. Some of the metrics measure distinct issues whereas others assess different causes of issues that can manifest in similar ways; collectively, they are intended to be comprehensive in their ability to discriminate for a range of issues. Trade-offs may exist between these metrics, and they need to be examined collectively to identify potential trade-offs or conflicts between them. This paper continues the example given in the preceding companion paper, demonstrating how these metrics of function allocation can be assessed from computational simulations of an air transport flight deck through the descent phase of flight.
In this paper, we identify the requirements for effective function allocation within teams of human and automated agents. These functions include all the activities in the team’s environment required to meet collective work goals, that is, taskwork functions. In addition, the allocation of taskwork functions then creates the need for additional teamwork functions to coordinate between agents. Key requirements include that each agent must be capable of each individual function it is allocated and must be capable of its collective set of functions, including teamwork. Of note, many important attributes may be observed only within the detailed dynamics of simulation or actual operations, particularly when a function allocation requires tightly coupled interactions and when teamwork (including human–automation interaction) may support or detract from effective performance. Finally, we note that function allocation is a key design decision that should be made deliberately. By addressing function allocation early in design, before technologies and interfaces are created, key trade-offs can be considered and fundamental concerns with human factors addressed.
Improving patient safety, within the context of a complex system, forms one of the key challenges in health care today. Cognitive work analysis (CWA) is one way to analyze complex systems, and although it has been applied to health care for 20 years, little is known about its effectiveness or future research needs. This article presents a review of CWA studies in health care, addressing questions of use, usefulness, challenges, and opportunities. Results of the review make clear that the research agenda is largely confined to acute care. Of the 39 articles reviewed, 28 relate to this setting. There appears to be a growing interest in medical informatics, error investigation, and decision support. Conversely, work in physiological monitoring has slowed, associated with the uncertainties of modeling "biological" systems. Studies related to "organic" social systems are similarly challenged, although there is a recognition that important opportunities exist, such as studying work flow processes between teams. Other opportunities relate to new methods to enhance CWA; new technologies, such as auditory displays; and new applications, such as requests for proposals and incident investigation. Ultimately, the capacity to foster an understanding into the deep structures of a system may prove to be the greatest contribution of CWA to health care today.
Human–automation interaction (HAI) takes place in virtually every high-technology domain under a variety of operational conditions. Because operators make HAI decisions such as which mode to use, and when to engage, disengage, monitor, or cross-check automation, it is important to understand their perceptions of how system and situational characteristics affect their interaction with automation. The objective of this study was to examine how systematic variations of automation interface, task and context features influence professional pilots’ judgments of HAI situations. Pilots received descriptions of crews interacting with flight deck automation in specific situations and were asked to rate cognitive demands and predict behaviors. Results reflect a complex interplay among automation features, task, and context. Automation features influenced judgments of workload, task management, and potential for automation-related errors; however, the impact of automation on situation awareness seems to be moderated by task features. Unanticipated tasks had broader effects on pilots’ judgments than operational stressors. Results suggest that although changes to automated systems may be small in technical terms, their cognitive and behavioral impact on operators may be significant. Performance effects of automation changes in aviation as well as other domains need to be addressed with reference to task characteristics and situational demands.