Keywords: aviation safety; risk analysis
A Probabilistic Decision Support System (PDSS) is being developed to assist NASA program managers with evaluating new technologies that are intended to lower the fatal aircraft accident rate. This decision support system utilizes the method of case-based reasoning in which case studies of specific accidents are modeled and analyzed.
One such case study is the crash of Air Ontario Flight 1363 which resulted from a loss of control due to ice contamination on the wings. The decision to take-off without deicing was the end result of a complex interaction of causal factors including crew resource management, worsening weather conditions, and organizational issues.
In this research work, accidents such as Air Ontario 1363 are modeled and the reduction in risk from the insertion of appropriate technologies is assessed. A user-friendly interface is being developed to create a software application that is flexible and easy to use.
There are several existing approaches for explaining accident causation, one of which is the Reason model of accident causation. The Reason model is a general model that traces the root cause(s) of an accident to organizational errors arising in the upper levels of any organization (Reason, 1995). This model is helpful in that it allows explanations of accidents to be based upon more than just individual operator performance. In large-scale technological systems such as those in the aviation industry, each situation or event is a result of multiple factors interacting within the organization. The accident causal chain as explained by the Reason model, sometimes called the Swiss Cheese Model, is depicted in Figure 1.
One of the disadvantages of the Reason model is that it does not account for the detailed interrelationships among causal factors. Without these distinct linkages, the results are too vague to be of significant practical use. It is for this reason that in this research, a combination of approaches and methodologies is used to create a more comprehensive method for evaluating aviation safety system risk.
1.1 Human Factors Analysis and Classification System (HFACS)
Between 70 and 80 percent of aviation accidents can be attributed, at least in part, to human error (Shappell and Wiegmann, 1996). Hence, the taxonomy described by the Human Factors Analysis and Classification System (HFACS) is used as a framework for the modeling and analysis of accidents.
HFACS uses Reason's model as a basis for describing the "holes" in the four levels of human failure, which are both individual and organizational in nature. The four levels are unsafe acts, preconditions for unsafe acts, unsafe supervision, and organizational influences. HFACS divides each level into a series of causal factors (Figure 2).
For instance, unsafe acts are broken into two causal categories: errors (light yellow) and violations (darker yellow). Each of these categories, in turn, is subdivided. In particular, within the "error" category, the scheme distinguishes decision errors, skill-based errors, and perceptual errors; while "violations" are described either as routine or exceptional.
A complete list of the individual causal factors in HFACS and an explanation of each can be found in Shappell and Wiegmann (2000).
1.2 Aviation Safety Program (AvSP) Technologies
The safety of flying aboard U.S. airlines has significantly improved over the past century. However, the aviation industry continues to grow and the cost of aviation accidents has risen steadily in lives and dollars (Shappell and Wiegmann, 2000). The high costs associated with even a single accident place great pressure on the National Airspace System (NAS) to reduce the number of accidents even further from the current level.
It is for this reason that the Aviation Safety Program (AvSP) Office at NASA is actively involved in the development of new technologies to reduce the fatal aircraft accident rate by 80% within ten years (NASA Product Dictionary, 2002). These technologies include advanced simulation training modules, improved vision systems, and enhanced equipment for turbulence detection during adverse weather conditions. NASA's AvSP Technologies can be grouped into three main categories as shown in Figure 3.
Unfortunately, many of these new technologies are so expensive that under current budget constraints, it would not be possible to install and implement each of them in every airline. In order for NASA to achieve its goal of an 80% reduction in the fatal aircraft accident rate, it is crucial to target the technologies (or combinations of technologies) that will have the greatest reduction in overall system risk.
2.0 Research Objective
Each technology has a target goal associated with it that is stated in terms of technical success such as an improvement in crash survivability or a percentage reduction in the number of weather-related accidents. Even though each new technology has an associated goal, these goals are estimations for individual technologies that in most cases do not indicate system risk reduction. In other words, the goals are not easily comparable since they are technology-specific and are not measured according to an overall reduction in the number of accidents. In addition, they do not provide an estimated reduction in system risk for combinations of technology insertions. These measurements are necessary for NASA to compare technologies and to determine which technologies to target for initial implementation.
The objective of this research project is to provide NASA with a Probabilistic Decision Support System (PDSS) to evaluate the projected impact upon system risk reduction of new technology insertions/interventions.
2.1 Analytical Modeling Approach
Our analytic modeling approach consists of the following stages:
We illustrate the use of the approach by applying it to the crash of Air Ontario Flight 1363.
3.0 Air Ontario 1363 Case Study
3.1 Accident Summary
The following summary is based on the description and analyses presented in Beyond Aviation Human Factors: Safety in High Technology Systems by Maurino et al. (1995).
On Friday, March 10, 1989, Captain George C. Morwood advanced the throttles of Air Ontario Flight 1363, initiating take- off from Dryden Airport, Ontario, Canada. Flight 1363 was the second part of the day's flying schedule which consisted of a Winnipeg to Thunder Bay round trip, with intermediate stops at Dryden.
Captain Morwood reviewed the operational status of the aircraft before departing Winnipeg and verified, among other maintenance deferred defects, that the Auxiliary Power Unit (APU) was unserviceable. This defect meant that either the engines had to be started from an external power unit or one engine had to be kept running to cross-start the other engine. If both engines were shut down at a station where no external power unit was available, the aircraft would be grounded until the APU was fixed or an external power unit was made available. There was no external power source at Dryden, and therefore, one engine would have to be kept running. The manufacturer, Fokker, and Air Ontario strictly prohibited de-icing with either engine running.
Because the original flight release from Thunder Bay to Dryden prepared by the Air Ontario Systems Operations Control (SOC) had not been updated, ten passengers were added to Flight 1363 after it had been refueled. Captain Morwood suggested off-loading the ten passengers and their baggage since the aircraft was overweight for take-off. However, the Air Ontario SOC duty manager overrode the Captain's decision and decided to reduce the weight by off-loading fuel. The de-fueling caused an additional 35-minute delay in the departure of Flight 1363 from Thunder Bay. Flight 1363 departed Thunder Bay and arrived in Dryden one hour behind schedule.
The hot refueling process started with passengers on board, which is considered to be an unsafe practice. This is unusual given that Captain Morwood was a conservative decision-maker and strictly adhered to rules and regulations. It was indicated by the Commission of Inquiry that Captain Morwood had a heated conversation with the SOC regarding the passenger load and weather conditions in Winnipeg prior to the departure from Dryden. It was visible that Captain Morwood became increasingly irritated while in the terminal, after his telephone contact with the SOC, and that as he briskly walked back to the aircraft, he showed signs of frustration.
When he returned to the aircraft, Captain Morwood asked the ground handler whether de-icing was available. He was told that it was, but he still did not request de-icing. Snow was falling heavily as the aircraft was about to leave the terminal, and the wings were covered in snow to depths varying from one-eighth to one-quarter of an inch.
The combination of the slush on the ground and the wet snow, which had frozen into ice on the wings, hindered the performance capabilities of the aircraft. After a longer than normal take-off roll, the aircraft rotated, lifted off slightly, began to shudder, and settled back onto the runway. It rotated again, lifting off at the 5,700 ft point of the 6,000 ft runway. It flew briefly, and cleared the end of the runway approximately 15 feet above the ground. It failed to gain altitude and crashed, coming to rest about one kilometer from the end of the runway.
This accident and a preliminary model were presented in "An Aviation System Risk Model (ASRM) Case Study: Air Ontario 1363" by Luxhøj and Maurino (2001). Since then, enhancements have been made to follow the HFACS taxonomy as mentioned previously and to reflect advice gathered from subject matter experts.
3.2 Influence Diagram For Air Ontario 1363
The following model (see Figure 5) depicts the causal factors and the interactions among them. A detailed description of these factors and interrelationships are described subsequently.
3.3 Node Explanations
3.4 Causal Factor Interactions
The interactions among the causal factors of this accident are depicted through the links among the nodes in Figure 1. This section explains the reasons for making the links.
Spending cuts imposed by the Canadian government meant that the number of people available and qualified to carry out both their original and new responsibilities acquired as a result of the merger was very much reduced. As a result, workloads increased.
The report discusses Air Canada's lack of support to Air Ontario during the introduction of the jet service, and compares standards in specific areas such as operational policies for dispatch with an unserviceable APU; minimum equipment lists; manuals; aircraft defects, hot refueling policies; de-icing policies, etc. The comparisons show that Air Canada and Air Ontario had differing safety standards.
After the merger, Air Ontario staff members were not provided with sufficient levels of training and resources. Some of the appointments made at Air Ontario, which included naming the president's close relatives to key managerial positions, were the subject of considerable discussion at the Air Ontario committee board meetings. Some of those newly appointed Air Ontario managers were confronted by demands for which their experience may not have been adequate.
The organization's tense atmosphere and poor delegation of authority allowed willful disregard of rules and regulations by supervisors. For example, the F-28 project manager had the responsibility to ensure that the implementation and operation of the F-28 program was properly monitored and supervised. The appointed manager, a relative of the president of the company, lacked experience in the F-28 program.
Inadequate standard operating procedures and factors, such as operational tempo and time pressures, adversely affected the mental states of the employees.
The report includes the contention that the working relationships among the pilots over the previous two days had probably not been cordial, with the consequent stress impact on the crew. Following the merger, there was a bitter strike that left residual feelings of ill will among pilot groups.
Lack of formal, written operating procedures resulted in supervisory inaction with regard to maintenance problems and frequent failures to de-ice.
CRM training aims at developing skills for the optimum utilization of the resources available to the captain as well as to other crewmembers. Air Ontario did not provide CRM training to its crew. Due to lack of training, the dispatcher failed to inform the rest of the crew of the freezing rain.
The weather forecast called for freezing rain at Dryden during the time span of operation of Flight 1363. This forecast was available to the Air Ontario SOC while Flight 1363 was still on the ground at Thunder Bay. This information, which could have induced Capt. Morwood to overfly Dryden, was never transmitted to the pilots due to lack of dispatcher training.
Some F-28 pilots used the Piedmont F-28 Operations Manual while others used the US Air F-28 Pilot's Handbook because Air Ontario did not have its own standardized F-28 operations manual. This lack created problems on the flight deck.
Air Ontario personnel often deferred maintenance 'snags' and flew the aircraft without fixing them. This practice was the result of inadequate supervision.
Failures to de-ice and correct known maintenance problems are routine violations.
Deficiencies in crew pairing resulted in miscommunication among the crew.
Failure to enforce rules and regulations and the actions of an unqualified dispatcher with respect to maintenance defects led to routine violations.
Stress due to numerous factors degrades the ability of humans to process information. Captain Morwood's final decision to take off was affected by events that caused high levels of stress.
A lack of situational awareness led to the pilots' disbelief in the need for de-icing.
The chief pilot's overconfidence led to ignoring maintenance defects on several occasions and to underestimate the need for de-icing.
Stress due to the merger and misplaced motivation (acting to avoid flight delays rather than to ensure safety) resulted in miscommunication among the crew.
Lack of information due to miscommunication and lack of training resulted in procedural errors.
Poor decision to take off with contaminated wings resulted in a loss of control.
The pilot's visual inspection from the cockpit led them to believe that there was no ice accumulation on the wings. This misperception led to the poor decision of taking off.
Several instances of pilots' failures to de-ice (e.g. 80-knot test) eventually resulted in LOC.
3.5 ASRM with AvSP Technology Insertions/Interventions
In addition to the enhancements made from the original Air Ontario 1363 model presented by Luxhøj and Maurino (2001), another model has been built to assess the possible risk-reducing effects of certain AvSP technologies. Upon discussions with subject matter experts and on our knowledge of the 41 proposed technologies, several AvSP technologies have been found to be appropriate for insertion into the model of Air Ontario Flight 1363. Figure 6 depicts the ASRM with relevant technology elements.
The following is a list of the technologies shown in the above influence diagram and the reasoning associated with each insertion. The definitions and descriptions in this section are based on the NASA Product Dictionary (2002).
This technology element involves upgrading the 24-year-old technology of the Aviation Safety Reporting System (ASRS) database to include: conversion of the ASRS legacy database to ORACLE, electronic submission of reports, and testing and evaluation of an analyst decision support system.
With the insertion of this technology, improvement in the ASRS report production and research processes is expected. Hence, this element is considered to have a potential impact on the following nodes: 'Organizational Processes', 'Decision Errors', and 'Transport Canada'.
This element aims at the permanent field implementation of a National Aviation Operational Monitoring Service (NAOMS) responsible for developing and maintaining a comprehensive and coherent survey of the safety and performance of the NAS from the perspective of front line personnel NAS wide.
The objective of this element is to create a mechanism for measuring the overall safety of the NAS in a quantitative, precise, and repeatable way on an ongoing basis. Thus, insertion of this technology in the 'Transport Canada' and 'Organizational Processes' nodes is considered to have a potential impact.
APMS is an integrated suite of tools to facilitate the implementation of routine flight-data analyses within each of the air-service providers. APMS develops and documents the software and procedures for data management and analyses of Flight Operational Quality Assurance data that enable users easily to interpret implications in safety and efficiency of flight.
Similar to ASMM-2, this technology aims to implement proactive management by monitoring and analyzing large flight- recorded databases on a continuous basis. The objective is to develop tools that convert data into information, merge databases, and create visualization capabilities by extending APMS to new applications. Therefore, insertion of this element in the 'Organizational Processes' and 'Transport Canada' nodes is expected to have an impact.
This technology element provides the capability to collect and process Air Traffic Control (ATC) operational data; compute quantitative operational performance measures on a regular basis relating to system safety, delay, flexibility, predictability and user accessibility; conduct causal analyses and operational problem identification and analyses; archive performance statistics and basic operational data for use in research, development and planning studies.
The objective here is to monitor performance metrics continuously to enable the implementation of a policy of proactive NAS management. Extending APMS concepts and approaching to the ATC environment is proposed. Iterative evaluation by ATC users is expected to improve the measurement and tool requirements. Thus, insertion of ASMM-4 would have a potential impact upon the 'Transport Canada' node.
AM-1 involves the development of advanced analysis modules such as Finite Element Modeling (FEM) elements in a functionally integrated suite of dynamic crash analysis software. The purpose of this software is to make the design process more efficient, integrated and standardized and allow certification of crashworthiness as a system rather than by individual component testing.
Since AM-1 concerns crash analysis codes, it is considered to be relevant to the 'LOC' node in the influence diagram.
AM-2 deals with development of designs for improved energy absorbing seats, restraints, and energy absorbing aircraft structures to improve survivability in accidents.
The objective in developing this element is to improve structural designs that help reduce crash loads and to develop human energy criteria for use in system design and ways to interpolate crash criteria data for a specific aircraft. Insertion of this technology element into the LOC node is expected to have an impact on the relative risk.
AM-4 involves the development of a handbook containing test results of materials tested and design guidelines for their use.
The objective of this element is to develop a Transport Crash Design Guide to target issues specific to the transport industry and to reduce the difficulty of validating crashworthiness. Similarly, AM-4 is considered to have a potential impact on the LOC node.
WxAP-1 concerns the development of substantiated aviation weather information system guidelines for flight deck user interface and for operational use.
The objective is to develop weather presentations that are easy for the users to understand, interpret, and act on. This technology is considered to have a potential impact on the 'Decision Errors' node.
WxAP-3 involves the development of datalink system and architecture guidelines supporting the transfer of weather information from ground-to-air. The targeted problem is the poor dissemination of weather information to the flight deck. This technology element is considered to have an impact on the 'Decision Errors' node.
3.6 Scenario Analysis
Combinations of new technologies could have a greater effect on risk reduction than the insertion of a single technology. Hence, it is important to evaluate several scenarios, each consisting of a risk assessment for different technologies and groups of technologies.
First , the risk is evaluated for the Baseline Scenario, in which there are no technology insertions. Then the risk is computed again, but for the opposite scenario, which involves the insertion of all relevant technologies and for others with the new technologies introduced singly. Additional calculations for each scenario with technology insertions include Absolute Percentage Risk Decrease and Relative Percentage Risk Decrease. Absolute % Risk Decrease is calculated as the difference in Risk Intensity Level from the baseline scenario, and Relative % Risk Decrease is calculated as (Absolute % Risk Decrease)/(Risk Intensity Level of baseline). The following is a summary of scenarios that have been generated based on the Air Ontario Flight 1363 case.
3.7 Software Interface
It is anticipated that once development of the ASRM software has been completed, it could have a wide user group. In addition to experienced NASA technicians, the software, with customizations, could be marketable to airline personnel.
In order to appeal to a larger market, however, the software should be flexible and easy to use. Hence, a user-friendly interface is being developed using the Visual Basic programming language. The software supplies the user with a multitude of results after the click of just a few buttons.
The interface will provide the user with the option of assessing risk for major categories of accidents including Maintenance, Loss of Control (LOC), and Controlled Flight into Terrain (CFIT), for example. Each of these categories will in turn have several cases built in, each of which is based on a past accident and its associated NTSB report. Figure 8 shows a sample display from which the user can choose a case to evaluate.
Once the user has chosen a case to evaluate, he or she can then begin to generate scenarios by checking combinations of technologies and compiling. The form will show Risk Intensity Level of the generated scenario, Absolute % Risk Decrease, and Relative % Risk Decrease as discussed previously. This form with a sample scenario is shown in Figure 9:
From this form, the user will also be able to view an Excel spreadsheet summary that captures the results from each scenario generated.
The analytical approach used in this research is a systematic method for modeling aircraft accidents and assessing risk reduction. It uses but is not confined to the HFACS taxonomy as a baseline framework to depict causal factors, and it combines statistical data with judgment of subject matter experts to obtain reasonable estimates of probabilities.
The Air Ontario 1363 case study demonstrates the importance of considering not only errors made by pilots and the cabin crew but also situational (e.g. weather), supervisory, and organizational factors.
Future work for this research involves the model-building, analysis, and probability elicitation of case studies in other accident categories such as Controlled Flight into Terrain (CFIT) and Runway Incursion.
Many parts of this research are done in collaboration among research team members. For help with research work discussed in this paper, the authors acknowledge the following Industrial and Systems Engineering graduate students: Muhammad Jalil, Erim Kardes, Ram Kuturu, and Ahmet Oztekin.
Maurino, D.E., J. Reason, N. Johnston and R.B. Lee (1995). Beyond Aviation Human Factors: Safety in High Technology Systems, Ashgate Publishing Limited, United Kingdom.
Luxhøj, James T. and Michele Maurino, "An Aviation System Risk Model (ASRM) Case Study: Air Ontario 1363," The Rutgers Scholar, Vol. 1 (2001), http://rutgersscholar.rutgers.edu.
NASA Product Dictionary (2002), February.
Reason, J. (1995). "A System Approach to Organizational Error," Ergonomics, 38, 1708-1721.
Shappell, S., and D. Wiegmann (1996). "U.S. Naval Aviation Mishaps 1977-92: Differences between single- and dual-piloted aircraft," Aviation, Space, and Environmental Medicine, 65-69.
Shappell, S., and D. Wiegmann (2000). "The Human Factors Analysis and Classification System (HFACS)," Office of Aviation Medicine, Federal Aviation Administration, 1-14.
Copyright 2003 by James T. Luxhøj.
Current URL: http://rutgersscholar.rutgers.edu/volume05/luxhoj-kauffeld/luxhoj-kauffeld