The Rutgers Scholar, electronic journal, Undergraduate Education, undergraduate research
The Rutgers Scholar, electronic journal, Undergraduate Education, undergraduate research

Analysis of a group decision support system (GDSS)
for aviation safety risk evaluation

Michele A. Maurino* and James T. Luxhøj
Department of Industrial and Systems Engineering,
Rutgers University, New Brunswick New Jersey 08901

*Rutgers Undergraduate Research Fellow

Keywords:


Abstract

Aviation is a highly dynamic, complex domain. The evaluation of aviation risk typically involves a group of decision makers assessing tradeoffs among competing attributes, such as safety, business concerns, and risk. This paper presents a Group Decision Support System (GDSS) that uses the method of the Analytic Hierarchy Process (AHP) to structure and evaluate a complex decision problem. The underlying method and GDSS are demonstrated with a hypothetical example involving the assessment of risks associated with aircraft repair facilities.


Introduction


When making decisions in a group, for example in an executive environment, complexities arise in assigning the proper weights to multiple criteria and in considering the different viewpoints of decision-makers. In situations where due regard for individual beliefs may be critical, decision makers need systems that integrate communication methods for the analysis of the questions at hand. The current methods for the analysis of these questions may be biased due to the fact that the "alternatives that receive very high scores from one expert could be the most favored alternatives, regardless of the opinions from the rest in the group" (Ammarapala and Luxhøj, 2002, p.1).

To address the aforementioned problems, the Federal Aviation Administration (FAA) has sought to develop a new method to aid in the evaluation of group decisions. One proposed method, known as the ClusterGroup technique, uses various clustering algorithms to aggregate similar opinions of groups of experts into "majority" and "minority" clusters. Ammarapala and Luxhoj, (2002, p.1) have recently compared the new ClusterGroup technique to more established methods, such as the Analytic Hierarchy Process (AHP) and the Team Expert Choice software.

The new ClusterGroup technique may help the FAA to efficiently identify the relative importance of aviation safety risk factors. This new decision-making process allows the FAA to gain an understanding of the rationale behind decisions made in situations involving risk. At this point, the FAA has chosen to direct its limited resources toward identifying the major risk associated areas, in the hope of learning from previous experience and preventing future incidents (Ammarapala and Luxhøj, 2002, p. 2).


Purpose


For the research involved in this paper, an understanding of multi-attribute decision making, as well as of the AHP, is needed. With this knowledge, the software program Team Expert Choice can be used to evaluate decisions and expert opinions on aviation safety issues. Once information is gathered on the current methods of group decision making, a step-by-step comparison to the proposed ClusterGroup technique can be made.

The remaining sections of this paper provide the background information about current methods and processes, and provide an example of group decision-making using the software program Team Expert Choice. Comparison of the group-decision making techniques, as well as the detailed analysis of the ClusterGroup method, is a process remaining for future FAA/Rutgers research assistants.


Multi-attribute decision making


When making decisions, several criteria are often involved. These criteria include tangibles, which can be measured by costs and profits, as well as intangibles, such as safety and customer service, which cannot be directly measured by costs and profits. Even though there may be readily definable objectives for making the decision, the attributes or criteria of the decision are not always clearly defined. Therefore, the goal of the analyst involved in multi-attribute decision-making should be to use reasonable methods to evaluate both the objectives and the criteria invoked in concluding that a particular course of action is likely to attain those objectives (Canada, Sullivan, and White, 1996, p. 465).

One of the key elements to the multi-attribute decision making process is the choice of the attributes involved. Each of the attributes is used to distinguish between at least two alternatives and involves a unique aspect of the problem at hand. Judgment is required to determine the correct number of attributes for the evaluation. The selection of these attributes is a result of a group consensus and is heavily influenced by the decision problem being considered in the process (Canada, Sullivan, and White, 1996, p. 467). For example, if the objective is career choice satisfaction, some attributes may be money, job security, family life, and work environment.

After the alternatives and attributes are chosen, an appropriate scale of measurement needs to be developed. The scale can be based on monetary values, on ratings ranging from "poor" to "excellent," or on many other measures. The scales chosen allow the various states of the attributes to be represented in the decision-making process (Canada, Sullivan, and White, 1996, p. 467). For example, for the attribute "money", the state "dollars" would be appropriate while for an attribute "noise pollution" the state "decibels" would me appropriate.

The next stage involved in multi-attribute decision-making is the selection of the best analysis technique. There are several techniques to choose from including:

  1. alternative-attributes score card
  2. ordinal scaling
  3. weighted evaluation of alternatives
  4. Brown-Gibson Model
  5. Analytic Hierarchy Process (AHP)
One of the methods in use, the Brown-Gibson Model, is a weighted evaluation developed in 1972 by P. Brown and D. Gibson. The model integrates both objective and subjective measures for decision risk factors and enables the decision-makers to become more aware of important criteria. As compared to the other methods, the Brown Gibson approach leads to a more realistic decision model (Canada, Sullivan, and White, 1996, p. 477). Although the Brown-Gibson Model is a popular method used in the multi-attribute decision making process and the subject of a previous technical report (Ammarapala and Luxhoj, 2000), the main focus of this research is the use of the Analytic Hierarchy Process for decision making.

Analytic hierarchy process (AHP)

The Analytic Hierarchy Process method was developed by T. Saaty in the 1970s and is widely used in a variety of areas, including transportation planning and corporate planning. The AHP is a method that structures "a complex, multiperson, multiattribute, and multiperiod problem hierarchically" (Canada, Sullivan, and White, 1996, p. 483).

The first stage of the AHP is to break down the decision problem into a hierarchy of decision elements. Figure 1 shows the standard form of the AHP hierarchy.

Figure 1

The objective is the top level, followed by the attributes considered important. Further dividing the attributes into sub-attributes creates the remaining levels, and finally, the alternatives are the bottom level.

After the hierarchy is established, weights must be assigned to each set of elements at the various levels. Pairwise comparisons of the attributes are made using various types of scales, indicating strength. The evaluations determine which element dominates the others with respect to the higher-level elements and all results are weighted and placed in a matrix (Canada, Sullivan, and White, 1996, p. 488).

The next step in the AHP is to "determine the priorities of each of the alternatives with respect to each of the attributes to which they relate in the next higher level" (Canada, Sullivan, And, 1996, p. 489). Once again, pairwise comparisons can be used in this stage. However, performance data can also be used to prioritize the alternatives with respect to attributes. The main advantage of this approach is its objectivity. Since the performance measure method assumes that a linear relationship exists between a performance value and its relative weight, the pairwise comparison method should be used when this linear relationship cannot be assumed for a given attribute (Canada, Sullivan, and White, 1996, p. 492). When using the pairwise comparison method, the relative nine-point scale is appropriate.

Once the judgments, made by different decision makes for the same issue, are all entered the consistency of the judgments can be measured. The ability of AHP to test for both local and global consistency is one of the method's greatest strengths. Consistency is based on the notion of cardinal transitivity. For example, if A is judged to be 2 times more important than B, and B is judged to be 3 times more important than C, then perfect cardinal consistency would imply that A be judged 6 times more important than C. If intransitivity is found based on the AHP consistency ratio metric, the decision-makers are made aware of the issue and can reevaluate the judgments made. Those involved in the decision making process may need to review the "hierarchical formulation" of the problem if the inconsistencies cannot be found or resolved (Canada, Sullivan, and White, 1996, p. 496, 498). At this point, all steps involved in multi-attribute decision-making using the AHP are complete.


The Expert choice software


Expert Choice (EC) is one of the software packages available that incorporates the ideas and methodology of the AHP. It is "designed for the analysis, synthesis, and justification of complex decisions and evaluations" and can be used in either individual or group situations (Quick Start Guide and Tutorials, 2001, p. 6). All necessary steps involved in the AHP are captured in the Expert Choice software. It includes the processes discussed in previous sections, as well as a brainstorming session, which can help determine objectives and alternatives.

Team Expert Choice, which allows decisions to be generated in a group setting, is another feature of the software. A group model allows individual judgments or expert opinions to be combined and evaluated, producing a group decision in the end. Team Expert Choice includes the same steps and features as Expert Choice.

One important application, which is discussed in the remaining sections, is the software's use in the evaluation of aviation safety. Given case studies and expert opinions, important aviation risk factors can be identified, possibly preventing future incidents and/or accidents.


Hypothetical example


The purpose of this section is to illustrate a hypothetical situation in which group decision-making tools or methods are useful. Following the example, the data analysis using the Team Expert Choice software is discussed.

In aviation safety, it is important to identify key internal areas of risk potential to repair stations. For example, areas of risk that may affect the overall functioning of repair stations, including corporate decisions and other organizational factors. At this point, a goal may be to formulate performance measures that reflect attention to the major areas of risk. The risk factors can be grouped into four categories: reliability, analysis, level of involvement, and surveillance. One main goal is to maximize flight safety, with decisions to be made regarding the allocation of limited resources. For Team Expert Choice evaluations, the categories above are considered the attributes. Each of these attributes has individual factors, sub-attributes, which are defined in the next sections.

Reliability

Reliability is defined as the overall dependability or solidity of a company and folds in many factors that may or may not have an obvious connection to the performance of repair stations. In AHP, a risk value to the repair station is assigned based on each change in overall reliability that the organization faces. For purposes of the hypothetical example, changes that are considered for the reliability factor include the following:

  • Change in ratings - reveals potential risk since a shift in operations of repair station may occur
  • Change in key personnel - change in management style may affect the business processes
  • Number of legal actions - may require a change in business processes to accommodate the financial impact of the legal actions
  • Change in certified mechanics/non-certified mechanics/repairmen - an increase or decrease in technical personnel has potential risk due to new training or less help
  • Number of years the repair station is certified by the FAA - indicates the stability or solidity of an organization
Analysis/investigation

At the organizational level the analysis attribute is defined as the investigations or inquiries performed. For the hypothetical example of risk to repair stations, "analysis" involves keeping track of the following information:

  • Certification revocation
  • Certification suspension
  • Fines greater than or equal to $10,000
  • Other enforcement investigative reports - warning letters, other fines
  • Parts Reporting System Measure - involves multiple calculations based on factors and level of seriousness
Level of involvement

At the level of the repair station, this measure reflects the number and the difficulty of the tasks that the repair station must carry out. It includes the number of major air carriers the organization provides service to, as well as the level of seriousness of the work performed. For purposes of the hypothetical example, the individual factors can be defined as the following:

  • Number of carriers - the number of major airlines with which a repair station is contracted (the level of involvement increases with the number of airlines)
  • Special Federal Aviation regulation - the ability to approve major repairs on a product (this authorization increases the level of involvement)
  • Joint Aviation Authorities-Listed - the organization complies with U.S. and European regulations, increasing the complexity/level of involvement
Surveillance

The surveillance attribute monitors the response of the organization to surveillance and reporting activities performed by the FAA. For the hypothetical example, the characteristics of this attribute are as follows:

  • 2 Year Program and Tracking Reporting System Unfavorable - risk factor based on the ratio of unfavorable results
With all four of the attributes and the individual risk factors defined, the Team Expert Choice model is constructed. The steps involved in this process are explained in the next section.


Data analysis - team expert choice


The previously mentioned attributes and risk factors associated with each can be converted into a Team Expert Choice model. To do this, data must be obtained from experts. In the hypothetical example, three confidential expert opinions are used. Table 1 is an example of expert data.

In Table 1 the numerical entries of the table indicate the relative importance that the expert assigns to any two criteria: a value of 0.200 indicates significantly less importance, 0.333, slightly less importance, 1.00 equal importance, 3.00 slightly more importance, and 5.00 significantly more importance. Thus, in the example given, the expert assigns slightly less importance to analysis than to reliability.

Table 1. Illustrative expert data


Reliability Analysis Level of involvement Surveillance
Reliability 1.000 0.333 1.000 1.000
Analysis 3.000 1.000 1.000 3.000
Level of involvement 1.000 1.000 1.000 1.000
Surveillance 1.000 0.333 1.000 1.000
Total 6.000 2.667 4.000 6.000


From the goal, attributes, and risk factors, the model is developed. The goal is to determine the weights of the individual attributes: reliability, analysis, level of involvement, and surveillance. Once the model is complete, pairwise comparisons are made between the attributes and the factors, as illustrated in Figure 2.

Figure 2. Team Expert Choice Model


Figure 2


The data obtained from experts must be properly converted before the numbers are input into Team Expert Choice as illustrated in Table 2.

Table 2. Conversion of Data


Number from Expert Number for TEC
0.200 -9
0.333 -5
1 1
3 5
5 9


The negative values indicate that the latter attribute is preferred to the first attribute. For example, based on the data conversion used for this example, a -5 is input if an expert believes surveillance is .333 times as important as analysis. Team Expert Choice indicates that this comparison has been made by colorizing the number line, as shown in Figure 3. On the table, a -5 is indicated by selecting the "blue" 5, emphasizing the fact that analysis is five times more important than reliability. (For expert 1, the "blue" 5 is shaded gray).

Figure 3. An Example of Pairwise Comparisons


Figure 3

The last stage in the Team Expert Choice evaluation is obtaining the weights for the individual attributes. This information is calculated once all pairwise comparisons are made, and for the hypothetical example, the weights are shown in Figure 4.

Figure 4 Team Expert Choice Weights for the Attributes


Figure 4


For the hypothetical example, the three experts evaluated the attribute of "analysis" with its associated risk factors to have the highest priority given the set of four attributes. The ratings of the three experts in this case are "averaged" since all experts are considered of equal weight, although different weights may be considered.


Discussion


The previous example is a hypothetical scenario used to show the application of Team Expert Choice to aviation safety issues. Ongoing research is being used to develop an improved method, the ClusterGroup technique, for group-decision making. Remaining stages to this research focus on the comparison of the current methods, such as AHP and the Team Expert Choice software, to this ClusterGroup technique.


Acknowledgment


The authors would like to acknowledge the support of the Federal Aviation Administration through grant number 00-G-006. This paper is based on research performed at Rutgers University. The contents of this paper reflect the views of the authors who are solely responsible for the accuracy of the facts, analyses, conclusions, and recommendations presented herein, and do not necessarily reflect the official views or policy of the Federal Aviation Administration.


References


Advanced Decision Support Software - Quick Start Guide and Tutorials (2001), Expert Choice Inc., Pennsylvania.

Ammarapala, Veeris and James T. Luxhøj (2000), "A Review of the Brown-Gibson Model for Multiattribute Decision Making," FAA Technical Report, July.

Ammarapala, Veeris and James T. Luxhøj (2002), "A Clustering Approach for Multiattribute Aviation Risk Evaluation," Proceedings of the 2002 International Conference on Industry, Engineering, and Management Systems, Cocoa Beach, Florida, March 11-13.

Canada, J. R., Sullivan W.G., White J. A. (1996), Capital Investment Analysis for Engineering and Management, 2nd edition, Prentice Hall, New Jersey.



Copyright 2002 by James T. Luxhøj
Current URL: http://rutgersscholar.rutgers.edu/volume04/maurluxh/maurluxh.htm