The Decision Making Trial and Evaluation Laboratory ( Dematel ) and Analytic Network Process ( ANP ) for Safety Management System Evaluation Performance

In order to improve airport safety management system (SMS) performance, an evaluation system is required to improve on current shortcomings and maximize safety. This study suggests the integration of the DEMATEL and ANP methods in decision making processes by analyzing causal relations between the relevant criteria and taking effective analysis-based decision. The DEMATEL method builds on the ANP method in identifying the interdependencies between criteria. The input data consists of questionnaire data obtained online and then stored in an online database. Furthermore, the questionnaire data is processed using DEMATEL and ANP methods to obtain the results of determining the relationship between criteria and criteria that need to be evaluated. The study cases on this evaluation system were Adi Sutjipto International Airport, Yogyakarta (JOG); Ahmad Yani International Airport, Semarang (SRG); and Adi Sumarmo International Airport, Surakarta (SOC). The integration grades SMS performance criterion weights in a descending order as follow: safety and destination policy, safety risk management, healthcare, and safety awareness. Sturges' formula classified the results into nine grades. JOG and SMG airports were in grade 8, while SOG airport was in grade 7.


Introduction
Evaluation systems have a vital role in assessing and improving quality service by certain criteria.This allows an organization to take decision or to formulate a policy that control, monitor, and improve the quality of service in the future.The Decision Making Trial and Evaluation Laboratory (DEMATEL) and Analytic Network Process (ANP) methods allow the modeling of the problems in a evaluation system.They also excel at measuring the interdependencies among criteria in a hierarchy or network.The results can be more objective than those of the other conventional Multiple Criteria Decision Making (MCDM) methods which assume independence among criteria.The DEMATEL and ANP methods in an evaluation system enable various trouble-shooting, such as supplier performance, learning materials, information security risks, company performance, etc [1].The results of the criteria analyses and grading using these methods show that the graded qualities of each criterion affect the result of an evaluation.
ANP grading presents the interdependencies among criteria and sub-criteria, giving a more general framework for decision making without assuming independency among elements [2].The ANP method, on the other hand, is unable to identify interdependencies among individual criteria; to do so requires in-depth discussion and expert opinions.The ANP method needs an additional instrument to present such interdependencies among criteria, such as the DEMATEL method.Additionally, the DEMATEL method makes apparent the interactions in a decision model, grades the weights of criteria, supports ANP decision making [1], and finds and analyses the dominant criterion in a system [3].
The implementation of DEMATEL and ANP aims at improving decision making processes by taking into account interdependencies among criteria [4].The DEMATEL method delineates the relations between various perspectives and helps the understanding of complex performance-related problems.It is these relations among the criteria acquired by the DEMATEL method that will be used to grade each criterion.The grading itself employs the ANP method that more effective results can be expected.The ANP method utilizes pair-wise comparison and offers a more accurate quantitative model of decision [5].
A performance evaluation system integrating the DEMATEL and ANP methods can be applied in aviation security management to minimize the loss of life in a situation, aviation property damage, and license revocation; to improve efficiency in capital outflow; and to reduce human and environmental externalities.The Safety Management System (SMS) is a systematic framework for safety management: it integrates performance-based approach to the regulation-based approach.Airport safety analysis is crucial in complying with the certification of aviation standards [6].
The airport SMS can help improve on existing shortcomings of an airport or provide quality and safe aviation training [7].The aviation evaluation system has been used in airport safety evaluation by analyzing relevant factors using the Fuzzy method [8], as well as in a study on airport SMS evaluation using the ANP and Fuzzy TOPSIS methods [9].The study highlights the need for an evaluation system in enhancing safety and in maintaining quality service to the highest aviation standards.
The current study will integrate the DEMATEL and ANP methods in evaluating airport SMS in its efforts to improve airport safety quality based on relevant factors.The result will grade the SMS-es in three Indonesian airports, namely Adi Sutjipto International Airport, Yogyakarta (JOG); Ahmad Yani International Airport, Semarang (SMG); and Adi Sumarmo International Airport, Surakarta (SOC).The grades rank the airport SMS-es that will motivate further improvements.

Safety management system
According to the International Civil Aviation Organization, SMS is an organized approach for safety management, which covers structured organization, accountability, policy, and procedures.SMS as a management instrument makes use of proactive components on safety performance, with a focus on the process and the identification of the risks of reducing the components [10].The Federal Aviation Administration, on the other hand, regards SMS as a formal, top-down business-like approach to risk management [11].
In Indonesia, safety management system is regulated in Act 1, 2009 on aviation and in Transportation Ministerial Regulation KM.20, 2009 on the creation of safety management system for airport operation: a safety management system has to cater to the size, nature, and complexity of the airport as well as the safety risks.Civil aviation safety is regulated in the Transportation Ministerial Regulation PM.55, 2015, chapter 139 on aeorodrome.
As for safety management system operation, it is regulated in Air Transportation Directorate Regulation KP.622, 2015 on the directives on technical safety regulations of civil aviation, chapter 139-08.Four components from these regulations will be treated as the criteria for SMS airport evaluation, as shown in Figure 1.

Safety Policy and objectivies
Management commitment and responsibility

Safety Accountability
The

DEMATEL
The DEMATEL method was developed by the Science and Human Affairs Program of the Battele Memorial Institute of Geneva between 1972 to 1976.It was intended to trouble-shoot and solve complex and interconnected groups systems [12].The DEMATEL method is a comprehensive method for building and analyzing structural models involving relations among factors [13].
The method aims at discovering immediate or immediate relations (dependency) in a system of variables.Steps in the DEMATEL method [1] The a ij value represents the influence of the ai criteria or cluster against the aj criteria or cluster and n represents the number of criteria. (1)

The calculation of normalization of the immediate
relation matrix uses the normalization factor S in equation ( 2). ( Immediate relation matrix normalization is signified with X and is taken from the multiplication immediate relation matrices A and S as in equation ( 3). (3)

Calculation total relation matrix (T) uses equation (4).
(4) I is the identity matrix.An essential feature of the DEMATEL method is the visualization of the most influenced and influential factor.After calculating the total relation matrix T, the number of rows and columns are available.The sum of the values in a row is signified with D, as in equation ( 5).The sum of the values in a column is signified with R, as in equation ( 6).To integrate the DEMATEL method to the ANP method, several additional steps are taken, as follow: 5. Creating a total relation matrix using equation ( 7).(7) is the total relation matrix for the criteria and is the level of influence criterion i to criterion j. 6. Summing up all row values on the total relation matrix using equation ( 8). (8) 7. Normalizing the total relation matrix using equation ( 9).
8. Transposing the matrix .The transposed matrix will be used as an input to the supermatrix in the ANP method. (9)

ANP
The Analytic Network Process (ANP) was introduced by Saaty as an improvement of the Analytical Hierarchy Process (AHP) [2].ANP builds on a shortcoming of AHP and introduces the ability to accommodate interdependencies among criteria or alternatives.There are two kinds of interdependencies in the ANP method, namely innerdependence (the interdependency elements of the same set) and outerdependence (the interdependency between different elements).These interdependencies make ANP more complex than AHP.
ANP is a general theory of relative measurement that can reduce the priority ratio from individual ratio scales.It reflects relative measurements of the influence of interacting elements against the control criteria.ANP provides a general framework for decision making without the assumption of independence of higher order elements [2].Steps in the ANP method are as follow: 1. Creating decision network hierarchy of decision factors.

Creating pair-wise comparison matrix.
The comparison of significance among all criteria is transformed into the form of matrix A. The value represents the relative level of significance of the element on row i on the element of column j.For example, .If there are an n number of element to compare, then the comparison matrix A is defined in equation (10).The relative significance values are represented in a ration scale from 1 to 9, as proposed by Saaty.
(10) 3. Measuring element weights.Calculation of the priority vector value w is the eigen vector, uses equation ( 11). ( A is a pair-wise comparison matrix, whereas w is the eigen vector and the maximum eigen value is represented with .4. Calculating consistency index using equation (12).

CI = (12)
CI is the consistency index and n represents the number of elements.By comparing the consistency index and the random index (RI), a threshold value becomes available to determine the consistency level of a matrix.This is called consistency ration (CR).CR is calculated using equation (13).

CR = (13)
Random index (RI) is the value of a matrix that measures the consistency of the results acquired up to this step.If the pair-wise comparison matrix has a value of less than 0.10, then the inconsistency of the opinions of decision makers is to be expected; if not, data assessment in the decision making process needs to be improved.If CR equals zero, the result is consistent, and if it is less than or equal to 0.1, the result is consistent enough. 5. Creating the supermatrix.
Supermatrix is a matrix of the results of the priority vector from pair-wise comparison between clusters, elements, and alternatives.There are three components of a supermatrix, namely: a. Unweighted supermatrix, acquired from the comparisons among clusters, criteria, and alternatives with the eigen vector column in the matrix.b.Weighted supermatrix, acquired by multiplying all elements in the unweighted supermatrix by the value of the transposed matrix in the corresponding DEMATEL operation.c.Limiting supermatrix, acquired by multiplying the supermatrix values with themselves until each row has the same value.

Methodologies
Data collection was through surveys with questioners on SMS performance.The study cases on this evaluation system were Adi Sutjipto International Airport, Yogyakarta (JOG); Ahmad Yani International Airport, Semarang (SRG); and Adi Sumarmo International Airport, Surakarta (SOG).The questioners were responded to by the SMS head officers and SMS officers.The questioner had three components, namely the levels of significance of criteria, levels of significance of sub-criteria, and airport SMS evaluation.
The criteria for the indicators in the evaluation process were based on the Air Transportation Directorate General Regulation KP.622, 2015 on the operations of airport safety management system.The following indicaters were considered: safety and destination policy, safety risk management, healthcare, and safety awareness.The steps in the evaluation process of airport SMS are shown in Figure 1.

Results and discussion
The DEMATEL method managed to improve the ANP method in identifying the interdependency relations of each criterion.In view of this, the DEMATEL and ANP methods can be integrated in modeling the problems in an evaluation system, with especial benefits in measuring the relations among criteria in a hierarchy or network.This integration can be implemented effectively on airport SMS evaluation system as it can determine the value and direction of the interdependencies among the criteria.This study results in an evaluation system that can be used to assess airport SMS performance and to improve it.This evaluation system presents relevant information on the grades of the criteria to be used in evaluating the SMS.The grades can in turn be evaluated and be used by stakeholders or the management in implementing airport SMS performance.This system also grades the results of the evaluation of SMS performance, to which the current airport SMS performance can be compared.Assessments of the levels of influence and levels of significance by the SMS head officers were conducted based on the proposed criteria and subcriteria.The results from the assessments were further analyzed using the DEMATEL and ANP methods.The DEMATEL method sees relations among criteria as shown by the Network Relation Map (NRM).The NRM reflecting the assessments is shown in Figure 2. The suggested model indicates that the criteria for SMS performance measurement were interdependent.The influencing and influenced criteria were taken from the total relation matrix from the DEMATEL method calculation.To find those criteria, the dispatcher vector (D) and the receiver vector (R) were calculated.Dispatcher vector is the number of rows in a total relation matrix, and the receiver vector the number of columns in a total relation matrix.Next, the (D+R) values are placed on the x axis of a impact-digraph map, while the (D-R) values on the y axis.Positive (D-R) values also put criteria as criteria dispatchers and negative (D-R) values put them as criteria receivers.D and R vectors are shown in Table 1.After determining the (D+R) and (D-R) values, the impact-digraph map can be created.The direction of the arrow (following the row-to-column direction) is represented on the matrix with values exceeding the threshold value.The threshold value qualifies a parameter forming a network.This value is the mean of the total matrix minus 0.1; in this case, the threshold value (α) was 1.8545.The resulting impact-digraph map is in Figure 3.To integrate the DEMATEL method to ANP, the values of the criteria matrix, or the Tαd matrix, were calculated The transposed results for Tαd matrix were used in calculating the supermatrix values in ANP, as shown in Table 2.After the levels of influence of the four criteria were calculated, the levels of significance among the subcriteria were calculated.The ANP method determined the levels of significance of the sub-criteria.The initial stage determined the pair-wise comparison matrix, giving the weight of each sub-criteria.Weight measurement was is intended to determine the dominant sub-criteria to be improved on in the SMS performance evaluation process.
The CR value, 0.0661, was acquired from the pairwise grade matrix.As the value is less than 0.10, the matrix is regarded as consistent.The implication is that the opinions gathered during the ANP-method assessment should be regarded.The final stage of ANP is the creation of the supermatrix.This creation is comprised of three components, namely the unweighted supermatrix, the weighted supermatrix, and the limiting supermatrix.The creation of the supermatrix is integrated into the transposed matrix Tαd (total relations) from the DEMATEL method.The unweighted supermatrix is shown in Table 3, the weighted supermatrix on Table 4, and the limiting supermatrix on Table 5.In calculating the supermatrices, an alternative ranking of weights is acquired from the sum of each row in the limiting supermatrix.This alternative weight is shown on Table 6.The results of the ANP calculation will also determine the weight of each sub-criterion to be used in the SMS performance evaluation.These results will determine the dominant criteria and sub-criteria or the priority to be improved in airport SMS performance, as represented with the lowest rank in the alternative weight ranking is shown in Table 6.It can be seen that the safety and destination policy has been graded with the maximum average value of 0.1053, while management commitment and responsibility 0.1852.
The average assessment figures from the evaluations by the SMS head officers and SMS officers in the three airports studied were multiplied with the weight of each sub-criterion.The results are as follow: the SMG airport was given a score of 4.1487, the JOG airport 4.3996, and the SOG airport 3.9851.
The results of the SMS performance evaluation were then classified using Sturges formula into nine grades.This classification would hopefully motivate each airport to improve the safety system performance.The Sturges's formula determined the number of classes in the classification.The formula yielded nine classes from observation data of all 298 airports in Indonesia.The between-class interval values of the nine classes (grades) are shown in Table 7.The classification of the three airports in the current study is shown in Table 8.

Conclusions
The DEMATEL and ANP methods can be integrated in modeling the problems of an evaluation system with the advantage on measuring the interdependencies among criteria in a hierarchy or network.The current study investigates the airport SMS performance evaluation system using those methods.It employs the DEMATEL and ANP methods in determining the priority of airport SMS performance improvement areas.This study yields several conclusions.First, the criteria used in the evaluation system were interdependent in terms of the levels of influence among criteria with the safety and destination criterion being innerdependent.Second, the airport SMS performance system can rank criteria and sub-criteria to be multiplied.The safety and destination policy has been graded with the maximum average value of 0.1053, while management commitment and responsibility 0.1852.Third, with the weight measurement by the ANP method, the results of SMS evaluation of the three airports are as follow: Adi Sutjipto International Airport, Yogyakarta, was graded 4.3995, Ahmad Yani International Airport, Semarang, 4.1487, and Adi Sumarmo International Airport, Surakarta, 3.9851.Regarding the nine classes acquired from the Sturges formula, these results put the Yogyakarta and Semarang airports in grade 8, while the Surakarta airport in grade 7.
figure shows the overall influence of other factors on the particular factor.Additionally, a visual causal diagram can be represented by the calculation of (D+R) and (D-R).The (D+R) values are on axis x, showing how important a criterion is, whereas the (D-R) values are placed on axis y and are termed relations.The relation values activate factors to be classified into cause and effect groups.If a relation value is positive, then it is classified into the cause group, and into the effect group otherwise.

Fig. 2 .
Fig. 2. The steps in the evaluation process of SMS.

Table 1 .
Vector dispatcher and vector receiver

Table 6 .
Alternative ranking weights

Table 7 .
Interval value between class of evaluation result

Table 8 .
Classification of the results class of Airport SMS evaluation