The Decision Making Trial and Evaluation Laboratory (Dematel) and Analytic Network Process (ANP) for Learning Material Evaluation System

. Evaluate the learning that will be given or given to diagnose success and failure in understanding teaching materials. Failure to provide learning can be caused by the use of inappropriate teaching or media methods, and difficult language to convey. By applying information technology in evaluating learning materials, a web-based evaluation system can be developed to conduct assessments and as a way to improve the quality of learning materials with certain criteria. So that the problem of failure in the learning process can be known based on the learning material used. This study integrates the DEMATEL and ANP methods that support decision making by evaluating lecturer learning material to students. Based on the problems and explanations described earlier, the use of the DEMATEL and ANP methods is expected to assist in evaluating learning materials with the aim of always trying to provide good learning materials based on the right factors or criteria. The results of making an evaluation system aim to assess the suitability of learning materials with the criteria used as further improvement efforts and to find out where the lack of success in learning. Thus it can be sustainable between the objectives and the final results of teaching.


Introduction
Learning Process is an activity carried out by individuals with groups and groups with other groups that contain educational values. Educative value is an activity that provides knowledge, understanding, and direction. It can also be interpreted as anything that can or can provide learning to someone in the form of knowledge so that it can be useful for the cognitive development of others. The learning activities that will be provided have been formulated or compiled beforehand in the hope that they are in accordance with the achievement of certain goals [1].
As an educator, it is expected that the success of the learning process delivered can go hand in hand with the target of learning achievement. Therefore it is necessary to evaluate the learning that is being or has been given to diagnose success and failure in understanding teaching materials. Failure to provide learning can be caused by the use of inappropriate teaching methods or media, and difficult language to understand. The implementation of the evaluation needs to be carried out in connection with Article 58 Paragraph (1) of the RI Law No.20 of 2003 National Education System which states "evaluation of student learning outcomes is carried out by educators to monitor the process, progress, and continuous improvement of student learning outcomes".
By implementing information technology in evaluating learning materials, an evaluation system can be developed to assess and improve the quality of learning materials with certain criteria. In making the learning material evaluation system can apply the Decision Making Trial and Evaluation Laboratory (DEMATEL) method and the Analytic Network Process (ANP) which measures the interrelationship between criteria with the objective results obtained more objectively. According to [2] the use of both DEMATEL and ANP methods can be used in building various evaluation systems such as evaluation of learning materials, evaluation of company/supplier performance, and evaluation of the level of information security risk. The output of the evaluation system will depend on the quality of the crystallization of the teaching material because the DEMATEL and ANP methods use the results of the analysis and weighting of the criteria in the calculation. The merging of these two methods is useful for complementary uses between methods, namely DEMATEL serves to analyze the relationship between criteria and the use of the ANP method to calculate the weights of each of these criteria [3].
The use of the ANP method does not require determining the level of the objectives, criteria, and alternatives, because this method uses a network approach by considering the inter-elements in the network. ANP makes a general scheme in making decisions without giving rise to assumptions about the independence of elements. This is because in weighting using ANP in the criteria and sub-criteria that are owned are interrelated [3]. To further strengthen the results of ANP processing, a DEMATEL method is needed that can present interdependence relationships between criteria or analyze dominant criteria [4].
Based on the problems and explanations described earlier, the use of the DEMATEL and ANP methods is expected to be able to assist in evaluating learning materials with the aim of always striving to provide good learning material based on the appropriate factors or criteria. The results of the evaluation system are aimed at evaluating the suitability of learning materials with the criteria used as further improvement efforts. Given the increasing number of educators and the growing development of education and knowledge in conveying information to students, it should always be evaluated for teaching materials that are being or have been given. Thus it can be sustainable between the goals and the final results of teaching.

Learning materials
In order to improve the quality of the learning process by an educational institution view teaching materials as an important component. Teaching materials are the most important material for instructors or instructors to assist in carrying out the learning process. Learning materials can be either printed or non-printed books that aim to create effective and educative learning situations. The notion that teaching materials are all forms of material (information, tools, or texts) that are systematic, describes the ultimate goals of students' competencies, and are used in the learning process [5]. There is also another understanding that all forms of material are in the form of a set of material that is systematically designed and used by lecturers/instructors in carrying out learning activities [6].
So that it can be concluded that the material is all learning tools or learning material that has been prepared with the aim of a systematic learning process. The existence of teaching materials is an inseparable part of the learning process. How teaching materials can be likened to the initial concept of learning not even be arranged systematically, the results of the learning process can be in accordance with expectations [3]. Teaching materials are derivatives of the curriculum in the form of subjects or fields of study with certain subject matter about sub-material [8]. Teaching materials are handbooks for a course written and compiled by experts in related fields and fulfilling the rules of the textbook and officially published and disseminated (Kepmendiknas No.36/D/O/2001, article 5 verse 9). Textbooks have differences with textbooks because teaching books are prepared based on specific provisions related to student learning.
In conducting an evaluation of learning material activities not only focus on materials and media used in learning. However, it is necessary to enter an assessment based on class action, where a lecturer discusses issues related to cases that occur in class. So that it can indirectly make students more active in the following learning and increasing thinking about the benefits going forward [9]. The development of problem-based learning also requires students to be more active and think critically [10].
Determining learning materials is the first step in learning. So that the learning process can run systematically and can get results in accordance with expectations. If viewed based on its function, learning materials include: (1) the means of developing materials and programs in the education curriculum, (2) the means to facilitate the learning process, (3) the means to facilitate learning goals, (4) the means to facilitate learning efficiency and effectiveness [11] Based on the understanding of the function of the learning material, the determination of learning materials becomes an important point for achieving learning objectives. The advantages obtained from learning materials include: (1) can present messages or information in large quantities, (2) messages or information can be learned by students according to their needs, interests, and speed, (3) can be learned at any time because easy to carry, (4) it will be more interesting if it is equipped with pictures and colors, (5) repairs / revisions are easy to do [12].
Based on the advantages of determining the learning material, the learning material itself can be grouped by type, namely (1) printed material including handouts, books, modules, student worksheets, brochures, leaflets, wallcharts, photos/pictures, models/models; (2) listening teaching materials (audio) such as tapes, radio, LPs and audio CDs; (3) listening material (audio-visual) such as video CDs, films; and (4) interactive teaching materials such as interactive CDs [13].

DEMATEL
The DEMATEL method was created by the Science and Human Affairs Program of the Battelle Memorial Institute of Genawa between 1972 and 1976. The method is an effective procedure for analyzing or studying the structure of problems and resolving the interrelationships between system factors [14]. Because the method is effective in the process of building and analyzing decision-making systems that depend on the ability to understand causal relationships between factors in the system [15].
The DEMATEL method functions to describe the basic perceptions of contextual relationships between elements of the system based on the values obtained with the aim of obtaining the power of influence between elements. So that results will get information related to direct or indirect relations (dependencies) between variable systems [2]. DEMATEL has been widely used and successfully used in the field of research, including business management processes, supplier selection, and green supply chain management [16]. When compared with existing multicriteria decision models, DEMATEL can provide complete information in decision-making 2 systems related to problems with complex interdependencies [17].
Procedural steps in the DEMATEL method are described in the following sections. Value of aij present the influence of the criteria/cluster ai on the criteria/cluster aj and n respectively 'no influence', 'little influence', 'Moderate influence', 'high influence', and 'Extremely high influence' [18].
M is a normalization factor. Normalization of the relationship matrix is directly denoted by X obtained from the matrix multiplication of direct relations A and M in the equation (3).
3. After getting the results of the normalization of the direct relationship matrix. Then to get the total matrix the relationship can be calculated by equation (4).
After the process of calculating the total matrix of relations T, the number of rows and columns can be obtained. The sum of the row values is denoted by D explained in equation (5).
While the number of columns denoted by R is explained in equation (6).
TD is a total relationship matrix for criteria, is the level of influence of criteria i on criteria j. 5. Calculates all row values in the total matrix relationship with the equation (8).
7. Transposes the matrix . The result matrix of the transpose matrix total relationship will be used as input to the super matrix on the ANP method.

ANP
Analytic Network Process (ANP) is an improvement of the Analytic Hierarchy Process, both methods were developed Saaty. The development of ANP aims to correct the weaknesses of AHP in the form of the ability to accommodate the interrelationships between criteria or alternatives. Therefore ANP is developed consisting of control hierarchies, clusters, elements, interrelationships between elements, and the interrelationships between clusters [19]. ANP is generally used to reduce the priority ratio composition of the individual ratio scale by reflecting the relative measurement of the influence of the elements related to the control criteria [3]. The steps for calculating the ANP method are as follows. 1. Make a network hierarchy of decisions between decision factors. This initial stage, choosing the goals or objectives to be achieved, criteria that refer to the criteria and alternative choices. It aims to identify alternatives that will be the most significant in decision making [3]. 9 developed by Saaty [19]. The pairwise comparison matrix is shown in equation (10).
3. Calculate element weights. Calculate the priority vector value w which is called the eigenvector using equations (11).
Description A = pairwise comparison matrix w = eigenvector (priority weight of a matrix used in super matrix preparation) = maximum eigenvalue 4. Calculate consistency index using equations (12).
Description CI = Consistency Index = maximum eigenvalue n = many elements By comparing the consistency index and the Random Index (RI), a benchmark is obtained to determine the consistency level of a matrix, called the Consistency ratio. Ratio (CR). To calculate Consistency Ratio (CR) with equations (13).

(13)
Description CR : Consistency Ratio CI : Consistency Index IR : Index Random Consistency Random index is a matrix value that has a value determined by Saaty to measure consistency of the results obtained. If the paired comparison matrix with a CR value is less than 0.10 then the inconsistency of opinion from decision makers can be accepted, if not then assessment of data decisions must be corrected. If CR = 0 it is said to be consistent and if CR ≤ 0.1 it is said to be quite consistent [4].

Super matrix Formation
Super matrix is a matrix of priority vector results from paired comparisons between clusters, elements and alternatives. Super matrix consists of three stages, namely: a. Unweighted super matrix, obtained based on comparisons between clusters, criteria and alternatives by inserting eigenvectors into the matrix corresponding to the cell. b. Weighted super matrix, obtained by multiplying all elements in the unweighted super matrix with the TaD matrix transpose value in the appropriate DEMATEL method. c. Limiting super matrix, obtained by increasing the weight of the super matrix by multiplying the super matrix by itself until the value of each row is the same.

Combination DEMATEL and ANP
In this research using a combination of ANP and DEMATEL methods to create and analyze learning materials. The combination of DEMATEL and ANP is nothing new, but there is a lot of literature that uses a combination of both methods. This combination is carried out by maximally taking advantage according to Pai [20]. DEMATEL is a method used to analyze causal relationships, and can quantitatively present criteria and consider related structural models. However, DEMATEL cannot determine the weight of individual criteria, where ANP will be useful. In cases where the evaluation criteria are diverse and complex, ANP can provide benefits in calculating criteria for criteria and relationships. Both methods provide support in dealing with complex problems.
DEMATEL and ANP joiners can provide processing that is useful for identifying critical attributes of policy implementation and calculating the criteria for weighting business environment [16]. Use of DEMATEL and ANP methods in the evaluation system used to solve various problems, such as supplier performance evaluations, evaluation of learning materials, evaluation of risk levels information security, company performance evaluation, and others [2,3].
In building a learning material evaluation system using the DEMATEL process by building a matrix of direct relations to the criteria, then normalization of the direct relationship matrix is carried out. The ANP process performs the initial stages of building a pairwise comparison matrix between existing sub-criteria. After the pairwise comparison matrix is obtained, then calculate the weight vector, measure consistency, and compare the value of the consistency ratio. The next stage forms a super matrix by entering vectors obtained from DEMATEL and consistency of the matrix. The results of this process are to obtain the weight of the criteria for evaluating the criteria for super matrix limiting.

Methodologies
The research developed is quantitative research. The initial stages are carried out by conducting a survey process using a questionnaire as input in the system processing. The questionnaire was used as an assessment of learning materials. An assessment of the questionnaire will be used as input to the system for DEMATEL processing. After obtaining preparations from DEMATEL, the next process from the processing of DEMATEL will be input into the ANP to form the super matrix. Questionnaires were distributed to experts to conduct evaluations. Evaluation of learning materials is carried out based on the criteria and sub criteria shown in Table 1.

Results and discussion
The results of this research are evaluation systems used to evaluate the learning materials used by the instructor as efforts to improve the quality of education and the quality of students. This evaluation system presents information regarding the feasibility of learning materials used by the instructor. The results of this system can be used as an evaluation material and help stakeholders (stakeholders) or guarantors of education quality in setting policies to improve the performance of the learning process.

Fig. 1. NRM DEMATEL
From the criteria and sub criteria data used, an assessment of the level of influence and level of importance is carried out by the education quality guarantor. The results of the assessment are processed using DEMATEL and ANP methods. The DEMATEL method serves to see the relationships between the criteria indicated by the Network Relations Map (NRM). The NRM generated from the experts' assessment is shown in Figure 1. The results of the NRM found a relationship between: 1. C1 has an inner dependency relationship, which is a relationship of interaction and feedback within a cluster.
2. C1 has an outer dependence relationship, namely the relationship of interaction, and feedback between clusters with C2, C3 and C4. 3. C2 has an outer dependency relationship to C1, C2, and C3. 4. C3 has an outer dependency relationship to C1, C2, and C3. 5. C4 has an outer dependency relationship to C1, C2, and C3 The proposed model shows that the evaluation criteria for learning materials are interdependent. Criteria that influence and are influenced are obtained from the total relationship matrix in the DEMATEL method calculation. To find out the influential and influencing criteria, a dispatcher vector (D) and receiver vector (R) is calculated.
The dispatcher vector is the number of rows of the total relationship matrix and the receiver vector is the sum of the columns of the total relationship matrix. Then the value (D + R) is placed horizontally representing the x axis on the impact-digraph map, while the value (D-R) is placed vertically representing the y axis on the impact in graph map. From the value (D-R) we can also classify the criteria into dispatcher criteria (cause) if it is positive and the receiver (effect) if it is negative. Vector D and vector R are shown in Table 2, while criteria group D and R are shown in Table 3.

C3 (Presentation Format) C4 (Personalization)
After getting the D + R and D-R values, an impactdigraph map can be made, with arrows represented by a value in the matrix that exceeds the threshold value (the direction of the arrow is represented from row to column). The threshold value is the threshold value of a parameter to form a network. The threshold value is obtained from the average value of the total matrix minus 0.1, so that the threshold value (α) is 1.8545. Then the impact-digraph map is obtained as shown in Fig. 2. To integrate the DEMATEL method into ANP, the calculation of the criteria matrix value is performed by Tαd matrix. The transpose for the Tαd matrix is used to calculate the super matrix value in the ANP shown in Table 4. After four criteria calculated the level of influence between criteria, then determine the level of importance between sub criteria. The ANP method is used to determine the level of importance of the sub criteria with the initial stage of determining a paired comparison matrix so as to produce weights from these sub criteria. The determination of weights is done to determine the dominant sub-criteria that need to be improved on the process of evaluating learning materials. From the paired level matrix, the value of the Consistency Ratio (CR) can be obtained, which is 0.0661. Because the CR value is less than 0.10, the matrix is consistent. So that the opinions of experts for processing with the ANP method are acceptable. The final stage of ANP is the super matrix formation. The super matrix formation consists of three stages, namely unweighted super matrix, weighted super matrix, and super matrix limiting. For the formation of the super matrix, it is integrated from the transpose Tαd matrix (total relationship) of the DEMATEL method. Unweighted super matrix is shown in Table 6, whereas for the weighted super matrix shown in Table 7 and for super matrix limiting shown in Table  8.
From the super matrix calculation, we obtained alternative ranking weights from the number of each row in super matrix limiting. Alternative ranking weights are shown in Table 5. The results of the assessment using ANP will be used to determine the weight of each sub criteria used in conducting the assessment SMS performance. The results of this weight will be used to determine the criteria and dominant sub-criteria or priorities that need to be corrected in determining learning materials that are presented with the lowest ranking on the alternative ranking weights.

Conclusion
Based on the results of research and discussion on learning material evaluation system for determining priorities for improving the quality of learning materials. some conclusions are obtained, namely: the criteria used in this evaluation system have the relationship between the level of influence between criteria and safety policy criteria and objectives has an inner dependency relationship; learning material evaluation system can rank the criteria and the sub criteria used with the material suitability criteria and objectives have the highest weight with a value of 0.5267 and the conformity subcategory of the material discussed has the highest weight of 0.1852; by weighting obtained from the ANP method, the results of evaluation of learning materials can be obtained.