Usability analysis of environmental disaster mitigation efforts in Lampung Province: case study of Jaring Bencana application

02001


Introduction
Indonesia has many natural disasters, such as earthquakes, tsunamis, floods, and landslides.The natural disasters that occurred in Indonesia in 2022 have caused enormous losses.Material losses reached 1.5 trillion rupiah [1].These losses are not only in the form of physical damage but also economic and social losses.The Indonesian government continues to strive to reduce the risk of natural disasters.These efforts include building disasterresistant infrastructure, increasing community preparedness, and strengthening early warning systems [2].However, these efforts are still insufficient to overcome Indonesia's high risks of natural disasters.Communities need to know the types of natural disasters that often occur in their area and ways to reduce the risk of these disasters [3].By increasing awareness and vigilance of natural disasters, Indonesian people can reduce the risk of life and material losses due to natural disasters.One of the efforts of various parties is the development of various disaster-related technologies, such as development of various digital applications.
Based on these efforts, researchers see that several disaster applications in Indonesia have features and functions that seek to contribute to strengthening disaster.Several experts have studied disaster applications, and the study provides an analysis of existing disaster risk assessment and response tools to enable the user community [4].Mobile applications can potentially assist disaster response by providing an avenue for distributing relevant and timecritical information to the public [5].Various disaster apps are already in the app market.However, engaging users in maintaining disastrous smartphone applications is a challenge [6].The research presents the author's experience implementing a chatbot system in Taiwan government departments for disaster response [7].Existing research has investigated various mobile applications that engage with crowds during disaster situations.The findings show that disaster applications can be general-purpose or specific for disaster purposes.This review further focuses on built-for-disaster-purpose applications and shows the various interactions these applications foster with the public and the added value contributions of applications throughout the disaster lifecycle [8].There are three areas of need for future research: app engagement prior to the disaster response stage, public behavior and motivation towards app use, and mobile app usability.
Therefore, based on the existing problems, the Jaring Bencana application needs the usability test, which is crucial and can be used as material in future application development plans [9].Several studies regarding usability have been carried out, and research conducted by Tan's research shows that five usability factors significantly influence continuance intentions.The main positive influences are (1) the user's perception of whether the application delivers its functionality (application utility), (2) whether it is reliable, and (3) whether it presents information that can be easily understood.Furthermore, too much focus on (4) user interface graphics and (5) user interface inputs can discourage continuation intentions.The challenge of finding essential usability factors for m-government applications and selecting the right factors for a particular m-government application [10].The results include six usability factors: learnability, simplicity, satisfaction, security, privacy, and memorability [11].Several other studies have compared user perceptions collected using a final questionnaire based on the System Usability Scale [10].This paper intends to use a similar approach to analyze the prototype of the Jaring Bencana application, which is to be introduced to disaster management groups in the regions.So, the problem in this paper is, how is the achievement of the usability value of the Jaring Bencana application?Moreover, what are the problems in testing the Jaring Bencana application faced by user groups?This study aims to examine the level of usability in disaster applications by measuring the variables of effectiveness, efficiency, and user satisfaction.

Research method
The research methodology contains the stages or description of the research conducted.The methodology is applicable in every research to make it easier.This research was conducted using a mixed method approach, which combined survey instruments and interviews with the target group of application users-adjusted research stages with stages on analysis and evaluation with the usability testing method on the Jaring Bencana application.

Data collection
Data collection consisted of three stages: user identification by testing 30 participants divided into three age groups; teens (13-17 years), college students / young adults (18-24 years), and adults (25-64 years).It is because the target user of the Jaring Bencana application is the public aged 15 -64 years old.After identification of the user, create a task scenario by creating a set of tasks for, the list of task scenarios can be seen in the table 1 below.

Job description
Task Description Task 1 (T1) Log in to the application Task 2 (T2) Try the report disaster info menu around Task 3 (T3) Try to find disaster response volunteers near Task 4 (T4) Trying to respond to the disaster report menu that appears Task 5 (T5) Find educational information features for disaster risk reduction (environmental conditions that have the potential for disaster or activities) Task 6 (T6) Find notifications of disaster location positions (earthquakes, floods, etc.) The final activity is to conduct interviews to gather the necessary in-depth information.At the same time, type data testing, which may be used to evaluate usability, contains all of the application's functionality.When users utilize the Jaring Bencana application, the performance measurement technique is employed to assess its efficacy and efficiency.Effectiveness measurement can be calculated based on the success and failure of each respondent's tasks.Measurement of effectiveness can be calculated based on the success and failure of the task scenario that each respondent has done.Effectiveness is calculated by setting the number binary "1" for respondents used in the study.It consists of quantitative data and qualitative data.Qualitative data results from Usability Testing using the Retrospective think-aloud (RTA) technique.

Analysis and Data Processing
The information gathered during the data gathering stage is subsequently processed.The processed data is then evaluated to produce findings.The following is how data is processed in usability study on the Jaring Bencana application: (a).Usability testing is a technique for evaluating the quality and usability of a service or product by directly participating application users to find out problem information on tested mobile device applications.In the usability success complete tasks scenario, there are ten ways and a "0" for respondents who failed.Equation ( 1) is used to compute the respondent's success rate, whereas equation ( 2) is used to calculate the respondent's failure rate.
Respondent success level is calculated by dividing the number of successfully completed scenario tasks by the total number of given task scenarios, and then multiplying the result by 100%.The failure rate of the respondent is computed by dividing the number of failed task scenarios by the total number of offered task scenarios and multiplying the result by 100%.Based on the average time to finish each work from all existing responses, the calculation results will highlight which tasks are most important to accomplish [11]2w.Meanwhile, the level of efficiency can be measured from the time of completion of the task with the success or failure of carrying out the task using the equation (3).
Whereas: N : task totals R : total participants Nij : Result of task i by user j; if the user completes the task successfully, then Nij = 1.If it does not work, then Nij= 0. Tij : Time taken by user j to complete task i; If the user does not complete the task, the time is measured until the user quits the task.
The System Usability Scale (SUS) will be used to assess the user satisfaction component.SUS is a well-known usability testing tool that is dependable, popular, efficient, and costeffective.When calculating the System Usability Scale (SUS) score, there are various rules to follow.The rules for calculating questionnaire scores according to the user's score decreased by one; and for each odd-numbered question received.Each question has an even number; the final score is determined by subtracting the value five from the user's question score.The SUS score is obtained by multiplying the sum of each question's scores by 2.5.Further calculations seek each respondent's SUS score by averaging all scores and dividing by the number of respondents.The average of SUS score from all respondents was derived, and the score was then adjusted to the SUS assessment.The average SUS score from the number of studies is 68, hence scores above 68 are considered above average, while scores below 68 are considered below normal.
Retrospective think-aloud analysis is also used to test a system that incorporates users, or end users, by verbalizing what users feel and think when utilizing the system in a sustainable manner.Verbalization is accomplished by conducting interviews with respondents to learn about their experiences and opinions while using the Jaring Bencana program.The verbalization comprised the respondent's impression of the message or challenges conveyed, as well as the comments made after the testing session.

Result and discussion
The usability testing was implemented on 30 people who had fulfilled the test requirements.The requirements are that test participants are of productive age; aged 15 -64 years, use the application on Android smartphones, and have never used Jaring Bencana.The usability test conducted in this study aims to calculate the effectiveness, efficiency, and satisfaction components using performance measurement and retrospective think-aloud techniques.This test is carried out by giving a series of tasks to the test participants to complete.The results of the data processing effectiveness and efficiency in the tests carried out are as follows.

Component effectiveness
The purpose of measuring the level of effectiveness is to provide an overview of the application's ability to support the convenience of test participants in using the application based on success in carrying out tasks.The results of the success rate are shown in form percentage.All data successful test takers tasks and test participants who failed to carry out the task are shown in Table 2.
Table 2 shows results from successful test takers in doing the task.Data in the table is represented using binary numbers 0 and 1 to simplify calculations.A value of '1' will be given if the participant can complete the assigned task correctly and mark '0' if t h e participant test failed the task.Most of the results from completing tasks are not happening or much failure.There are several anomalies in tasks T2 and T3.Some test takers experienced difficulties in finishing tasks T2 and T3.After the interview, the participants tested had an understanding of task 2. However, the problem is that some test participants are less focused and less aware of the location of the hazard info button and its recommendations.The result of calculating the success rate of using this application is 89% above average success, i.e., 78%.This figure concluded that the application is practical and has a good level of learnability.

Component efficiency
To measure component efficiency, the time needed by the test to complete the assigned task is recorded and recorded.The time spent completing the task is counted from when the researcher finishes giving the assignment, and the test participants start using the application.
The time calculation will be stopped when the test participant states that the task has been completed or the test participant gives up on completing the task.Table 3 shows the times needed by every participant test to complete the task.If seen from component efficiency, the teens' category performs more efficiently than other categories.In comparison, the adult category performs better than the college students category.Efficiency calculation is based on calculated time using the success data of participants in completing the tasks listed in Table 4 and the time data needed by participants to complete the tasks in Table 5.The calculation results show the time-based efficiency value of the Jaring Bencana Application is 18.0 second or 0.24 goals /sec.

Component Satisfaction
The satisfaction component in this study was measured by filling out an online questionnaire given to 30 users of the Jaring Bencana application.SUS is a measurement of the level of user satisfaction by giving questionnaires to users after the application (Brooke, 2013).
Respondents were asked to fill in data self and ten statements from System Usability Scales (SUS) with a scale Likert 1-5.Based on the calculation, the SUS obtained from the respondents was 68.Score SUS on 68, so it would be considered above average, and anything below 68 below average.The SUS score on the Jaring Bencanas application is still below the average and has received a grade scale D with adjectival ratings "OK" and the acceptability range entered on the category "marginally acceptable."

Retrospective Think Aloud
Retrospective think-aloud data was obtained by conducting interviews with participants test.An interview was conducted to find out test participants' experiences and opinions while carrying out scenario assignments using the Jaring Bencana application.The list of questions used follows the interview framework that has been made before.Questions were further filed by the answers of participants to explore the problem.From the results of interviews with all test participants, several problems were found, and the test participants gave suggestions.A list of problems can be seen in Table 4.The application requires a stable internet network because it has to load GPS 6 Application loading feels heavy when using low HP specifications.

7
I found some bugs in some menu features. 8 There is no offline feature for loading maps yet.
Based on Table 4, it is necessary to improve and strengthen several functions in the application.A more concise guide that can be effectively understood by new users is needed.Improvement and strengthening of this feature will later strengthen the benefits of this application for its user group.

Conclusion
Based on the analysis, it can be concluded that the results of the evaluation of the Jaring Bencana application have complied with the criteria of usability in effectiveness and efficiency, as well as user satisfaction in using the application.The results evaluation used performance measurement techniques that measure components' effectiveness and efficiency to 30 test participants, resulting in an effectiveness component value of 89% and an efficiency of 0.24/sec.The value shows that the level of learnability of the application Jaring Bencana is on average.While the value of the satisfaction component is 68, it is included in the grade scale category D with the information "OK."Then, the retrospective think-aloud technique was conducted through interviews with test participants to produce several identifications of usability problems applications.

Table 1 .
Task Description

Table 3 .
Results for participant test in completing tasks (units/seconds)

Table 4 .
Problems and suggestions for all test participants