Usability measurement of telecommunications testing website PT. Telkom Indonesia using user centered design

. The telecommunication testing website developed and manage by PT Telekomunikasi Indonesia which aims to enable admin and customer to do the online test for their product. Online test save time and cost of testing. Some website functions found out did not operate effectively and efficiently. Usability Testing using User Centered Design (UCD) method is a research instruments in the form of draft interview, task scenario, and questionnaire. Variables used are Completion rate and Error to assess Website effectiveness, Task Time variable to assess website efficiency, and Satisfaction score variables by subjective questionnaire. Result of WEBUSE questionnaire, usability of website is 0.5825 categorized as moderate usability level (0.4 <x <= 0.6). The lowest usability is the User Interface Design category (0.55). The result conclude that the website need to be evaluate and improved. The time completion rate is 850 seconds and the error rate is high for task certificate download and complete the QA document. The highest task time is complete the registration form.


Introduction
PT Telkom is the largest telecommunication company and network service provider in Indonesia. Telkom Group serves millions of customers with a complete range of telecommunication services and various services in the fields of information, media and edutainment. PT Telkom Indonesia has various fields that support the business process of the company, one of them is the Digital Service Division (DDS). DDS offered service on technology, infrastructure, products and new business, TIMES (Telecommunication, Information, Media Edutainment & Service), and Product Development Center). DDS offered service through innovation incubation management, and issued a certificate of readiness to implement technology, infrastructure and products.
Infrastructure Assurance (IAS) as part of DDS, handles infrastructure testing. User Relation (UREL) is one part of the IAS that coordinate testing between the User and the laboratory. The test is required by PT Telkom Indonesia for its suppliers to get a good quality on infrastructure. The infrastructure is manufactured by the vendor which try to sell their products to PT Telkom Indonesia. The product must have a Quality Assurance (QA) certificate by pass through the laboratory test in IAS. The flow of testing process is done manually where the customer must come to the laboratory and carry out the whole process. This manual process has the risk of missing files due to inadequate documentation process. Also, wasting time in file searching and result on customer dissatisfaction due to lack of information about the stages of test they already performed.
Division Digital Service create a website http://www.telkomtesthouse.co.id to reduce the risk and as a means of testing the telecommunications equipment and its supporting devices. Tests are available for QA (Quality Assurance), VT (Voluntray Test), TA (Type Approval), and CAL (Calibration). The website was created in 2016 and launched to the customer in 2017. The website still need to be develop by analyzed the web usability.
Some companies have successfully implemented a website to support the business process by created a web-based financial reporting applications or an accounting reports. The site should contains various things required by investors such as information about financial statements, international prices, announcements and describes the various types of products offered. The Web site of a company is becoming one of the important tools in a competitive environment, but it's no guaranty that a company will get competitive advantage in the market by creating a Web site [1].
Website quality defines as how well a website is designed and how well the design meets with user satisfaction. The website which has a good usability can provide convenience and increased value of the company. Telkom try to digitalize its business processes so that it can reduce the difficulties and errors that occur on manual process. The testing website handles the entire flow from the customer registration to the certificate taking. The customer facilitated by running 2 Literature review Powell [2] describes website design as a multidisciplinary approach related to web planning and production. It scope are on technical development, information structure, visual design, and network delivery. Website design is multi-disciplinary, it is often to draw ideas and theories from related fields of science. Rosinski [3] mentions HCI has a significant focus on pre-design and post-design tasks oriented to the user that is (1) User requirements gathering, (2) Prototyping and (3) Usability Testing. So, usability testing is an important part after the design of the website. Similarly, Hewett et al [4] explains that Human-Computer Interaction is a discipline that deals with the design, evaluation, and implementation of interactive computer systems for humans and the study of phenomena common in surrounding.
Based on ISO 9241-11 describes usability is a development in which a product can be used by certain users to achieve its goals effectively, efficiently and satisfaction within the scope of certain uses. ISO / IEC 9126-1 explains that usability is the ability of a software product to be understood, studied, used and attracted when used in certain circumstances. While IEEE std.610.12 states usability is the ease with which the user can learn to operate, prepare input and interpret the output of a system or component [5]. Usability is the most traditional concept in HCI research. Usability can be defined as a "measurable characteristic of a product's user interface that is present to a greater or lesser degree" [6].
Usability is the rate at which a product can be used by a particular user to achieve a certain goal effectively, efficiently, and gain satisfaction in the context of its use [7]. Usability means the effective, efficient and safe in product use with its easy to learn / understand, remember, and good utility. Usability has several components that are traditionally associated with five attributes; learningability, efficiency, memorability, error level and satisfaction [8]. There are various methods to assest usability. Interactive Educational Game Design Based on Motion Gesture Technology used methods of interview and observation the users [9]. Wang develop a comprehensive model and instrument for measuring customer information satisfaction (CIS) for web sites of digital marketing products and services [10]. Others research which did an usability assessment are Website Analysis and Measurement Inventory or WAMMI [11], User Centered Design approach to improve the Website usability with scenario based [12].
UCD is a design philosophy that users as a center of system development process. Techniques, methods, tools, procedures and processes of design interactive systems build based on the user experience. Another method on usability assessment is WEBUSE. The WEBUSE method [13] focuses on developing a Webbased Usability evaluation system for measuring websites from the usability aspect. Usability evaluation based on user perception by a set of questionnaire use representative samples of end users who represent the actual working environment and observation using or reviewing product representations. The benefits of Usability Testing are make highlight on benchmarking for the future, minimize service and support costs, increase sales and possible repeat sales, increase consumers satisfaction, gain competitive advantage and minimize risk. Customer Satisfaction Rate of the user will greatly affect the company's brand image.

Research Methods
Test of usability must involves users of website. The data collected from 10 respondents of students of Computer Engineering Diponegoro University who operate the Telkom website test house. The measured variables refer to the research parameters in UCD and WEBUSE. The determination on the number of respondent based on the quota restrictions provided by companies. Testing simulation will go directly into the system and business processes of the company, so it is not possible to use many respondents.
Research variable is determined based on ISO 9241 pt. 11 (Guidance on Usability, 1998) and ANSI (Common Industry Format for Usability Test Reports, 2001). There are five steps in doing the usability test in this research.

Define user specification context
The objective of this stage is to know the menu or operation which process needs to be a priority testing because it is often used by users and important. Conducted by interviews and direct observation. After known the frequency of users using the menu or operation, then the information are used to prepare a tasks scenarios.

Requirement specification
This stage try to identify the user needs of the website features. Identification was done by distributing the online WEBUSE questionnaire specification to respondents. WEBUSE questionnaire is used to determine the needs of users on 4 categories: a. Content, Organization, and Readability b. Navigation and Links c. User interface design d. Performance and Effectiveness The data collection from this stage is the respondent perception on the quality of website on some aspect. The questionnaire used the yes or no question. Number of yes answer divided by total respondent define how complete the website feature to serves the user.

Task scenario simulation
The respondent was asked to do the operation on the website. The task used in this research are 7 task below.
1. Sign in to the website (registration) 2. Login to the website used specific account 3. Order of operation needed if forget password 4. Complete the Telecommunication specification (STEL) document 5. Quality Assurance registration 6. Tracking Process to find out the stages of test their product already performed 7. Download the QA Certificate

Data collection from task scenario
The data collected from the task scenario simulation are Task time, Task Completion Rate, Error on Operation, Task Time, and User Satisfaction. Data was collected on each given task directly while respondent operating the website. Effectiveness is measured by Completion Rate and Error, efficiency is measured by Task Time, and satisfaction scores are measured by knowing respondent perception on the level of website ease of used. a. Completion Rate is used to determine the effectiveness of website. Completion Rate is a condition that indicates the respondent can or cannot complete the task. Shown with the number 1 (one) for the respondent who can complete the task and the number 0 (zero) for respondents who cannot complete the task b. Error is used to find out the effectiveness of website. Error is the total number of mistakes made by respondents when carrying out a given task. The error show the number of mismatch steps done by the respondent in completing the task c. Task Time is a parameter used to determine the efficiency level of website. Task Time is the time takes by a respondent to complete each task (in seconds) in the task scenario. d. The satisfaction score on this assessment shows the respondent's impression of the level of ease in completing the task. There are 7 levels should be chooses by a respondent. The number 1 indicates that the impressions obtained by respondents is very easy in completing this task. Contrary, number 7 indicates that the respondent impressions in completing the task is very difficult.

Data processing and analysis
a. The efficiency of website measure by task time. Task time is the time needed by user to complete a task. A task has sequence operation. The average of task time give the information on how long the time needed by a user to complete a task. Total of task time mean the time needed by a user to complete all the task on QA Test. b. The effectiveness measured by error rate and completion rate. Error rate is define by the number of error operation made by a user while trying to complete a task which consist of some operation. The completion rate show the number of task which is finished by a user. Complete a task is one value and un-complete will get zero value. c. User satisfaction measure difficulty of the website while operate by a user used a questionnaire. The difficulty is measure on scale 1 to 7. The user must give a score 7 means the web is easy to be operate and one for opposite condition (difficult to operate) in a question of questionnaire. The average score on each task define the user satisfaction on a task.

Result and discussion
Based on interviews with stakeholders, it is known that the highest frequency of testing is the QA (Quality Assurance) Test. The greatest revenue of DDS is obtained from this type of testing. QA testing has longer procedures compare to other tests. This study do usability testing that focuses on QA testing service. The first step is define the user specification context to identify the menu or operation which is often used by users and important. The website features category often used by users are displayed in table 1. The value obtain by spread the WEBUSE questionnaire to users. There are 6 questions on the WEBUSE questionnaire. The respondent fill their approval to the question on scale 0 to 1. Score 0 mean they disagree with the statement and score 1 means they really agree to the statement. Mean value describe the user perception on their approval. Mean value of four category is 0.5825. The value categorized the website usability based on WEBUSE questionnaire as medium quality. It means only a half of the users (respondent) perceived that the website are complete and well operate. One of usability indicator is efficiency, measured by Task Time. Task time is the time needed by user to complete a task, display in seconds. The average of task time from 10 users describe on graphic 1. Graphic 1. Average Task time Graphic 1 shows that task 5 has the longest task time. Task 5 do the QA registration, user must complete a registration form. Task two, login on the web used specific account has the lowest task time. The average of the completion time is total of task time, it reach 850 seconds for one user to complete all the task. The lower the completion time means that the web is working more efficient. Graphic 1 give a clue that the registration form need to be simplified in order to decrease the task time.
Effectiveness in usability is measured by the error rate and task completion. Error rate show how many error operation made by user while trying to complete one task. One task has some sequence operation. The average of error for each task is displayed in graphic 2.
Graphic 2. Average of error on task scenario Graphic 2 shows that there was no user did an error operation on task one. Task one has zero on its average error. Task seven, download the QA certificate has the highest error average. Some recommendation are needed to improve the clarity of the menu of download certificate.
Also, task 4, complete the Telecommunication specification (STEL) document still has high error rate. The number of error made by a user show that the user face the difficulty caused by misunderstanding to the information given by the web menu. Improvement on error rate is interface design to be more user friendly and unambiguous.
Another indicator of web effectiveness is completion rate. The completion rate describe the number of task complete by a user. After user complete the task the value of completion is one, if the task still un-complete the value is zero. Task 1, 2, 3 and 6 has the value 1 for 10 respondent. It means that task 1, 2, 3 and 6 have completion rate 100%. There was 4 users didn't finished task four, then the completion value is 60 %. Two respondent didn't finished task 5 and 7 respondent didn't finished task 7. Task 7 completion rate is only 30%. Completion rate data shows that the task 7 often unfinished by user.
User satisfaction is the only indicator of sustainability which is measured by a subjective methods. This indicator show the difficulties of the website. The user satisfaction value is given on graphic 3.

Graphic 3. Average of user satisfaction value
The objective of the research is to formulate a recommendation to improve the website usability. Recommendations are formulate based on the data processing result above, the video which record the task scenario, and the interview with users. The recommendations are displayed on table 2.
Some contradiction found on the research result. The task 7 which is has the lowest completion rate (30%) and highest error value (8 operation) got the user satisfaction on 3.5 (average category). Also, task 4 which has completion rate 60 % and error rate 3.5 operation got the user satisfaction value 4.08. It means that there is a low consistency on user. The task time, completion rate and error rate measure from an objective methods by collecting data from task scenario. The user satisfaction value resulted from a subjective methods by ask the user perception on how difficult a task.

Conclusion
Some conclusion as a research finding are the measurement result of usability assessment. The usability assessment of website used WEBUSE questionnaire categorized the website usability as medium quality. The score is 0.5825 that mean only a half of user perceived that the website already well designed and has a complete feature that needed by users. The result of assessment used the variable from ISO 1998 found some task are still not ease to use. The completion time is high on task to complete the QA online test especially for task complete the registration form. The error rate for task 7 (download the certificate) and task 4 (complete the QA document). Contrary to the objective data, the user satisfaction which measure used questionnaire show a good result for task 7 and task 4.