Indonesian version of FASER LX scale to measure Learner Experience: A validation study

. This study aims to test and validate the FASER LX Scale, an instrument developed to measure e-learning learner experience. Online survey to 365 respondents selected randomly form Komunitas Guru Belajar, a virtual learning community in Indonesia was applied. Statistical analysis was firstly performed by optimizing Confirmatory Factor Analysis to confirm the compatibility between factor structure from existing data and its model specification as proposed in the previous research. Result of CFA showed that model fit criterion is failed to achieve. So, the five dimensions model of learner experience cannot be confirmed. Thus, an additional EFA was conducted to find the underlying factor structure in the existing data. EFA revealed a 32-item, 5-factor structure solution which proven to be valid and reliable. Further analysis of this finding stated that this instrument successfully explained 61,239% of total variance. This study recommends the use of this instrument to measure learner experience in order to evaluate e-learning program as one of the alternative way to support SDGs good education mission.


Background
Education is one of seventeen key points addressed by SDGs, in which Information and Communication Technology (ICT) plays a pivotal role in assisting to address that challenge [1,2]. Various attempts have been implemented to ensure the quality and enlarge the accessibility of education. This can also be witnessed in the field of e-learning, which has been pushed to spread all over the world especially after the outbreak of Coronavirus [1,2]. The implementation of physical and social distancing to prevent the transmission of the virus, caused face to face learning activities unable to be carried out. Thus, e-learning option in which mediated online interaction [3] happened, has later become an alternative learning method adopted in many sectors. E-learning is actually refers to all types of learning mediated by digital technology supported by web-based system innovations [4]. Transfer of knowledge in this certain context occurs through internet, audio visual medium, learning software, or other communication media [5]. This learning innovation in a way, opens up the possibility of learning by lessening geographical limitations. Learning can be carried out without necessarily being in the same time and place. In some cases, it can even include a more personalized design [4], where the learners have more freedom in managing their learning journey by choosing the appropriate time, the place to learn, and by what device they want to engage to the class they attend. Although beneficial, it is still arguably hard to determine whether the e-learning activity they implement can be called successful and positively contribute to improve the learners capacity and fulfilled their learning needs [6].
Previous studies have been dedicated to fill this gap [6][7][8]. In 2019, FASER LX, a scale intended to measure e-learning learner experience was first proposed in academic literature [6]. This scale enables the both industry and academic party to evaluate the implementation of their program by observing five elements, namely learner characteristics, instructor characteristics, technological characteristics, course characteristics, and social aspects of the e-learning activity. However, until this study is conducted, the scale has not been tested yet. Considering the potential contribution in the wider use of the scale, this study is conducted to test and validate the FASER LX Scale and provide an Indonesian version of it. So that, the instrument will be more suitable to be used further in Indonesian cultural context.
To help readers understand better, this paper is organized into four main sections consists of 1) introduction and literature review, 2) explanation of method, 3) result and discussion, and 4) conclusion.

Literature review
The concept of learner experience (LX) is derived from User Experience (UX) in e-learning context [6]. User in LX, is specifically a learner, who can actively contribute to the creation, dissemination and sharing of information [9]. Experience refers to the details of interactions between users and products, including the feelings that appear when interacting, the understanding about how the product works, as well as the suitability of products and systems with the goals, needs and expectations of using the product [10]. It is built on a combination of cognitive, social, and affective elements [11], achieved through the interaction of the five senses with the environment [12]. Thus, learner experience can be defined as the wholeness of cognitive, social, and affective experiences, which learner gets as a result of interactions with e-learning products, services, and institutions, in an effort to meet certain expectations, needs, and learning goals.
Experience is personal, unique, and represents the subjective interpretation of someone resulted from his interaction with the world [13]. Although different learners are involved within the same learning program, their differences in expectations, beliefs, learning strategies, accessibility [14]; learning dynamics [15]; previous experience and understanding [16]; and a combination of different technology used to support the learning process [17], will bring out different learner experiences. This indicates that learner is the best experts in their respective learning experiences [15]. Therefore, a study of the learner aspects of elearning is better explored and understood from their own perspective [18].
From academic point of view, understanding learner experience can help the development of better and more effective learning program both intellectually and emotionally, in order to increase interest, motivation, and learning outcomes [13]. From the industrial point of view, understanding learner experience provides many benefits for the development of e-learning programs [19]. Practically by this understanding, industrial sector becomes easier to assess the impact of investment, summarize and find interesting things such as emotional issues from the learner's point of view [14], evaluate the effectiveness of programs and services [20], provide recommendations for development programs and services [14,19], recognize the needs, desires, and abilities of users in utilizing products and systems created [21], and help to accommodate in order to create positive experiences for learners. This positive experience can influence attitudes [22], trust [23], to decision-making tendencies [24] related to the use of the system or product offered.

The instrument
The FASER LX Scale is a scale developed to measure the learner experience of e-learning program. This semantic differential questionnaire was originally written in France and English in 2019 [6], contains of 30 indicators included in five main dimensions: learner, instructor, system/technology and course/program characteristics, and social aspects. Learner characteristics consists of seven indicators, namely computer self-efficiency (autonomousnon-autonomous), self-enjoyment (unpleasant -pleasant), perceived usefulness (boringcaptivating), self-effort (undemanding -demanding), self-regulation (free use -compulsory use), self-security (confident -distrustful), and perceived anxiety (calming -stressing). Instructor characteristics consists of four indicators, namely communication ability (easy communication -difficult communication), responsiveness (high reactivity -low reactivity), informativeness (not comprehensible -comprehensible), and fairness (unfair -highly fair). System characteristics consists of eight indicators, namely connection access quality (slowfast), device and context independence (device dependent -device independent), efficiency (tedious -efficient), security (unreliable -very reliable), perceived ease-of-use (difficult learning -easy learning), availability (not available -very available), interactivity (not interactive -very interactive), and personalization (customizable -not customizable). Course characteristics consists of six indicators namely course quality (confused -clear), content diversity (not diversified -very divided), course flexibility (rigid -flexible), design and system quality (pleasant -unpleasant), up-to-dateless (static -dynamic), and diversity in assessments (diversified assessment -not diversified assessment). While social aspects consists of five indicators namely subjective norm (recommendable -not recommendable), self-image (valuable -non-valuable), learner-learner interaction (gets closer to learnersseparates me from learners), learner-learner interaction (gets closer to learners -separates me from learners), learner-instructor interaction (get closer to teachers -separate me from teachers), and Instructor-instructor interaction (getting teachers together -separate teachers).
However, due to the case of double barrel on two items: device independence and context, as well as design and system quality, this study uses 32 pairs of bipolar words on a scale of 1-7 [25]. Bipolar adjectives are placed on an extreme polar scale, indicates an evaluation of positive and negative experiences [26]. The placement of bipolar adjectives is randomized according to the original questionnaire [6] to avoid the tendency to fill in scheme when filling out questionnaires [26]. The randomized items later being reversed in coding phase by replacing the obtained item score with its opposite to ensure that negative words representing negative experience are all in the left continuum and vice versa [6,26]. The higher the score achieved, the more positive the learner experience is [6].
Modification of scale is needed to ensure a better understanding of the targeted population [27], before widely used in other cultural context [28]. This study modifies the scale to provide Indonesian version of it. To increase the content validity, this study applies three steps: back translation [29] that consists of forward translation and backward translation [30], preliminary test, and field test [28]. After being translated, a target population review to improve the suitability of the instrument and ensure that this instrument can easily be understood by the observed population [31] is applied. The draft is consulted with three reviewers from three different regions who are listed as members of the observed community, regarding word choices and delivery technique of the questionnaire.
A small preliminary test was then conducted [28]. In this phase, 40 respondents [32] were involved to fill out the questionnaire with additional open ended questions to accommodate suggestions for questionnaire. Preliminary tests were carried out to improve the clarity of items, measure processing time, and reduce the burden on respondents when filling out questionnaires during field testing [28]. Inputs obtained from the target population review and preliminary tests were considered to improve the questionnaire that will be used in field test. Indonesian version of FASER LX scale is presented in Table 1.

Data collection
Data are collected from 365 respondents who listed as member in a virtual learning community in Indonesia. Respondents are randomly selected from 3360 total members of the observed community reached by online questionnaire distributed via google forms. The ideal response rate for online survey is 70% [33]. However, in its implementation, a response rate of 30% is acceptable [34]. Sample is calculated with a web-based sample calculator to ensure the accuracy [35]. With a population of 3360, a confidence level of 99% [36], and 5% margin of error, the required sample size is 556 respondents. In this case, 365 participants achieved over four weeks period, represents a 66% response rate. This has exceeded the number of respondents required in measuring psychometric validity, where 4-10 respondents required to validate each item [37]. Referring to this threshold, the minimum number of respondents needed to validate 32 indicators in this study is between 128 -320 respondents.

Social Aspects
Subjective norm recommendable -not recommendable Self-image valuable -non-valuable

Learner-learner interaction gets closer to learners -separates me from learners
Learner-instructor interaction get closer to teachers -separate me from teachers Instructor-instructor interaction getting teachers together -separate teachers

Statistical analysis
Construct validation of the scale was conducted through both Confirmatory (CFA) and Exploratory Factor Analysis (EFA) [38]. CFA was first used to test the model fit between acquired data and its five dimensions specification model [38,39] as proposed in previous study [6]. It was performed with the assistance of IBM SPSS AMOS 2.5 software and applied a combination of five goodness of fit indicators, namely X 2 which acceptable below 3, CFI, NFI, and TLI that should be greater than 0.90, and RMSEA is acceptable ≤ 0.10 [40]. EFA was used to identify the possible emergence of new structural model from existing data [38], through the assistance of IBM SPSS 24. This study used Principal Axis Factoring (PAF) Extraction because the data is not normally distributed [41]. Total variance explained and scree plot combination was used to determine number of factors to extract [42]. Only factors with eigenvalue > 1.00 and located on the upper side of the scree line were considered. Five factors were extracted and proceed with varimax rotation. Reliability was measured by the Cronbach coefficient with minimum reliability threshold 0.75 [43].

Results
Online questionnaires were distributed to 556 randomly selected respondents from a virtual learning community concentrates in teacher's capacity development in Indonesia. From that number, a total 365 responses returned with no missing data existed. Respondent demographical background were described in five characteristics, which are sex, age, institution, degree level, and origin. The majority of respondents were identified as female (69,3%), age range between 21 -30 years old (46,30%), teach in primary level (27,4%), hold a bachelor degree (75,5%), and live in Java Island (67,4%). Factor analysis results are described as follow.
Firstly, CFA conducted to test the suitability of existing data with the original conceptualization of learner experience [6], stated to cover five dimensions, the learner, instructor, system, and course characteristics, as well as the social aspects. Result of the study showed that only two of five goodness of fit indicators are acceptable, that are X 2 (1,698) and RMSEA (0,087). Three other indicators are posited slightly below the minimum threshold CFI (0,821), TLI (0,805), and NFI (0,772). These values indicate that the fit between data and proposed model were not adequate enough. In this case, a model modification was needed to be done in order to achieve better theoretical structure. However, after undertaking four modifications as described in Table 2, the model fit was still failed to achieve. To respond it, EFA was conducted to identify factor structure based on the existing data [44]. Secondly, EFA was carried out with which results shown the evidence of five factor structure generated. Each factor was over Kaiser's criterion of 1.00 eigen values and located upper the straight line of scree plot. The new factor model collectively explained 61,239% of the variance. The Kaiser-Meyer-Olkin (KMO) test result showed the value of 0,951, representing sampling adequacy. This was supported by the Bartlett (p) value 0,000 which showed the high correlation between variables/items. Thus, all variables can be involved in factor analysis [45]. Varimax rotation showed that each variable has a loading value above 0.32, which indicates that the factor model is satisfactory [42]. Reliability testing shows the Alpha coefficient value of 0.940. The value in the Cronbach's Alpha if Item Deleted column shows that each indicator has a value above 0.937, which indicates a very high reliability rate [46,47], so there are no indicators need to be deleted. The comparison between factors loaded in CFA and EFA, as well as the highest loading value for each item in EFA are presented in Table 3.

Discussion
The presented work in this paper is the first study dedicated to test and validate the FASER LX Scale. Although this study has applied an adequate number of sample (n=365) that help the study to present better quality data, the lack of previous validation and measurement study made it hard to compare and gain deeper insight from previous works. The findings contribute significantly in supporting existing literatures on e-learning and e-learning program evaluation, especially in Indonesian context. This study provides an Indonesian validated version of the questionnaire which can be further used and replicate for future researches. Findings from CFA stage showed that existing data is failed to fit the specification model proposed previously [6], even after four times modified. Five model fit indices were applied to determine the goodness of fit. However, only X 2 and RMSEA values showed acceptable fit, while the other three indicated slightly unfit model. Theoretically, model specification suggests that every observed variable can only correlate with one latent variable. So, the failure in performing acceptable goodness of fit indicated that the hypothesis of the factor structure as proposed in the model specification failed to be confirmed. This can occur when the assumed factors do not appear in the data [48], or the questions raised do not measure what is assumed to be measured through these questions [44].
In this condition, EFA was carried out to identify the factor structure which fit the data better [44]. The use of CFA as the initial step of testing, followed by EFA to identify the structure of the factors that display data like this has been done in previous studies on the Teacher Reporting Attitude Scale [28] and Disease Impact on Daily Life [38]. Findings of this stage indicated a newly factor structure consisted of five factors solution. Varimax rotation results showed that each variable has a loading value above 0.32. This indicates that the existing factor model is satisfactory [42].
The highest loading value for each variable is presented in the right side of table 3. Eleven variables were related to factor 1, thirteen variables on factor 2, four variables on factor 3, and two variables are proven to be related to factor 4 and factor 5 for each. Considering the loading values obtained, almost all variables can be explained by one factor. However, factor 4 and factor 5 tend to be a weak factor. These two factors only have two variables. Whereas a satisfactory factor ideally consists of more than two variables [49,50]. However, 32 of 32 variables showed loading values above the minimum threshold, so that no variables need to be excluded in the analysis. Cumulatively, the five factor structures explained 61.239% of the total variance. This means that more than a half variance has been revealed. However, there are still 38.761% variations that have not been explained in the study.

Conclusion
A validation study of FASER LX Scale was conducted in this study. Indonesian sample of virtual learning community did not achieve the model fit presented through the CFA. Additional EFA findings revealed five factors solution to represent better factor structure of existing data. Although the results were different, application of CFA and EFA stages validated that FASER LX Scale is suitable to be used in evaluating e-learning learner experience. 32 of 32 indicators loaded to five new factors were proven to be valid and reliable. Moreover, the findings showed that these indicators successfully explained 61.239% of total variance in data. It is possible to find the other 38.761% variations that have not been explained in the study. That is why, further examination and development of the FASER LX Scale is continuously needed to increase the total variance explained as well as the internal validity of the applied construct.