Q methodology to determine distinguishing and consensus factors (a case study of university students’ ecoliteracy on disaster risk reduction)

Q methodology is to measure the diversity of human subjectivity, such as perspective, mindset, preferences, opinions, and attitudes. It is one of the approaches applied in mixed methods research. This paper is part of a preliminary study that aims to examine the application of Q methodology. Fourteen undergraduate students from the Faculty of Geography, Universitas Gadjah Mada, participated in this study. Data collected with Q sorting techniques using the Q-Sortware, and analyzed with Principal Component Analysis using KADE desktop. Both are software used specifically in Q methodology research. The method implementation aims to determine the distinguishing factors and consensus of students' ecoliteracy components on disaster risk reduction. The research showed that Q-Sortware tends to facilitate researchers in collecting data. However, it is quite complicated and less practical for participants. Meanwhile, the KADE desktop has an attractive interface with a tutorial that makes it easy to use. By analyzing the case study, the result showed that the students' ecoliteracy component is represented by five distinguishing factors and two consensus statements, with cumulative variations explained by 68 %. This study is, however, limited by the performance of quantitative procedures without in-depth qualitative data interpretation.


Introduction
In 1935, William Stephenson discovered Q methodology. Traditionally, the use of Q methodology is often associated with quantitative methods because of factor analysis involvement [1]. Furthermore, this method receives much attention in social research because it has a strong statistical component to examine human subjectivity. It plays a major role in combining qualitative and quantitative techniques as well as also one among the significant approaches for mixed methods research [1][2][3].
Q methodology has been widely used since 1985 in psychology, communication, political science, health, environment, and other related fields [4]. It measures the diversity of subjectivity that represents an individual's feelings, experiences, opinions, perspectives, or preferences. It also identifies similarities, investigates the construction of a phenomenon, and the relationship between categories in a concurrent manner [5]. Yet, this method does not aim at measuring sample or population statistically but rather exploring the possibilities of various perspectives and consensus within the group about specific topics being investigated [6].
What This method does is clustering people and not clustering the variables. Therefore, the researcher must * Corresponding author: aldila.rahma@mail.ugm.ac.id have sufficient variables to find differences between participants, rather than aiming for a sufficient number of participants to determine the difference between variables as well as in factor analysis [3]. In the factor analysis, the number of samples is useful for controlling errors, while perspective is considered as an external component [7]. Q method, however, only utilizes a small number of participants in order to provide various viewpoints, comments, and opinions from various perspectives [5].
For the application, this method requires a Q grid for placing unit statements according to the participants' preferences. Additional tools are required, either traditional paper-based or softwarebased. Traditional Paper-based Q Methods (TPQM) typically using cartons and cards with statements [8]. Various studies with Q method approaches have utilized software-based tools such as Visual Q method (VQMethod), Q-Assesor, and FlashQ software for collecting data [8][9][10]. For data analysis, many studies employ different software such as QUANAL, PCQ for Windows Academic Edition, PQMethod, Ken-Q Online Analysis, and KADE Dekstop [6,11,12]. This paper is a preliminary study aimed to examine the Q methodology application to determine the distinguishing factor and consensus using university students' ecoliteracy as a case study. It includes the examination of using Q-sortware and KADE as software for data collection, and data analysis, respectively. However, the qualitative procedure is not included in this study. Hopefully, this study provides additional methodology information for research in the environmental science and disaster management field.

Data collecting technique
This research was carried out in December 2019 -April 2020. Fourteen undergraduate students from the Faculty of Geography, Universitas Gadjah Mada, took part in this study. In the study, Q Methodology employed generally consists of the following six steps [13,14]: 1. Developing a comprehensive set of statements (Q set) from various sources related to disaster risk reduction, climate change, and ecological understanding topics. This set of statements called concourse [15]. A total of 52 statements were arranged related to these topics, where the statements must be "opinion" and not a fact [4]. 2. Sampling (P-set). Q participant is a person selected to do the sorting process. Unlike the respondents represent the population, Q participants were chosen to represent the breadth of opinions in the population target. The most important thing is that the number of Q participants should be fewer than the Q statement. It usually uses a ratio of 3:1, as in three statements require only a single participant. 3. Statements distribution (Q sorting). Each participant was asked to distribute statements into the Q grid, based on their preferences. The ranking process is similar to the Likert scale format. The Q sort design is shaped like an inverted pyramid and applying forced distribution and using quasi-normal distribution ( Figure 1) [14]. This step used Q-Sortware online software [16]. Participants were invited by e-mail. 4. Conducting interviews with selected participants who have the "strongest" opinions. It is to explore and obtain a deeper understanding of the Q sort profile. 5. Analyzing and interpreting the data obtained.

Data analysis technique
Data from Q sorting were analyzed using KADE (Ken-Q Analysis Desktop Edition version 1.1.0) [17]. The steps are as follows: 1. Inputting Data 2. Correlation Matrix 3. Extraction Factor (PCA/ Principal Component Analysis) 4. Rotation factor (Varimax rotation) 5. Loadings Factor 6. Output Data This technique requires factor analysis to analyze groups of people who share their views. Each factor represents a unique viewpoint out of the research topic. Quantitative parts in Q methodology depend heavily on factor analysis and factor rotation. Factor analysis represents the heterogeneity of participant viewpoint. The final step is interpreting interviews result of participants. The results are verified, disassociated, and communicated to the related parties [18].
The reliability test of the Q method usually using a replicability type. However, because it does not aim to generalize the population, the reliability does not become a concern in this method [6,8]. This study only conducted a single measurement or one-shot. As for PCA, besides being used for factor analysis, it can also be used for the validity test. For one factor to be distinguished by another, the correlation coefficient score between them should be low [19].

Q-Sortware for collecting data in Q methodology
Collecting data (Q Sorting) in this study was using software called Q-Sortware. The researcher created and designed procedures in advance using the software and then sent the procedures and instructions to each participant's e-mail address of the invited participants, after which the participants obtained a personal link to do Q sorting.
There are two main steps in the sorting process using Q-sortware. First, participants classified statements of agree, neutral, and disagree. The aim is they took a "practice" before doing the actual Q sorting.
According to Figure 2, the screen consists of an instruction column on top and statement column below. The statements (in the yellow box) appeared one by one, and then participants were to drag statements one by one to the column below. After all the statements already moved, participants clicked the "continue" icon and proceeded to the second step. In this step, participants sorted the statements that have been classified previously (disagree, neutral, agree), into more detailed categories. The procedure is still the same as the previous step. Figure 3 is a screenshot of Q-Sortware second step. After finishing the entire sorting process, participants clicked the "continue" icon, and data would be sent immediately to the researchers' account. Data from Q-Sortware will be exported in .csv type files and analyzed using KADE.

KADE desktop for data analysis in Q Methodology
The data analysis process using KADE is rather simple. Figure 4 shows the interface of KADE. During the process, the researcher needs to input data files and follow the steps. KADE will calculate the correlation. Similar variables represented by correlations between them [20]. A high correlation score will construct a group of factors. The second step is factor extraction. KADE provides Centroid and PCA (Principal Component Analysis) methods that commonly used in Factor analysis (R). The Centroid is a preferred extraction method that is not too familiar in general statistical analysis [21]. Traditional factor analysis techniques as Centroid did not automatically provide mathematical solutions, so PCA was developed to assist [22]. The result of PCA is a group of factors. For the optimum model, the percentage of variance must be higher than 60. However, the percentage limit of variance for social research is acceptable between 50 % to 60 %.
The third step is the rotation. There are two rotation options in KADE, varimax, and hand rotation (judgmental). Varimax was chosen because it allowed the researcher to produce simple structures and maximize Eigenvalue for each factor. It can maximize the amount of variance from the factor extraction [22].
The fourth step is factor loading. The researcher selected individuals who can represent each factor called flagging. Representative individuals marked with (X) (correlation score > 0.5). Factor description and analysis are determined by participants who were flagged on certain factors.
Output is the final result of the analysis process. It shows the tables of factors. Researchers conducted interviews with representatives' participants to explain the reasons why they arranged those statements. Lastly, the researchers interpreted each factor and coded it.

Case study analysis results
The results of the university students' ecoliteracy case study analysis using KADE are shown as follows. Correlation score describes variables that have similarities-high score causing these variables grouping and construct factors [20].  Table 1 is the result of the PCA analysis where maximum number of factors that can be extracted are eight factors. Reducing and simplifying multivariate techniques data is applied in PCA. Thus, PCA extracts essential information from data tables and also analyzes observable structures of variables [12,22].
PCA was analyzed based on the Eigenvalue. It indicates how many variables can explain a factor. The default value of Eigenvalue is set on > 1 (less than 1 is ignored because it cannot construct a factor) [4,18,22]. Therefore, table 1 only shows a factor 1 to 5. The cumulative explained variance is 68 %. Five distinguishing factors represent the ecoliteracy component of the students.
Varimax rotation was applied in this study. Factors that can be rotated by Varimax have eigenvalues >1. Varimax is maximizing Eigenvalue on each factor [15]. Table 2 shows a result of varimax rotation. Flagging (X) is for selecting representative participants of each factor. Auto Flagging is based on the high significance of loading factor criteria (p <0.05) [12]. Selected participants are those with scores >0.5 because it meets the requirements of commonality based on the assumptions of factor analysis [24]. Loading factor is simply the correlation coefficient. A higher score of loading factor indicates a high correlation of participants with their represented factors [7].

Distinguishing statement
The final results of the factor analysis display the distinguishing statements. According to table 3, the relationship between factors is less than 0.399. Based on the Pearson r correlation coefficient classification, the coefficient interval between 0.2 to 0.399 indicates a low relationship level [24]. Also, a negative mark (-) indicates a negative correlation. The correlation between factors should not be high, because of the higher correlating factors, the lesser distinguishing factors (which can distinguish between factors) [19]. The composite reliability of each factor is above average (0.8). Factor 3 shows the highest score (0.95). A high-reliability above the average indicates the instrument is consistent and reliable. However, validity and reliability, not being a concern in Q because participants' subjectivity is more emphasized in this method [6]. Table 4 compares the characteristics between factors. Five participants represented factor 3. The factor is determined by some participants who "load" in it. One factor that has at least 5 participants who define it, then the reliability, can reach 0.95 [19]. Output analysis also shows a table of distinguishing statements (p <0.05). The table will later be useful in interpreting the results. Due to page limitations, this paper only provides 1 example out of 5 total tables.  Table 5 is the distinguishing statement of factor 3. Z score describes the place of each statement in the Q sort table [3]. The most extreme z value (its position increasingly towards the edge of Q sort table) is the most useful in interpreting factors. According to Table 5, 3 statements distinguish this factor from another. A total of 5 participants agreed with statement 6 and disagreed with statement 18 and 23.

Consensus statement (Agreement)
Consensus statements are useful to determine statements agreed by all participants. Consensus provides insight into how participants perceive the problems, express their logic, and put the essential issues in the context. This method intentionally involves people with different points of view to fill the vital information gap [14]. According to table 6, two statements become consensus. All participants agreed to statements 10 and 15. These two statements were not statistically significant at both p< 0.01 and p<0.05.

Discussion
Q statements usually consist of 30 to 60 units as representative of a concourse [3]. The concourse obtained from primary sources (direct conversations, participant interviews, and observations) and secondary sources (literature, popular media) [4,5]. It depends on the researcher's objectives and considers whether it can explain the perceptions of all participants. The unit statements in this study were arranged to explain students' ecoliteracy components on DRR. It consists of affect, knowledge, cognitive, and behavior components [25]. P sets or participants in the Q method, are not the same as the "sample size" in studies with other methods. Participants are chosen because they have a different and well-formed opinion. People with a well-formed opinion will find it easier to do Q sort and tend to produce reliable results [13].
Q-Sortware is a software created by Alessio Pruneddu, with intellectual property rights belongs to the University of York United Kingdom [16]. The researcher made registers first on Q-Sortware website to create an account, then arrange the instrument. Using web-based software ease the researchers in the sorting process, rather than using the Traditional Paper-Based Q Method (TPQM). Practically, TPQM requires direct meetings with participants. Also, it has greater difficulty and requires more resources [8].
On the other hand, Q-Sortware helps researchers to collect data more efficiently. Once the participants finished the sorting, data can be submitted immediately to be analyzed. Researchers can also get information on participants who have completed the sorting, date of data submission, and time needed for the process.
The disadvantage of using Q-Sortware is that it may be rather confusing and impractical for some people. This software is web-based, so it requires stable internet access. It is also in the form of a desktop-based software that cannot be operated on a smartphone. In operating the instrument, it requires participants to drag statements into columns. This process cannot be done on a smartphone, where people more often use it for practical reasons. Participants have to use a laptop or PC to conduct the sorting. This software also has a slightly complicated interface. The instruction column is too small, so that most of the participants might not be aware and tend to miss the instructions. It merges instructions and sorting columns in only one screen page. Figure 3 gives the appearance of the interface, which is too "crowded." Participants also met difficulties in the sorting process. They never did sort with forced distributions before. They were more familiar with the open questionnaire method, such as Likert scale or checklist questionnaire. So even though they were doing online sorting, the researcher must watch the process to minimize the error. The obstacles like this become a challenge in collecting data using Q-sortware.
Other software that can be used for sorting procedure is Visual Q method (VQMethod). VQMethod allows the addition of new dimensions such as video and audio replacements for statements. VQMethod is more thorough and provides more reliable data compared to TPQM [8]. Another study was carried out using Q-Assesor's paid licensing software. It has complete validation features. This software allows the researcher to send instructions to ease the sorting steps. The researcher obtained data through an e-mail sent by participants [9]. Another study used FlashQ software. This software also consists of a 2-stage process like Q-Sortware, but it looks much simpler. Participants can directly arrange statemet cards into the Q sort table as if they were arranging a puzzle [10]. However, FlashQ software requires particular expertise in creating HTML (HyperText Markup Language) versions. So, it can be concluded that using software is helpful to facilitate the sorting process and making it more interactive.
The data analysis process in this study used KADE. KADE is a desktop-based Q Methodology analysis application that can run in Microsoft Windows, Apple OS, and Linux OS. KADE interface looks much simpler and easier to operate and also provides interactive features that are helpful for the analysis process [17]. It also provides templates, demo data, and manuals, to help researchers understand its use. This software does not require an installation process (portable) and free from the internet. KADE only analyzes statistically, and the results are in the forms of numbers. The researchers need to interpret the table meaning for qualitative analysis. There is no specific strategy to interpret the formed factors as it depends on the researcher's objective [19].
Other commonly used software for Q method analysis is QUANAL, PCQ for Windows Academic Edition (paid license), PQMethod, Ken-Q online Analysis, and KADE Desktop edition [11]. Among those software, PQMethod and KADE are often used extensively for data analysis. PQMethod has the same analysis steps as KADE, but its operating system based on DOS [6,12].
Q methodology can determine variations in students' epistemological views. It can also avoid a lengthy interview process, which is typical of qualitative research [15]. This study helps to construct the concourse from the statements. Data interpretation is a part of the hermeneutic process involving an interpretive perspective from the researcher. However, it is still limited by the participant's subjective factors during the sorting process [5].
The result of the Q method is that there are diverse points of view on a topic that may be operant or measurable. The results cannot become a representative of the sample or population. Reliability statistics work to generalize samples to be representative of a population, not become a concern in this method [7].

Conclusion
Q Methodology is useful in research involving subjectivity, especially in social research. The steps start from arranging unit statements, determining participants, sorting process, and data analysis. The analysis process consists of correlation, factor analysis, rotation, loading, flagging, and data interpretation steps.
Q-Sortware facilitates the researcher to collect participant data. However, for the participants, it is quite tricky, less practical as it cannot be operated on a smartphone, and has a "crowded" looks interface. Meanwhile, KADE has a simple interface. KADE also has an easy-to-understand demo data and manual, making it easier for the researcher to do data analysis.
According to the study, there are five distinguishing factors with the composite reliability of each factor between 0.89-0.95 and also two consensus statements agreed by all participants. Cumulative explained variance of extraction factors is 68 %, which is acceptable for social research.
This preliminary study is limited to the procedure for operating Q methodology and its statistical analysis. Also, it reveals a problem that occurs in the process. Factors interpretation, which is the part of the qualitative analysis, does not include in this paper. The researchers recommend further research to conduct replicability tests to ensure the reliability of Q Methodology. Besides, use comparisons with other tools, either traditional paper-based or software-based Q method for Q sorting process.