THE USE OF A SCORECARD TO EVALUATE THE PUBLIC’S COMPREHENSION OF COVID-19: A PILOT STUDY

The rapid spread of information and infodemic might result in public confusion and hinder the handling of the COVID-19 pandemic. Public comprehension of COVID-19 as part of health literacy is an important determinant to filter hoaxes from facts. Therefore, a scoring card called the Karlivid (the COVID-19 literacy and public vaccination scorecard) was developed to evaluate the individual’s comprehension level of COVID-19. A pilot study was conducted with this scoring card. The participants were recruited via consecutive random sampling by using emails from the researcher’s co ntact list (n=92). A total of 78.3% of the respondents were considered to have an adequate comprehension level. Approximately 77% of all respondents agreed that this card could help them know their comprehension level, 81.5% agreed that this card could improve their comprehension, 81.5% agreed that the items in this card could help them screen facts from hoaxes, and 81.5% agreed that the language used was easily understood by the laypersons. Therefore, the Karlivid is a valid and reliable scorecard that can be used to evaluate public comprehension of COVID-19. Most of the respondents also had a good level of comprehension of this assigned topic.


INTRODUCTION
Literacy can be defined as an individual's capability to write and read as well as the ability to synthesize information and knowledge in managing daily life. Since the development of the internet, reliable information can be more easily obtained by filtering correct the sources. Digital literacy has become a necessity to filter hoaxes and falses (Sianturi et al., 2021). Eisenberg, Johnson, and Berkowitz (2010) created the information literacy model called "The Big 6 Skills". This is formed by formulating the problems, identifying the information needed, developing a strategy of information mining to determine good information sources, organizing and presenting the information, and evaluating the efficiency and effectiveness of the mined information. Additionally, Shapiro and Hughes (1996) in their article titled "Information Literacy as a Liberal Art" stated that information literacy can be used as a foundation to build a literacy model that can be adjusted to users' needs (Shapiro & Hughes, 1996). In Indonesia, literacy on COVID-19 and its vaccination still need to be nurtured. Previous studies conducted in Semarang (n=400) and the greater Jakarta area (n=839) have shown that public compliance with health protocols and government regulations is still quite low, causing outbreaks in several areas (Erawati, 2021;Rosha et al., 2021). Some people are hesitant to receive the COVID-19 vaccines due to various reasons, i.e., the conspiracy theory of certain parties to monopolize the vaccine and the potential side effects of vaccination (Akther & Nur, 2022;Rahmawati et al., 2021;Z. Yang et al., 2021). These reasons lead to the public's distrust of the COVID-19 vaccine, thereby hindering government targets to achieve wider vaccination coverage to achieve herd immunity (Batrinca & Treleaven, 2015;Okan et al., 2020).
These phenomena are believed to arise from the unpredictable nature of COVID-19 and an ongoing pandemic of quickly-spreading misinformation and hoaxes-so-called infodemic-via various platforms including the internet. Furthermore, news on how people who have been vaccinated could still be infected may alter the perception of the COVID-19 vaccination (Ifroh & Asrianti, 2020;Sagan et al., 2021;Sridhar, 2020).
Conversely, the growing number of novel phrases that arise during the pandemic can be posed as additional challenges for laypersons, i.e., positivity rate, confirmed antigen swab and/or RT-PCR, variants of mutations, m-RNA basedvaccine versus weakened virus based-vaccine, social and physical distancing, spike proteins, SARS-Cov2 virus, and cytokine storm (Adhikari et al., 2020;Hu et al., 2021). Validated information must be obtained via government channels and/or scientific journals. However, the availability and growing development of research on these topics have been vast and dynamic. Thus, people need to know how to correctly screen the information they obtain. Therefore, this pilot study was conducted to determine the validity and reliability of the self-developed scorecard (Karlivid) as a measuring tool for people's comprehension of COVID-19 (Harapan et al., 2020;Rachmani et al., 2019).

Study design
This study used a cross-sectional survey to determine the respondents' comprehension of COVID-19. Data were obtained via the self-filled scoring card. The development of the scoring card and the analysis of the responses of respondents were performed between October 2021 and February 2022.

Sample
The respondents were openly recruited via email by consecutive sampling methods. The email addresses were obtained through the researcher's contact list. The inclusion criteria were healthy adults aged more than 18 years old, able to read and comprehend the survey, native speakers of Bahasa Indonesia, have internet access, and agree and consent for the results to be published anonymously. In this study, a sample-to-item ratio of 10:1 was used with a minimum of 90 participants selected (Memon et al., 2020).

Instrument
A scorecard containing 9 questions related to COVID-19 and its vaccination was developed by a team of medical doctors and students. The contents of the instruments were developed from several published works of the literature (Adhikari et al., 2020;Erawati, 2021;Harapan et al., 2020). All items on the scorecard were selected to assess the public's health comprehension of COVID-19, where content validity and face validity were carefully performed (Adhikari et al., 2020;Erawati, 2021;Harapan et al., 2020). A total of 2 questions were scored from 2-0 (0=disagree, 1=neutral, 2=agree), while the other 7 questions were scored from 0-2 (0=agree, 1=neutral, 2=disagree). The 9 questions were formulated to capture the respondents' comprehension of COVID-19 and its vaccination; each question was pre-tested and thus could be understood by laypersons and completed by them anonymously. The total point was then summed; an individual with a total score of 12 or more was considered to have an adequate comprehension level, while a lower score was considered inadequate (Wijaya & Kloping, 2021). The language used in this card is Bahasa Indonesia as the main target was to provide a self-assessment of an individual's comprehension of COVID-19 among Indonesian people. The English translation is shown in Table 1.

Table 1. The scorecard on COVID-19 and its vaccination information literacy (a self-sufficient method to gauge your comprehension of the aforementioned topics) No.
Question Value Score 1 Comorbidities are diseases and/or abnormal conditions, such as being overweight, obesity, hypertension, diabetes mellitus, cancer, and autoimmune disorders, and these might affect the illness severity of COVID-19.
Agree=2, Neutral=1, Disagree=0. 2 Information on COVID-19 and its vaccination is dynamic and follows updated situations and conditions. 3 It is not mandatory for individuals who have been confirmed with COVID-19 to uphold health protocols due to acquired immunity.

4
Vaccination makes a person immune to the disease. 5 The infection risk of COVID-19 does not affect the number of people who have been vaccinated. 6 The brand of the COVID-19 vaccine determines its effectiveness due to the country that produced the vaccine. 7 COVID-19 vaccination is not needed for people who had influenza and/or pneumonia vaccination. 8 It is not mandatory for individuals who have been vaccinated with COVID-19 to uphold health protocols (wearing a face mask, washing hands, physical distancing, and avoiding crowds) due to their body's immunity. 9 Any information on COVID-19 and its vaccination from the internet is correct. TOTAL (More than 12: good, less than 12: insufficient) Four questions were asked to determine the usefulness of Karlivid, i.e., whether they agreed that this card could help them determine their comprehension level, whether they agreed that this card could improve their comprehension, whether they agreed that the items in this card could help them screen facts from hoaxes, and whether they agreed that the language used were easily understood by laypersons.

Data collection
The electronic version of the scorecard was uploaded as a Google Form, and the hyperlink was attached to the with national-based coverage called PeduliLindungi application; as well as peer-reviewed scientific articles on the Pubmed and Science Direct databases. Thus, it is expected that internet users who are looking for information related to COVID-19 can find our Blogspot page and fill out the scorecard.

Data analysis
Incomplete responses or multiple takers were excluded. The statistical analysis of the results and survey items was conducted using SPSS 17.0 (IBM, Chicago, IL, USA). Descriptive analysis was performed on the sociodemographic aspects of all respondents, in addition to their individual opinions on the Karlivid. The frequency of each item's response was also calculated for males and females. A Chisquare test was performed to determine any differences between sex groups with the significance level defined as a p-value of <0.05 (SPSS 17.0, USA).

Ethical considerations
This study was targeted as a community service activity and ethical clearance was obtained from the Faculty of Medicine, Universitas Airlangga No.145/EC/KEPK/FKUA/2021, dated 26-07-2021. Informed consent was obtained by clicking the agreement button.

RESULTS
The development of Karlivid as a scorecard to evaluate individual comprehension of COVID-19 took approximately one month before its upload to a research blog (https://COVID-19fkua.blogspot.com/). Public comprehend sion of COVID-19 is vital for individuals to understand their health situation during the pandemic and act accordingly. In this study, the age of the respondents was between 18-30 years. Sex difference has been reported to affect these values (Flor et al., 2022;Galasso et al., 2020;Rosha et al., 2021) and therefore analyzed in our pilot study. All items have passed the validity test (n=76, r table of 0.05 is 0.223) and the reliability test (n=76, r table of 0.05 is 0.223, Cronbach's alpha > 0.6). All respondents agreed that the items were easily understood by laypersons (81.5%). After completion of the scorecard, the respondents claimed that they understood the importance of COVID-19 and its vaccination (77%); the items in the scoring cards helped them to become aware of the importance of news screening on the assigned topics (81.5%); and that the scorecard was useful for increasing their comprehension level on COVID-19 and its vaccination (81.5%). After these steps, we recruited respondents for the pilot study to evaluate public comprehension levels via Karlivid. The total number of respondents was 92 (29 males and 63 females).
It was observed that 73 respondents had an adequate level of health comprehension (79.3%). The frequency of each response for each item was analyzed to understand potential differences in the comprehension levels amongst these respondents. For item number 1 (comorbidities are diseases and/or abnormal conditions such as being overweight, obesity, hypertension, diabetes mellitus, cancer, autoimmune disorders, and these might affect the illness severity of COVID-19), 76% of respondents agreed; for the 2 nd item (information on COVID-19 and its vaccination is dynamic and follows the updated situations and conditions), 90% agreed; for the 3 rd item (it is not mandatory for individuals who have been confirmed with COVID-19 to uphold the health protocols due to acquired immunity), 16% agreed. For item number 4 (vaccination makes a person immune to the disease), 32% agreed; for item number 5 (the infection risk of COVID-19 does not affect the number of people who have been vaccinated), 40% agreed and 45% disagreed; for item number 6 (the brand of COVID-19 vaccine determines the vaccination effectiveness due to the country that produced the vaccine), 42% agreed. Next, for item number 7 (COVID-19 vaccination is not needed for people who had influenza and/or pneumonia vaccinations), 67% disagreed; for item number 8 (It is not mandatory for individuals who have been vaccinated with the COVID-19 vaccination to uphold health protocols, e.g., wearing face masks, washing hands, physical distancing, and avoiding crowds, due to their body's immunity), 94% disagreed; for item number 9 (any information on COVID-19 and its vaccination from the internet are correct), 86% disagreed. Furthermore, there are no statistically significant differences in the response type of each item between men and women (p>0.05). The analysis of the participant's responses is shown in Table 2. Any information on COVID-19 and its vaccination from the internet is correct.

DISCUSSION
In this study, a higher percentage of the respondents had adequate health comprehension of COVID-19 and its vaccine compared to those with inadequate comprehension. This is in line with a study conducted during the early pandemic period of the COVID-19 pandemic in Indonesia (Triyanto & Kusumawardani, 2020). However, contrary to a previous survey about COVID-19 health literacy conducted in East Kalimantan (Ifroh & Asrianti, 2020), we found no significant differences in the type of responses between men and women who answered each item in the scorecard; both groups generally had a good understanding of the subjects. In terms of the usefulness of the scorecard, most respondents agreed that this card contained items with easily understood language and could raise their awareness of the importance of filtering any information regarding COVID-19 and its vaccination. Most of them also agreed that by filling in this card, they obtained a general idea of their health comprehension level and gained some good information as well.
Literacy regarding COVID-19 and its vaccination can be affected by several factors, including education level and the individual's experience on the related topics. Moreover, the language and content of the information also play a vital role. Other factors determine this literacy, such as the individual's environment, their technology adeptness, their personal needs and interests, socio-culture, habits, and beliefs (Li & Liu, 2020;Pechrapa et al., 2021;P. Yang et al., 2021). A reading habit can be developed to help people raise their literacy on certain subjects, including COVID-19 and its vaccination. Nowadays, for many people, the internet is their main source of information. While almost unlimited data might be sought via the internet, filtering facts from hoaxes can be quite a challenge. Furthermore, the confirmation of information to authority and/or field experts may be difficult, especially for the elderly and children (Brashier & Schacter, 2020;Herrero-Diz et al., 2020).
In this study, we explored balanced opinions on whether the brand of certain vaccines would determine their efficacy. However, this result must be taken wisely due to the relatively small sample and the limited variables studied. People's opinions might also change due to the availability of news and data they obtain, and the data on the COVID-19 vaccine is still growing.
Information literacy is a part of a basic process that is required to produce a high-quality workforce in various departments including in the social, political, economic, and cultural areas (Tilvawala et al., 2009). At the individual level, this soft skill is particularly useful because adequate health literacy on COVID-19 and its vaccination would determine how a person reacts in daily life. For example, it would affect how they would uphold health protocols as a form of protection for themselves as well as for others and to become vaccinated despite what other people may think or say. This attitude could be spread to others and might be good for the growth of new values, a new way of thinking, and a new way of life (Saad-Roy et al., 2020).
This study found that the larger part of respondents had a good comprehension of how comorbidities may lead to worse COVID-19 symptoms. They mostly agreed that the dynamic nature of information regarding COVID-19 and its vaccination must be taken into consideration, that vaccination could raise their immunity against COVID-19, and that health protocols must be upheld by all, including those who have contracted the disease and those whose have been vaccinated. However, lots of these people have not yet comprehended the importance of vaccination as a vital part of the public effort to reduce the risk of infection from COVID-19, thus decreasing the positivity rate (WHO, 2021).
In Indonesia, the distribution of the COVID-19 vaccine has been a national project that started in February 2021. This activity is provided via health facilities in all district areas in Indonesia, with priority given to health providers and public servants, followed by all clusters of the community and children up to six years old (Isbaniah et al., 2020). By January 2022, according to the Ministry of Health's report (Kemenkes RI, 2022), approximately 180 million 1 st doses and 125 million 2 nd doses of the COVID-19 vaccine have been administered in Indonesia. At the beginning of January 2022, the third dose of booster vaccine has also been provided to the public. Although this may seem like a large statistic, the end target is a long way to go, as the country's total population reaches approximately 270 million people spread over the archipelago. The vaccination was given for free as part of the government's program to combat the disease (Rahmanti et al., 2021;Setiati & Azwar, 2020). By knowing the information literacy levels on these topics, people would be more aware of the situation and hopefully would act accordingly.
This was a pilot study that served dual purposes, namely, to build the Karlivid scorecard with items that had acceptable validity and reliability levels, and to also determine the public's initial comprehension of the COVID-19 virus on a small scale.
There are several limitations to our study. Firstly, this research has a relatively small number of respondents due to its nature as a pilot study. The content of the blog is also still evolving to include various types of educational methods on these subjects. Moreover, a consecutive random sampling approach was applied and the majority of the respondents were between the ages of 18-30 years old. In this study, the difference of each item was analyzed only between sexes, while other factors, i.e., age group, educational level, economic status, occupation, and digital literacy level, could also be taken into consideration for its relationship with health literacy.

CONCLUSION AND RECOMMENDATION
We succeeded to develop the Karlivid scorecard with good validity and reliability levels. This scorecard is novel in Indonesia and was uploaded to a research blog. In this pilot study, we observed good comprehension levels regarding COVID-19 among the majority of the respondents, which suggests a positive value for the current quality of public health literacy. Future studies with a larger number of respondents are necessary to evaluate the Indonesian people's health literacy on COVID-19.