Reliability and reproducibility of an Italian questionnaire on “Knowledge of high social impact Eye Diseases” (KED-IT)

Participants and setting

An opportunistic sample was recruited. Subjects were contacted in the waiting rooms of the Ophthalmology Clinic and in the administrative offices of the Umberto I Hospital in Rome (Italy). The questionnaire was offered only to subjects >18 years of age who were Italian native speakers. Subjects studying or working in ophthalmology, orthoptics or optics were excluded.

Questionnaire development content and face validity

To ensure construct validity, each item in the questionnaire was assessed by five experts in the fields of epidemiology, community health, and ophthalmology. The experts had an academic and clinical background (Sapienza University of Rome, University San Raffaele University of Rome) or were executive officers of an experienced non-profit eye care organization operating in Italy (Italian Branch of the International Agency for the Prevention of Blindness - IAPB Italia ONLUS); the final supervision was provided by a Full Professor of Ophthalmology from Sapienza University of Rome. The experts assessed the adequacy of questions on macular degeneration, glaucoma and diabetic retinopathy based on the 2005 survey of public knowledge, attitudes, and practices related to eye health and disease KAP (National Eye Institute & Lions Clubs International Foundation, 2007) and proposed new questions on keratoconus.

The questionnaire was reviewed to ensure that (i) the questionnaire reflected the theoretical framework used, (ii) the items were appropriate for the construct being measured, and (iii) the items adequately covered the main study objectives. Feedback was recorded on a five-point rating checklist.

The KED-IT consisted of three sections:

• Sociodemographic: three items (gender at birth, age, education level).

• Hearsay: two items “Have you received information about eye diseases in the past year?” (y/n) and where (newspapers, television, internet, social media, parents/friends, etc.).

• Knowledge: 14 items on four eye diseases with high social impact. Eleven items were taken from the KAP 2005 (National Eye Institute & Lions Clubs International Foundation, 2007) covering three eye diseases with a significant global impact: glaucoma (Tham et al., 2014), macular degeneration (Fleckenstein, Schmitz-Valckenberg & Chakravarthy, 2024), diabetic retinopathy (Wong & Sabanayagam, 2019). The KAP2005 was written in English, so the items included in the KED were translated into Italian using the Brislin back-translation method by two translators who were linguistic and subject matter experts. Their translations from English were compared to obtain a common Italian version. A third researcher also translated the common version into English to assess the overlap with the original English version. KED-IT also includes three items on a fourth eye disease, keratoconus, which was not present in the KAP2005 and is the leading cause of corneal transplant in Italy (Frigo et al., 2015). All 14 items ask about knowledge of risk factors, possible early symptoms, risk of vision loss and possible treatments. The 14 items allowed responses in the form of “false”, “true”, or “not sure”. A score of “1” was given for a correct response, and a score of “0” was given for an incorrect or “unsure” response. The KED-IT is shown in Table 1.

The questionnaire was pre-tested on 5 respondents to ensure consistency of wording. Some words were rephrased in layman’s terms (e.g., “blindness” or “visual damage” instead of “vision loss”, “to keep under control” instead of ”to treat”) and a simple definition of central vision was added to better explain the symptoms of macular degeneration. A direct true/false question was preferred to direct assertion to actively engage the participant and encourage a thoughtful response. A yes/no format could have been used as well, as long as both options are similar and understandable in the Italian language.

The revised items were pre-tested again with the same respondents to ensure their understanding of the revised wording. Respondents who participated in the pre-test were not included in the pilot test.

Study administration

Researchers introduced themselves to potential participants, who were informed of the purpose of the pilot study. Written informed consent was obtained from those who agreed to participate. The questionnaire was self-administered, with assistance from the researchers to clarify items and ensure that the questionnaire was fully completed. The questionnaire was administered twice, 5 to 8 days apart (T0 and T1).

Statistical analysis

Data were analysed using PSPP version 1.6 for Windows.

Participants’ demographic background and practice information were analysed using descriptive statistics. Data were presented as means (SD) for numerical and normally distributed data, or as frequencies and percentages (%) for categorical data. The normal distribution of continuous variables in the sample was assessed using the Kolmogorov–Smirnov test.

Difficulty and discrimination indices were assessed to identify items that should be acceptable (Mitra et al., 2009).

The difficulty index (DI) was estimated according to the Sim & Rasiah approach (Sim & Rasiah, 2006). According to Mitra et al. (2009) the difficulty index value was considered acceptable between 30% and 70% (Rahim, 2010; Mamot et al., 2021).

The discrimination index indicates how well an instrument discriminates between high and low scorers (Sim & Rasiah, 2006). Its computation was performed according to Rahim (2010). The interpretation of the discrimination index score can vary, in the present study was considered low discrimination ≤0.19; good discrimination ≥0.2 according to Rahim (2010).

Regarding the internal consistency reliability test of the KED-IT questionnaire, the KR-20 reliability coefficient, which is a special form of Cronbach’s alpha, was conducted because the responses of the questionnaire were dichotomous in nature (Tan, 2009). KR-20 reliability coefficients of <0.50 were interpreted as low, 0.50−0.80 as moderate, and >0.80 as high (Tan, 2009).

An overall KR-20 coefficient and KR-20 coefficients when an item was deleted were calculated.

For test-retest reliability, at least 26 participants (50%) should complete the questionnaire again within 5–8 days. Due to the dichotomous nature of the items responses, the robust statistical method Cohen’s Kappa statistic (K) was used to estimate the reliability (Sim & Wright, 2005). The K result was interpreted as follows: ≤ 0 no agreement; 0.01−0.20 none to slight; 0.21−0.40 fair; 0.41−0.60 moderate agreement; 0.61−0.80 substantial agreement; 0.81−1.00 near perfect agreement (Sim & Wright, 2005). A K value for each item and an average kappa value were calculated.

A p-value <0.05 was considered statistically significant.

Sample Size

The sample size for the internal consistency reliability analysis using the KR-20 was calculated based on the subject to item ratio, using a subject to item ratio of 4:1 (Tan, 2009). Therefore, a minimum of 56 participants was required (14 items × 4 = 56).

In terms of test-retest reliability, the sample size for testing Cohen’s kappa agreement was determined to be 26, which is 50% of the total number of participants (Bujang & Baharum, 2022).

Ethics

The study was conducted in conformity with the Declaration of Helsinki (The World Medical Association, 2008).

The protocol of this study was reviewed and approved by the Institutional Review Board of Sapienza University of Rome, Department of Psychology (Approval number: 742).

Respondent characteristics

68 individuals were invited to participate; 60 of them agreed to participate (88.2%). The mean age of the 8 individuals who did not participate in the study was 41 years (SD =11), and 5 (62.5%) were male. The age distribution of the respondents was Normal (the Kolmogorov Smirnov test’s significant level was p = 0.459), the mean age was 41.6 (SD = 15.6) years, and more than half (58.3%) were female. 45% of the respondents had completed higher education (university degree) and 55% had completed high school.

The mean time to complete the KED-IT, as assessed by the questionnaire administrator, was 7 min ± 2 min.

Reliability, difficulty and discrimination of KED-IT

Table 2 shows the distribution of correct answers, the Difficulty Index (DI) and Discrimination Index for each KED-IT item.

In terms of DI, 1 item was high difficulty, (“Does glaucoma have any early symptoms that I can notice?”); 7 items were acceptable (“Can vitamins and zinc help with macular degeneration?”, “Can macular degeneration run in families?”, “Can a person have macular degeneration and not know it?”, “Central vision is what allows you to read, recognize people’s faces, and see the food on your plate. Knowing this, do you think macular degeneration can affect central vision?”, “Can keratoconus seriously affect vision?”, “Does keratoconus have any early symptoms that I can notice?” and “Are there therapies for keratoconus?”) ; 6 items were easy (“Can glaucoma cause blindness?”, “Can visual damage caused by glaucoma be prevented?”, “Are there therapies available to keep glaucoma under control?”, “Do people with diabetes have a higher risk of eye disease than people without diabetes?”, “Is it true that people with diabetes should have an eye exam with pupil dilating drops at least once a year?” and “Is it true that there are NO medications or other therapies to control eye disease caused by diabetes?”).

A good discrimination index (>0.19) was reported by 11 out of 14 items. Item 2 (“Does glaucoma have early symptoms that I can notice?”), item 3 (“Can visual impairment caused by glaucoma be prevented?”) and item 4 (“Are there therapies available to keep glaucoma under control?”) reported a high discrimination index of 0.08, 0.10 and 0.13, respectively.

The KR-20 internal consistency reliability coefficient of the 14-item KED-IT questionnaire was 0.878 (Table 2). The KR-20 alpha coefficient was good. All items showed the same contribution to the overall reliability: when one item was removed, the KR-20 was essentially the same: it ranged from 0.863 to 0.889.

Reproducibility of KED-IT

The 60 subjects who answered the questionnaire (T0) were asked to answer it again 5 to 8 days later (T1). The test-retest was fulfilled for 55 subjects with a response rate of 92%. 5 subjects were no longer available, and it was not possible to administer the T1 questionnaire: they were four females, and the mean age was 61 years (SD =5.1 years).

The results of the test-retest reliability of the KED-IT questionnaire are shown in Table 3.

Correlation coefficients between the two administrations (T0 versus T1) ranged from 0.50 to 0.99, with 8 out of 14 items showing more than 0.7, indicating very good reproducibility (Table 3).

The average kappa value for all 14 items of the KED-IT questionnaire was k = 0.747, of which 3 items showed moderate agreement (k: 0.4−0.6), 6 good (k: 0.61−0.8) and 5 excellent agreement (k > 0.8).