A novel approach to improving colonoscopy learning efficiency through a colonoscope roaming system: randomized controlled trial

Recruitment

We recruited participants in hospitals by posting notices about the experiment.

Inclusion criteria were as follows: (1) voluntarily participate in the study and sign the informed document; (2) be able to complete gastroscopy independently without systematic study of colonoscopy; (3) own computer that can support CRS operation and have basic computer operation skills; and (4) complete the pretest questionnaire.

Exclusion criteria (those who met one of the criteria were excluded): (1) quit during the experiment or fail to complete the corresponding tasks as required; (2) the score of the pretest was too high (the accuracy rate was higher than 80%) because the CRS was designed for beginners who had not previously mastered colonoscopy knowledge (Nakanishi et al., 2017).

The experiment was carried out from May 2020 to May 2021, with Fig. 1 illustrating the flow. Following the 30-minute pretest by the 84 eligible volunteers, elimination was performed based on the exclusion criteria. IBM SPSS version 24 was employed to facilitate the random assignment of the qualified participants into two groups: the e-learning group (EG) and the control group (CG: non-e-learning). At the on-campus Endoscopy Centre of the Second Affiliated Hospital of HMU, all enrolled students completed 2 months of theoretical and practical training in colonoscopy. In the process, each EG participant received an account number and corresponding password to facilitate logging into the CRS during the trial. Following the training, the study subjects completed the posttest and colonoscopy examination assessment two months later. The EG participants also completed a satisfaction questionnaire regarding CRS. This study was approved by Harbin Medical University Institutional Research Board (Approval Number HMUIRB2019008PRE). Participants gave written informed consent to participate in the study.

Developing CRS and its description

We developed CRS in collaboration with Harbin University of Science and Technology. The specific development process is shown in Fig. 2. First, a 3-dimensional model (Figs. 3A–3B) of the LDT was obtained after a review of various data sources (mostly photographs and films of colonoscopy examinations from the Digestive Endoscopy Centre of the Second Affiliated Hospital of Harbin Medical University (HMU)). The initial structure of the model was created using Autodesk Maya software (Figs. 3C–3H), while the detailed structure of the colon cavity was finalized with ZBrush software (Fig. 3I). The 3D model was then transformed into an operating scene to simulate and render the physiological functions of the LDT. Second, the user interface of the system, including the log-in interface (Fig. 4A), scene interface (Fig. 4B), exit interface, etc., was designed using Unity 3D software. The C# programming language was used, giving the CRS the functions of interacting, cognition, and roaming (Figs. 4C–4G). CRS was built to be bilingual in Chinese and English and consists of two modes: focal trigger mode (mode A) (Fig. 4H) and structural exploration mode (mode B) (Fig. 4I).

Learning outcome evaluation strategy

In the 1950s, Kirkpatrick designed a well-known framework for the evaluation of learning effects (Giray, 2021) that could be used to assess the effectiveness of the CRS (Alliger & Janak, 1989; Ward et al., 2001; Galloway, 2005). The framework encompasses these four aspects of assessment: reaction, learning, behaviour, and results (Abdelhai et al., 2012).

Reaction level scoring

The reaction level was assessed through a satisfaction questionnaire. We used a questionnaire developed by Wang (2003), which has been widely cited in evaluations of e-learning satisfaction. This questionnaire has been shown to have a reliability (Cronbach’s alpha) of 95. Since the current CRS does not contain the function of online mutual communication, the evaluation section of the ‘learning community’ in the original questionnaire was removed in this experiment. The 20 questions were divided into four modules: content satisfaction, interface satisfaction, test satisfaction and personalized learning satisfaction. A 5-point Likert scale was applied to score each question (strongly disagree 1, disagree 2, neutral 3, agree 4, strongly agree 5). The students’ evaluation of each module and the overall evaluation were calculated. All EG participants finished the satisfaction questionnaire after training completion. Because of the COVID-19 pandemic and the paperless procedures, we created quick response (QR) codes for the participants to answer the questions. The QR codes and the questionnaire are provided in File S1.

Assessing the learning level

The most routine and reasonable way to assess learning levels is to compare test scores before and after the intervention. As mentioned above, the pretest was performed prior to colonoscopy learning, and the posttest was performed following the 2 months of training. To ensure that the pretest and posttest are of the same difficulty, both tests have the same questions, but the order of the questions is different (Oliveira, Mattos & Coimbra, 2017). All of the colonoscopy images used in the questions were from the Department of Gastroenterology, the Second Affiliated Hospital of HMU. The test was compiled by experts from our endoscopy centre. The QR codes and the test (posttest version) are available in File S2.

Behaviour level

As routine colonoscopy training was provided to all participants, we assessed behavioural levels by measuring participants’ actual colonoscopy performance. All the patients participating in the colonoscopy tests had made appointments for a general colonoscopy at the outpatient department and had agreed to have their colonoscopies performed by beginner colonoscopists. All students needed to perform 20 colonoscopies after completing the entire learning process. During the procedures, the instructor recorded the student’s procedures, and the video was used to assess the following: (1) the times to pass through the LDT and to reach the sigmoid colon, splenic flexure, hepatic flexure and cecum; (2) the overall colonoscopy examination time and fluency; (3) the clarity of the collected images; (4) diagnostic accuracy; (5) the patient’s comfort and satisfaction; and (6) the detection rate of colonic polyps. After three failed attempts in the above procedures, the expert endoscopists performed the procedures and recorded it as a failure for the student (Cass et al., 1993). The entire operation process of the trainees was video-recorded. Each participant had an opportunity to select the performance that he or she thought was the most satisfactory for archiving. A total of five expert colonoscopists (from the Second Affiliated Hospital of HMU) who were unaware of the participants’ grouping scored the students’ colonoscopy examination videos based on their experience in colonoscopy. The experts collaborated to develop the examination standards. The mean score was obtained. The QR codes and the grading table for colonoscopy examinations are provided in File S3.

Result level scoring

To evaluate the results, we developed a course called ‘Integrated Online and Offline Teaching of Digestive Endoscopy Class’ for the standardized training of residents at the Second Affiliated Hospital of HMU. The course certificate is shown in File S4.

Statistical analysis

R Studio 3.6.3 was used for data analysis. Measurement data following a normal distribution are described as the means and standard deviations; measurement data not following a normal distribution are described as the medians and upper and lower quartiles. For measurement data following a normal distribution that had homogeneous variance between groups, Student’s t test was employed for comparisons between the two groups. If data did not follow a normal distribution or the variances between the two groups were not uniform, the comparison between the groups was performed by using the nonparametric Wilcoxon rank sum test. Enumeration data are described as the frequencies (%), and the χ2 test was used for intergroup comparisons. A P value of < 0.05 was considered to be statistically significant.

Sample description

Of the 84 eligible students from the Second Affiliated Hospital of HMU, two participants were excluded based on the exclusion criteria. One student secured > 90% in the pretest, and another did not complete the pretest. Eighty-two (97%) fit the inclusion criteria after completing the pretest. Random assignment of all 82 participants was done to the EG and the CG on average. In the EG, one dropped out during the training period. Therefore, there were 40 participants in the EG and 41 participants in the CG. Both groups displayed similar baseline characteristics, as depicted in Table 1.

Analysing the examination scores

The pretest and posttest scores in the EG group and the CG group were not normally distributed, so they are represented as the medians and upper and lower quantiles. A nonparametric Wilcoxon rank sum test was used to perform comparisons between the two groups. The pretest score in the EG was 48 (48, 50) and increased to 83 (80, 86) on the posttest (p < 0.0001). In contrast, the pretest score in the CG was 50 (46, 50) and increased to 80 (76, 84) on the posttest (p < 0.0001). The results showed that there were no significant differences in the pretest scores between the EG and CG (p > 0.05). In addition, after 2 months of training, the posttest results of the EG were better than those of the CG (p < 0.001) (Fig. 5A).

For the colonoscopy examination scores, a total of 800 colonoscopies were performed among 40 students in the EG. There were 617, 559, 526 and 275 instances of successful passage of the colonoscope through the sigmoid colon, splenic flexure, hepatic flexure and cecum, respectively. In the CG, 41 students performed a total of 820 colonoscopies, among which there were 587, 526, 491 and 229 successful instances of passage of the colonoscope through the sigmoid colon, splenic flexure, hepatic flexure and cecum, respectively. There were statistically significant differences in the number of successful instances of passage of the colonoscope through the sigmoid colon, splenic flexure, hepatic flexure and cecum between the EG and the CG, with the EG having a higher number of successful instances than the CG (Fig. 5C). The detection rates of colon polyps in the EG and CG were 66 (8.25%) and 38 (4.63%), respectively, which was a significant difference (p < 0.01) (Fig. 5C). The detection rate of polyps in the EG was higher than that in the CG. The numbers of correct diagnoses during colonoscopy in the EG and CG were 620 (77.5%) and 581 (70.85%), respectively, which was a statistically significant difference (p < 0.01) (Fig. 5C). The correct rate of disease diagnosis in the EG was higher than that in the CG (Fig. 5C). The colonoscopy training score of the students in the EG was 91 (89, 93) points and that of the CG was 89 (86, 90) points. The difference between the two groups was statistically significant, with the EG having a higher score than the CG (Fig. 5B). The time to reach the sigmoid colon, splenic flexure, hepatic flexure and cecum in the EG and the CG were determined, and the time to reach each segment of the colon was shorter in the EG (Fig. 5D). Table 2 summarizes the examination scores.

User satisfaction of the CRS

The satisfaction levels are detailed in four areas. Regarding the content quality, 87% of questions raised in Q1-Q5 demonstrated that the course was up-to-date, useful, and sufficient and fit their needs (Fig. 6A). Regarding the interface quality, 83% of questions raised in Q6-Q10 showed that the students were in favour of the speed, stability and user-friendly approach with ease of finding the necessary content (Fig. 6B). Regarding the testing quality, 86.5% of questions raised in Q11-Q15 reveled that the course testing was prompt, secure, fair and easy to complete and understand (Fig. 6C). Regarding the quality of personalization, 88.5% of questions raised in Q16-Q20 claimed that the CRS facilitated personalized learning, controlled their learning progress, selected what to learn, recorded their learning progress, and helped them learn the necessary content (Fig. 6D). Overall, 86.25% of questions raised in Q1-Q20 were satisfied with the CRS and considered it successful.

To further validate students’feedback, we conducted a satisfaction survey with patients. Both groups displayed similar baseline characteristics, as depicted in Table 3. All the patients in the e-learning group participated in the satisfaction survey, which showed a high level of satisfaction (Fig. 7).

Principal results

To date, textbooks and other written materials are considered the most widely spread knowledge in traditional colonoscopy training (Trelease, 2016), but in the field of medical skill operations, the traditional mode does not provide three-dimensional thinking of the digestive tract structure and operational experience. In addition, the COVID-19 pandemic has necessitated mandatory e-learning in medical education. In this study, we developed, evaluated and applied e-learning software named the CRS during the COVID-19 pandemic. We developed the CRS for student learning colonoscopy in collaboration with Harbin University of Science and Technology, with the CRS being the first e-learning program for colonoscopy teaching. Furthermore, the CRS was developed to be used in either Chinese or English; therefore, the system can facilitate colonoscopy training in China and worldwide.

After two months of learning, the theoretical test scores of all the students significantly increased, with an EG score greater than that of the CG (p < 0.001) (Fig. 5A). This shows that CRS is more effective in improving theoretical knowledge. The success rate of EG was higher than that of CG in the four key points of colonoscopy operation, namely, the sigmoid colon, splenic curvature, hepatic flexure and cecum. CRS provides beginners with a highly simulated virtual environment of LDT, enabling them to freely explore the structure of LDT and then fully master it through repeated practice so that they can have a good understanding of the actual operation and effectively improve the success rate of operation. In the endoscopic diagnosis and the polyp detection rate, the EG was significantly higher than the CG (p < 0.01), indicating that the students had enriched their experience in disease identification by learning endoscopic images of common LDT diseases embedded in CRS. We further submitted the operation videos of the students to expert colonoscopists for scoring and found that the scores of EG were higher than those of CG (p < 0.01), which fully demonstrated that the success rate of colonoscopy operation can be improved through CRS in a short period of time. It was verified that CRS enabled beginners to learn colonoscopy better and faster after they fully mastered the anatomical structure of LDT, theoretical knowledge and endoscopic manifestations of common diseases.

However, the CRS is not meant to be a complete colonoscopy training but a complementary training. E-learning is not a substitute for traditional learning, but an effective supplement to be used in combination with traditional learning mode.

The results of the satisfaction questionnaire show that the overall satisfaction rate of the students in the EG was very high (Fig. 6). Based on Wang’s questionnaire, we obtained satisfaction information with each aspect of the training. In terms of interface quality (Fig. 4B), the CRS’s operating interface was considered complex and somewhat difficult to understand. The reason may be that the software design uses ‘Z/X’ to represent inhalation and inflation, respectively. In the process of the insertion and passage of the colonoscope through the digestive tract, the intestinal cavity must be continuously inflated; otherwise, the colonoscope cannot pass through. We upgraded and improved the software version based on feedback from the trainees and added detailed instructions for the procedures.

Limitations

First, the small sample size at a single institution affects the generalizability of the results. The main reason for the current study design was that our CRS software was still in its initial stage. In the future, we will integrate endoscopy examination and treatment to establish a complete endoscopy e-learning teaching system. Residents should be promoted to this system through standardized training. Second, during the evaluation of the actual colonoscopy procedures, there was no way to ensure the same difficulty level among students because of the variations in the LDT structure and the differences in pain tolerance in each patient.