Interaction of guidance types and the Need for Cognitive Closure in wiki-based learning

Measuring process variables

Both self-developed wiki environments were designed to record participants’ selection behaviour by measuring individual clicks on the article and discussion tabs as well as on individual discussion thread titles. On the talk page, clicking on a title was necessary to select and expand a thread and thus unveil its contents (topic selection). By design only one topic could be open for reading at a time and had to be collapsed by clicking again before proceeding to the next topic of interest. For further processing click counts that triggered only the expanding/opening events were recorded in the log. Furthermore, the environment recorded the times of events such as time spent on the article page, time spent on individual discussions and time spent to write a reply to a topic (topic contribution time). Discussions’ reading times were measured by calculating the differences between thread opening and closing times. If a topic was opened and closed more than once, the environment also recorded cumulative reading times for each discussion thread. Correspondingly, for the article page the environment’s logging system also recorded the overall time spent with the article. Participants had to edit and reply three times during the study and the system recorded the number of replies to each topic and which kind of controversy it was (topic reply frequency).

Measuring learning success

To measure individual learning success about the study’s subject matter, a post-experimental multiple-choice knowledge test was developed. Such tests are still widely used to quantify learning results in collaborative and individual settings (Kent, Laslo & Rafaeli, 2016). In total, the test comprised 18 questions about different forms of energy, such as different types of renewable energy sources (e.g., “What is the efficiency of water?”, “What are the negative effects of wind turbines?”). Six of these questions were designed to be answerable with only the information provided in the original base article. Therefore, they were practically solvable without having read any of the discussion threads. The remaining 12 questions were constructed in a way that exactly one question covered one of the controversial discussion topics. Every multiple-choice question had four answering options comprised of up to three distractors and at least one attractor. The test’s overall sum of correct answer options was used as a general indication for individual learning success about the study’s subject matter. The theoretical maximum score a student could reach was 32.

Measuring further potential influences

Individual epistemic curiosity was measured with the Epistemic Curiosity Scale (Renner, 2006). This validated questionnaire with a total of 10 items measures the two dimensions, diversive exploration and specific curiosity. Each of the two subscales consists of five statements (e.g., ‘When I learn something new, I like to learn even more about it.’) and had to be rated on a 4-point scale ranging from “fully disagree” to “fully agree”. In addition to that, participants were also asked to rate their further experience with wikis in a short self-developed questionnaire with six items on a 4-point scale ranging from “not correct at all” to “fully correct”. Items covered questions regarding passive use of and active participation in wikis.

Measuring user experience

To measure the acceptance of the two guidance implementations which have been implemented in the experimental wikis, the User Experience Questionnaire (UEQ) was used (Laugwitz, Held & Schrepp, 2008). This questionnaire consists of 26 items measuring the perceived usefulness of user interfaces and their implementations. The UEQ does not provide a total score of the user experience, instead the construct is made up of six dimensions that are considered individually. Overall, values smaller than M =  − 0.8 are considered negative evaluations, values between −0.8 < M < 0.8 as neutral and values greater than M = 0.8 as positive evaluations of a tool on the respective dimension.

Remarks on statistical analyses

For the following analyses, equivalence hypothesis tests have been used with the TOST (two one-sided t-Tests) procedure. This testing procedure is very useful when a null or very small effect is expected and has in general greater statistical power for gathering evidence for or against the absence of effects. It applies two directional one-sided t-tests against a priori specified lower and upper effect size or raw score bounds. Simultaneously, a classic NHST (Null Hypothesis Significance Testing) t-test of differences from a null effect is applied. Within the TOST framework, the use of confidence intervals is more prevalent than p-values. Highly simplified, 95% CIs including zero correspond to non-significant p-values, whereas intervals excluding zero correspond to significant results (Lakens, 2017a). Equivalence hypothesis tests were performed with the R package TOSTER (version 0.3.3) by Lakens (2017b). The underlying two one-sided t-test (TOST) procedure requires to determine a priori the smallest effect size of interest (SESOI) for the specification of equivalence bounds (Hauck & Anderson, 1984; Schuirmann, 1987). Applying the TOST procedure, in conjunction with a t-test of differences within the null hypothesis significance testing (NHST) framework, can yield four possible outcomes for an effect: (a) statistically equivalent and not different, (b) not equivalent and statistically different, (c) statistically equivalent and different, and (d) not equivalent and not different.

In the paragraphs regarding the analyses of interaction patterns, at first a 2 ×  2 MANCOVA was used with the guidance type (implicit vs explicit) as group factor and the median split NCC (low vs high) as second factor. The dependant variables were the selection and reading behaviours of resolved and unresolved discussion topics. Participants’ self-assessed subjective prior knowledge was used as covariate in all the following analyses because substantial differences between the experimental groups have been identified, t(173.08) = 2.37, p = .019, d = 0.35. As subsequent analyses, hierarchical linear regressions were used to account for the full continuous data spectrum of the Need for Cognitive Closure scale. For the analysis of the reply frequency as dependant variable, a 2 ×  2 ANCOVA has been used instead of a MANCOVA.

Topic selection

Participants in the implicit guidance wiki selected on average M = 4.21 (SD = 2.75) resolved topics in comparison to M = 4.60 (SD = 2.86) resolved topic selections in the explicit guidance wiki. The mean selection difference of 0.39 for resolved topics is equivalent in a 90% CI [−1.17–0.21] and not significant in a 95% CI [−1.30–0.34] within raw equivalence bounds of −1.40 and 1.40 resolved topic selections. Unresolved discussion topics have been selected on average M = 4.84 (SD = 3.88) times in the implicit guidance wiki and M = 4.24 (SD = 3.21) in the explicit guidance wiki. The mean difference of 0.60 topic selections between the guided wiki groups is equivalent in a 90% CI [−0.28–1.47] and not significant in a 95% CI [−0.45–1.64] within raw equivalence bounds of −1.78 and 1.78 unresolved topic selections.

Topic reply frequency

Participants in the wiki with implicit guidance replied on average M = 1.42 (SD = 0.92) times to resolved controversies. In the explicit DDR script wiki, they replied M = 1.80 (SD = 0.74) times to these controversial discussion topics. The mean difference in reply frequency between groups of 0.38 replies is not equivalent in a 90% CI [−0.58–−0.18] and statistically significant in a 95% CI [−0.63, −0.14] within raw equivalence bounds of −0.42 and 0.42. Since the number of total discussion replies was fixed to n = 3, analyses of the reply frequency to unresolved controversies would be completely redundant. Subsumed, the results in this and the previous paragraph provide evidence for hypothesis H1a where we assumed equivalence of implicit and explicit guidance wikis regarding the first set of process variables (topic selection and topic reply frequency).

Topic contribution times

In the implicit guidance wiki participants spent on average M = 386.46 (SD = 246.55) seconds on contributions to resolved controversial topics, whereas in the explicit guidance wiki M = 481.11 (SD = 270.70) seconds were spent on contributing to these controversies. This mean contribution time difference of 94.64 seconds between both groups is not equivalent in a 90% CI [−158.30, −30.98] and also statistically significant in a 95% CI [−170.62, −18.67] within raw equivalence bounds of −129.45 and 129.45 s. Regarding unresolved controversial topics, participants in the implicit guidance group spent M = 436.21 (SD = 298.56) seconds on contributions. In the explicit guidance wiki they spent M = 341.37 (SD = 228.61) seconds on contributing to unresolved controversies. The mean difference in contribution times for unresolved controversial topics of 94.84 seconds is not equivalent in a 90% CI [29.51–160.17] and also statistically significant in a 95% CI [16.86–172.82] within raw equivalence bounds of −132.95 and 132.95 s. The second part of the analysis regarding unresolved controversies contradicts our assumption in hypothesis H1b that explicit guidance leads to generally longer contribution times.

Knowledge test

In the knowledge test, participants in the implicit controversy highlight group scored on average M = 15.84 (SD = 3.43) in comparison to an average score of M = 15.47 (SD = 3.60) in the explicit scripting group. A mean test score difference between both guidance groups of 0.37 points is equivalent in a 90% CI [−0.50–1.23] and not significant in a 95% CI [−0.66, 1.40] within raw equivalence bounds of −1.76 and 1.76 points (Fig. 3). This result supports hypothesis H1c where we expected equivalence in learning outcomes on a group level, meaning that no guidance type is superior per se.

Topic selection

In the MANCOVA model, the interaction effect of the experimental groups and the NCC level on topic selection behaviour was very small to virtually non-existent, λ = .99, F(2, 175) = 0.83, p = .440, ${\eta }_p^2<$ .01, 90% CI [.00–.04]. The follow-up regression analysis with the full continuous NCC data showed small effects for the interaction with the grouping variable on the selection of resolved topics with b = 0.02 (SE = 0.04), t(176) = 0.46. This first analysis of time spent with resolved controversies topics supports hypothesis H1b, whereas t p = .644 within a small total effect model, F(4, 176) = 0.72, p = .583, R² = .02, 90% CI [.00–.04]. A similar pattern with small effects was found for the selection of unresolved topics with b = −0.02 (SE = 0.05), t(176) = −0.47, p = .640 in the total effect model, F(4, 176) = 1.05, p = .385, R² = .02, 90% CI [.00–.05].

Topic reply frequency

Due to the limitation of exactly n = 3 replies, instead of a MANCOVA on two reply variables a 2 ×  2 ANCOVA has been used for analysing the interaction effect of guidance and NCC on resolved topic reply frequency. The effect with the dichotomized NCC was very small, F(2, 175) = 1.91, p = .169, ${\eta }_p^2$ = .01, 90% CI [.00–.06]. The regression model with the continuous NCC showed virtually the same pattern with a small effect with b = 0.02 (SE = 0.01), t(176) = 1.32, p = .188 within a rather large total effect model, F(4, 176) = 6.24, p < .001, R² = .12, 90% CI [.04–.19].

Topic contribution times

For contribution times, the MANCOVA model showed an extremely small effect for the interaction of guidance type and the NCC, λ = .99, F(2, 175) = 1.10, p = .335, ${\eta }_p^2<$ .01, 90% CI [.00–.04]. Follow-up regressions with the NCC as continuous variable suggest some influence of the NCC interaction on the resolved topic contribution time with b = 4.58 (SE = 3.61), t(176) = 1.27, p = .206 within a medium-sized total effect model, F(4, 176) = 3.03, p = .019, R² = .06, 90% CI [.01–.11]. A much weaker effect was found for the regression on unresolved topic contributions time of b = 1.12 (SE = 3.70), t(176) = 0.30, p = .762 that was also in a medium-sized total effect model, F(4, 176) = 3.07, p = .018, R² = .07, 90% CI [.01–.12]. Hypotheses H2a and H2b are not supported by the data. The raw data of the process variables suggests that prior knowledge is primarily responsible for differences in replying frequency and contribution times for both resolved and unresolved controversial discussions.

Knowledge test

Regarding the potential effects of the guidance type and the NCC level on the learning outcome, with a 2 ×  2 ANCOVA a statistically significant small to moderate effect was found, F(2, 175) = 5.30, p = .023, ${\eta }_p^2$ = .03, 90% CI [.01–.11]. The Shapiro–Wilk test for normality suggests that the residuals in both experimental groups were close to normally distributed for (1) the knowledge test scores with W = 0.98, p = .153 (implicit), W = 0.98, p = .150 (explicit) and (2) for the NCC with W = 0.99, p = .661 (implicit), W = 0.98, p = .275 (explicit). Using the full NCC data spectrum with a hierarchical linear regression model, the effect of the interaction itself was weakened, b =  − 0.03 (SE = 0.05), t(176) = −0.67, p = .502 within a moderate total effect model, F(4, 176) = 3.20, p = .014, R² = .07, 90% CI [.01–.12]. Both analyses provide evidence for the anticipated directional effects in hypothesis H2c, when controlling for prior knowledge differences. Figure 4 shows the interaction diagrams according to the conducted variance and regression analyses.

Testing equivalence between NCC levels

The following analyses on the process and outcome variables with the NCC as a dichotomised grouping factor are exploratory and were not part of our main research questions and hypotheses. We have performed them systematically in the same manner as we have done in the previous subsection with the main grouping factor (guidance type). We were interested to explore if there are any meaningful any differences in the dependent variables when the NCC is used as a main grouping factor.

Topic contribution times.

Low NCC participants spent on average M = 427.15 (SD = 245.71) seconds on contributions to resolved controversial topics, whereas high NCC participants spent M = 440.11 (SD = 279.94) s on contributing to these controversies. This mean contribution time difference of 12.96 s between both groups is equivalent in a 90% CI [−77.78–51.87] and not significant in a 95% CI [−90.33–64.41] within raw equivalence bounds of −131.69 and 131.69 s. Regarding unresolved controversial topics, low NCC participants spent an average time of M = 409.79 (SD = 298.79) seconds on contributions. High NCC participants spent M = 367.61 (SD = 235.40) seconds on contributing to unresolved controversies. The mean difference in contribution times for unresolved controversial topics of 42.17 s is equivalent in a 90% CI [−23.83–108.18] and not significant in a 95% CI [−36.61–120.96] within raw equivalence bounds of −134.49 and 134.49 s.

Epistemic curiosity

Epistemic Curiosity was additionally explored as a construct that is related to the Need for Cognitive Closure. There was a small positive direct effect on the knowledge test scores, b = 0.09 (SE = 0.06), t(176) = 1.54, p = .124. Participants scoring high on Epistemic Curiosity perform minimally better in the knowledge test. The overall effect model in the hierarchical linear regression was medium-sized, F(4, 176) = 4.65, p = .004, R² = .07, 90% CI [.01–.13]. A relatively large portion of the effect is due to differences in prior knowledge. The data of analysing the interaction of Epistemic Curiosity and guidance suggests a small positive effect on learning outcomes, b = 0.15 (SE = 0.11), t(176) = 1.40, p = .164. Participants scoring low on the used curiosity scale perform slightly better with controversy awareness highlights for implicit guidance. In contrast to that, participants on the higher end of the curiosity scale perform slightly better with the DDR script for explicit guidance. The overall effect model of the interaction hierarchical was medium-sized, F(4, 176) = 3.99, p = .004, R² = .08, 90% CI [.02–.14]. As before, some portion of the effect in the overall model is due to prior knowledge.

User experience

Overall, participants in the group with controversy awareness for implicit guidance rated the status highlights in five out of six UEQ dimensions higher than the collaboration script group for explicit guidance rated the DDR script. The largest difference on all six UEQ dimensions between both groups was on rating the Efficiency of their respective wiki environment, t(179) = 6.44, p < .001, d = 0.96. Students in the implicitly guided group rated their controversy status visualisations as rather positive in terms of efficiency whereas the explicitly guided experimental group gave even negative scores on the efficiency scale for the collaboration script. The only dimension where the explicitly guided group rated their wiki modification higher was the aspect of Novelty, t(179) = −2.11, p = .037, d = 0.31. See Fig. 6 for the detailed comparisons of all six UEQ dimensions between the experimental groups.