The validity and reliability of the Test of Memory Strategies among Italian healthy adults

Introduction

Previous research has shown that executive functions are related to performance in episodic memory (M) tasks (Brooks, Weaver & Scialfa, 2006). It is always hard to interpret whether a failure in a memory test is mainly due to a primary memory deficit or to an impairment of executive functions (EF). Previous literature has observed that the EF has a vital role in the performance of M tasks (Brooks, Weaver & Scialfa, 2006; Li et al., 2016). Craik et al. (2018) conducted two studies investigating these two important aspects of memory problems in older adults who have difficulty retrieving recent episodic events and an often-transient inability to retrieve names and other well-known facts from semantic memory. In fact, one of the questions they asked was whether these age-related difficulties reflected a common cause: a recovery problem related to inefficient EF. From their studies it emerges that no task is a pure measure of the theoretical constructs of EF or of episodic and semantic memory. Their studies showed that individual differences in EF in older adults correlate with the effectiveness of recovery in both episodic and semantic memory, but also that these relationships depend on the specific tests chosen to represent both EF and the memory recovery (Craik et al., 2018). Therefore, executive function and memory deficits occur in most neurodegenerative and psychiatric disorders, and many other diseases (Arevalo-Rodriguez et al., 2015; Khan et al., 2014; Litvan et al., 1994). A mediation analysis revealed that the EF network had an indirect positive effect on episodic memory performance in the amnestic mild cognitive impairment patients (Yuan et al., 2016). These findings provide new insights into the neural mechanisms underlying the interaction between impaired EF and M deficits in amnestic mild cognitive impairment patients and suggest that the EF network may mediate episodic memory performance in patients with mild cognitive impairment (Yuan et al., 2016).

Other studies, however, such as Moro et al. (2015) utilized a training program to teach specific strategies and metacognitive skills to enable patients to perform attention and executive tasks. When training was performed on attention and executive functions, the results showed generalized improvements also to memory (Moro et al., 2015). Despite brain imaging and neuropsychology findings, there is a shortage of psychometric tests that examine the interaction between EF and M constructs. Until recently, there were not many neuropsychological tests to evaluate the interaction between EF and M functions. The assessment of this interaction would be helpful to understand whether a bad performance on an episodic memory test is due to a primary impairment on executive functions or memory (Bäckman, 2008; Saenz et al., 2015; Stramaccia et al., 2018; Yubero et al., 2011). With few exceptions, such as the Rey Auditory Verbal Learning Test (RAVLT, Carlesimo, 2014), California Verbal Learning Test (CVLT-II, Woods et al., 2006), short story recall (Carlesimo et al., 2002), and Wechsler Memory Scale (Dumont et al., 2014), classical neuropsychology views have promoted the use of separate tests (Goretti et al., 2014; Mattioli et al., 2014) to assess M and EF in different conditions and diseases (Lis et al., 2008; Vaccaro et al., 2018). However, none of them assess the interaction between EF and memory in a parametric way. To fill this gap, Yubero et al. (2011) built the Test of Memory Strategies (TMS) to evaluate the impact of EF on performing a memory task. This tool allows us to test whether a deficit found in a memory task could be associated with a primary memory problem or whether that deficit could be due to an executive problem affecting memory performance. TMS presents a type of verbal learning task in which, through consecutive stages, the need to enact internal memory strategies should be progressively reduced. Yubero et al. (2011) in their study hypothesized that patients with severe EF damage would score worse when more internal memorization strategies were needed to complete a task; conversely, participants with a greater deficit in memory functions would have scored worse when memory involvement was predominant. While participants with impaired EF and memory would have scored very low at any stage of the TMS (Yubero et al., 2011).

In detail, Yubero and colleagues in their study applied TMS to four groups of elderly patients with varying levels of cognitive impairment. Under conditions with low material organization, the multidomain groups of mild cognitive impairment and vascular cognitive impairment showed greater impairment of executive function. However, as the material was progressively organized, the multidomain mild cognitive impairment and vascular cognitive impairment groups improved their performance. The study results confirm executive functions appear to influence performance in memory tasks (Yubero et al., 2011).

In 2018, Fernandes et al. (2018) the authors translated and validated the TMS for use on the Portuguese population. The focus of Fernandes et al. (2018) was to analyze the performance of TMS in healthy individuals of different ages, to study the test’s ability to discriminate between different age groups and education levels. In their study, Fernandes and colleagues hypothesized that the growing external organization of TMS activities would affect the response of the older sample in a negative way.

Fernandes et al. (2018), like Yubero et al. (2011), highlighted the importance of the external organization of activities in the different phases of the TMS. In addition, we expected groups with higher formal education levels to perform better in TMS than those with fewer years of formal schooling. Finally, they showed adequate psychometric properties of TMS (Fernandes et al., 2018). The results showed the reliability of TMS scores based on internal consistency analysis. Factor analysis of the TMS scores revealed the test produced two factors, one capturing executive function and the other capturing memory. Correlations with classical neuropsychological tests supported the convergent and discriminant validity of the TMS scores. The older groups had greater difficulty in creating and mobilizing strategic memory than the younger group after controlling for the influence of education, although both groups experienced performance improvements in the five TMS sub-tests. Fernandes and colleagues, stated that the results of their study suggest that TMS is an adequate measure for assessing memory and EF while applying it to a Portuguese sample (Fernandes et al., 2018).

Some tests investigate the executive functions and memory separately (such as Rey Auditory Verbal Test; etc.), but this does not allow to verify if during execution the subject fails because of a slowdown in memory or planning or problem-solving strategies. Instead, the TMS is structured to capture which phase of the test the subject increased the number of errors. A failure in the initial condition would but a clear improvement after diminished executive functions would signify an executive deficit more than a memory problem. If the participant does not improve across time, then a primary memory problem can be noticed.

In the adult and childhood clinical population, it is difficult to understand whether there is a memory or executive function impairment or both just from the interpretation of traditional neuropsychological tests. This study aimed to improve and increase the neuropsychological assessment tools available in Italy to cognitive diagnosis using the TMS, a neuropsychological test already validated in Spanish and Portuguese (Yubero et al., 2011; Fernandes et al., 2018). There is no neuropsychological test for evaluating the interaction between memory skills and executive functions simultaneously in the Italian context. This preliminary study on the Italian population could fill the gap in the neuropsychological test resources for improving diagnosis. The ability to generate memory strategies is a key factor in the performance of episodic memory tests. Fernandes et al. (2018) argue that there is evidence about the inefficient use of memory strategies in older adults. However, it is not clear whether a worse performance on a memory test in older people might be attributed to an inability to mobilize cognitive strategies or to an episodic memory deficit (Fernandes et al., 2018; Yubero et al., 2011). We contend that in the Italian literature there are no validated tests adapted for an Italian sample that measures memory and executive functions simultaneously and allows to detect, in specific conditions, when, for example, there is primary damage of memory or executive functions. This study tried to address the question by validating the TMS—which parametrically reduces the need for executive functions on memory tests—on an Italian sample. The main aim is to have a version of the TMS available for the Italian population to help achieve better cognitive diagnosis.

Materials and Methods

Results

Descriptive analysis about lists of TMS tests shows how the mean number of words repeated by subjects increased from TMS-1 to TMS-4 and how it decreased from TMS-4 to TMS-5, as shown in Table 1. The average amount of words that subjects repeated but that did not belong to any TMS word list, called “Interference” words, was also measured (Interference List 1 = 0.298 ± 0.542; Interference List 2 = 0.388 ± 0.583; Interference List 3 = 0.339 ± 0.571; Interference List 4 = 0.504 ± 0.732; Interference List 5 = 0.645 ± 0.773). Table 2 shows descriptive analysis about neuropsychological tests, in detail the mean and standard deviation of the subjects’ scores on each neuropsychological test.

A significant correlation was found between the total score of each of the word lists (TMS tot List), the score of each list, and age and education level (Table 3). As shown in Table 4, from the comparison between models, our hypothesized bi-dimensional is the one that outperforms tridimensional and the unidimensional model in terms of fit indices. Furthermore, when comparing the hypothesized model to the alternative models using the Bayesian Information Criterion BIC (Schwarz, 1978) the hypothesized model showed the lowest BIC (2173.484 vs. ≥ 2178.213). Furthermore, as shown in Table 4, the hypothesized model had excellent fit indexes. In Fig. 1, we show the path diagram of the assumed models (Panel A, B, C, D, and E). In Table 5, we show the factor loading about our hypothesized two factors model.

In terms of internal consistency, the EF subscale showed an acceptable reliability (ω_t = 0.74), whereas the M subscale showed a good reliability (ω_t = 0.87).

We found that both M and EF subscales were correlated with the scores obtained by the participants in each specific test for memory and executive functions (Table 6).

The ANOVA on the EF and M subscales as dependent variables for age groups showed that there was significant difference F (28.5) p = < 0.001 among groups for EF, and that there was significant difference F (29.1) p = < 0.001 among groups for M. Regarding the post-hoc results, Table 7 showed that only in the comparison between the participants aged between 18–30 (G1) and 31–50 (G2) there was no significance for the EF subscales, so also with regard to the post hoc relative to the M subscales, not there was significance in the group of participants between the ages of G1 and G2. Finally, we found that the covariation between the latent variable reflecting EF in the TMS and the latent variable reflecting EF in the Classical neuropsychological tests (0.94) was higher than the covariation between the latent variable reflecting EF in the TMS and the latent variable reflecting M in the Classical neuropsychological tests (0.84, diff = 0.10, z = 1.69, p = 0.045). Furthermore, the covariation between the latent variable reflecting M in the TMS and the latent variable reflecting M in the Classical neuropsychological tests (0.81) was higher than the covariation between the latent variable reflecting EF in the TMS and the latent variable reflecting M in the Classical neuropsychological tests (0.66, diff = 0.15, z = 2.59, p = 0.005).

Discussion

This study examined the validity and reliability of the TMS among Italian healthy adults. Results of this study suggested that the different performance throughout the progressive external organization of the TMS word lists, as expected, we found a general increase of the total remembered words from TMS-1 to TMS-4. This is an indication that the progressive external organization of the material alleviates from memory strategies and improves memory performance, as demonstrated as well in the Spanish and Portuguese populations (Yubero et al., 2011; Fernandes et al., 2018).

There was a tendency of a drop in performance between TMS-4 and 5. There is a well-known phenomenon in psychological task performance called the “practice effect” (Donovan & Radosevich, 1999; French et al., 2016; Goldstein et al., 2007). The more the subject repeats a task, the better is the performance on that task. Because in the last condition, the participants were informed about the existence of two categories, there were different instructions to the task, the participants could think that the last condition (TMS-5) is different from the previous ones, losing both the practice effect and the focus on task since the instructions are different. Indeed, some participants say they focused on understanding what the categories were. This hypothesis arises from the “clinical” observation made during the administration of the test.

Another possibility is that the categories used in TMS-5 were much harder to memorize than those in TMS-4, but this is unlikely, as words in all these categories had a similar frequency of use; or another possibility is the possible interference of words between the lists. Indeed, the number of intrusions increases as you get closer to List 5.

The hypothesis of identifying sample heterogeneity as one of the causes could also be considered. For all age groups, the score of the words remembered from the TMS 1 list to the TMS 4 list increases, as already highlighted. From the TMS 4 list to the TMS 5 list, the number of words remembered decreases for all groups but especially for the older ones. These results are consistent with the ones found in Yubero et al. (2011), in which none of the groups showed statistically significant differences between TMS-4 and TMS-5, although there was a reduction in the number of words recalled in TMS-5. Instead, Fernandes et al. (2018) find a general increase in the total words remembered from TMS-1 to TMS-5. Like those of Fernandes, our results seem to be consistent with those of Bor et al. (2003), who demonstrated that coding strategies improve memory performance in complex tasks.

Concerning the structure of the test, the results of CFA suggested that the two-factor model is an acceptable one, with TMS list 1 and TMS list 2 reflecting one factor, and TMS list 3, TMS list 4, and TMS list 5 reflecting another factor. In line with Yubero et al. (2011) and Fernandes et al. (2018), our results support the idea of the TMS as a measure of memory and executive functions. This may indicate that the use of the total score could be appropriate, calculating both M and EF score simultaneously. Consequently, the TMS appears to be particularly suitable for research and clinical purposes because it offers an efficient way to measure general EF and M functions. Concerning the psychometric properties, the Italian TMS showed internal consistency values that ranged from good to excellent and in line with those reported for the original TMS (Fernandes et al., 2018; Yubero et al., 2011).

Our analyses showed that the latent variable reflected by the EF TMS items correlated more with the latent variable reflected by the EF Classical neuropsychological tests items than with the Memory Classical neuropsychological tests. Vice versa, the latent variable reflected by the Memory TMS items correlated more with the latent variable reflected by the Memory Classical neuropsychological tests items than with the EF Classical neuropsychological tests. Such a pattern of results supports the construct validity in terms of convergent and discriminant validity.

Yubero et al. (2011) found that the level of education in the age grouping influences the score obtained on the TMS and the elderly population has difficulty performing tasks where a self-initiation process is required for efficient coding and information retrieval. In our case, an analysis was not carried out by dividing by age groups because the sample size did not allow it.

As indicated before, classical test of episodic memory does not consider the separate assessment of memory and executive functions and therefore they are deeply influenced by the patient’s ability to mobilize complex memory strategies. The TMS is trying to avoid this source of confusion to interpret the results of an episodic memory test by progressively reducing the need of executive functions. Other memory test as the California Verbal Learning test is evaluating the use of memory strategies as well. However, the CVLT, is assessing the use of memory strategies by the subjects’ answers, but not by the manipulation of the verbal material itself. The TMS is progressively organizing the verbal material to reduce at minimum the need of EF and evaluate episodic memory without the influence of internal cognitive strategies. As an example, in the last condition of the TMS two categories are presented one after the other and the participant is aware about that external organization of the material to avoid the use of any internal memory strategy. In this sense, the material organization performed in the TMS directly evaluates the memory abilities expecting having worse performance in patients with a primary memory deficit.

Some shortcomings of the present study must be mentioned. First, the sample is small, it is not strongly representative of the entire Italian population, and the distribution of the sample is too homogeneous. Indeed, one of our future objectives is to expand the sample and make it representative of the healthy Italian and, subsequently, also add a clinical population. Additionally, the findings regarding the CFA must be interpreted with caution, and future studies on measurement invariance to the clinical vs. non-clinical population are therefore needed (see Yubero et al., 2011 for a clinical study). Considering the discrepancy between the models developed by us and the one proposed by Fernandes et al. (2018), In the future it would be useful to carry out an administration to a large Spanish and Italian sample in order to analyze both and detect the cause of this differentiation.

In conclusion, our preliminary study shows that the Italian version of TMS is a relatively valid and reliable measure of executive and memory functions in its Italian version. TMS is useful for improving cognitive diagnosis in patients with neurodegenerative diseases when a memory failure can be due to both executive dysfunction and a primary memory problem. This test is useful for detecting slight differences in cognitive functioning, especially because other neuropsychological instruments do not test the relationships between memory and executive functions. The profiles from the TMS can be easily used for the development of a personalized or even more tailored rehabilitation program.

The TMS could be useful in prospective studies in children and school assessments to understand if a learning deficit is memory or executive function related and, therefore, help structure the school educational-rehabilitation plans. TMS could have excellent practical implications in the clinical field. For example, it could be very useful in those neurodegenerative diseases in which it is still difficult to make a differential diagnosis, such as the subtypes of progressive nuclear paralysis. In these subtypes of PSP and diseases for example, there is still a lot of confusion about the characteristic cognitive deficits. The greatest confusion is precisely between deficit to executive functions and memory deficits, for example (Vaccaro et al., 2020). Typically, low scores in an episodic memory test are used for starting an episodic memory intervention. However, this intervention could fail if the primary deficit is an executive function deficit rather than memory. Therefore, TMS could help clinicians to find a more appropriate intervention approach improving the rate of success in neuropsychological rehabilitation.

Therefore, it might be used in future studies where it is crucial to either measure or control for the joint and differential contribution of memory and executive functions (El Haj et al., 2016; Gray et al., 2012; Novellino et al., 2019; Polak et al., 2012; Vaccaro et al., 2018; Sarica et al., 2021; Vaccaro et al., 2020).

Supplemental Information