Tolerogenic dendritic cell reporting: Has a minimum information model made a difference?

Introduction

A large amount of background information is required to fully understand the context, methods, data and conclusions that pertain to an experiment.  Minimum information models (MIMs) provide a framework for reporting all the essential information (metadata) about experimental work and they have become popular among the scientific community. They are designed to make research more reproducible; following their guidelines ensures that the minimum necessary information about the experimental work is presented in a report or publication, so that other researchers can re-use or re-purpose the data and methods.

From 2014 to 2016, a group of researchers working in the field of tolerogenic dendritic cells (tolDCs) came together to generate a MIM for their field. tolDCs are antigen-presenting cells that induce or restore immune tolerance (Fucikova et al., 2019). These cells show great promise as a therapeutic tool for the treatment of conditions caused by a breach in immune tolerance (e.g., autoimmune diseases) or for the prevention of graft-rejection. However, the diverse range of experimental designs and reagents used to generate tolDCs precluded meaningful comparisons between different tolDC types. It was anticipated that a MIM would improve the transparency, reproducibility and data interpretation of the different tolDC types. MITAP (Minimum Information about Tolerogenic Antigen-Presenting cells) was published in 2016 (Lord et al., 2016); the more general name was chosen to allow for the inclusion of other tolerogenic antigen-presenting cell types, for example regulatory macrophages (MRegs).

In total MITAP took 18 months to create as a collaboration by experts in the tolDC field. The whole procedure involved a series of interactive workshops in which several exercises were conducted to gather the relevant basic vocabulary used within the community; to obtain feedback on the draft reporting guidelines, and finally to test its comprehensibility by the end-users. The full MITAP document, including the checklist, can be found on https://w3id.org/ontolink/mitap.

MITAP consists of 4 sections that were considered to be essential by the experts for reporting all necessary information about the generation of tolDCs and other tolerogenic antigen-presenting cell (tolAPC) products. The most crucial stages of the production process are encapsulated in these sections in an orderly manner. They are summarized below:

Section 1. Cells before: This section describes characteristics of the cells before they undergo any manipulation such as (a) essential information about the donor, (b) source of the cells, (c) the cell extraction method (d), cell phenotype after extraction and (e) cell number and viability.

Section 2. Differentiation and induction of tolerogenicity: This section describes the protocol that has been used to differentiate and/or induce tolerogenicity in the cells described in section 1. There are five subsections giving details on (a) preculture conditions, (b) culture conditions, (c) the protocol used to induce differentiation and tolerogenicity of the cells, (d) loading of cells antigen, and (e) the way cells are stored immediately after culture.

Section 3. Cells after: This section describes the characteristics and state of the cells after the differentiation/induction of tolerogenicity process has taken place. This section has three subsections, two of which provide similar basic details as in section 1 on the (a) cell phenotype and (c) cell number and viability. Another part (b) focuses on functional in vitro assays such as migratory capacity of the cells, or their ability to induce T regulatory cells.

Section 4. About the protocol: The final section of MITAP describes the general features of the protocol as a whole, such as (a) any external regulatory authorities or guidelines followed; (b) purpose of the cells; (c) whether cell product is applied in an autologous, allogeneic, xenogeneic or syngeneic manner, and finally (d) contact details of the authors.

Here, we examined the usage of MITAP in the tolDC community. More specifically, we addressed the question of whether these reporting guidelines improved the provision of relevant metadata in papers published after the MITAP publication date.

Methods and Results

How many papers could have used MITAP?

While it is useful to know the actual penetrance of MITAP, we also wished to understand how many papers that are reporting results about tolDCs could have used MITAP but did not. This question is harder to answer exactly, but we calculated an estimate. Our overall approach is depicted in Fig. 1. We assumed that most of the papers related to tolDCs are available on PubMed. We therefore searched PubMed with a variety of keywords appropriate for papers related to the MITAP subject matter, restricting the search to papers that were published after MITAP itself was published.

Of course, to use the number of search results as an estimate for the total number of papers, we must be sure of our query terms. We achieved this by testing the performance of each query by calculating the precision (how accurate the results were) and recall (how complete the results were). We could test for recall because we had a set of papers that directly cited MITAP; these therefore fall into the category of papers that could (and in this case did) use MITAP. If our query has 100% recall, therefore, all of these papers (and others) should be retrieved. We calculated the precision by simply reading a subset of 20 papers from each query and making a expert judgement about whether they were relevant.

To provide a base line for our queries, we initially generated keywords using inverse document frequency (IDF) from the titles and abstracts of 10 relevant MITAP citations. All the queries were refined by putting a ‘date of publication’ filter, so that only papers published after MITAP were retrieved, since no papers published before MITAP could have used it. In addition, the review papers were also excluded by applying a “NOT” filter on the query, since these were not relevant to this investigation. A query using only keywords that were given in the original MITAP paper retrieved a smaller number of papers, but the recall was also lower (Table 1). Finally, we modified these queries based on expert judgement of the authors resulting in the manual adjusted queries. The precision and recall were higher, but the number of retrieved papers was large. Therefore, the query was further refined by adding keywords such as “generate”, “produce” or “induce” to get only those papers that reported on tolDCs that were produced as part of an experiment. The precision and recall of manual adjustment-2 retrieved results were found appropriate among other queries. Thus, this list of 72 papers was decided to be the total number of tolAPC papers published after MITAP (of which the majority focused on tolDCs, as expected). Hence only (10/72) 14% of researchers have used MITAP so far to report their data, while the remaining 62 tolAPC papers did not use MITAP for data reporting.

Table 1:

SQL queries on PubMed for retrieving MITAP related publications.

The “*” at the end of words is to represent all versions of that word such as “induce*” refers to induces, induced, induction etc. In addition, a filter to exclude review papers “NOT (review[Title/Abstract])” was also added for all queries.

	Query	Number of Papers	Precision	Recall
IDF via Titles	(”tolerogenic dendritic” OR ”derived dendritic” OR ”tolerogenic*” OR ”regulatory cell*” OR ”regulatory-macrophage*”)	921	0.3	0.75
IDF via Abstracts	(”tolerogenic dendritic” OR ”derived dendritic” OR ”regulatory cell*” OR ”tolerogenic*” OR ”regulatory macrophag*” OR autoimmun* OR antigen-presenting)	8,992	0.25	0.71
IDF viaMITAP keywords	(“Tolerogenic dendritic cell” OR “Regulatory dendritic cell” OR “Tolerogenic antigen-presenting cell” OR “Regulatory macrophage” OR “Tolerogenic dendritic cells” OR “Regulatory dendritic cells” OR “Tolerogenic antigen-presenting cells” OR “Regulatory macrophages”)	129	0.45	0.69
manual adjustment version 1	(((”tolerogenic antigen presenting” OR ”tolerogenic-dendritic-cell*” OR ”derived-dendritic*” OR ”regulatory macrophag*”)	410	0.5	0.69
manual adjustment version 2	(”antigen-presenting” OR ”dendritic cell*” OR ”derived-dendritic*” OR ”tolerogenic*” OR ”regulatory macrophag*”) AND (”induc*”[Title] OR ”generat*” OR ”develop*”[Title] OR ”produc*”)	72	0.75	0.83

DOI: 10.7717/peerj.15352/table-1

Comparison of papers with and without MITAP

The purpose of the MITAP document is to ensure that authors provide sufficient basic information about the generation of tolDCs or other types of tolAPC. As shown in the original MITAP paper, complete information is not being provided routinely. 5 years after the MITAP publication, we wanted to re-check the status of information provided by the tolDC community. We therefore applied the MITAP checklist to 10 papers that used MITAP and 10 papers that could have used MITAP but did not. These papers are listed in the Further Reading section, but the sequence in Fig. 2 is different and anonymized. Red sections in the heatmap represent the information missing from the papers, green areas show the information directly provided in the paper and yellow areas show that the information is partially available (e.g., information is available in a referenced paper but not in the paper itself), or not relevant to the paper. Figure 2 shows that not all the sections of the MITAP checklist were completed by both category of papers. We therefore carried out further analysis to check the percentage of reported parameters for each section and performed statistical analysis. As the data is categorical and does not follow any distribution, the non-parametric McNemar test was carried out, using python 3.9 mlxtend library. For both MITAP and non-MITAP papers, Section 1 is the least reported section with less than 40% of the parameters reported, whereas for all other sections >50% of the parameters were reported. MITAP papers performed marginally better than non-MITAP papers for sections 1, 2 and 4; there were no significant differences for section 3 (Fig. 3 and Table 2).

Table 2:

Significant reporting differences between MITAP and Non-MITAP papers by McNemar’s test.

Section	Subsection
1	a	Characteristics of the organism—health
	b	Source of cell material—quantity (vol, size, weight)
	c	Cell separation—equipment used
		Cell separation—tissue condition between retrieval and cell separation
		Cell separation—methodology
2	b	Culture conditions—source of medium
	c	Differentiation protocol—source of cytokines/other agents
	e	Storage—Fluid
		Storage—Container
4	a	External authority approved
		Does protocol follow GMP?
	c	Allogenic/Autologous/Xenogeneic/Syngeneic

DOI: 10.7717/peerj.15352/table-2

Comparison of MITAP with other MIMs

While it is known that high quality data reporting is important and essential for reproducible research, as we have shown, MITAP is currently used by only 14% of tolDC publications. There has been much recent interest on the lack of reproducibility in many areas of biology and immunology, so we doubt that these figures for the tolDC community are unusual. This may be because following standards for the design and subsequent reporting of experiments and subsequent analysis is complex or time consuming. There are a number of ways to reduce the effort of this data reporting; of these, MIMs are seen as an effective mechanism for standardisation, as evidenced by the number of them that being introduced into the medical field (Fuchs et al., 2018). Figure 4 shows an increasing trend of MIMs in the medical field in the last 20 years.

Because MITAP was used by a smaller number of published papers than we might have hoped, we analysed and compared usage of other MIMs to MITAP. MITAP was published in August 2016. Figure 4 shows that five other MIMs were also published in 2016. Looking at the citations of these other five MIMs, MITAP performs to a similar level and is even in the top three most cited MIMs published in 2016 (Fig. 5).

In addition to comparing MITAP with MIMs in 2016, it is useful to compare it with other relevant MIMs. MIAME and MIGS are well-established MIMs in the medical field, published in 2001 and 2008, respectively (Brazma et al., 2001; Field et al., 2008). As dendritic cells are a type of immune cells, we looked at other immunology related MIMs such as MIATA (Minimum Information about T cell Assays) and MITREG (Minimum Information about T Regulatory Cells) (Britten et al., 2011; Fuchs et al., 2018). Figure 6 shows the citation statistics of these five MIMs compared to MITAP. MIGS and MIAME are MIMs for ”omics” data and are both broadly applied in many areas of biology, so might be expected to be widely used, but it can be clearly seen that the citations are still limited. Similarly, MIATA and T cells also cover substantially bigger and older research fields than the tolDC field but show maximum 16 citations only in the peak year.

These citation statistics suggest that MITAP is performing well and is a sufficient tool to implement standardisation in the field of dendritic cells. Even when the MITAP checklist is not followed completely, it still fulfils the requirement for providing a minimum set of most important entities for tolDCs, as well as increasing the data reporting.

Discussion

Over 5 years ago, investigators in the field of tolDC came together to create MITAP, a minimum guidelines tool for reporting of the protocols to generate these cells. By using MITAP on extant papers that were published before MITAP came into existence, we showed that a large proportion of papers lacked sufficient data required to interpret and reproduce the generation of these tolerogenic cells (Lord et al., 2016). Here, we investigated usage of MITAP, and whether it improved the description of the four protocol sections: (1) Cells before; (2) differentiation and induction of tolerogenicity; (3) cells after, and (4) about the protocol.

Despite the highly collaborative nature of building MITAP, which involved many experts in the field, we have found that the use of MITAP is surprisingly low: only 14% of research papers that could have used MITAP, actually did. Although encouragingly, those papers that used MITAP performed slightly better on reporting relevant data across three of the four protocol sections, experimental details remained under-reported especially in section 1 (‘Cells before’) and 2 (‘Induction’).

There were no significant differences in reporting on section 3 (‘Cells after’) between MITAP- and non-MITAP papers. This can most likely be explained by the fact that section 3 deals with the final tolDC product itself, as these cells are the main focus of these papers, details on these cells are usually well reported. In contrast, section 1 and 2, which describe the source and features of the cells before (section 1) they undergo the differentiation process to become tolDCs (section 2), were significantly under-reported in non-MITAP papers compared to MITAP papers.

It is not known whether this is because the authors either did not have the relevant information or data available or just did not report it. Nevertheless, under-reporting of these sections will ultimately make it more difficult for investigators to reproduce published data and/or to make comparisons between different types of tolDCs. As it is becoming increasingly clear that considerable heterogeneity exists between different tolDC products, but also between tolDC products derived from different patients’ groups (Navarro-Barriuso, Mansilla & Martínez-Cáceres, 2018; Navarro-Barriuso et al., 2019), it is becoming even more important to improve reporting on the cell source and culture protocol to generate tolDCs.

There is a growing recognition that open data reporting and standardisation are necessary for repurposing or reusing data to discover valuable insights from past work, thus promoting research transparency. In this context, the 14% usage of MITAP might seem low or disappointing. However, our experience shows that the take up of MIMs takes time, despite the general appreciation of their importance. A possible reason for this could be the disparity between effort and reward: for the person producing the data, standardisation may be time-consuming with low reward; conversely, for those consuming the data, they provide high reward, maximize repurposing and reuse, with much less effort that otherwise; a MIM can provide standardisation and tooling can help to reduce the effort for data producers, but it is a high effort activity for those experts developing the MIMs. We also note that, from an informal analysis of the papers, that MITAP usage seems higher among papers coming from European authors; this perhaps suggests that more outreach work is needed, with scientists in other parts of the world.

Perhaps the main way to increase the penetrance of a MIM is the availability of a relevant data repository. MIAME is one of the most successful MIMs and has relevant repositories such as ArrayExpress or GEO requiring MIAME compliant data submission. As a result, many journals indirectly also require researchers to follow MIAME to submit the data. Other factors to the success of MIM can be the availability of software to record the metadata combined with a database to retrieve the data along with the metadata efficiently.

Our experiments demonstrate that MITAP is sufficient for the reporting of data; as a result, it has fulfilled its key objective. However, to make its use more widespread in the field, we need to introduce an online platform where researchers can record, edit, and save the metadata in a user-friendly manner.

Supplemental Information