Where is all the research software? An analysis of software in UK academic repositories

Software: a definition

What exactly do we mean by software in the context of research? The COAR Controlled Vocabularies for Repositories defines a resource type named software as ‘A computer program in source code (text) or compiled form’ (COAR, 2022a), which is sourced directly from the DCMI term (DCMI, 2020) and similar to the US Library of Congress’s ‘computer program’ (Library of Congress, 2021). More specifically, both research software (COAR, 2022b) and source code (COAR, 2022c) are defined as specific entities in and of themselves within this vocabulary. Schema.org (2023) a community body established by Google, Microsoft, Yahoo and Yandex, seeks to define schemas for structuring data on the internet. This too includes an entity of software application, defined as a subclass of a creative work, which is in turn a subclass of a ‘thing’.

Software as research output

REF, which aims to assess UK higher education research quality and impact, describes research outputs as the ‘published or publicly available products of research, which can take many forms. These include ... software’ (Research Excellence Framework, 2022). Sompel et al. (2004) enumerate the four functions of scientific communication as proposed by Roosendaal & Geurts (1998) and append a fifth:

1. Registration: the claim of precedence of a research finding is made.

2. Certification: the validity of the registered claim is established e.g., peer review.

3. Awareness: the registered claim is disseminated to others.

4. Archiving: the record of the registered claim is preserved.

5. Rewarding: the participants in the communication system somehow benefit from derived metrics.

Sompel et al. (2004) go on to say:

‘The system should consider datasets, simulations, software, and dynamic knowledge representations as units of communication in their own right.’

It is hard to find where research software does not fit with these five function statements as part of the overall purpose and ethos of scientific communication. However, there have been claims of reproducibility crises across science for several years, with particular emphasis on the components of the research process, including research software. For example, Baker (2016) surveyed 1,500 scientists regarding reproducibility within their research fields. When asked what factors contribute to irreproducible research, over 81% said that methods or code being unavailable ‘always’ (13.8%), ‘very often’ (31.6%), or ‘sometimes’ (36.3%) contributes to the problem.

Specific journals exist to peer-review research software and publish it (e.g., Journal of Open Source Software, Journal of Open Research Software, Software Impacts etc.), along with a growing number of ‘artefacts tracks’ at large conferences. Prestigious conferences from organisations such as IEEE, USENIX and ACM have begun adding ‘badges’ to deposited software within these tracks, awarding levels of reusability based on the code quality. Publishing code under this peer review process forces the author to write better code to a higher standard. Ultimately, better software means better research: greater research integrity, reproducibility and rigour.

Software is specifically mentioned by the San Francisco Declaration on Research Assessment, known as DORA (Declaration on Research Assessment, 2012) as a legitimate research output, with a recommendation to research institutions and publishers that “For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications.”(DORA, Recommendations 3 & 5).

UK Research and Innovation (UKRI) is the UK’s national funding agency, tasked with investing over £7.9 billion annually in science and research across the UK. The body also sets out terms and requirements for research outputs supported by its funding, including the management of Open Access (OA) publications and materials that support research. Their OA Policy (UKRI Open Research Team, 2022) states that, in order to meet the Concordat on Research Data:

“...it is a requirement for in-scope research articles to contain a data access statement. This informs readers where the underlying research materials associated with a paper are available...Underlying research materials are research data, as defined in the Concordat on Open Research Data, and can include code, software... ”(UKRI Open Research Team, 2022, pp. 10).

The Concordat (UKRI, 2016) lays out principles to ensure that the research data yielded from UK research is made openly available for use by others wherever possible. It notably includes reference to not only the data that underpins a publication, but the software required to generate the results of the output:

“...it is vital that the data supporting and underlying published research findings should, as far as possible, be made open by the time the findings are published and be preserved for an appropriate period. This could be achieved by depositing and providing access to relevant data and associated software (where possible) via a repository owned or operated by a discipline-specific research community and its funding bodies, a publisher, a research institution, a subject association, a learned society, national deposit libraries or a commercial organisation; or via other mechanisms that provide appropriate and sustainable services.” (UKRI Concordat on Research Data, Principle 8, pp 16)

In essence, this concordat requires that the research software must be correctly stored and made accessible for UKRI-funded research, in order to allow for reproducibility, replication and sustainable research. In the 20th anniversary update (BOAI, 2022) to the 2002 Budapest Open Access Initiative (BOAI) (BOAI, 2002), the authors added an explicit statement on how open access research software should be stored, with particular reference to the risk posed by employing commercial platforms, such as GitHub, to do so:

“Host OA research on open infrastructure. Host and publish OA texts, data, metadata, code, and other digital research outputs on open, community-controlled infrastructure. Use infrastructure that minimizes the risk of future access restrictions or control by commercial organizations.” (BOAI, 2022)

Efforts have been made to promote the application of the FAIR (Findable, Accessible, Interoperable, Reusable) principles for scientific data management and stewardship (Wilkinson et al., 2016) to research software (e.g., Chue Hong & Barker, 2021, Lamprecht et al., 2020, Clément-Fontaine et al. 2019). The FAIR for Research Software (FAIR4RS) working group published their FAIR Principles for Research Software v1.0 (Chue Hong et al., 2021), a community-sourced conversion and reapplication of the FAIR principles to the specific features and requirements of research software.

One factor pertinent to this issue is the common conflation of research software with research data or methods. Some authors claim that software is not just inextricably linked to data, but that it is a form of data in and of itself (Marcus & Menzies, 2010). Others specify that, while technically a form of data as it can be processed by a computer, software is different as it is executable and should be considered as a creative tool that operates on data (Katz et al., 2022). While it is clear why software, and other essential components, should be retained as artefacts of the research process, the how of this principle remains both rather unclear and highly variable.

While the RSE movement has certainly grown rapidly within UK universities, it is not clear as to any effects this has had on the presence of software within UK academic repositories. Similarly, when investigating the role of the institutional repository in the low return REF return rates for software, differences may be found in line with the research-intensity of the institution. For example, The Russell Group is composed of 24 research-intensive universities in the UK, producing more than two-thirds of the world-leading research output of UK universities (Russell Group, 2022). Our investigation seeks to examine such variables as potentially correlated with the level of software within the IRs.

Software as a REF-returnable item

As previously mentioned, the REF exercise assesses research outputs from UK Universities for quality and impact. Outputs that are eligible for submission to the institution’s REF process are termed REF-returnable. Strict open access guidelines for certain outputs (e.g., journal articles) are in place to maintain REF-returnability, which partly drove the rapid adoption of IRs by UK universities. However, REF exempts non-textual outputs, including software, from this open access policy (Research Excellence Framework, 2019). The REF submission guidance does state in its definition of software that it ‘...has been made publicly available’. A written description of the software must be provided, along with how the software (including source code if applicable) can be accessed (Research Excellence Framework, 2019, pp. 110). Mandating that the software is ‘publicly available’ is rather ambiguous and could be met by a wide variety of solutions, not necessarily indexed e.g., Google Drive is publicly available with the correct link, but could not be described as publicly discoverable, which a repository can facilitate. This suggests that there is no requirement for software to be deposited in the IR to be REF-returnable. However, part of the institutional management process for REF might only take consideration of items deposited in the IR. For example, Queen’s University Belfast guidance states that a module within the IR, using the Pure platform, is responsible for administering the selection process for the institution’s REF return, suggesting that only items in the IR are ‘visible’ to the process (Queen’s University Belfast, 2019, p. 4.4.2).

Repositories

Since the milestone manifesto of BOAI (2002), the Open Access movement has transformed the age-old faculty tradition of self-archiving research outputs into the de rigueur workflow for academia. With funders beginning to mandate open access requirements for research, the institutional repository quickly became the essential self-archiving platform. Crow (2002) define the institutional repository as:

“...digital collections capturing and preserving the intellectual output of a single or multi-university community...”(Crow, 2002, pp. 1)

Institutional repositories, typically called Research Information Systems (RIS), contain permanent records of research output from employees of the university. While these exist largely to fulfil open access requirements for funders, they also provide metadata on their contents for services that aggregate harvested content, the primary one of which is described in the following section.

There is an important distinction to be made between types of repositories that are used by academic software developers as mentioned in the literature. Code repositories, such as GitHub, GitLab etc., offer collaborative development tools for open-source software e.g., issue-tracking, versioning, release management. While mainly focussed on active development of software, they can accommodate archiving of code that is not under active development, but is still usable. Archival repositories provide accurate long term immutable references to any form of digital object, tracked by a permanent DOI. While this format enables long-term reference storage with FAIR attributes, it is not suitable for active and collaborative development of software and is targeted more towards traditional paper formats. However, some archival repositories (e.g., Zenodo) are software-friendly, with interactive integrations to GitHub etc. Further to this, Garijo et al. (2022) make a distinction between research software registries and repositories, with the former indexing code without storing it and the latter doing both.

As described by the CodeMeta Project (Jones et al., 2017), the infrastructure underpinning the description, indexing, archiving and discoverability of research software is far behind that of traditional research outputs, though not for technological reasons. The issue, the authors state, is in the interoperability of disparate platforms, scientific disciplines and descriptors for research artefacts, i.e., individual repositories cannot communicate in a common language to describe their data to each other. The authors cite an example of workflow integration that allows persistence for both code and state from GitHub on the former within the archival format of FigShare. However, in order to attain a DOI for the output, the pair must go to a third body (DataCite) to have one minted. The three offer differing metadata, meaning the detailed descriptors can be lost to more generic, less granular equivalents in translation. This metadata is crucial in making software FAIR, but the variance among standards seems to be holding the FAIR movement back in this regard.

Metadata & protocols

When examining how types of research output are stored, it is important to examine what they are labelled as. This work is undertaken by the metadata format used to describe the object through a codified list of identifiers and across a variety of descriptors. Different metadata formats offer differing vocabularies, as identified above, which will ultimately determine the classification of research output into discrete categories. For example, if a piece of research software is described under a metadata format without ‘software’ as a specific type of output, it cannot be described as ‘software’ using that format and will typically be referred to as ‘other’. The Dublin Core™ Metadata Element Set (commonly referred to as the Dublin Core or DC) is the metadata format, decided upon with international consensus, that has been formally standardised as ANSI/NISO Z39.85 and ISO 15836. It contains 15 core properties to describe resources. The DC label ‘Type’ is defined as the ‘nature or genre of the resource’ (DCMI, 2020a) and it is recommended to use a fixed or controlled vocabulary to define types, such as the DCMI type vocabulary, which includes a fixed type named Software (DCMI, 2020b). This is an important point to note, as many repositories using metadata formats that inherit from DC actually do not include software as a defined type of output, as discussed later in the paper.

The Research Information Systems (.ris) file format is a standard text-based tagged template for bibliographic information, including a fixed vocabulary of 35 types of publication, one of which is the COMP (Computer Program) type of reference (RIS File Format, 2012). While this is a very common format, supported by popular digital libraries and reference managers, there is no obligation on the importing agent to enforce tags of types. For example, it does not appear to be accepted by resource list management platform Talis Aspire, which omits it from a list recognised types (Hodson, 2019) and states non-conforming types will be rejected from import.

RIOXX: The Research Outputs Metadata Schema was originally developed to meet the then-newly created reporting requirements of Research Councils UK (now UKRI). The schema was designed to allow institutional repositories to share metadata and enable consistency in the tracking of items of open access research outputs across disparate systems. Version 1 appears to have allowed free text with the ‘type’ description (Rioxx: The Research Outputs Metadata Schema, 2023a). The current v2 (Rioxx: The Research Outputs Metadata Schema, 2023b) mandates one of 14 types from a controlled vocabulary, none of which include software or similar, meaning repositories that use the RIOXX format cannot explicitly label software by type. However, the draft third version of the metadata format has a proposal to include the vocabulary of COAR within its own type category (Walk & Brown, 2023).

Jisc maintains the Institutional Repository Usage Statistics (IRUS) service, which allows users to gather and share comparative statistics on research item usage. This service did not include software as a type in the original definition of IRUS item types from 2012. This omission was maintained through two further updates in 2015 and 2020, until the most recent definition in August 2022. Jisc uses the DC Type fields, which include software, but maintains its own vocabulary. Why, then, was software not counted within this service originally? Jisc explains in great detail the evolution of their policies on item types in (IRUS-UK, 2022). The original 25 item types were distilled from over 700 types identified by the Intute Repository Search (IRS) project (Jisc, 2016). This project mapped all item types in use in UK repositories, which included software (Reed, 2014), and the most populous were selected. Jisc’s report also illustrates the variance and inconsistency in item types, noting that DCMITYPE (DSpace), Sympletic, Pure, Jisc Infokit List 1, CERIF and REF2014 all included the type Software, with Code recommended by Schema, though these were mapped to Other in the iterations up until August 2022. The report states:

“In addition, we are starting a pending list of item types that we are considering for inclusion in the future if their usage becomes significant. We have been asked to consider the inclusion of ‘Code’ as an item type. None of our current participants uses that but some use the term ‘Software’ so we have added this to our pending list.” (Reed, 2014, p. 13)

It appears that the attempts to standardise metadata for research outputs have not always been adequate for consideration of software as a legitimate output, either in how they were designed or in how they have been implemented.

The open archives initiative protocol for metadata harvesting (OAI-PMH)

The OAI-PMH protocol was designed to provide a framework to enable metadata harvesting between online repositories, e-print servers, digital libraries etc., in a manner that is application-independent and applicable across many archive types. The first version of the specification of the OAI-PMH interoperability architecture was released to the public in January 2001 (OAI, 2001), with 2002′s v2 being the current implementation. It uses XML over HTTP, with an XML schema associated with the OAI_DC metadata prefix (Johnston, 2002). Each implementation of OAI-PMH must support the Dublin Core metadata format as a minimum, but may also support other formats too.

OAI-PMH also allows selective harvesting based on ‘sets’, i.e., preconfigured collections of records that are grouped together along some common attribute. The sets for each OAI-PMH server can be determined with a simple API call, which must be implemented for all OAI-PMH instances. However, IRs are not obligated to actually use sets and can return empty lists. This is discussed further in the methodology section below.

Preliminary search

Several services exist to ingest, correlate and index metadata on research outputs from a wide variety of sources. The key starting point in our census was to examine search-enabled registries and repositories as central data stores for the existence of research software. While this approach is limited by the fact that these searches index more than institutional repositories, it can provide an indication of the wider approach to software depositing.

The Open Science Framework (OSF) shows 6,146 works¹ globally with the tag ‘software’, out of a total collection of 4.7 million. Open Access Infrastructure for Research in Europe (OpenAIRE.eu), which enables compliance with the European Commission open access policy, has specific categories for research software along with datasets and publications. It lists 688 pieces of research software from the UK out of 304,376 listed globally. The total record of all ‘research products’ is currently over 162.4 million, with 5.6 million attributed to the UK² . The Bielefeld Academic Search Engine (BASE) lists 3,165 hits from 313.8 million documents for software in the UK, though 2,204 are from BioMedCentral³ . While the search engine has a specific selector for software, the records also include items such as digitally-held video documentaries. This highlights an issue with categorisation from primary sources when picked up by harvesting entities, which is discussed further in the ‘Background and wider context: Software as research output’ subsection. CORE.ac.uk is the UK aggregator for research outputs (Knoth & Zdrahal, 2012), operated as a not-for-profit service delivered by the Open University. CORE is based on a collection of 243 million searchable research ‘papers’ (CORE, 2022). Within CORE’s search API, the ‘type’ parameter is limited to only ‘Research’, ‘Thesis’, ‘Unknown’ and ‘Slides’. This again highlights the issue of missing, erroneous or incomplete metadata from the source records. CORE includes RIOXX v2 as a metadata profile, which again does not allow software as a type. Further, the difference between the software as a record in and of itself and the more traditional ‘paper presenting software’ is not clear cut in many cases. The OpenDOAR search service from Jisc allows detailed searches to be made of contributory repositories, including per repository type and filtering to include those that contain software. When queried for institutional repositories that list software, 23 IRs were returned⁴ . The returned IRs did not include some known to contain software (e.g., our own institution was not included) and so this information was deemed incomplete.

While these searches were comprehensive in many ways, they did not provide the quantity, quality or granularity necessary for the current investigation and a full specific search of UK institutional repositories was required, as described in the following section.

Surveying academic institutional repositories

Additional variables

Along with general descriptive metadata, the data retrieved using the CORE API also included details of the RIS platform software and the metadata format used, providing the fields ‘ris_software’ and ‘metadataFormat’. A series of additional variables were added to the main dataset to investigate potential correlations between these and the presence of software. The 24 Russell Group universities were marked within the dataset as a dichotomous variable ‘Russell_member’ using the list available at The Russell Group (2022). While the RSE movement has certainly grown rapidly within UK universities, it is not clear as to any effects this has had on the presence of software within UK academic repositories. To test this, the RSE groups mapped by the UK Society of Research Software Engineering (Society of Research Software Engineering, 2022) were manually added to the dataset under a dichotomous ‘RSE_group’ variable.

The extent of software in UK Academic IRs

To examine the overall prevalence of software in the repositories of UK academic institutions, the server responses were categorised into three subsets:

1. Does not contain software: those that contained software as an explicit type of entry in the repository, but did not have any records matching the type i.e., capable of containing software as a specified category, but doesn’t have any yet.

2. Contains software: those that contained software as an explicit type of entry in the repository and also have one or more records matching the type i.e., can and does have defined software records.

3. No direct software search capability: does not have a specified entry type of software or similar i.e., cannot contain an explicit entry of type software. Note, this does not mean there is no software in the repository, as each repository may have catch-all categories, such as ‘other’ or ‘non-categorised’.

As depicted in Fig. 1, over 63% of the 182 polled IRs cannot contain software as a defined type. Less than 28% contain software and the remaining 8.8% have no records containing software, but are capable of categorising them as such. As each institution can have more than one repository, this data is re-presented in Fig. 2 to restrict one category per institution. For example, if an institution has two IRs, one containing software, then the institution is labelled as containing software. This analysis shows a similar order as per-repository, with an increased presence of software capability to an approximate 60-30-10 split.

Figure 3 shows the software records per institution, of those that do contain software. The percentage of software records is heavily weighted towards the top 10 institutions ranked by percentage contribution to the overall corpus of software records. For example, the top ranked institution contributes just under a fifth of all software records, with the top five responsible for just over half (50.93%). Among those repositories with software, the average number of software records was 30.24, though only 11 of 50 IRs were above that.

Software records per research information system type

The brand or type of RIS was examined as a possible explanatory variable for the presence of software in the repository. The repositories were grouped by the overall framework, including all versions (e.g., EPrints = EPrints3, EPrints3.1 etc.). The count of repositories using each type of framework is given in Table 1. The majority of repositories used the open-source EPrints software (57.14%), approximately 31% was made up of DSpace and Pure, and the remaining seven platforms accounted for just under 11% of total repositories combined. As depicted in Fig. 4, of the ten types reported, half had no software record capability. To examine independence of the two variables, a Chi-square test of independence was performed using cross-tabulated categorical variables ‘software records’ and ‘RIS framework’. This failed to reject the null hypothesis that both variables were independent of each other: chisq = 9.048, p = 0.9586, dof = 18.

Metadata formats

When the dataset was examined for metadata formats, 106 entries used the OAI_DC format and 63 used RIOXX, with 13 missing values. To examine whether either format was correlated with software records being present, a cross-tabulation was conducted with these two variables (see Fig. 5). A Chi-square test of independence was performed using the cross-tabulated categorical variables, which failed to reject the null hypothesis that both variables were independent of each other: χ² = 0.0608, p = 0.9701, dof = 2, indicating software records and metadata type are independent with p > 97%.

Russell Group membership

Membership of the Russell Group was cross-tabulated with the presence of software records in the dataset for all institutions, shown in Fig. 6. Russell Group universities were split approximately 42% containing software, to 58% without and no instances of the type being included without records. Conversely, non-member universities had a 24.5% to 65% split, and over 10% with the ability to store a software record but none recorded. However, a Chi-square test of independence was performed using the cross-tabulated categorical variables, which failed to reject the null hypothesis that both variables were independent of each other: χ² = 0.1555, p = 0.9252, dof = 2, indicating software records and membership of the Russell Group are independent with p > 92.52%.

RSE Group at institution

The proportion of universities that employed RSE groups having software in their institutional repositories was twice as high than universities without RSE groups (22.5% vs 45%). However, more than half (52.5%) of repositories at a university containing an RSE group still had no software record capability. For a non-RSE institution, this was 66.9%. The cross-tabulated results can be seen in Table 2.

The independence of these two categorical variables was again tested using a chi-square test of independence, which failed to reject the null hypothesis that the two variables were independent (chi2 = 0.142, p = 0.9315, dof = 2), indicating software records and RSE Group co-located at institution are independent with p > 93.1%.

RQ1: to what extent is software published in academic repositories?

It is quite clear from the results presented in this research that there is a concerning lack of software stored in institutional repositories in the UK. More than 72% of the IRs polled had no software as a defined type within their repository and over 63% could not include it as specific output. When institutions, as opposed to repositories, were examined, nearly 70% had no software deposits. While it is certain that some research software will be held in other forms (personal GitHub/Lab, Innersource, private lab storage), this limits the reusability of the research code. Furthermore, a service like GitHub is a privately owned business, with no guarantee that it would not disappear tomorrow. While UKRI does not specify exact repositories in which software vital to the research should be deposited, it should fall under FAIR principles and essentially be governed by the same standards as a full journal article. It is more difficult to replicate and reproduce results (that are likely published in a more traditional format) if the code needed to produce and/or analyse the data is withheld, or obscured. Since most research is funded via taxpayer money, funders decree that everyone should have access to the data and resultant code. The enterprise and business development teams in universities may wish to exploit potential Intellectual Property (IP), which is well within the remit of universities. However, this can be easily managed with licensing and the fact that records in the institutional repositories do not need to contain the code itself, but even a pointer to the code location would be an improvement. This would operate as a code ‘registry’, but with the benefit of being maintained within the institution, non-volatile and respecting research software as legitimate research output, worthy of institutional archiving.

RQ2: are there any explanatory variables associated with whether software is included in a repository?

None of the examined traits of the repository: RIS platform, metadata format used, RSE group in the same institution or Russell group membership showed correlational relationships with the presence of software in the repository. Certain RIS platforms do not contain software as a searchable item for any of the instances stored, though numerically these are not popular when compared to the three main RIS platforms, thus not impacting the result of this analysis. The three most popular open-source platforms (Pure, DSpace and EPrints) have instances in all three categories of our analysis, possibly due to the ease of changing configurations in the open source platforms.

In theory, metadata format could play a role in whether an IR contains software due the self-limiting nature of not having software as a type within the metadata vocabulary. While initially unexpected, our result indicating that metadata format does not correlate with the presence of software in an IR can be explained with some further analysis. Firstly, the source of the data indicating the metadata format was derived from the CORE dataset on data providers, not the individual institutions and this data is possibly incomplete. Many OAI-PMH servers offer multiple metadata formats in their responses, which can indeed affect the detail in the response. For example an OAI-PMH query (https://kar.kent.ac.uk/cgi/oai2?verb=GetRecordidentifier=oai:kar.kent.ac.uk:82465metadataPrefix=rioxx) requested in RIOXX format to a known software record (i.e., listed as such on the public-facing webpage (https://kar.kent.ac.uk/82465/) lists the type as: <rioxxterms:type>other</rioxxterms:type>, with no indication it is software. However, changing the metadata format to oai_dc in the OAI-PMH gives the detail sought: <dc:type>Software</dc:type>. Further, when the examining the source code of the item’s webpage, it appears to present the information in DC format: <meta name=”DC.type” content=”Software” />. This example IR offers seven different options for metadata format, each with varying descriptors, though in the CORE dataset its format is listed as RIOXX. We consider this a limitation of the current study and future work could provide more granular data on the metadata types, particularly when public-facing webpages are used for the query rather than OAI-PMH.

The presence of an RSE group at the institution was investigated as a potential correlating variable as it may indicate the prevailing attitude to the importance of research software at the institution. This did not correlate with the presence of software. Large academic institutions are far from homogeneous in their views, processes and decision making. The decision to include software as a type of research output within a repository (or wilfully omit it) could well be taken in complete isolation from other professional services functions within the institute, with no relation to the willingness to invest in professional research software engineers. Similarly, membership of the Russell Group of leading research universities did not correlate with software being present in their repositories. Again, this could well be symptomatic of the planning and management of the IR being entirely detached from other aspects of the university.

RQ3: are there barriers preventing repositories from storing records of software as distinct research outputs?

There are likely many interacting contributing reasons for the lack of software in IRs, which we categorise as technical infrastructure, institutional and cultural in wider academia.

Technical infrastructure

When the three most common repositories are examined (EPrints, DSpace and Pure), both Pure and DSpace contain software as an explicitly activated type by default. DSpace v7.4 has a default configuration that allows for software as a type (Dspace, 2022) and mappings to the DC terms for type. However, in our analysis, over 69% of the DSpace instances did not have software as a defined type in their search. Software as a type may have been purposely removed as a type of output, whether as an individual item, part of a larger reduction of items or a whole replacement of the default configuration. This is despite the output type’s explicit definition within the DCMI (DCMI, 2020). Future work could investigate this or other potential reasons for the omission, as our data cannot provide definitive analysis. Pure has the default type ‘software’ under ‘non-textual form’ according to internal Elsevier support documents for Pure admins. The majority of Pure instances we polled have software as a searchable type, however we observed a case where it appeared to have been removed for that instance, suggesting again that it is an institutional and cultural issue, rather than technical. EPrints, which accounts for over 60% of repositories, does not have a default type for software. In communication with the development team, they elaborated that it is trivial to add new types of output, with a word in a configuration file (EPrints, 2022) enabling the type. However, even if institutional repositories include software as a type, CORE (the UK’s repository aggregator), does not currently ingest this specific information and so it is lost in aggregation. This may change in the near future, with the updating of the RIOXX metadata protocol, upon which CORE relies, to include software. Given the default or trivial amendments needed to enable software as a type, we must consider other non-technical reasons for their omission or removal.

Institutional

Academia can be slow to change long-held habits, with peer-reviewed journal articles still the prevailing currency for credit and career progression. For this reason, software can often be released along with the results in a traditional journal or conference article, sometimes with directions from the authors for users to cite their article, not the software. Thus the line between the article and the software becomes blurred. Specific journals have been established to make this process easier for those who write software, not articles (e.g., Journal of Open Source Software). While benevolent, the focus again switches to an article with a tacit acknowledgement of its status. The use of an article as the output to describe a secondary output (i.e., the software) will likely explain some of the occurrences of software being omitted from the IR. Future work could examine this using text-mining approaches to determine the extent of this practise.

Academic culture

It is entirely possible that a degree of intransigence exists around accepting software as valid scientific output in its own right, despite the firm mandate it receives from funders, policy-makers and researchers themselves. With the rate of software being submitted to REF dropping 90.2% to a mere 11 submissions since 2008, even if the infrastructure is in place to accept software, institutional norms may still hamstring the return rates regardless. There are many recommendations for hybrid pairs of repositories for different purposes, but if software and datasets are valid scientific output on a par with journal articles etc., why hold records of them in different, potentially unofficial, repositories? It would also appear that researchers who do deposit their research code in the likes of GitHub do not mirror this with entries in their institutional repositories, even when available. While the quantity of research software repositories on public hosts like GitHub may be hard to enumerate, several papers reviewed here have discussed growth in such hosting, with strong recommendations on the positive aspects. This trend may also explain, in part, declining institutional repository submissions of software.