Composite environmental indices—a case of rickety rankings

Introduction

Composite environmental indices attempt to communicate complex information by combining multiple indicators into a single score. They provide an overall snapshot of some features of environmental systems and allow for comparisons between environmental systems (Oţoiu & Grădinaru, 2018). Environmental indices can be useful tools to measure stability and change over time (improvement or decline), easily communicate information to the general public and policymakers, drive accountability between countries, encourage accountability of a nation’s government to its citizens, and facilitate public engagement (Dobbie & Dail, 2012; Nardo et al., 2005). Environmental indices can also help inform and support policy decisions, as well as generate political and media awareness of environmental issues (Haberland, 2008). However, if indices are poorly constructed or communicated, they can hinder efforts to identify and address environmental failings and may mislead policy messages and decisions (Alberti & Parker, 1991; Haberland, 2008).

The popularity of environmental indices has grown in recent years. Environmental indices have been used to measure and compare results for individual nations in global analyses (Bradshaw, Giam & Sodhi, 2010; Wackernagel, Beyers & Rout, 2019; Wendling, Emerson & Esty, 2020), for clusters of nations (Cook et al., 2017; Halkos & Zisiadou, 2018), for regions within a nation (Mukherjee & Kathuria, 2006; Universiti Teknologi, 2018), and to measure the state of specific ecosystems (Blumetto et al., 2019). Most commonly, however, environmental indices are used in global studies to compare results across nations, because national policy formulation and implementation are centred at the national level (Roberts, 2011; Usubiaga-Liaño & Ekins, 2021). Therefore, in this review, the geographic scale is for individual nations within global analyses.

There is a diverse range of available environmental indices, but country rankings differ considerably in these different indices. To illustrate, national rankings vary by an average of 45 places between the 2018 versions of the Environmental Performance Index (EPI) and ecological footprint (EF) (ranks normalized 0–100, File S1). Such large rank variations, particularly for low income and high income countries, have led to questions about the legitimacy and validity of rankings (Morse & Fraser, 2005; Press Information Bureau Government of India, 2022). Some environmental indices include indicators such as access to clean drinking water and sanitation, mortality, and per capita income for which high income nations will score well and low income countries will score poorly. The inclusion of such indicators may provide a more or less optimistic view of the environment of a nation, depending on its development status (e.g., Global North, Global South) (Bradshaw, Giam & Sodhi, 2010; Haberland, 2008).

There is extensive literature examining metrics for sustainable development and environmental sustainability (Alberti & Parker, 1991; Böhringer & Jochem, 2007; Bradshaw, Giam & Sodhi, 2010; Dobbie & Dail, 2012; Ebert & Welsch, 2004; Kwatra, Kumar & Sharma, 2020; Mallett, 1999; Oţoiu & Grădinaru, 2018; Siche et al., 2008). However, past work has largely focused on the need for credible environmental metrics (Alberti & Parker, 1991; Hák, Janoušková & Moldan, 2016), principles for constructing composite indices (Dobbie & Dail, 2012; Floridi et al., 2011; Nardo et al., 2005), and weighting, aggregation, and uncertainty analysis methods (Burgass et al., 2017; Gan et al., 2017; Greco et al., 2019; Morse & Fraser, 2005; Tripathi & Singal, 2019). Research into rank differences between existing environmental indices is relatively sparse. Research into rank variations in other fields, such as university rankings and sports leagues, is also limited and has largely concentrated on construction methods and policy implications (Garcia-Zorita et al., 2018; Saisana, d’Hombres & Saltelli, 2011).

The aim of this review is to answer the following questions: Which conceptual framework(s) or indicators inform the development of the country rankings? What effect do different conceptual frameworks (and indicators included have on global rankings? To what extent do the different environmental indices adequately capture environmental issues in low income and high income countries? In doing so, this review provides insights into how environmental indices are constructed and provide recommendations for best practices. In the present study, we have identified ten environmental indices for which country rankings can be compared to improve our knowledge of how conceptual frameworks and methodological choices (e.g., weighting and aggregation methods) may influence rank variation. This article provides a novel contribution to the development and use of composite environmental indices. This review is intended for environmental scientists interested in environmental assessment and protection, conservation executors, environmental analysts, and policymakers at both the national and international levels.

The article is organized as follows. First, we discuss how different understandings of sustainable development, environmental sustainability, and natural capital have informed environmental indices. Second, we discuss differences in the conceptual frameworks and methodological choices used to develop environmental indices. Third, we summarize existing environmental indices and the conceptual frameworks and construction choices behind them. Fourth, we examine correlations between ranks in global environmental indices. Fifth, we examine ranks for two high income countries (Germany and New Zealand) and two low income nations (Mongolia and Niger) and identify potential reasons for wide differences in rank for low and high income nations generally. Finally, we discuss our key findings and outline how conceptual framework and methodological choices may affect the ranking of nations in environmental indices.

Review of Global Environmental Indices

The Brundtland Commission (Brundtland, 1987) defined sustainable development as ‘development that meets the needs of the present without compromising the ability of future generations to meet their own needs’. The three pillars (or dimensions) of sustainability are economy, environment, and society (see Fig. 1). The aim of sustainable development is to use economic development to promote fairer societies within the ecological carrying capacity of the planet (Baker, 2016; Strange & Bayley, 2008).

There are complex connections and trade-offs between economic growth, improvements in social state, and environmental quality (Strange & Bayley, 2008). The Environmental Kuznets Curve (EKC) hypothesises that environmental degradation initially increases with income, then declines (Stern, Common & Barbier, 1996). However, critics have highlighted that the EKC model is debateable, particularly when considering different indicators of environmental degradation. For example, biodiversity may not improve with rising income (Mills & Waite, 2009). Several studies have demonstrated that economic growth has a strong positive effect on carbon dioxide, sulphur dioxide, and industrial greenhouse gas (GHG) emissions, but weaker effects on particulate matter (PM) and non-industrial GHG emissions (Holtz-Eakin & Seldon, 1995; Stern, 2017). The constructors of the pENV investigated relationships between per capita wealth and environmental impact and found that EKC predictions were not supported (Bradshaw, Giam & Sodhi, 2010). The EPI team, however, found a positive correlation between EPI scores and country wealth—indicating that rising GDP per capita is associated with higher EPI scores (Wolf et al., 2022a; Wolf et al., 2022b).

While complex interactions exist between the three pillars of sustainability, environmental sustainability can be distinguished from social sustainability and economic sustainability (Goodland, 1995; Usubiaga-Liaño & Ekins, 2021). Several different definitions for environmental sustainability exist (Table 1). A common theme in these definitions is the need for some qualities or functions of natural capital to be sustained for future generations. Natural capital (the natural environment) is comprised of natural resource stocks (e.g., water, forests, soil, wetlands, atmosphere), which provide a flow of goods and services. Goods and services can be renewable and non-renewable with and without marketed value (Ekins et al., 2003; Helm, 2015; OECD, 2008). Indicators representing the qualities or functions of natural capital can be linked to environmental pressures, states, and impacts, as well as social states where their functions are associated with human health and welfare (Ekins et al., 2003; Usubiaga-Liaño & Ekins, 2021).

Environmental metrics represent the environmental dimension of sustainable development and use a variety of indicators related to the qualities or aspects of natural capital (Usubiaga-Liaño & Ekins, 2021). For the purposes of this review, we categorize environmental indices into two broad categories. Environment-only indices focus exclusively on aspects of natural capital that represent pressures, states, or impacts and exclude indicators related to environmental policies or human health and welfare. Thus, environment-only indices only include indicators that fall under the environmental pillar of sustainable development (Fig. 1). In line with research by Bradshaw, Giam & Sodhi (2010), multidimensional indices use indicators that represent aspects of natural capital linked to environmental pressures, states, impacts as well as indicators related to environmental policies or human health and welfare. Human health and welfare functions provide services to humans which maintain health and contribute to human wellbeing in other ways—both economic and noneconomic (Usubiaga-Liaño & Ekins, 2021). Multidimensional indices typically incorporate (either directly or as calculation components) indicators such as human access to safe sanitation and drinking water, exposure to pollutants, and per capita income. Unlike environment-only indices, multidimensional indices include indicators that fall under all three pillars of sustainable development (Fig. 1).

Rank Correlations in Global Environmental Indices

There are considerable differences in rank among existing environmental indices. Here, we compare country ranks for each of the indices identified in ‘Summary of ten national-level indices’, and explore potential explanations for differences in ranking, such as conceptual framework and construction choices.

Methods

First, we examined rank correlations for global environmental indices. Data for the EPI (2020), LPI-NE (2021), EF per capita (EFpc) and EF total (EFtot) (2018) were sourced from official websites (Global Footprint Network, 2022; Legatum Institute, 2021; Yale Center for Environmental Law and Policy, 2023). Data for ESSI (2018), EVI (1999), aENV and pENV (2010) were obtained from academic publications (Bradshaw, Giam & Sodhi, 2010; Kaly, Pratt & Mitchell, 2004; Oţoiu & Grădinaru, 2018). Data for the CIEP (2015) was obtained directly from Dr Thiago Almeida (Campina Grande Federal University, Brazil), one of the developers of the CIEP (Almeida, 2015a).

Table 5:

A. Degree of Correlation. B. Correlation Matrix for EPI (2020), ESSI (2018), CIEP (2015), LPI-NE (2021), EVI (1999), EWI (2001), EF tot (2018), EF pp (2018), aENV (2010), pENV (2010).

(a) Degree of correlation.

(b) Correlation Matrix for EPI (2020), ESSI (2018), CIEP (2015), LPI-NE (2021), EVI (1999), EWI (2001), EF tot (2018), EF pp (2018), aENV (2010), pENV (2010).

DOI: 10.7717/peerj.16325/table-5

Notes:

*Correlation is significant at the 0.05 level (2-tailed). **Correlation is significant at the 0.01 level (2-tailed).

Abbreviations

CIEP

Composite Index of Environmental Performance

EWI

Ecosystem Wellbeing Index

EF tot

Ecological Footprint total

EF pc

Ecological Footprint per capita

EPI

Environmental Performance Index

ESSI

Environmental State & Sustainability and Index

EVI

Environmental Vulnerability Index

LPI-NE

Legatum Prosperity Index - Natural Environment Pillar

Ranks were compiled for each of the following indices: CIEP, EPI, ESSI, LPI-NE, aENV, pENV, EFpc and EFtot, EVI, EWI. Countries were only included if there was a value for each index, resulting in a total of 114 countries. Original rankings were normalized using the following formula: ${\displaystyle z_i=\frac{\left(x_i-min\left(x\right)\right)}{\left(max\left(x\right)-min\left(x\right)\right)}\ast 100}$ where z_i is the ith normalised value in the set, x_i is the ith value in the dataset, min(x) is the minimum value in the dataset, and max(x) is the maximum value in the dataset.

To determine the degree of agreement between ranks we used Kendall’s τ coefficient, which is a commonly used measure of rank correlation (Field, 2018). Because several ranks were tied, we chose Kendall’s τ coefficient over Spearman’s rank correlation because it is considered more appropriate in the case of tied ranks in the data set (Field, 2018). Kendall’s τ was calculated using IBM SPSS Statistics (Statistical Package for Social Sciences) for Windows, Version 28. If these distinct environmental indices assess similar measurable aspects of environmental features of countries, moderate-to-high positive correlations are to be expected.

Second, we investigated rank correlations for indices that were published in the same year. Original ranks were normalized using the method outlined previously and Kendall’s τ coefficient was used to determine the degree of agreement between ranks for: the pENV and EPI (2010), the aENV and EPI (2010), the EFpc and EPI (2018), EFtot and EPI (2018).

Third, we examined rank correlations for pillars within the EPI. For the 2010 version of the EPI, the correlation between the environmental health and ecosystem vitality pillars could be tested because the environmental health pillar consists exclusively of human health and welfare indicators, while the ecosystem vitality pillar consists exclusively of indicators of ecosystem pressures, states, and impacts. For the most recent versions of the EPI, human health and welfare indicators are no longer the sole components of the environmental health pillar. Thus, a simple correlation analysis for the pillars is not possible.

Finally, we compiled rank summaries for four countries to investigate rank patterns for low and high income countries. Germany and New Zealand represent high income countries in the Europe and Pacific regions, respectively. The countries of Mongolia and Niger represent low income countries in Asia and Africa, respectively. The four countries face a range of different environmental issues. Germany has a high emissions profile and faces issues with air and water pollution as well as biodiversity loss (European Environment Agency, 2023; OECD, 2023). Environmental issues in New Zealand include water pollution, soil degradation, biodiversity loss, and a high emissions profile (OECD, 2017; for Environment, 2022). In Niger, environmental issues include land degradation and desertification, pressure on natural resources from population growth, and pollution from mining (Larsen & Mamosso, 2013; The World Bank, 2021). Environmental issues in Mongolia include habitat and biodiversity loss, and pollution from mining (Rossabi, 2021).

Results

If these distinct environmental indices assess similar measurable aspects of environmental features of countries, moderate-to-high positive correlations are to be expected. However, we found positive correlations between multidimensional indices, positive or no correlations between environment-only indices, and negative or no correlations between multidimensional and environment-only indices.

Rank correlations for ten national level indices

Correlations ranged from moderate negative to strong positive (Table 5A and 5B). The highest positive correlation was between the EPI and ESSI, which indicates that the increase in a country’s EPI rank was associated with an increase in ESSI rank. There were moderate positive correlations between the EPI and LPI-NE, EPI and CIEP, CIEP and ESSI, CIEP and LPI-NE, ESSI and LPI-NE, aENV and EFtot, pENV and EVI, EVI and EWI. There were also weak positive correlations between the EFpc and EWI, EFtot and EWI, aENV and pENV, EWI and pENV.

There was a moderate negative correlation between the EPI and EFpc. This negative correlation indicates there was an inverse relationship between the two indices, i.e., the increase in a country’s rank in the EFpc was associated with a decrease in rank in the EPI (or vice versa). There were also weak negative correlations between the EFpc and ESSI, Efpc and CIEP, aENV and EPI, EWI and EPI, EFpc and LPI-NE, EWI and ESSI, EVI and ESSI, EWI and CIEP, and EVI and EPI.

Rank correlations for indices published in the same year

There was a moderate negative correlation between the 2018 versions of the EFpc and the EPI (Kendall τ = −0.483, P < 0.01). Ranks varied up to 92 places (normalized 0–100). There was a very weak negative correlation between country ranks for the 2018 versions of the EFtot and the EPI (Kendall τ = −0.109, P < 0.05) and country ranks varied by up to 93 places (normalized 0–100) (File S1).

There was a very weak negative correlation between country ranks for the 2010 versions of the aENV and EPI (Kendall τ = −0.072, P < 0.05) and country ranks varied up to 86 places (normalized 0–100). There was also a very weak negative correlation between ranks for the 2010 versions of the pENV and EPI (Kendall τ = −0.156, P < 0.01) and country ranks varied up to 99 places (normalized 0–100) (File S1).

Rank correlations for pillars within EPI (2010)

There was a weak negative correlation between the Ecosystem Vitality and Environmental Health pillars of the 2010 version of the EPI (Kendall τ = −0.219, P < 0.01). This indicates that an increase in a country’s environmental health ranking was associated with a decrease in its ecosystem vitality ranking (or vice versa) (File S1).

Summary of rankings for Germany, New Zealand, Mongolia and Niger

When normalized 0–100, New Zealand’s rank varied by up to 88.6 places (Fig. 2A) and Germany’s rank varied by 79 places (Fig. 2B). Germany and New Zealand achieved the highest scores in multidimensional indices, and the lowest scores in environment-only indices. Niger’s rank varied by up to 94 places (Fig. 2A), while Mongolia’s rank varied up to 80.3 places (Fig. 2B). Mongolia and Niger achieved the highest scores in environment-only indices, and the lowest scores in multidimensional indices.

Of note, when considering weighting methods there was no clear pattern across the different indices. For example, rankings for New Zealand and Niger were quite disparate in the ESSI and pENV (20.9 and 90.4; 95.7 and 2.3 respectively), even though both indices use statistics-based weighting procedures. However, when considering Germany, New Zealand, Mongolia, and Niger’s rankings across multidimensional and environment-only indices clear patterns emerge. Mongolia and Niger scored well in environment-only indices, while New Zealand and Germany scored well in multidimensional indices.

Discussion

Composite indices have been widely used to rank the environmental performance of nations. Despite their benefits, environmental indices can have numerous drawbacks; poorly constructed or communicated indices can undermine efforts to identify and address environmental failings and misguide policy messages and decisions (Böhringer & Jochem, 2007; Bradshaw, Giam & Sodhi, 2010; Haberland, 2008; Morse & Fraser, 2005). This review sought to answer three questions. The first question asked: which conceptual framework(s) or indicators inform the development of country rankings in environmental indices? We split environmental indices into two broad categories according to their conceptual frameworks and the indicators included. Multidimensional indices measure environmental performance in relation to human health and welfare functions, and typically incorporate indicators such as human access to safe sanitation, exposure to pollutants, and per capita income. Environment-only indices, on the other hand, focus exclusively on functions of natural capital that represent pressures, states, or impacts and explicitly exclude indicators related to human health and welfare. Both multidimensional and environment-only indices use a range of weighting and aggregation methods.

Second, we sought to understand the impact of different conceptual frameworks (and indicators) on global rankings. We examined rank correlation for ten national level environmental indices, and where possible, we investigated rank correlations for indices published in the same year. While previous research has demonstrated the sensitivity of rankings to different weighting methods (Halkos & Zisiadou, 2018; Pinar, 2022), in our analysis we did not observe any clear patterns across the indices. Our results indicate the following patterns: (1) multidimensional indices (CIEP, EPI, ESSI, LPI-NE) were positively correlated with one another, (2) environment-only indices (EFpc, EFtot, aENV, pENV, EVI, EWI) were positively correlated with one another, or there is no correlation, and (3) multidimensional indices were either negatively correlated with environment-only indices, or there is no correlation. Researchers have used a range of weighting methods. The multidimensional indices included in this study showed much stronger positive correlations with each other, than do environment-only indices. This may be due to similarities in the indicators and datasets used. The ESSI, for example, was developed to guide revisions of the EPI and uses similar indicators but with a statistics-based weighting system to account for relationships between them. However, the general pattern of correlations observed for multidimensional and environment-only indices indicate that conceptual framework and indicator choice may influence a country’s rank among different indices.

As a complement to our analysis of rank variation for existing national-level indices, we examined rank correlations within the EPI’s pillars. The ecosystem vitality pillar includes indicators that fall under the environmental pillar of sustainable development, whereas the environmental health pillar includes indicators that fall under the social and economic pillars of sustainability. The negative correlation between the environmental health and ecosystem vitality pillars of the 2010 version indicates that an increase in a country’s environmental health rank is associated with a decrease in the ecosystem vitality ranking. Our findings support previous research (Bradshaw, Giam & Sodhi, 2010) and demonstrates that environmental sustainability can be distinguished from social and economic sustainability. This highlights how the inclusion of human health and welfare indicators may influence a country’s overall rank with the potential to misrepresent the environmental sustainability of different countries. The environmental health pillar of most recent version of the EPI includes indicators relating to waste management as well as human health and welfare indicators. Thus, it is not straightforward to compare ranks for the environmental health and ecosystem vitality pillars. While it is beyond the scope of this article to disaggregate and re-weight the indicators and issue categories of the most recent version of the EPI, this could be explored in future research.

One limitation in our correlation study is that the release dates for the indices range over several years. Unfortunately, some indices only have one iteration, and it was therefore not possible to conduct a year-on-year comparison for every index. While this is an important limitation, analyses of indices such as the EPI have confirmed strong positive correlations between ranks over multiple iterations (File S2). This indicates that country ranks within the same index do not vary markedly over time and provides some assurance in the results reported here. In particular, the EPI (and other indices such as the LPI-NE) could be constructed without indicators related to human health and welfare, and have ranks compared to those in existing environment-only indices, such as the Ecological Footprint. As the EPI uses expert opinion-based weighting, different weighting procedures (e.g., multivariate analysis) could also be considered along with sensitivity analysis. Additionally, subsets of data for the same years could be analysed for a more direct comparison.

Thirdly, we sought to understand the extent to which the different environmental indices capture environmental problems in low and high income countries. We have used the examples of Germany, New Zealand, Mongolia, and Niger to illustrate the complexities involved when interpreting ranks among different indices. Low income countries, such as Mongolia and Niger, often score poorly in multidimensional indices and well in environment-only indices. In Niger 42.9% of the population live in extreme poverty (The World Bank, 2021) and only 13% of the population has access to basic sanitation services (UNICEF, 2020). In Mongolia, 28% of the population lives in poverty (The World Bank, 2021) and only 27% of the population has access to sanitation (UNICEF, 2017). Thus, the inclusion of indicators such as access to sanitation and GDP in multidimensional indices complicates interpretation, and ultimately leads to a less optimistic view of Mongolia and Niger’s environments when compared to high income nations. By contrast Germany and New Zealand both score well in multidimensional indices. Germany and New Zealand are classified as a high income nations and this is reflected in indicators such as access to clean drinking water and sanitation (UNDESA, 2021; World Bank, 2023a; World Bank, 2023b; World Bank, 2023c). The inclusion of such indicators provides a more optimistic view of German and New Zealand’s environments when compared to lower income nations. Because interactions between economic growth, improvements in human health and welfare, and environmental quality are complex, it is important to treat multidimensional indices with caution—multidimensional indices require more layers of interpretation than environment-only indices.

Research looking at rank differences between environmental indices and socio-economic has largely focussed on relationships between the EPI and GDP per capita, human development (Lai & Chen, 2020; Tektüfekçi & Kutay, 2016; Wolf et al., 2022a; Wolf et al., 2022b; Wurie & Pillai, 2014). While the scope of this article was concentrated on examining correlations in environmental indices, the dataset collated for the present research could be utilised to extend research into correlations between multidimensional and environment-only indices and other composite indices or indicators of human health and welfare such as GDP. These may include sustainability and climate indices such as the HDI, Global Sustainability Index (GSI), Sustainable Development Index (SDI), and Global Social Mobility Index.

Our article provides insight into the benefits and challenges of constructing environmental indices, which can help inform the future design of (improved) indices. It is important to assess whether the conceptual framework and indicators included are appropriate. If an index is intended to measure the qualities or functions of natural capital linked to environmental as well as human health and welfare or policy processes, then it may be appropriate to include indicators such as access to sanitation, indoor air pollution, and per capita income. If the index is intended to measure the environment-only, then functions linked to human health and welfare, or policy should be excluded. If such indicators are included, the actual state of natural environments may not be accurately represented.

In this article we have: (1) examined existing environmental indices and quantified rank variations, (2) identified patterns across multidimensional and environment-only indices, and (3) examined how conceptual frameworks and methodological choices influence global rankings. Our findings highlight wide variations in country ranks among existing environmental indices. Specifically, our results suggest that conceptual framework and indicator choice can affect ranking and produce a more, or less, optimistic view of a country’s environment. While weighting procedures and sensitivity analysis may have some impact on overall ranking, it is likely that conceptual framework and indicator choice have a larger effect.

Supplemental Information