Prognosis predictive value of the Oxford Acute Severity of Illness Score for sepsis: a retrospective cohort study

Introduction

Scoring systems for outcome prediction have been developed in intensive care medicine for a long time, and some of them have been widely used in the intensive care unit (ICU) (Rapsang & Shyam, 2014). For patients with sepsis, the Sepsis-related Organ Failure Assessment (SOFA) score, which was initially designed to evaluate the severity of organ dysfunction in patients who were critically ill from sepsis (Vincent et al., 1998), has been proposed as a tool to facilitate the identification of patients at risk of dying from sepsis (Singer et al., 2016). The SOFA system uses a few measurements of major organ function to calculate a severity score, which contains some laboratory results such as platelets and bilirubin. Other scoring systems which consisted of more complex parameters such as the Simplified Acute Physiology Score II (SAPS II) (Le Gall, Lemeshow & Saulnier, 1993) were also widely used in ICU practice. However, as we know, clinicians usually prefer an instrument that is not laboratory-dependent and is easy to use. The quick SOFA (qSOFA) score, which consists of only three parameters and can be easily measured at the bedside, has also been proposed as a tool to help identify patients with early sepsis outside of the ICU (Seymour et al., 2016), but it was reported to have poor accuracy for predicting 28-day mortality in critically ill septic patients (Hwang et al., 2018). Johnson, Kramer & Clifford (2013) developed a new reduced severity of illness score using machine-learning algorithms, the Oxford Acute Severity of Illness Score (OASIS), which contained 10 parameters without any laboratory tests and had discrimination and calibration equivalent to more complex existing models. Given that the predictive value of the OASIS was only validated in mixed ICU patient populations, its performance in septic patients remains unknown. In this study, we evaluated the association of the OASIS with outcomes of septic patients in ICU, and examined its predictive value mainly by comparison with the SOFA score.

Methods

Results

Association of the OASIS with clinical outcomes of septic patients

As shown in Table 1, non-survivors had significantly higher OASIS on ICU admission than survivors (38 vs 33, p < 0.001). The distributions of the OASIS and the SOFA score with corresponding hospital mortality are presented in Fig. 1. As each score increased, the hospital mortality of the patients approximately increased accordingly. According to the result of univariable logistic regression analyses for variables associated with hospital mortality which was presented in Table S2, age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30) were included in the multivariable regression analyses. Results of curve fitting of the relationship between hospital mortality and several continuous variables were presented in Fig. S1, which indicated that the independent variables were linearly related to the log odds. Schoenfeld residual plots shown in Fig. S2 supported the proportional hazard assumption in Cox models. After adjusting the variables above, the OASIS was significantly associated with hospital mortality (Odds ratio (OR) 1.07 per one-point increase, 95% confidence interval (CI) [1.06–1.08], p < 0.0001), ICU mortality (OR 1.07 per one-point increase, 95% CI [1.06–1.08], p < 0.001), and 28-day mortality (hazard ratio (HR) 1.06 per one-point increase, 95% CI [1.05–1.06], p < 0.001) (Table 2). Results of analyses of the SOFA score and the SAPS II are presented in Tables S3 and S4, respectively.

Table 2:

Association of OASIS with hospital mortality, ICU mortality, and 28-day mortality.

Outcomes	OR/HR	95% CI	p
Hospital mortality
Non-adjusted	1.07	[1.06–1.08]	<0.001
Adjusted	1.07	[1.06–1.08]	<0.001
ICU mortality
Non-adjusted	1.09	[1.08–1.10]	<0.001
Adjusted	1.07	[1.06–1.08]	<0.001
28-day mortality
Non-adjusted	1.06	[1.05–1.06]	<0.001
Adjusted	1.06	[1.05–1.06]	<0.001

DOI: 10.7717/peerj.7083/table-2

Notes:

Associations of OASIS with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of OASIS with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Statistical significance (p < 0.05) is shown in bold.

OASIS, the Oxford Acute Severity of Illness Score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.

Discriminatory power of the OASIS in septic patients

As shown in Fig. 2, the AUC of the OASIS for predicting hospital mortality was 0.652 (95% CI [0.636–0.668]), which was significantly lower than that of the SOFA score (AUC 0.682, 95% CI [0.666–0.699], p < 0.001). The best threshold of OASIS was 34.5 with a specificity of 55.80%, a sensitivity of 64.93%, a positive likelihood ratio of 1.47 and a negative likelihood ratio of 0.63. However, the discriminatory power of both scores were all significantly lower than that of the SAPS II (AUC 0.739, 95% CI [0.725–0.753]). Results of comparison of the three scores were similar when they were used to predict ICU mortality (Fig. 2). According to the Kaplan–Meier curves presented in Figs. S3, S4 and S5, patients with higher severity scores had shorter survival time regardless of which scoring system was used.

Sensitive analyses

Results of the sensitive analyses were consistent. The OASIS was still significantly associated with hospital mortality (OR = 1.03, 95% CI [1.01–1.05], p < 0.001) and 28-day mortality (HR = 1.03, 95% CI [1.01–1.04], p < 0.001) when patients with an ICD-9-CM code for sepsis were included only, but no significant association was found between the OASIS and ICU mortality (OR = 1.02, 95% CI [1.00–1.04], p = 0.064) (See Table 3). Association of the other two scores with outcomes were presented in Tables S5 and S6, respectively. In the sensitive analyses, the OASIS still had the lowest discriminatory power for predicting hospital mortality (AUC 0.586, 95% CI [0.555–0.616]) and ICU mortality (AUC 0.608, 95% CI [0.569–0.646]) when compared with the SOFA score and the SAPS II (Fig. 3).

Table 3:

Sensitive analysis of association of OASIS with hospital mortality, ICU mortality, and 28-day mortality.

Outcomes	OR/HR	95% CI	p
Hospital mortality
Non-adjusted	1.04	[1.02–1.05]	<0.001
Adjusted	1.03	[1.01–1.05]	<0.001
ICU mortality
Non-adjusted	1.04	[1.03–1.06]	<0.001
Adjusted	1.02	[1.00–1.04]	0.064
28-day mortality
Non-adjusted	1.03	[1.02–1.04]	<0.001
Adjusted	1.03	[1.01–1.04]	<0.001

DOI: 10.7717/peerj.7083/table-3

Notes:

Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of OASIS with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of OASIS with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Statistical significance (p < 0.05) is shown in bold.

OASIS, Oxford Acute Severity of Illness Score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.

Subgroup analyses

Associations of the OASIS with hospital mortality were analyzed using logistic regression models across different subgroups, and the result presented in Table S7 found several variables had a p-value of interaction less than 0.05. After professional judgement, only the SAPS II was chosen as a potential interaction modifier and examined in the subgroup analysis further. As shown in Fig. 4 and Table S8, there was no statistically significant difference in the discriminatory power of the OASIS and the SOFA score for predicting hospital mortality when a SAPS II was lower than 50. In addition, the SOFA score had significantly higher AUC than that of the OASIS for predicting ICU mortality only when a SAPS II was higher than 35.

Discussion

In the study, we evaluated the performance of the OASIS for predicting clinical outcomes of patients with sepsis in ICU. Results of our study indicated that the OASIS on ICU admission was significantly associated with short-term clinical outcomes of patients with sepsis, but its discriminatory abilities of hospital mortality and ICU mortality were statistically significantly lower than that of SAPS II. However, an interesting finding was that there was no statistically significant difference between the mortality discriminatory power of the OASIS and that of the SOFA score when the septic patients were not that severe (evaluated by their SAPS II). Even when the patients were more severe, it should be noticed that the absolute AUC difference between the OASIS and the SOFA score was rather small and there was also an overlap between the 95% CIs of the two scores, so it should be cautious when interpreting the clinical significance of the results. Nevertheless, as far as we know, this is the first study that assesses the prognostic predictive value of the OASIS in patients with sepsis in ICU, which might help clinicians to make good use of this newly developed scoring system.

Since the prediction of ICU outcomes might contribute to clinical research and individual patient management (Power & Harrison, 2014), many scoring systems for ICU outcome prediction have been developed and some of them have been routinely used. In terms of sepsis, the most famous scoring system is the SOFA score, an indicator of identifying organ dysfunction and part of the diagnosis criteria of sepsis-3 (Singer et al., 2016). The association of the SOFA score with ICU outcomes has been assessed either in mixed ICU patients or septic patients (Ferreira et al., 2001; Jones, Trzeciak & Kline, 2009; Lie et al., 2018), and it has been reported to be a valuable predictor of ICU outcomes. As shown in Table S3 and Fig. 2, results of our study about the SOFA score were consistent. Unlike the SOFA score, the SAPS II is a widely used disease severity scoring system aimed at mixed ICU patients, and it has been reported to perform better for predicting hospital mortality and 90-day mortality than the SOFA score (Granholm et al., 2016). Similar results were also observed in our study that the SAPS II had the best discriminatory power for hospital mortality and ICU mortality among the three scoring systems analyzed. It is not surprising to find the results, since the SAPS II scoring system has more complicated parameters than the other two systems. However, the complexity of the SAPS II scoring system might be a challenge in daily clinical assessment, which impelled us to investigate the performance of the OASIS.

The OASIS system only consists of 10 easily accessible parameters including length of hospital stay prior ICU admission, age, Glasgow Coma Score (Teasdale & Jennett, 1974), heart rate, mean arterial pressure, respiratory rate, temperature, urine output, ventilation, and elective surgery. In fact, the OASIS system was developed from the popular severity scoring systems APACHE IV (Acute Physiology, Age, and Chronic Health Evaluation VI) with the aim of reducing the number of parameters without losing predictive accuracy (Johnson, Kramer & Clifford, 2013). The goal has been achieved to some extent, however, as far as we know, the discriminatory ability of the OASIS for outcomes of septic patients has not been examined especially when compared with the SOFA score, which has similar numbers of items. Before the study we assumed that the OASIS had poorer prognostic predictive value that that of the SOFA score, for the reason that the OASIS does not contain any laboratory parameters. However, surprisingly, we found that the performance of the OASIS was not different from that of the SOFA score when the septic patients were of mild conditions. Although it is impractical to choose and use the OASIS after the SAPS II of a septic patient has been calculated to determine the severity of the patient, we believe that the OASIS is still a promising tool. Clearly, the simple components of the OASIS make it possible to be calculated automatically, which may provide constructive information especially when few laboratory examinations have been done. In addition, in our previous study (Chen et al., 2018) we have found the OASIS might serve as a supplement to the qSOFA score and help to identify more septic patients. Therefore, we reason that perhaps it might also serve as a supplement to the SOFA score to predict outcomes of septic patients. However, further researches are needed to explore the possibilities.

Another issue needed to be discussed is the criteria we used to identify sepsis. In the study, we initially identified septic patients from the MIMIC-3 database using Sepsis-3 criteria. However, this method assumed a causation from infection to organ dysfunction (the SOFA score ≥2) while in a lot of cases there might be only associations, which might lead to an overestimation of case numbers and an underestimation of mortality (Fleischmann-Struzek et al., 2018). Thus, we then performed sensitive analyses using the ICD-9-CM codes to identify sepsis. Results of the sensitive analyses were consistent, indicating the robustness of our conclusions. However, it should be noticed that the two criteria might represent two distinct populations of sepsis, since the Sepsis-3 criteria was developed in 2016. In our study, the patients identified by the ICD-9-CM codes had higher hospital mortality rate (23.41%) with a median SAPS II of 49.5 and a median SOFA score of 6, so it is not strange to find a lower AUC of the OASIS for hospital mortality in the sensitive analyses since they were more severe.

Several limitations of the study should be noticed. First of all, given the observational nature of our study, selection bias was inevitable although we adjusted a few potential confounders. Second, only the discriminatory power of the OASIS was evaluated, we did not assess its calibration. Third, several variables were not considered. The actual admission years of the patients in the database had been shifted to protect patient confidentiality, so we did not include it in the regression models. Since the database included patients from 2001 to 2012, there might be some differences in outcomes of the patients treated in the early 2000s and later, but we did not take therapy as a covariate into consideration. Last but not least, since all the data in the study was treated as a single dataset, the performance of the OASIS was inherently optimistic.

Conclusions

To sum up, the retrospective observational study validated significant associations of the OASIS on ICU admission with short-term outcomes of septic patients, and found that although the discriminatory power of the OASIS for hospital mortality was statistically significantly lower than that of the SOFA score, the significant difference existed only when a SAPS II was higher than 50, which suggesting that the OASIS might serve as an initial predictor of clinical outcomes for septic patients, but one should be circumspect when it is applied to more severe patients.

Supplemental Information