Systematic reviews and the Journal of Antimicrobial Chemotherapy; past, present and future. A systematic reappraisal

Introduction

There have been >200 systematic reviews (SRs) published in the Journal of Antimicrobial Chemotherapy (JAC) over the past 20 years. A SR attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer specific research questions. Researchers conducting SRs use explicit, systematic methods to minimize bias, in producing robust findings to inform decision making.1Statistical methods (meta-analysis) may or may not be used to analyse and summarize the results of the included studies. SRs are placed high in the evidence-based hierarchy.

When this topic was previously reviewed in 2005, fewer than 10 SRs had appeared in JAC and both ‘systematic review’ and ‘meta-analysis’ were novel techniques applied in deriving summary estimates of treatment effects from published randomized controlled trials (RCTs) towards the development of evidence-based treatment guidelines.2 Today there would be few interventions examined in an RCT that have not been included within several SRs.

The expectations for how SRs should be properly undertaken has been considered previously. Since then, new challenges have emerged that have led some to question the value of SRs and how they might best serve evidence-based medicine.3–5 Some have stated that ‘…a sizeable proportion are flawed beyond repair…’3 Checklists, such as the PRISMA2020 checklist, improve the reproducibility of an SR.6

This review identifies several emerging challenges and suggests that better methods of displaying and contextualizing the study results with external benchmarks will be required for SRs to remain high within the evidence hierarchy.

Past

Here, I review some SRs published in JAC. The search, and citation count, were obtained using Google scholar for publications in JAC with the term ‘systematic review’ in the title (Figure 1). Half of the SRs have appeared in only the past 7 years with 11 relating to COVID.

Open multimedia modal

Several were highly cited with 17 generating follow up comment or correspondence in the journal. This suggests that SRs addressing interventions that are either controversial, emerging or for which the evidence is mixed are of great interest.

One of the earliest SRs examined the evidence for the association of specific antibiotics with hospital-acquired Clostridium difficile-associated diarrhoea. This SR (cited >400 times) noted that many of the primary studies had limitations with use of incorrect control groups, the presence of bias, inadequate control of confounding and small sample sizes and hence was unable to establish causal relationships.7

The most widely cited SR (>850 citations) examined the association of extended-spectrum β-lactamase production on mortality and the delay in effective therapy in Enterobacteriaceae bacteraemia. This SR likewise acknowledged that the lack of controlled studies limits any interpretation regarding causality in this association.8

The SR with the most included studies (n = 318) examined the question of drug–drug interactions among patients receiving antibacterials for COPD exacerbations. As the authors note, many of the included studies were observational in nature.9

One notable SR examined the question of drug dosing for septic patients with acute renal failure receiving continuous renal replacement therapy. This SR found that out of 64 potentially eligible studies, none reported sufficient pharmacokinetic information, such as volume of distribution and clearance, on which to base drug dosing regimens.10 The paucity of suitable RCTs on which to base clinical practice guidelines has been noted elsewhere.11

Two SRs examined diagnostic approaches for the rapid detection of antimicrobial resistance in Mycobacterium tuberculosis.12^,13 These meta-analyses used summary receiver operating characteristic curves to summarize the diagnostic accuracy.14

Other SRs address questions relating to aetiology, prevalence and adverse effects as well antimicrobial chemotherapy used as prevention or treatment of infectious diseases for individual patients.

Several SRs have addressed research questions related to populations. For example, a SR including >170 studies of carbapenem resistance among clinical isolates of Acinetobacter baumannii was able to display the estimated resistance rates by country on a map of the world.15

Present

The poor quality of the primary studies, as highlighted by several examples before, limits the validity of SRs. Traditionally, SRs have address research questions framed using PICO components to specify the population (P), intervention (I), comparison (C) and outcome (O). The ideal study design towards estimating causal effects of interventions is an RCT, adequately powered and free from bias. The few published studies attaining this ideal are generally outnumbered by those that are either non-randomized, uncontrolled or studies with before–after designs. What should an SR do with these non-RCT observations?

A serious challenge, which goes beyond SRs, is fraudulent data. There is no simple solution to this problem except constant vigilance by authors, editors, reviewers and readers.16–18

Another challenge is the increasing number of potential studies that an SR might include. There have been 12 SRs in JAC with >100 included studies of which 11 were published in the last decade. How best to present large amounts of data has emerging solutions. For example, an SR of antibiotic prescribing in COVID-19 patients in China including >250 studies used violin plots, which are a hybrid of a box plot and a kernel density plot, to summarize the large amount of study data to enable visual contrasts of the distribution of numeric data across several groups.19

Even within studies that achieve the RCT ideal, there may be either more than one treatment group or more than one control group. Often SRs will drop these additional arms from the analysis. Is this loss of information, sometimes substantial, avoidable?

Which approach to meta-analysis is best? A viewpoint regarding the safety of cefipime use as part of empiric therapy for febrile neutropenia highlighted that reanalysing data published by the US FDA using Bayesian methods of meta-analysis leads to inferences contrary to that obtained using the traditional frequentist methods.20 This potential for differing inferences from Bayesian versus frequentist methods applied to the same data is not unique to meta-analysis.21 Frequentist methods derive inference from imaginary repetitions of the same experiment projected as a data distribution. With Bayesian methods, by contrast, the inference is derived from the investigator’s prior belief, whether enthusiastic versus sceptical, as modified by the observed data.21

There are many important research questions that would be difficult to address with an RCT, such as those relating to potential adverse effects or resulting in patient harm, as in the cefipime example, or where the outcome in question is rare. How do we assess evidence towards the development of guidelines for these questions?

Stable unit treatment variable assumption

Research questions estimating intervention effects for individuals use an RCT design wherein individual patients are randomized. Estimating intervention effects for populations requires a cluster randomized trial (CRT) design wherein whole populations are randomized.22 The RCT versus CRT approach will answer the same research question at different levels, a distinction that is especially relevant for research questions relating to the population effects of antimicrobial chemotherapy.23 For example, the potential for acquiring resistance in the population deserves more attention in SRs of antimicrobial use.24

Of note, the CRT, but not the RCT, avoids any spillover of intervention effect between control and intervention groups. Any potential for spillover invalidates the stable unit treatment variable assumption (SUTVA). SUTVA, although fundamental to causal inference conferred by randomized treatment allocation, is rarely mentioned.

SUTVA can only be verified by referencing data external to the study. For example, RCTs of vaccine interventions for transmissible infections might show seemingly ineffective effect sizes (ESs), as spillover of prevention effect to concurrent unvaccinated control groups bias the study result towards the null. Observing lower infection rates in both control and intervention groups versus comparable non-exposed populations suggests spillover of prevention effects occurred within the RCT.25 Conversely, the potential for spillover of harmful effects from intervention to control groups should also be considered, as this could inflate the apparent ES away from the null.

Risk of bias

The summary findings within SRs are increasingly accompanied by certainty ratings based on assessments of risk of bias (ROB), including the directness of the evidence, the consistency of the results, the precision of the estimates and the risk of publication bias. How reliable are these certainty ratings in addressing the inclusion of suboptimal studies within SRs? Moreover, certainty is a loaded term that implies certainty in the findings beyond merely observations deemed to be reliable.

The ROB assessed for each individual study includes elements such as sequence generation and allocation concealment (selection bias), blinding of participants and providers (performance bias), blinding of outcome assessors (detection bias), incomplete outcome data (attrition bias) and selective outcome reporting (reporting bias). The process of assessing ROB for both randomized and non-randomized studies is comprehensively detailed in The Cochrane Handbook.26 Of note, the ROB does not assess SUTVA, as this requires reference to observations external to the RCT.

ROS scores may provide a basis for exploring how the study ES estimates vary by study quality. This may be especially relevant for study end points that are subjective, such as the diagnosis of ventilator associated pneumonia (VAP) among mechanically ventilated ICU patients.27

Heterogeneity and I²

Heterogeneity within SRs is often represented by the I² metric, which is the proportion of total variability in ES due to between study heterogeneity and judged on a scale with arbitrary levels of concern. The I² metric is widely misunderstood.28^,29I² is not an absolute measure of heterogeneity. Being a proportion, I² is a relative measure. Heterogeneity is poorly estimated where there are few studies and provides false reassurance when heterogeneity appears absent. The L’abbe plots display the relationship between event rates in the control and intervention groups and provide a visual indication of heterogeneity. However, as with the I² metric, heterogeneity inferences for the same data in a L’abbe plot will differ depending on whether the ES is derived as odds ratios, risk ratios or risk differences.30

A better guide than the I² metric to gauge heterogeneity among the individual study results within a meta-analysis is the 95% prediction interval (95% PI) associated with the summary ES.31^,32 The 95% PI incorporates the between study variance (tau²) in its calculation and is wider than the 95% confidence interval (95% CI). The 95% PI indicates where the ES of 95 of the next hypothetical 100 studies would probably be found whereas the 95% CI, which is narrower than the 95% PI, represents the limits between which 95 of the next hypothetical 100 replications of summary ES estimates would be expected to lie.

Contrast- versus arm-based meta-analysis

Traditionally, the meta-analysis ES of interest arise from the contrast between the pair of singleton control and intervention groups within each RCT. For data originating from ‘ideal’ RCT designs, a causal inference can be inferred from this contrast-based analysis. Increasingly there is a need to include studies that may have multiple treatment or control groups. Techniques are emerging to accommodate these multi-arm studies in arm-based analyses. One recent example is a network meta-analysis (NMA) of data from 70 multi-arm studies that examined whether vancomycin plus piperacillin/tazobactam increases nephrotoxicity compared with other anti-pseudomonal beta-lactams such as cefipime or meropenem.33

The merits and limitations of contrast-based versus the emerging arm-based methods are controversial but the latter are gaining increasing acceptance.34 The strength of contrast-based methods relies on the causal inference conferred by random assignment, which is lost in an arms-based analysis. Of note, arm-based methods are accepted and widely used in the meta-analysis of diagnostic tests as the interest is usually in the arm-based properties of a test, being summary sensitivity and specificity.14

A case study: ‘VAP zero’

The findings of several SRs of >200 RCTs suggest that VAP is preventable among mechanical ventilated ICU patients through various intervention methods.35–47 The summary findings of 13 Cochrane reviews of VAP prevention interventions imply that topical antibiotic prophylaxis (TAP), with protocolized parenteral antibiotic prophylaxis (PPAP) as selective digestive decontamination (SDD) or without PPAP, as selective oropharyngeal decontamination, is superior to topical chlorhexidine which in turn is superior to various non-antimicrobial interventions, which are summarized here as a composite (Table 1). Moreover, the TAP + PPAP combination appears to prevent mortality, a finding that is rated with high certainty. Are inferences from these antimicrobial SRs safe? Moreover, do they allow projections towards attaining ‘zero VAP’ among ICU populations?49

Open multimedia modal

The methods, techniques of literature searching, ROB assessments and data extraction among these SRs, being as prescribed by The Cochrane Handbook,1 are exemplary. Furthermore, Cochrane SRs are distinguished by their rigorous attention to detail, providing study data that allows for the replication of each meta-analysis in both contrast-based and arms-based analyses.48^,50–53

Among Cochrane reviews relating to VAP prevention are two pairs relating to topical chlorhexidine, appearing in 2017 and 2020, and two relating to TAP ± PPAP, appearing in 2009 and 2021. Each pair included similar primary studies and share authorship. There are two specific minor concerns and five major concerns (Tables S1–S3, available as Supplementary data at JAC Online). Moreover, several underlying assumptions differ, which illustrate how inferences from these SRs pivot on specific, and unstated, assumptions.

First, of minor concern, the more recent SR of topical chlorhexidine excludes or reclassifies eight studies included in the earlier SR without ROB concerns identified. While these exclusions are acknowledged in the more recent SR, the resulting summary VAP prevention ES calculated is slightly magnified, but the certainty is downrated.

Second, the heterogeneity metrics associated with all summary ESs substantially misrepresent the study heterogeneity. These metrics, which appear acceptable, merely estimate the statistical heterogeneity in each ES. With studies of >15 regimens among diverse patient populations and ICU settings included among these SRs, which are the summary ES representative of? Moreover, the substantial dispersion in VAP proportion among the component groups has not been considered (Figure 2).

Open multimedia modal

The first major concern is that the most recent SR of TAP + PAPP includes one large CRT57 of TAP + PPAP (risk ratio ES 1.01; 0.87–1.16), which contributes 14% by weight to, but differs significantly from, the overall mortality prevention summary ES (0.84; 0.73–0.96), which is rated with high certainty. This discrepancy is not discussed within this SR. Is this high certainty rating justified if the summary ES fails to predict the results of the largest component study?

Likewise, another SR58 found the summary ES for mortality prevention derived from 27 studies with individual patient randomization (5699 patients), most published before 2009, was an apparent prevention effect (log risk ratio −0.16; −0.26 to −0.06) whereas the summary ES from three cluster randomized studies with randomization of populations (18 335 patients), all published since 2009, showed zero prevention effect (log risk ratio 0; −0.24 to +0.26). Why the difference?

Second, the pneumonia proportions are unusually high. The expert range for VAP proportion, as acknowledged in these SRs, is 5% to 40%. However, among these SRs, the VAP incidence exceeds 40% for the control groups of 30% and 50% of the chlorhexidine and TAP ± PPAP studies, respectively (Figure 2).54–56 These high pneumonia proportions, noted in the median event rates in the SR summary tables, originate from studies without major ROB concerns identified and remain unexplained. They become apparent only in an arms-based reanalysis of the VAP data from all 13 SRs (Figures 2 and 3). Surprisingly, the mean pneumonia proportions and 95% PIs of the TAP ± PPAP, chlorhexidine and non-antimicrobial intervention groups align closely not only with each other but also with the expert range and also align with the 95% PIs of the non-antibiotic control groups (Figure 2).

Open multimedia modal

Likewise, other end points among control groups of decontamination studies are unusually high whereas those among intervention groups are not unusually low when compared with external benchmarks. The bacteraemia proportions among control groups of studies of TAP ± PPAP as summarized in another SR,58 are also unusually high.59 The mortality incidence, using even the conservative counts from Minozzi47 and even after adjusting for group mean age, among concurrent control groups of antimicrobial-based decontamination studies, is five percentage points higher than expected and showed greater dispersion, whereas that among the intervention groups is comparable to what is expected.60

Third, the pneumonia proportions among the chlorhexidine and TAP ± PPAP intervention groups are not unusually low with only two of 20 and three of 30, respectively, being <5% versus 10 of 115 among the non-antimicrobial intervention groups (Figure 2).

Fourth, there are major differences in numerator and denominator counts on which the summary ES estimates are based (Tables S2 and S3) with several differences exceeding 20% between counts in the Liberati46 versus Minozzi47 SRs. As noted in the SRs, these differences can be attributed to whether the counts were obtained from the studies as published (Minozzi47) or from the authors by personal communication (Liberati46). These differences presumably originate from assumptions in the early SDD studies wherein pneumonia rather than mortality was the primary end point of interest, and the TAP intervention was believed to be most effective towards preventing infection acquired after the first 4 days of application. On that basis, excluding the early counts to obtain per-protocol data towards pneumonia ES estimates was logical.61 However, the discrepancies are not acknowledged in the most recent SR, nor are they obvious to an inexpert reader, and excluding early mortality counts is problematic. Moreover, more recent studies of the TAP intervention use intention to treat data without excluding early counts.57

Finally, the most serious concern relates to whether the SUTVA assumption is reasonable in the ICU setting? The first study of TAP,62 and others,57 thought not and presumed that the results of TAP intervention would differ for CRT versus RCT studies. SUTVA has not been considered, let alone tested in these SRs. If spillover is possible, it is neither logical nor safe to use per-protocol data.

The inference pivot

Valid causal inferences from the SRs of chlorhexidine and TAP ± PPAP pivot on the required assumptions, being that SUTVA is valid, there is no spillover, the ROB and heterogeneity are acceptable, and that the data as published are fit for the PICO research question, which relates to VAP prevention interventions targeted at individual patients. Guidelines for zero-pneumonia can build on these causal inferences only if the assumptions are deemed tenable. However, the unusually high VAP incidences, together with the other unacknowledged data discrepancies, cast doubt on the ‘certainty’ of the SR findings.

Does the ES from RCTs vary with underlying risk?63 Differences in underlying risk were found to explain the disparate results between the preclinical and clinical trials of anti-inflammatory agents for sepsis.64

The studies of chlorhexidine and TAP interventions that have control groups with unusually high VAP proportions might differ for inapparent reasons from those with VAP within the expected range. Recalculating the summary ES separately for subgroups of studies with control group VAP proportions greater than versus less than 40% for all antimicrobial (chlorhexidine and TAP ± PPAP) studies gives a summary ES (and 95% CI) of 0.26; 0.18–0.35 (n = 35) versus 0.59; 0.48–0.73 (n = 33), respectively (Figure 3) whereas the mean and 95% PI for the two strata of intervention groups in these studies, being 25; 9.2–52 and 13; 4.2–35, respectively, each align with the expected VAP range.48 This does not augur well for zero VAP.

However, when the RCT incidence data is benchmarked against incidence data observed outside the RCTs, such as the VAP range issued by expert opinion, the assumptions underlying the antimicrobial SRs are questionable. If the assumptions are deemed not tenable, are the inferences from these SRs ‘flawed beyond repair’?3 Moreover, for population-based initiatives, such as zero-Pneumonia, evaluation of these interventions within CRTs, wherein ICU populations are randomized, would be preferrable.

The causal inference pivots if viewed from a population perspective. Spillover, which is harmful, occurring in RCTs of topical antimicrobial interventions in the ICU setting would explain the unusually high VAP proportions in RCTs. Moreover, the absence of spillover in CRTs would explain the zero ES of decontamination interventions there. Of note, these two inferences, with benefit at the individual level versus harm at the population level, are not incompatible.

Future

To remain relevant to evidence-based medicine, SRs will need to recognize and validate the key underlying assumptions. In developing guidelines for ‘pneumonia zero’, are the findings of SRs of various VAP prevention interventions safe? Are they useable? The case study previously raises a more general question for any SR. How can SRs best recognize and validate the underlying assumptions? Increasingly, this will require data from external sources relevant to the PICO research question and scrutiny of the study data versus external benchmarks if available.65^,66

NMA is an emerging approach to deal with several of the challenges raised previously. NMA requires careful attention to ensure the comparator groups are truly comparable. The findings of an NMA for antibiotic prophylaxis in patients with cancer differed depending on how the comparator and control groups were grouped.67 NMA has additional assumptions, particularly that of transitivity, being that any effect modifiers are similarly distributed across the observed trial comparisons in the analysis network.67–70 One NMA example is that of antibiotic choice for the medical versus surgical management of acute appendicitis. This NMA enabled broad comparisons using data obtained from several RCTs.68 However, for this NMA, the RCT data were limited so observational studies were incorporated using a Bayesian approach.

Graphical displays, such as the caterpillar plot, are better than the traditional forest plot for visualizing data dispersion and the balance between potential outlier versus inlier study results towards the summary effect in SRs. There are >200 types of graphical display available.65^,66 The L’abbe plot has long been used for this role but has deficiencies, especially where there are limited data.30 There are newer techniques to display the data that will help to identify outlier results.

Another emerging technique is that of the directed acyclic graph to objectively depict the assumptions and potential confounders that underlie the data generating mechanisms across different studies.71 A recent example has considered the application of directed acyclic graphs for assessing the impact of time varying confounders on the relationship between antibiotic de-escalation and patient mortality.72

Another tool that may emerge is structural equation modelling, which potentially might enable univariate and multivariate meta-analysis to enable comparisons of several treatments.73

Conclusion

SRs have evolved over the past 25 years. SRs are powerful tools for summarizing a diverse and sometimes controversial literature. The strengths and weaknesses of SRs critically depend on the validity of their assumptions. As with any other research tool, the safety of SR inferences and their application to evidence-based medicine need care. Increasingly, SRs will need to account for the primary study data provenance beyond any ROB assessments. Methods to visualize the data in relation to external benchmarks, if available, will help ensure this.