Introduction
Facing the difficult task of approving new therapies, policy makers often find themselves navigating complex choices under time pressure and limited evidence. When new information emerges on the clinical effectiveness of treatment options, they can decide to outright approve or reject potential treatments. Alternatively, they may also opt for demanding further trials, thereby delaying the approval process (usage only in research settings [OIR] or Investigational New Drug status), or grant emergency use authorization (EUA) while still conducting research (approval with research; AWR)., Balancing these choices has high stakes, in which the aim is to weigh maximizing certainty about treatment effectiveness against the risks of delay or implementing ineffective treatments.
The COVID-19 pandemic vividly underscored these challenges. When the pandemic was declared in 2020, there were no specific treatments available to treat patients affected by the SARS-CoV-2 virus. Early guidelines relied heavily on expert opinions and experiences from previous pandemics. Meanwhile, each decision delay affected thousands of lives, with 3,000 to 5,500 daily hospitalizations in the United States alone.
The toolset available for making such pivotal decisions is multifaceted. Policy makers may lean on individual trials for guidance or, when multiple trials have been conducted, aggregate the available evidence through meta-analyses. These methods, despite being common practice, often hinge on statistical significance. They frequently suggest further research is needed but fall short in quantifying decision-making consequences. An alternative approach is to use value-of-information (VOI) analyses. This decision-analytic framework accounts for current uncertainties tied to the decision and forecasts the potential implications of deciding on approval with and without further research.– VOI presents a more holistic perspective, contrasting the potential benefits of more research against possible lost opportunities due to delays and research costs.
In this study, we identify decisions drawn from each of these approaches across the COVID-19 timeline, including policy decisions as they were made, decisions suggested by cumulative meta-analysis (CMA), and decisions informed by prospective VOI with information available then. We then compare these to a reference standard, represented by the VOI-based approach equipped with the benefit of hindsight (retrospective). We apply these methods to the example of monoclonal antibodies (MAbs), which are designed to mimic the body’s natural immune response, in the treatment of hospitalized COVID-19 patients.
Methods
We compared 3 distinct approaches to guide policy makers in their choices using an example of the recommendations on implementation and further research of treatment with MAbs to usual care for hospitalized COVID-19 patients across the timeline of accrual of evidence:
The optimal strategy based on policy recommendations on drug approval. Here we use the decisions on EUA or approval made by the US Food and Drug Administration (FDA). We consider the strategy the FDA implemented to be the optimal strategy from their perspective.
The optimal strategy based on CMA. In this approach, the decision is based solely on the rolling result of the CMA. A mortality reduction with a significance level P ≤ 0.05 leads to the decision “approve” being considered as optimal by the CMA and P > 0.05 to “AWR,” while an increase in mortality with a significance level P > 0.05 leads to the decision “OIR” and P ≤ 0.05 to “reject.”
The optimal strategy based on prospective decision-analytic and VOI results. In this approach, we use only information available at the time point of the decision. The optimal strategy is the strategy for which the VOI identifies the highest expected value.
In our analysis, we assumed that the retrospective decision-analytic and VOI results (with information available in hindsight) represent the reference standard and that the optimal strategy identified with this approach is the true optimal decision. This optimal strategy is the strategy that yielded the highest expected value (EV), given that we now know how many patients would truly be affected by the decision throughout the pandemic. Thus, we used this fourth approach as a comparator to evaluate the results of each of the 3 examined approaches. By comparing the EV of the suggested strategy at that time for each approach to the true optimal strategy, we estimate the value loss.
The sections below provide methodological details on each of the evaluated approaches. Publications are referred to in the analysis and figures by their respective trial names or, if no trial name was assigned, by the first author’s name.
Approach 1: Policy Recommendations on Drug Approval
To evaluate the first approach, which proposes the optimal strategy is represented by the actual policy decision, we reviewed the press releases by the FDA for information on the EUA or approval of MAbs in the treatment of COVID-19.
Approach 2: CMA
To evaluate the effectiveness of the second approach, which proposes to base the decision making on CMA, we performed a systematic review and CMA of all randomized controlled trials (RCTs) investigating MAbs. Our objective is to identify the relative risk of mortality at day 28 of MAbs versus usual care in hospitalized COVID-19 patients.
Eligibility criteria
Population: The study population included hospitalized adult (≥18 y) COVID-19 patients.
Intervention: The intervention was treatment with any form of Mabs.
Control: The study included usual care as the control arm condition.
Outcome: Study outcomes reported mortality at 1 mo (28 d to 31 d after Rx).
Study design: The study was an RCT published in English between December 2019 and June 2021.
Search strategy
The literature search was conducted and optimized with the help of an information specialist from the Erasmus MC Medical Library. Citations were identified from 6 databases:
Excerpta Medica dataBASE (EMBASE)
Medical Literature Analysis and Retrieval System Online (MEDLINE) via OVID
Cochrane Central Register of Controlled Trials
Web of Science core collection
World Health Organization COVID-19 database
Google scholar
We reviewed the references of systematic reviews identified by the search for additional citations. The full search strategy is provided in Appendix Tables 1 and 2. We followed the PRISMA reporting recommendations.
Study selection, data extraction, and bias assessment
Two researchers independently screened articles based on title and abstract and full text and performed data extraction. The Rayyan QCRI data management tool was used to store, organize, and manage all references. Discrepancies were resolved by consensus or with a third reviewer. We assessed the quality of included studies using the Cochrane Risk of Bias-2 tool and reviewed the presence of publication bias through a Funnel plot.
Publication date
The publication date is noted as the first date the article became available online as a preprint (identified on medRXiv or Research Square using the trial ID) or published in a peer-reviewed journal. We assume this date is most in line with when the information became available to decision makers. We justified this choice based on previous studies that concluded that the main RCT results were consistent between preprint and printed journal articles and because some articles were delayed in publication. As our search strategy included trial registers, studies that were published beyond the last search date (June 2021) were eligible for inclusion if they were identified through the initial search.
Statistical analysis
The effect estimate for mortality was expressed as the relative risk (RR) of dying in the intervention arm compared with the control arm. We assume that, despite the diverse biological pathways targeted by MAbs, their mechanism of action is homogenous enough for meaningful meta-analysis. We performed a CMA of all included articles using a random-effects model. We used the Mantel-Haenszel method for pooling and quantifying expected heterogeneity with an I statistic. Between-study variance was quantified using the tau statistic, and its uncertainty was adjusted using the Knapp-Hartung adjustment.
In addition, we performed a meta-regression with severity as a predictor using articles that reported sufficient information. For this analysis, patients not on supplemental oxygen or oxygen delivery by face mask/nasal cannula were deemed nonsevere and assumed to be treated in the ward, while patients on noninvasive ventilation, invasive mechanical ventilation, or extracorporeal membrane oxygenation were categorized as severe and assumed to be treated in an intensive care unit (ICU) setting.
All statistical analyses were performed with R (version 4.1.2.).
Approach 3: Decision-Analytic Model and VOI Analysis
Model description
To evaluate the third approach (i.e., prospective VOI that proposes to base decision making on decision-analytic methods), we used a decision-analytic model and performed VOI analyses.
The state-transition cohort model consists of 4 health states: 1) hospitalized, 2) recovered from the ICU as the highest level of care, 3) recovered from the hospital ward as the highest level of care, 4) dead. The model structure is provided in Appendix Figure 7, and further details on this decision-analytic model are available in Dijk et al. The model was developed based on the DARTH framework.– We followed the CHEERS, CHEERS-VOI, and ISPOR reporting recommendations.
Whenever a new study is incorporated into the CMA, the decision model recalculates the pooled effect. Based on these updated data, it then estimates incremental cost-effectiveness ratios of treatment with the drug versus usual care without the drug.
All other model parameters were based on best available evidence as of December 2021. An overview of these parameters is provided in the supplementary materials (Supplementary File: Excel). We applied the cost and length of stay of tocilizumab as the most frequently used drug in the meta-analysis as the cost of treatment. Our analysis was conducted from the US health-system perspective with a $100,000/quality-adjusted life-year (QALY) willingness-to-pay (WTP) threshold– with a lifetime horizon. Costs were estimated in 2020 US dollars ($). We applied a 3% annual discount rate for both costs and effects, and probabilistic analysis was performed with 10,000 iterations.
A VOI analysis is a decision-analytic method that helps assess and quantify decision uncertainty and determine whether the available evidence is sufficient to make an immediate decision or if further research is needed. For the purpose of our evaluation, we calculated the expected added value of performing an RCT to reduce the uncertainty surrounding the treatment effect on mortality as partial perfect information (EVPPI) using a linear-regression meta-model. For time points with a positive value of further research (EVPPI > 0), we performed an expected value of sample information (EVSI) estimation using a Gaussian approximation approach as proposed by Jalal and Alarid-Escudero.– We next identified the optimal sample size of a new RCT. Further details on VOI methodology is provided in the published literature,,,,,– including in the context of COVID-19.
Prospective VOI analysis
The prospective VOI approach mimics a situation in which we had used VOI to inform decisions across the timeline. This analysis uses the number of patients affected by the decision based on the available information at that time point. In other words, the analysis presents the results we would have received at that time.
This analysis uses the sum of the number of daily hospitalizations forecasted by the Institute for Health Metrics and Evaluation (IHME), as of the latest available data set at the time of publication of the last study added to the CMA until the end of the forecast (Appendix Figure 9). When IHME predictions ended mid-peak, we extended these predictions under the assumption that peaks during a pandemic have symmetric inclines and declines, which would be more realistic than a sudden drop in patients at those time points. A depiction of the unextended and extended predictions can be found in Appendix Figure 10.
The net benefit obtained with EUA of treatments while performing further RCTs was determined for the expected number hospitalizations in the United States while awaiting trial results and their implementation (current patients) over 2 mo. We chose 2 mo based on the availability of predictions: a minimum of half of the prediction time needs to contain predictions on future patients. We calculated our results for a maximum feasible sample size of 2,500 patients (main analyses) and the optimal sample size capped at 10,000.
The net value of each strategy was calculated according to the equations listed in Figure 1.
Figure 1
Decision matrix. This 2 × 2 figure shows the 4 potential combined research-treatment strategies, their advantages and disadvantages, the decision rule when this strategy would be selected as the optimal strategy, and how the expected net monetary benefit (ENB) would be calculated for the value-of-information (VOI) analyses to determine the optimal strategy according to the VOI. Difference retrospective/prospective VOI: the number of patients is based on the projections on the date of evaluation (prospective) or on the last data set on the timeline (retrospective). iNB, incremental net (monetary) benefit (NB treat – NB control); p, proportion of patients n in a new randomized controlled trial (RCT) randomly assigned to treatment; 1 − p, proportion of patients n in a new RCT randomly assigned to control; EVSI, expected value of sample information, calculated in comparison with the optimal treatment (i.e., over and above the iNB gained if iNB > 0); EVSI(n), EVSI dependent on sample size n of the new RCT; cost RCT(n), fixed cost + variable cost of performing a new RCT dependent on sample size n. The optimal sample size is determined by maximizing the function in the quadrant with respect to n. Implementation and reversal costs are assumed to be 0 given that the intervention concerns a guideline change.
Reference standard: retrospective VOI analysis
The reference standard, also referred to as the retrospective VOI, mimics a situation in which we compare VOI analysis to make decisions at various time points across a timeline with hindsight knowledge. Namely, in this analysis, we use the sum of the number of daily hospitalizations observed and forecasted in the IHME data set of the last date prior to the end of our timeline (December 2021) (Appendix Figure 8). We consider retrospective VOI as the reference standard since it contains the most information and considers both uncertainty and the consequences of making a suboptimal decision using information in hindsight.
Sensitivity analyses
We provide additional cost-effectiveness acceptability curves across time points illustrating how the EVPPI results are influenced by the chosen WTP threshold. We performed several scenario analyses, in which we
increased the WTP threshold to $150,000,
increased the trial duration from a 2-mo to a 3-mo period,
included only the 4 trials listed in the FDA EUA press release,
did not extrapolate data sets that ended in peak predictions,
use noncumulative individual study data,
conduct the meta-analysis without the Knapp-Hartung adjustment,
conduct the meta-analysis based on a subgroup of only interleukin (IL)–6-(R-) inhibitor MAbs. We performed a subgroup analysis for this set of MAbs with a similar working mechanism as 16 articles that were included and could be meta-analyzed. Other MAbs included in our study were investigated only in single trials, for which the results can be found in the noncumulative individual study data scenario.
EV Loss due to Suboptimal Choice
Finally, we calculated the loss in EV for each of the possible approaches. We calculated the expected loss of the strategies proposed by each approach (policy, CMA, or prospective VOI) by comparing their EVs to the EVs of the retrospective VOI (i.e., reference standard). For example, if CMA suggests AWR as the optimal strategy and retrospective VOI suggests approve, then we calculate the net value of the retrospective VOI EV for approve minus the retrospective VOI EV for AWR.
Results
Approach 1: Policy Recommendations on Drug Approval
The FDA announced EUA on June 24, 2021, for tocilizumab, a MAb that reduces inflammation by blocking the IL-6 receptor. The press release states that 4 clinical trials contributed to this decision: RECOVERY, EMPACTA, COVACTA, and REMDACTA. The EUA statement notes that the known and potential benefits of this drug outweigh the known and potential risks of this treatment. The announcement does not specify the methodological approach to combining or comparing the results of these 4 trials but notes that RECOVERY and EMPACTA provided the most important scientific evidence on the potential benefit. On December 21, 2022, tocilizumab was approved; however, this date falls beyond the timeline of our study.
Approach 2: CMA
Our search identified 10,633 articles, of which 18 entailing 22 treatment comparisons and 10,031 patients were included in the meta-analysis.–,– A flowchart detailing the search results is shown in Appendix Figure 1. Study characteristics and quality assessment, a funnel plot, and a global distribution of included studies and meta-regression based on severity can be found in Appendix Tables 3 and 4 and in Appendix Figures 3 and 6.
Ten of the included studies investigated the use of tocilizumab, whereas the remaining studies investigated the use of sarilumab, vilobelimab, itolizumab, mavrilimumab, otilimab, levilimab, siltuximab, and lenzilumab. The total number of events (deaths) was 2,368, of which 1,097 occurred in the intervention group and 1,271 in the control group. The largest study was the RECOVERY trial with 4,116 subjects (41% of the total), and the smallest study was PANAMO (30 subjects). Seven studies had fewer than 10 deaths in the intervention arm, and 10 studies had fewer than 10 deaths in the control arm.
Figure 2 shows the results of the CMA in a forest plot, with an overall RR of 0.90 (95% confidence interval 0.82, 0.98) after all studies were included. The traditional (noncumulative) meta-analysis is provided in Appendix Figure 4.
Figure 2
Forest plot of the cumulative meta-analysis. Pooled relative risk for mortality with treatment with monoclonal antibodies versus control arms. At each time point the trial results were published: the relative risk is a pooled result of the newly included study and the evidence thus far accrued.
K, number of included studies; Sari, sarilumab; TCZ/Toci, tocilizumab.
Approaches 3 and 4: Decision-Analytic Model and VOI Analysis
Figure 3 shows the cost-effectiveness plane over time. After an initial increase in uncertainty once the results from the second trial (PANAMO) were added, the uncertainty surrounding the estimated cost-effectiveness increases then decreases again with accrual of further evidence. The figure illustrates that for most time points, the mean and majority of iterations fall in the upper right quadrant, in which both QALYs and costs are higher for MAbs.
Figure 3
Incremental cost-effectiveness planes of the monoclonal antibody (MAb) treatment versus usual care over time. The x-axes show incremental effectiveness, whereas the y-axes show incremental costs in USD. This grid slot shows the cost-effectiveness (CE) plot at each time point at which a new study (k) is added to the cumulative meta-analysis treatment effectiveness estimate. The plots are based on the results of 10,000 iterations. The yellow circle depicts the estimate for quality-adjusted life-years (QALYs) and the gray for LY. The dotted line surrounding the mean estimate reflects the uncertainty. Other parameter inputs remained constant over time.
Results of the cost-effectiveness analysis (CEA), probabilistic analysis, prospective and retrospective VOI, parameters for the ENB calculations, and the ENB results of each potential strategy are provided in Table 1.
Supportive figures are provided in the appendix, including CE planes (Appendix Figures 11 and 12), CEA curves and frontiers (Appendix Figure 13), expected value of partial sample information plots (Appendix Figure 14), and full results overview for all strategies and including optimal sample size (Appendix Figures 15 and 16).
Timeline of Approaches to Decision Making
We summarized publication dates and the optimal strategies as suggested by each of the approaches in Figure 4. Included articles were published between September 2020 and December 2021.
Figure 4
Timeline of publications and 4 approaches to decision making. The plot represents the publication of the results of each study and the optimal strategy as suggested by the 4 approaches (cumulative meta-analysis, policy set by the Food and Drug Administration, prospective value of information [VOI] and retrospective VOI).
The first policy decision (EUA) was made in June 2021 after 18 of 22 treatment comparisons from this article were included. Drug approval took place in December 2022, after the last article of our meta-analysis was included, and is therefore not shown in this figure. The CMA suggested AWR earlier and more frequently than prospective or retrospective VOI. Both CMA and prospective VOI suggest approve between the 15th and 18th treatment comparison. The final suggested strategy based on the CMA is approve, in contrast to prospective and retrospective VOIs, which suggest OIR. Retrospective VOI suggests OIR and AWR strategies across the entire timeline but never approval. None of the studies suggest outright rejection.
In Figure 5, we have depicted the loss that would result from a suboptimal choice in strategy in comparison with the reference standard. This loss is calculated based on the values as presented in Appendix Figure 16. Figure 5 shows that the expected loss is highest when decision making is guided by CMA alone, particularly in early stages (k2-k4). Here, CMA suggests AWR while prospective and retrospective VOIs suggest that OIR is the optimal strategy (Box 1).
Figure 5
Expected value loss from choosing the suboptimal strategy according to the gold standard compared with the other approaches. This is calculated by taking the EV of the optimal strategy—the EV of the strategy suggested by each respective approach (policy, CMA, or prospective VOI). The EV is provided by the gold standard, as this approach is closest to knowing the true EV as the largest proportion of hospitalizations have already been observed. The smaller figure in the top right corner is the same plot with a zoomed-in version of the plot with a y-axis set to max $25,000 million. The highest losses for each strategy were CMA approach: $269 billion; policy approach: $16 billion; prospective VOI approach: $2 billion. CMA, cumulative meta-analysis; EV, expected value; VOI, value of information.
Sensitivity Analyses
The figures illustrating the results of the sensitivity analyses are provided in the Appendix.
A WTP threshold of $150,000/QALY led to time point K9 now being considered cost-effective and additional prospective VOI time points suggesting approval (K11-K17+K19); however, retrospective VOI remained unchanged (Appendix Table 5).
Extended trial duration resulted in more frequent approval, attributed to the few remaining future patients after the trial ends. For example, the number of future patients calculated in K11-K13, K15, and K19 was between only 1,000 and 3,000. When few patients benefited from future research, the costs of a new trial did not outweigh the benefits even when EVPPI was high. Loss, therefore, increased for the prospective VOI approach (Appendix Table 6).
Including only the trials listed by the FDA’s press release showed that the FDA’s decision for EUA was inconsistent with both CMA and VOI or could be interpreted as late versus the CMA results if we extrapolate our analysis beyond our timeline until approval in December 2022. Prospective VOI did initially recommend EUA at the same time as CMA but did not recommend approval at any time point (Appendix Table 7).
Not extending the IHME predictions that ended mid-peak (stop-and-drop assumption after the IHME predictions) did not alter the suggested optimal strategies but reduced the number of patients who could benefit at those time points and related values (Appendix Table 8).
The traditional meta-analysis forest plot illustrates the individual contribution of each study. When examined in the model, the timeline plot now shows the results of the optimal suggested strategy if the recommendation of each approach was based on any of the individual studies (Appendix Table 9).
The CMA was run without using the Knapp-Hartung adjustment, which increases the uncertainty intervals, especially when few studies have been included. This led to approval being suggested at earlier time points. As uncertainty was smaller, EVPPI was lower, and loss from choosing the suboptimal strategies at approval time points decreased (Appendix Table 10).
Finally, in the subgroup analysis in which only IL-6-(R-) inhibitors tocilizumab, sarilumab, and siltuximab articles were eligible, 16 articles were included. The overall pooled result was more uncertain, resulting in a continued AWR recommendation in the CMA approach. Loss was largest in either the policy or CMA approach, depending on the time point (Appendix Table 11).
Discussion
Policy makers face complex decisions under conditions of imperfect information. However, we can distinguish different approaches to aid their choices and maximize the chances of evidence-based decisions that consider the consequences of those decisions.
Our analysis demonstrated that FDA decisions were not always optimal, based on both reference standard hindsight knowledge and real-time VOI analysis. Our analysis demonstrates the consequences of delay in policy making: not making a decision should be considered an active decision to maintain the status quo. This means that when the FDA takes longer to provide EUA than VOI approaches, an active decision is taken to continue using MAbs only in research settings, foregoing the potential benefits to the population when treatment seems promising.
Guiding decision making based on CMA alone would have led to the largest opportunity loss of all strategies. Had we used VOI analysis prospectively, suboptimal decisions would still have been made due to imperfect information, but the expected loss in value would have been substantially reduced. The reference standard, retrospective VOI analysis, demonstrated that either OIR or AWR was the optimal strategy depending on the point in time.
The difference in recommendations between the retrospective and prospective VOI is driven by the number of anticipated current (those admitted to the hospital while the new trial is ongoing) and future patients (those who would benefit from the new trial’s results). Therefore, the discrepancy between these 2 approaches (K15-K17), in which the prospective VOI suggests approval and retrospective VOI suggests AWR, stems from hospitalization forecasts. This underscores the importance of accurate forecasts when considering the value of future research or at least the acknowledgment of the uncertainty in our predictions. Even when a previous time point may suggest approval, it is vital to reassess if there are changes in other parameters such as the estimated number of patients. The need to revisit and update the analysis becomes especially relevant when dealing with rapidly changing forecasts during a pandemic. Had we relied solely on CMA, our decision to approve without further research would have been independent of any forecast; thus, we may not have found new reasons to revisit our decision, and approval might not have been reversed. According to the reference standard, the final choice at the end of our timeline was to withdraw EUA.
Strengths
To the best of our understanding, this is the first time these 3 real-time approaches—policy, CMA, and prospective VOI—were applied together across a timeline of evidence accrual along with a fourth, reference standard retrospective VOI approach. Our novel comparison to a reference standard helps us better understand the added value of VOI methods in decision-making processes. Implementing these methods as extensions to meta-analyses can facilitate a deeper understanding of the consequences of historical decisions and provide valuable insights for future decision-making methodologies, within a pandemic scenario and beyond.
The integration of CEA and VOI in treatment approval and future research decisions presents an important advantage over the reliance solely on CMA and statistical significance considerations.,
These techniques provide a more comprehensive evaluation by factoring in costs, the probability of treatment being cost-effective, additional uncertainties surrounding other parameters, and the consequences of each potential decision.,
By using CMA rather than traditional meta-analysis, we were able to show evidence accrual over time while considering the full body of available evidence. Notably, our exhaustive CMA did not exhibit publication bias. This contrasts with our sensitivity analysis that includes only the 4 trials mentioned in the FDA’s press release that announces EUA for tocilizumab.
Limitations
In addition to the limitations underlying the decision-analytic model and VOI highlighted in Dijk et al., this study brings its own set of methodological and practical limitations to consider.
Methodological limitations
Our study is a methodological showcase highlighting the choice of strategy in a timeline in the past, which limits its direct practical relevance. We have simplified the analysis where we deemed appropriate for a methodological article. We focused VOI on the treatment effect measure that would be investigated with future RCTs, but other parameters could also be chosen as targets to reduce uncertainty.
Our meta-analysis included some articles that made multiple treatment comparisons, such as dosage A versus usual care and dosage B versus usual care. By incorporating these 2 comparisons in our meta-analysis, we have double counted the number of controls, artificially increasing our sample size.
Our article calculates EVs of new decisions made immediately when new evidence becomes available. However, decision making and implementation take time and resources. Our analysis does not consider the cost to conduct CMA or VOI analyses themselves. Although our conclusion states that the policy approach did not make the appropriate decisions and delayed EUA, we do not consider the time frame it would take to act on VOI recommendations or the time required to develop a working decision analytic model. The first COVID-19 VOI model preprint became available only March 2022, beyond the scope of this timeline. By providing our full code, collected data, and this framework, we hope that analyses performed in future crisis situations can build on this existing work and enable more rapid decision making.
Given the distinct nature of each method, the comparisons are influenced by the inherent limitations of each approach.
This article provides evidence that prospective VOI provides additional value above more traditional approaches to decision making. Yet we also acknowledge that the prospective VOI was the most similar of all 3 approaches to what we set to be the reference standard. The fact that this approach considers prospective VOI to be the most effective strategy could be criticized as a self-fulfilling prophecy. Our choice of reference standard may not be universally applicable or agreed upon, which affects the interpretation of our results.
Lastly, the methods discussed in this article form a simplification of complex decisions. Our VOI assumes that decisions are based mainly on expected monetary and health outcomes but ignores other potential factors such as political, social, or resource-constraining considerations.
Limitations for practical implications and generalizability
While this article is presented as a methodological article, there are important assumptions and limitations that should be noted before considering any implications to practice.
The parameters in our analysis that change over time are treatment effectiveness (CMA) and predicted number of patients. We deliberately chose to vary only these parameters to underscore the differences between the approaches. However, other parameters that may change over time such as utility estimates, patient demographics, vaccination status, circulating SARS-CoV-2 variants, and changing usual care can also influence the analysis. Neither our CMA nor VOI explicitly captures all time-varying treatment effects due to evolving treatment protocols or patient characteristics. New developments and updates to the analysis with alternative scenarios remain important when new information arises. While VOI offers an advantage in adjusting model inputs to explore their impact on the suggested optimal strategy, it is crucial to acknowledge that both VOI and meta-analysis primarily focus on reducing statistical uncertainty and are less adept at detecting sources of bias or predicting unforeseen circumstances.
MAbs specifically are at even more risk than other treatments in the COVID-19 armamentarium for changing effectiveness over time. New variants of the SARS-CoV-2 virus can emerge due to mutations in the virus’s genome. An important practical limitation of our study is that our timeline ends on January 1, 2022. The FDA has since approved tocilizumab for COVID-19 and provided EUA to some other MAbs mainly for nonhospitalized patients but also revoked authorizations of MAbs such as sotrovimab and bebtelovimab. An important reason for discontinuation of EUA was when new variants, for example, Omicron subvariants BQ.1 and BQ1.1, showed a large reduction in (expected) susceptibility.
Our analysis in a pandemic setting differs in several aspects from that in a nonpandemic setting while also sharing some commonalities. While we saw successful examples of multicenter trials such as RECOVERY,, the pandemic also introduced a surge in numerous, small, underpowered trials, creating a unique landscape. Nonpandemic settings may allow for more efficient coordination of large, multicenter RCTs. In the pandemic, the highest number of patients was admitted to the hospital in the short term (during the new trial), whereas in general settings, more patients are expected in the future, even if the trial results will likely take longer to publish their results. These differences will influence the EV of early implementation and the collection of further evidence. A better stability in the predicted number of patients in the future will likely also reduce the fluctuation between optimal strategies in prospective VOI in nonpandemic settings. While we anticipate partial generalizability of our findings to nonpandemic settings, applications in other settings are needed to support these expectations.
We used a health care perspective with mainly US-based parameters. Although our modeling approach can be applied in different settings, our results cannot immediately be applied to different countries or globally, as for example, the costs of care, forecasts, and WTP thresholds will likely differ.
Our model identifies optimal strategies for research and treatment on the assumption that decision making and consequence bearing are centralized in the same entity. However, in reality this is not always the case. Specifically, an agency such as the FDA is not necessarily the entity deciding on research funding or the entity deciding on the incorporation of newly approved drugs in treatment guidelines. The decision-making landscape is in fact a complex network of various organizations, each with its own agenda and cost-outcome considerations. Our model may therefore not fully capture the diversity and nuance of the multistakeholder context within the health care system.
In our meta-analysis, we pooled results from all articles investigating MAbs eligible for inclusion, whereas from a pharmacologic standpoint, various types of MAbs can be distinguished along the pathways on which they work. Thus, we assumed sufficient homogeneity across these studies to allow for pooling of the results but used a random effects model to account for heterogeneity and performed sensitivity analysis to evaluate the effect of our assumption. Our search did not identify MAbs belonging to the neutralizing MAbs, likely because they are applied in outpatient settings. We performed a sensitivity analysis for the second major group, immune modulators, working through the IL-6-(receptor-)inhibition. In this analysis, the final CMA recommendation was AWR instead of approval due to remaining statistical uncertainty.
While treatment effectiveness is based on several different MAbs, we adopt the other parameters such as treatment cost and hospital stay in the decision-analytic model based on the most frequently cited treatment tocilizumab. In addition, the policy decision also applies to the EUA and approval of tocilizumab; however, in reality, each MAb treatment requires its own approval process in treating COVID-19, even if prior off-label use was possible.
Our analysis uses a trial duration of 2 months in our VOI, which can be subject to debate. This modeling choice had practical roots in the fact that prospective IHME predictions often projected only 4 months into the future, and we felt that at least half of these predictions should consider future patients. It would technically be possible to collect 28-d mortality figures from a new trial; however, this would require rapid decisions and trial rollouts to accomplish, and we will likely underestimate the number of patients in our model. The need for rapid trial rollout and large sample sizes as suggested by the EVSI supports the call for globally optimal trial design using for example a platform design, which could increase the speed at which decisions are made, reduce fixed trial costs, and increase the strength and implementation of evidence with large definitive trials.
In addition, by choosing mortality as a singular endpoint for further investigation in further RCTs, we neglect the value that could be gained in collecting further information about other relevant endpoints such as morbidity or length of hospital stay. Neither do we consider alternative study types than RCTs that could investigate consequences such as long-term quality of life or costs.
Finally, our analysis assumes that the costs associated with the implementation and reversal of new decisions are negligible, as they concern guidelines and policy changes rather than physical interventions. Nevertheless, this assumption might overly simplify the real-world scenario, as we neglected costs related to policy change execution and time lags between the decision-making processes, its implementation, and the observable effects of those changes. We did not consider the consequences for trials that had already been initiated, including early termination when new evidence became available.
Future Directions
This article, along with the accompanying code and data, offers a promising framework for employing and comparing multiple evidence-assessment methodologies including CMA and VOI. Given that this article pioneers this approach, it necessitates further validation across diverse settings or by applying it to alternative models. Further developments that could enhance the practical implementation of this model include its integration with a foundational susceptible-exposed-infectious-recovered (SEIR) model, which would anticipate the count of future patients. This integration would enable rapid updates to the results when forecasts change. Our code and data are made available to this end.
Conclusion
Our retrospective VOI analysis (reference standard) showed that across the 2020 to 2022 timeline, either OIR or AWR was the strategy that yielded the highest expected net benefit among potential combined approval-research strategies. The discrepancy between the suggested strategy by the policy and retrospective VOI approaches indicates that resources were used suboptimally. Using only CMA to inform decisions on treatment research-approval would have led to the largest opportunity loss of all strategies, followed by the current FDA policy approach. The use of (prospective) VOI in future research-approval decisions should be considered as an extension of meta-analyses that also incorporates the consequences of each decision. Although suboptimal strategies would still have been suggested at certain time points in our MAb example, EV loss would have been reduced substantially.
The authors would like to thank the Society for Medical Decision Making COVID-19 Decision and the Gordon and Betty Moore Foundation for their support through the COVID-19 Decision Modeling Initiative. We additionally thank the Decision Analysis in R for Technologies in Health (DARTH) workgroup and the Collaborative Network for Value of Information (ConVOI) for providing the coding framework in R. Lastly, we thank Dr. Maarten Engel, information specialist at the Erasmus University Medical Center medical library, for aiding in the creation of the meta-analysis search strategy.
Author Contributions Concept and design: Dijk, Hunink; acquisition of data: Dijk, Pandit, Lu; analysis and interpretation of data: Dijk, Krijkamp, Kunst, Gross, Labrecque, Visser, Wong, Hunink; drafting of the manuscript: Dijk, Hunink; critical revision of the article for important intellectual content: Dijk, Krijkamp, Kunst, Gross, Labrecque, Pandit, Lu, Visser, Wong, Hunink; obtaining funding: Dijk, Krijkamp, Kunst, Hunink; supervision: Hunink, Wong.
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr. Dijk reports grants from the Gordon and Betty Moore Foundation during the conduct of the study and grants from the German Innovation Fund outside the submitted work. Dr. Krijkamp reports grants and personal fees from the Society for Medical Decision Making fellowship through a grant from the Gordon and Betty Moore Foundation (GBMF7853) outside the submitted work. Dr. Kunst has nothing to disclose. Dr. Gross reports grants from the American Cancer Society, Johnson & Johnson, Pfizer, Flatiron Health, and Genentech outside the submitted work. Dr. Labrecque is supported by an NWO/ZonMW Veni grant (09150162010213). Mrs. Pandit has nothing to disclose. Ms. Lu has nothing to disclose. Dr. Visser has nothing to disclose. Dr. Wong has nothing to disclose. Dr. Hunink reports grants from the Gordon and Betty Moore Foundation during the conduct of the study; other support from the European Society of Radiology, the European Institute for Biomedical Imaging Research, and Cambridge University Press; grants from the American Diabetes Association, the Netherlands Organization for Health Research and Development, the German Innovation Fund, and the Netherlands Educational Grant (“Studie Voorschot Middelen”) outside the submitted work. The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Gordon and Betty Moore Foundation through grant GBMF9634 to Johns Hopkins University to support the work of the Society for Medical Decision Making COVID-19 Decision Modeling Initiative. The funding sources played no role in the writing or submission of this article. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
Ethics Approval No data from participants were collected in this study. No medical ethical review board approval was required since we performed a mathematical modeling study and used previously published data.
Protocol Registration The Systematic Review and Meta-Analysis protocol was registered internally within the Erasmus University Medical Center system and is available upon request.
Reporting Guidelines A filled-out PRISMA Statement is provided in Appendix Table 12 and CHEERS VOI Statement in Appendix Table 13.
Stijntje W. Dijk
https://orcid.org/0000-0003-2905-4128
Eline Krijkamp
https://orcid.org/0000-0003-3970-2205
Natalia Kunst
https://orcid.org/0000-0002-2409-4246
Data-Sharing Statement Key input parameters are provided in the article and Appendix. Input and code for the meta-analysis, decision-analytic and VOI code, data files containing all input data and distributions, and additional supplementary graphs and figures are publicly available via GitHub at https://github.com/krijkamp/Emerging-Therapies-for-COVID-19.
References
- 1. Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute for Clinical Excellence (NICE). Lancet. 2002;360(9334):711–5.
- 2. Claxton K, Palmer S, Longworth L, et al. A comprehensive algorithm for approval of health technologies with, without, or only in research: the key principles for informing coverage decisions. Value Health. 2016;19(6):885–91.
- 3. Dijk SW, Krijkamp EM, Kunst N, Gross CP, Wong JB, Hunink MGM. Emerging therapies for COVID-19: the value of information from more clinical trials. Value Health. 2022;25(8):1268–80.
- 4. Johns Hopkins University. COVID-19 data repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University. 2020. Available from: https://github.com/CSSEGISandData/COVID-19 [Accessed 11 November, 2020].
- 5. Claxton K, Griffin S, Koffijberg H, McKenna C. How to estimate the health benefits of additional research and changing clinical practice. BMJ. 2015;351:h5987.
- 6. Heath A, Manolopoulou I, Baio G. A review of methods for analysis of the expected value of information. Med Decis Mak. 2017;37(7):747–58.
- 7. Heath A, Hunink M, Krijkamp E, Pechlivanoglou P. Prioritisation and design of clinical trials. Eur J Epidemiol. 2021;36(11):1111–21.
- 8. Keisler JM, Collier ZA, Chu E, Sinatra N, Linkov I. Value of information analysis: the state of application. Environ Syst Decis. 2014;34(1):3–23.
- 9. Eckermann S, Willan AR. Expected value of information and decision making in HTA. Health Econ. 2007;16(2):195–209.
- 10. Lloyd EC, Gandhi TN, Petty LA. Monoclonal antibodies for COVID-19. JAMA. 2021;325(10):1015. DOI:
- 11. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.
- 12. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a Web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210.
- 13. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.
- 14. Bai AD, Jiang Y, Nguyen DL, et al. Comparison of preprint postings of randomized clinical trials on COVID-19 and corresponding published journal articles: a systematic review. JAMA Netw Open. 2023;6(1):e2253301.
- 15. DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: an update. Contemp Clin Trials. 2007;28(2):105–14. DOI:
- 16. IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol. 2014;14:25.
- 17. Rstudio Team. RStudio: Integrated Development for R. Boston (MA): RStudio, Inc; 2015. Available from: http://www.rstudio.com
- 18. R Core Team. R: A Language and Environment for Statistical Computing. Vienna (Austria): R Core Team; 2013. Available from: https://www.r-project.org/
- 19. Alarid-Escudero F, Krijkamp EM, Pechlivanoglou P, et al. A need for change! A coding framework for improving transparency in decision modeling. Pharmacoeconomics. 2019;37(11):1329–39.
- 20. Alarid-Escudero F, Krijkamp EM, Enns EA, et al. A tutorial on time-dependent cohort state-transition models in R using a cost-effectiveness analysis example. arXiv Preprint arXiv210813552, 2021.
- 21. Husereau D, Drummond M, Petrou S, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. Eur J Health Econ. 2013;14(3):367–72.
- 22. Kunst N, Siu A, Drummond M, et al. CHEERS value of information (CHEERS-VOI) reporting standards—explanation and elaboration. Value Health. 2023;26(10):1461–1473.
- 23. Briggs AH, Weinstein MC, Fenwick EAL, et al. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6. Value Health. 2012;15(6):835–42.
- 24. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316:1093–103.
- 25. Weinstein MC. How much are Americans willing to pay for a quality-adjusted life year? Med Care. 2008;46(4):343–5.
- 26. Braithwaite RS, Meltzer DO, King JT Jr, Leslie D, Roberts MS. What does the value of modern medicine say about the $50,000 per quality-adjusted life-year decision rule? Med Care. 2008;46(4):349–56.
- 27. Neumann PJ, Cohen JT, Weinstein MC. Updating cost-effectiveness—the curious resilience of the $50,000-per-QALY threshold. N Engl J Med. 2014;371(9):796–7.
- 28. Fenwick E, Steuten L, Knies S, et al. Value of information analysis for research decisions—an introduction: report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value Health. 2020;23(2):139–50.
- 29. Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Mak. 2014;34(3):311–26.
- 30. Jalal H, Alarid-Escudero F. A Gaussian approximation approach for value of information analysis. Med Decis Mak. 2018;38(2):174–88.
- 31. Kunst N, Wilson ECF, Glynn D, et al. Computing the expected value of sample information efficiently: practical guidance and recommendations for four model-based methods. Value Health. 2020;23(6):734–42.
- 32. Jalal H, Goldhaber-Fiebert JD, Kuntz KM. Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling. Med Decis Mak. 2015;35(5):584–95.
- 33. McKenna C, Soares M, Claxton K, et al. Unifying research and reimbursement decisions: case studies demonstrating the sequence of assessment and judgments required. Value Health. 2015;18(6):865–75.
- 34. Conti S, Claxton K. Dimensions of design space: a decision-theoretic approach to optimal research design. Med Decis Mak. 2009;29(6):643–60.
- 35. Rothery C, Strong M, Koffijberg HE, et al. Value of information analytical methods: report 2 of the ISPOR value of information analysis emerging good practices task force. Value Health. 2020;23(3):277–86.
- 36. IHME Team COVID-19 Health Service Utilization Forecasting Team, Murray CJL. Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months. MedRxiv. 2003–20, 2020.
- 37. Institute of Health Metrics and Evaluation. COVID-19 projections—USA. 2021. Available from: https://covid19.healthdata.org/
- 38. US Food and Drug Administration (FDA). Coronavirus (COVID-19) update: FDA authorizes drug for treatment of COVID-19. FDA News Release. Available from: https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-authorizes-drug-treatment-covid-19
- 39. Abani O, Abbas A, Abbas F, et al. Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial. Lancet. 2021;397(10285):1637–45.
- 40. Salama C, Han J, Yau L, et al. Tocilizumab in patients hospitalized with COVID-19 pneumonia. N Engl J Med. 2021;384(1):20–30.
- 41. Rosas I. Tocilizumab in patients hospitalized with COVID-19 pneumonia. N Engl J Med. 2021;384(15):1473–4. DOI:
- 42. Rosas IO, Diaz G, Gottlieb RL, et al. Tocilizumab and remdesivir in hospitalized patients with severe COVID-19 pneumonia: a randomized clinical trial. Intensive Care Med. 2021;47:1258–70.
- 43. US Food and Drug Administration (FDA). Frequently asked questions on the emergency use authorization for Actemra (tocilizumab) for treatment of COVID-19. 2022. Available from: https://www.fda.gov
- 44. Vlaar APJ, de Bruin S, Busch M, et al. Anti-C5a antibody IFX-1 (vilobelimab) treatment versus best supportive care for patients with severe COVID-19 (PANAMO): an exploratory, open-label, phase 2 randomised controlled trial. Lancet Rheumatol. 2020;2(12):e764–73.
- 45. Hermine O, Mariette X, Tharaux P-L, et al. Effect of tocilizumab vs usual care in adults hospitalized with COVID-19 and moderate or severe pneumonia: a randomized clinical trial. JAMA Intern Med. 2021;181(1):32–40.
- 46. REMAP-CAP Investigators, Gordon AC, Mouncey PR, et al. Interleukin-6 receptor antagonists in critically ill patients with COVID-19. N Engl J Med. 2021;384(16):1491–502.
- 47. Soin AS, Kumar K, Choudhary NS, et al. Tocilizumab plus standard care versus standard care in patients in India with moderate to severe COVID-19-associated cytokine release syndrome (COVINTOC): an open-label, multicentre, randomised, controlled, phase 3 trial. Lancet Respir Med. 2021;9(5):511–21.
- 48. Lescure F-X, Honda H, Fowler RA, et al. Sarilumab Treatment of Hospitalised Patients with Severe or Critical COVID-19: A Multinational, Randomised, Adaptive, Phase 3, Double-Blind, Placebo-Controlled Trial. Cold Spring Harbor Laboratory; 2021. DOI:
- 49. Kumar S, De Souza R, Nadkar M, et al. A two-arm, randomized, controlled, multi-centric, open-label phase-2 study to evaluate the efficacy and safety of itolizumab in moderate to severe ARDS patients due to COVID-19. Expert Opin Biol Ther. 2021;21(5):675–86.
- 50. Cremer PC, Abbate A, Hudock K, et al. Mavrilimumab in patients with severe COVID-19 pneumonia and systemic hyperinflammation (MASH-COVID): an investigator initiated, multicentre, double-blind, randomised, placebo-controlled trial. Lancet Rheumatol. 2021;3(6):e410–8.
- 51. Patel J, Beishuizen A, Ruiz XB, et al. A Randomized Trial of Otilimab in Severe COVID-19 Pneumonia (OSCAR). Cold Spring Harbor Laboratory; 2021. DOI:
- 52. Boyapati A, Wipperman MF, Ehmann PJ, et al. Baseline severe acute respiratory syndrome viral load is associated with coronavirus disease 2019 severity and clinical outcomes: post hoc analyses of a phase 2/3 trial. J Infect Dis. 2021;224(11):1830–8.
- 53. Lomakin NV, Bakirov BA, Protsenko DN, et al. The efficacy and safety of levilimab in severely ill COVID-19 patients not requiring mechanical ventilation: results of a multicenter randomized double-blind placebo-controlled phase III CORONA clinical study. Inflamm Res. 2021;70(10–12):1233–46.
- 54. Declercq J, Van Damme KFA, De Leeuw E, et al. Effect of anti-interleukin drugs in patients with COVID-19 and signs of cytokine release syndrome (COV-AID): a factorial, randomised, controlled trial. Lancet Respir Med. 2021;9(12):1427–38.
- 55. Temesgem Z, Burger C, Baker J, et al. 40. Lenzilumab efficacy and safety in newly hospitalized COVID-19 subjects: results from a phase 3 randomized double-blind placebo-controlled trial. Open Forum Infect Dis. 2021;8(suppl 1):S29. DOI:
- 56. Veiga VC, Prats JAGG, Farias DLC, et al. Effect of tocilizumab on clinical outcomes at 15 days in patients with severe or critical coronavirus disease 2019: randomised controlled trial. BMJ. 2021;372:n84.
- 57. Stone JH, Frigault MJ, Serling-Boyd NJ, et al. Efficacy of tocilizumab in patients hospitalized with COVID-19. N Engl J Med. 2020;383(24):2333–44.
- 58. Salvarani C, Dolci G, Massari M, et al. Effect of tocilizumab vs standard care on clinical worsening in patients hospitalized with COVID-19 pneumonia: a randomized clinical trial. JAMA Intern Med. 2021;181(1):24–31.
- 59. Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341–64.
- 60. US Food and Drug Administration. FDA announces bebtelovimab is not currently authorized in any US region. Available from: https://www.fda.gov/drugs/drug-safety-and-availability/fda-announces-bebtelovimab-not-currently-authorized-any-us-region
- 61. Eckermann S. Globally optimal trial design and risk sharing arrangements are key to avoiding opportunity cost of delay and enabling equitable, feasible and effective global vaccine research and implementation in current or future pandemics. Front Public Health. 2022;10:1085319.