Vaccination is one of the most cost-effective public health interventions for preventing infectious diseases, and in turn, minimizing lost wages and productivity losses due to illness and death []. In any setting, these benefits can only be realized and evaluated when burden of disease data are available and can be used to guide vaccine introduction and evaluation []. A lack of disease burden data limits public health capacity to guide these interventions and to develop the necessary policy to support the process. Therefore, research studies are critical for generating data to inform the creation and rollout of public health interventions at the country, regional, and global levels [, ].
A systematic multicountry research approach offers the advantage of generating standardized burden of disease data that allow for comparison between countries and can be (cautiously) generalized to a region. Regulated clinical trials, interventional studies that will provide data in support of licensure of vaccines or drugs, follow an international set of guidelines known as Good Clinical Practice (GCP) []. Epidemiological studies, however, are not subject to GCP requirements, but ensure data quality. Ensuring human safety, data quality and compatibility in multicountry studies is essential for the correct interpretation and applicability of the data. Standardization of the study protocol and standard operating procedures (SOPs) define a pathway for multicountry comparability. Monitoring during study implementation ensures the protection of human subjects and the correct application of such protocols and SOPs while identifying gaps and opportunities for improvement in quality.
While monitoring and evaluation are mandatory in clinical trials [, ], with the US Food and Drug Administration (FDA) recommending a risk-based monitoring strategy [], there is limited reported experience in monitoring best practices in epidemiological studies. The European Centre for Disease Control and Prevention (ECDC) and the World Health Organization (WHO) have outlined strategies for monitoring and evaluation [, ]; these guidelines, however, are largely for surveillance in routine healthcare systems and focus on data quality monitoring. Here, we present the monitoring and evaluation procedures implemented in the multicountry Severe Typhoid Fever in Africa (SETA) program [], adapted from the FDA and the ECDC recommendations on monitoring clinical trials and data quality, respectively. We also present preliminary data of the monitoring impact from two of the six countries participating in SETA.
METHODS
Description of the SETA Program
The SETA [] program is a study that was implemented in six countries in sub-Saharan Africa: Burkina Faso, Democratic Republic of Congo, Ethiopia, Ghana, Madagascar, and Nigeria. The aim of the study was to investigate the burden, severity, and long-term sequelae of typhoid fever and other invasive Salmonella infections with the purpose of contributing data for the decision-making process of typhoid vaccine introduction in the African region. The protocol of the study has been published elsewhere [].
The SETA surveillance system was established to identify invasive Salmonella infections among patients of all ages with prolonged fever in the defined catchment areas (Table 1). Patients meeting the inclusion criteria (Table 1) were enrolled into the study upon providing voluntary written informed consent. Blood, an oropharyngeal swab, a stool sample, a urine sample, and in cases of intestinal perforation, tissue sample(s), were taken from each enrolled patient. Samples were transported to the study laboratory for processing, analysis/testing, and storage. Confirmation of invasive bacterial disease was accomplished by blood culture. Patients testing positive for any Salmonella (cases) or those with intestinal perforation (special cases) were enrolled into long-term follow-up studies. In these long-term follow-up studies, two or three immediate household contacts and between two and four neighborhood controls meeting the inclusion criteria (Table 1) were identified, consented, and followed up over a period of 360 days. During the follow-up period, cost of illness (conducted in four of the six countries), long-term disease outcomes, and immune responses of cases were assessed.
Overview of FDA and ECDC Recommendations for Monitoring
The FDA recommends developing a monitoring plan that is tailored to the needs of a study. As such, no single approach to monitoring is necessary for every study, and a mix of centralized and on-site monitoring practices can be part of a risk-based plan []. Overall, critical data and processes should be identified and a risk assessment should be conducted before developing a monitoring plan that focuses on the important and likely risk to data and processes (Figure 1).
Figure 1
Aspects to consider when developing a monitoring plan tailored to study needs.
The ECDC recommends monitoring the surveillance system and data quality []. To achieve this, it uses an approach that is attribute-oriented focusing on the surveillance system’s attributes such as data completeness and validity, timeliness, usefulness, representativeness, simplicity, flexibility, and so forth. To quantitatively measure these attributes, ECDC encourages the development of data quality indicators (e.g., for completeness, completeness of data reported; for validity, proportion of cases complying with case definition) and settings targets for monitoring. These indicators could be prioritized and be part of a data quality monitoring report.
Development of SETA Monitoring Plan
Elements from both the FDA and ECDC recommendations were taken into account and applied in the development of the SETA monitoring plan.
Using the FDA framework, we identified the SETA critical processes that if inaccurate, not performed, or performed incorrectly would affect the delivery of outcomes from the three first SETA objectives (Table 1). We referred to these processes as SETA core activities (Figure 2). They include patient screening and enrollment, study form completion, sample collection, sample transportation, Salmonella cases/special cases notification, follow-up, storage and sample shipment, sample processing, data capturing, data entry, and data management.
Figure 2
Core activities in the SETA program.
From the risk assessment, we identified the type of data to be collected to deliver the outcomes from the three first SETA objectives (Table 1) and the specific core activities/processes required to collect these data. Using this information, a list of key indicators was developed to be monitored using on-site and centralized monitoring methods. Table 2 presents the list of data indicators monitored using a centralized method and Table 3 shows the list of processes or core activities monitored using on-site monitoring. Additionally, a team was created to dedicate fully to monitoring activities on-site. SETA databases were used in centralized monitoring at the International Vaccine Institute (IVI).
On-site Monitoring of Core Activities
The aim of monitoring of core activities in the field was to carefully observe the adherence of the study procedures during visits by the IVI monitoring team (Table 3). The monitoring of core activities in each field visit was organized into three major areas: (1) healthcare facilities, (2) community, and (3) laboratory and data management.
Healthcare Facilities
The procedures to identify eligible patients according to the study protocol included obtaining voluntary informed consent, enrollment, collection of samples, and documentation of observations at the healthcare facility. These procedures took place at the entry points within the specified healthcare facility in outpatient/inpatient departments. Where the source documents were patient charts, they were checked to corroborate the information written in the study case report forms. To ensure harmonization of patient identification and recruitment procedures across all sites and within enrollment points, a screening and recruitment log (Supplementary Figure 1) was developed and deployed at each site’s entry point.
Community
The procedures observed in the field included enrollment of household contacts, neighborhood controls, and follow-up visits of the cases, contacts, and controls. At each follow-up visit, the cases, household contacts, and neighborhood controls were visited at the designated location. Voluntary informed consent was obtained when enrolling controls and contacts. Follow-up and laboratory forms were completed and samples collected.
Laboratory and Data Management
The procedures for the laboratory and data management included sample transportation (Supplementary Figure 2), processing, storage, and documentation. This major area also included procedures for paper-based or electronic data capture and the management of data safety and entry. These procedures (Table 3) were monitored to ensure that they are conducted in accordance with the appropriate SOPs. An external microbiologist consultant was contracted to undertake the quality assurance process at the participating laboratories at least once every year. The consultant reviewed all the sample processing procedures on-site and flagged issues that required improvement and follow-up. The tool used to monitor and document findings (Supplementary Figure 3) was adapted from the International Organization for Standardization (standard 15189) [].
A checklist for each procedure was developed and used to guide the monitoring (Table 3). The list details activities that at a minimum were observed during each visit regarding every aspect of the study, with the core activities acting as a guide (Figure 2). Activities were considered satisfactory if implemented according to the protocol and unsatisfactory if they required improvement.
Central Monitoring of SETA Databases at IVI
Monitoring of SETA databases at IVI ensured the completeness, consistency, and quality of data recorded in databases. The data team checked all data from the field sites, both paper-based data collection and electronic data collection. A report for each site was generated for identified study indicators based on the objectives (Table 2) and shared with the principal investigator(s) (PI) at each specific site on a monthly basis. The report generated counts and proportions of indicators based on the study objectives. These included the number of patients screened, enrolled, and sampled, results of tests done, and contacts and controls under follow-up (Table 2). The report also outlined queries that needed to be addressed by the study team.
The observations made in the field complement the monitoring of databases, providing a comprehensive assessment of the surveillance performance and implementation of harmonized procedures at each site. All monitoring tools were subjected to pilot test at each site. This step was performed to assess and improve the tools and also to identify the unique cultural and geographical context of each site. These characteristics were considered during the implementation of the surveillance monitoring procedures.
Implementation of Monitoring
Site monitoring visits were scheduled at least twice per year by dedicated IVI SETA staff; additional visits were scheduled when necessary. An agenda based on the core activities (Figure 1) was sent to the PI of each country a month in advance of the visit to provide flexibility for the necessary logistical planning or to modify the agenda to accommodate unique issues in each site. Upon arrival at the respective site, a brief meeting was held with the PI and members of the study team to finalize the agenda.
A process audit of the relevant procedures for each core activity was observed, documented, and was assessed in and outside the healthcare facilities, using the protocol and specific SOPs. A full day was spent in each healthcare facility to observe all study procedures and their implementation by study staff, starting with eligible patients identification, to transportation of samples to the laboratory for processing. Study documentation, including screening and enrollment logs, study forms (5%–10%) of newly collected forms since the last visit randomly were selected, and all informed consent forms, were reviewed for completeness and accuracy. Key procedures such as phlebotomy and blood culture were observed to verify whether they were being implemented per SOPs.
In the study laboratories, the processing of the samples was observed from the time they were received and processed/tested until point of storing. Accompanying documentation was also reviewed for accuracy and completeness. The forms reviewed in the laboratory included laboratory forms, samples transfer logs, samples logs documenting the performed assays, and the corresponding results.
For sites using paper data capture, dual data entry, tracked error correction, data protection, and dual password entry to access the computer were monitored. Secure storage of the complete forms was checked to determine whether the site is lockable and only accessible by key study staff. In sites capturing data electronically on a digital platform designed for the study called SETACollect, real-time data entry was observed.
Follow-up procedures, including control identification and enrollment and the sampling and packing of samples collected, were observed and documented. In instances when it was not possible to observe a particular procedure (e.g., unavailability of a case for follow-up), the activity was scheduled for a subsequent visit.
For each visit, a comprehensive report was written and shared with the respective PI. The report included the objectives of the visit, a list of personnel encountered at site, an executive summary, findings during the visit, and actions for immediate implementation and long-term follow-up. Issues requiring immediate attention were addressed with the study staff. Other corrective measures were highlighted to the in-country team during debrief at the end of the monitoring visit. An action plan (Supplementary Figure 4) was developed and discussed with the SETA team, especially the country focal point, requiring their support for implementation and follow-up. The action plan acted as a guide to track the progress of the recommendations made following each monitoring visit.
External Monitoring
Additional monitoring was performed by the Scientific Advisory Process for Optimal Research on Typhoid burden of disease (SAPORT), on behalf of the Bill & Melinda Gates Foundation (BMGF). Elements monitored by SAPORT included project management and staffing, data management, enrollment of patients and controls, sample collection and transport, laboratory quality assurance/control, and follow-up visits. SAPORT, with a Secretariat at the Emory Global Health Institute, contracted by BMGF, provided direct research oversight for SETA [] and SEAP (Surveillance for Enteric Fever in Asia Project) []. The SAPORT group was comprised of experts in the field of global health, infectious diseases, epidemiology and microbiology, biostatistics and modeling, and policy. They were identified through consultation of the SETA and SEAP PIs and the chairperson of the SAPORT group identified by BMGF. Part of their activities included yearly formal site monitoring visits and review of quarterly monitoring reports with key project indicators, and annual meetings with network PIs (SETA and SEAP) to deliberate on findings from the visits. These SAPORT activities ensured that surveillance and research were streamlined and harmonized between SETA and SEAP.
Impact and Preliminary Results
Complete monitoring results will be published at the end of the study, but preliminary findings show the value of SETA monitoring and evaluation. One example is summarized here. The proportion of contaminated blood cultures is one of the data indicators used to measure the impact of monitoring. This indicator is important because the standard method for confirming typhoid fever in SETA is blood culture, and contamination in blood culture indicates whether blood culture collection procedures are being implemented according to SOPs.
Figure 3 shows the proportion of blood cultures contaminated by month during 2017 in two SETA sites. The graph shows that contamination decreases after a monitoring visit, suggesting that phlebotomy techniques improve following support of on-site monitoring activities. However, specific events can trigger an increase in blood culture contamination. We observed such an increase in Nigeria when the trained study phlebotomist took leave; the proportion of contaminated blood cultures increased from 5.7% to 12.2% over two months (Figure 3). Results of a monitoring field visit included the implementation of additional training and close follow-up of phlebotomy procedures. The site elected to develop and use a phlebotomy checklist to identify areas where enhanced training and oversight were needed to improve the phlebotomy process. These activities resulted in a reduction in the proportion of contaminated blood cultures and lower dependence on a single study member for aseptic blood culture collection, providing an example of the positive impact of monitoring both data and field activities and the ongoing communication with the local study staff.
Figure 3
Proportion of blood cultures contaminated by month during 2017 in two SETA sites. Abbreviations: APR, April; AUG, August; JUL, July; NOV, November.
CONCLUSIONS
Monitoring of epidemiological studies is important for data quality and to allow comparability across sites in a multicounty, multisite setting. Existing clinical trials and routine surveillance monitoring tools can be adapted for use in multicounty epidemiological studies. In implementing monitoring procedures, we observed that the environment and conditions of the countries participating in SETA vary as is likely to be the case for many multicountry, multisite epidemiological studies. Flexibility is required to navigate the dynamics of each study site and customized approaches have been implemented to adapt study procedures and overcome challenges in these individual conditions. Some of these challenges include political uncertainty and institutional and government bureaucracy. Nevertheless, rollout of core activities in line with the protocol and study SOPs required for data comparability across sites remained paramount, robust, and harmonized. The monitoring procedures and activities implemented have demonstrated value for the assessment of site performance and the identification of opportunities for improvement through on-site interaction with the site staff to ensure that recommendations made are addressed. It also allowed the creation of teams to identify solutions to the various challenges, making continuous feedback and team participation an important element in the success of monitoring activities. This underscores the value of considering routine monitoring of epidemiological studies.
Notes
Acknowledgments. The authors thank the following site faculty and staff for their contributions to the monitoring and evaluation of the Severe Typhoid Fever in Africa program (SETA): Ini Adebiyi, Adegoke Adeoye, Sarah Agbi, Eunice Aroyewun, Halima Babalola, Aderemi Kehinde, Abigail Kogsey, and Oluwafemi Popoola, as well as the rest of the country team that facilitated the activity. The authors are also grateful to the International Vaccine Institute staff for their programmatic support, including Ji Hyun Han, Hyon Jin Jeon, and Soo-Young Kwon. Finally, the authors are grateful to Adwoa Bentsi-Enchill, World Health Organisation; Dennis Chao, Institute for Disease Modeling; Hope Johnson, GAVI; Gagandeep Kang, Translational Health Science and Technology Institute; Samuel Kariuki, Kenya Medical Research Institute; Eric Mintz, Centers for Disease Control and Prevention and, Jeffrey Stanaway, University of Washington, who were part of the SAPORT team.
Financial support. This research was funded by the Bill & Melinda Gates Foundation [OPP1127988]. The International Vaccine Institute acknowledges its donors including the Republic of Korea and the Swedish International Development Cooperation Agency. This publication was made possible through a grant from the Bill & Melinda Gates Foundation [OPP1201031].
Supplement sponsorship. This article was published as part of the supplement “Severe Typhoid Fever in Africa (SETA) Program” sponsored by the International Vaccine Institute.
Potential conflicts of interest. All authors report no potential conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.
References
- 1. Duclos P, Okwo-Bele JM, Gacic-Dobo M, Cherian T. Global immunization: status, progress, challenges and future. BMC Int Health Hum Rights2009; 9(Suppl 1):S2.
- 2. Onwujekwe O, Uguru N, Russo G, et al Role and use of evidence in policymaking: an analysis of case studies from the health sector in Nigeria. Health Res Policy Syst2015; 13:46.
- 3. Maddalena V. Evidence-based decision-making 8: health policy, a primer for researchers. Methods Mol Biol2015; 1281:501–17.
- 4.
- 5. Artinian NT, Froelicher ES, Vander Wal JS. Data and safety monitoring during randomized controlled trials of nursing interventions. Nurs Res2004; 53:414–8.
- 6. Fleming TR, DeMets DL. Monitoring of clinical trials: issues and recommendations. Control Clin Trials1993; 14:183–97.
- 7. US Food and Drug Administration. Guidance for industry. Oversight of clinical investigations—a risk-based approach to monitoring. Silver Spring, MD: FDA, 2013.
- 8.
- 9.
- 10. Se Eun Park TT, Cruz Espinoza LM, Panzner U, et al The Severe Typhoid in Africa program protocol: assessing the burden, severity, host immunity, and carriage associated with invasive salmonellosis using prospective sentinel-based passive surveillance, healthcare utilization surveys, and prospective case-controlled and cohort study designs. Clin Infect Dis2019; 69(Suppl 6):S413–6.
- 11. Schneider F, Maurer C, Friedberg RC. International Organization for Standardization (ISO) 15189. Ann Lab Med2017; 37:365–70.
- 12. Barkume C, Date K, Saha SK, et al Phase I of the surveillance for Enteric Fever in Asia Project (SEAP): an overview and lessons learned. J Infect Dis2018; 218:188–94.