Lessons learned about adaptive research design during the pandemic

One of the most common terms used to describe the pandemic is that it has been ‘unprecedented’. This unusual and rapidly changing situation is somewhat at odds with standard ways of doing research, which normally requires detailed protocol development at the beginning and then following the planned process step by step until its completion. In this blog we reflect on how to strengthen research conducted during dynamic outbreaks or crises by drawing on our experience of doing a process evaluation of the Community Perception Tracker (CPT) approach in Lebanon and Zimbabwe. The CPT approach was developed by Oxfam during the Ebola outbreak in the Democratic Republic of the Congo and was adapted and scaled up for COVID-19 response. It provides a more systematic way of doing community engagement by allowing programme staff to routinely collect community perceptions and then analyse how trends change over time so that they can adapt programming or advocate on the behalf of communities.

Using a ‘Phased evaluation’ to strengthen uptake and use of results

Process evaluations are designed to assess whether a programme was implemented as intended and to monitor each part of the causal mechanisms in a theory of change. In our case the process evaluation was designed to document whether the CPT was a useful approach for allowing humanitarian organisations to learn from outbreak-affected populations in real time, and whether these community-generated insights were able to be used to improve programming so that it was more acceptable and relevant. Typically process evaluations share their findings with operational staff at the end of the research with the aim of the lessons learned being used to inform future programmes. However, the circumstances around COVID-19 were so unique that we felt that this traditional model of waiting until the end to share findings no longer made sense. So we adapted to a ‘phased’ way of working whereby we added in learning workshops at key time points which brought together research and operational staff. At these we discussed findings to date so that the operational partners could define a key set of actions to work on for the next period. Agreed actions were then actively measured in our next period of data collection. We feel this resulted in the best outcomes for crisis-affected populations, implementers and still met the requirements of a comprehensive evaluation.

Adapting while keeping focused on the research objectives

Our initial plans had been to conduct the same set of research methods in both Lebanon and Zimbabwe and for data collection rounds to occur at the same time. However due to unforeseen events (e.g. the explosion in Lebanon), rolling COVID-19 lockdowns and restrictions, and delays associated with ethical approvals, our workplans have had to change frequently. As timelines and methods were modified in both countries we have had to keep focused on the ultimate aims of the project and making sure we can deliver as best as possible against these. For example, we initially proposed that our work would involve 4 methods: 1) analysis of the community perceptions entered into the CPT, 2) repeated rounds of phone interviews with populations, 3) repeated interviews with CPT staff and 4) structured observations of COVID-19 prevention behaviours. Ultimately, we had to drop the observations at public handwashing facilities in both countries due to feasibility and safety concerns. To compensate for this, we diversified the ways we were capturing self-reported behaviour in the phone interviews with populations. This will allow us to achieve our objective of having some measures of behaviour over time. Another example is that that we decided to condense the data collection rounds due to delays associated with ethical approvals. Pleasingly this doesn’t appear to have caused compromises to data quality and in fact it has allowed our research teams to develop a stronger rapport with participants.

Making the most of the opportunity to do operational research differently

Historically research in humanitarian crises has been dominated by academics from high-income countries. Research often involved local partners as data collectors, but these individuals are often not invited to be part of the analysis and dissemination of results, and therefore often don’t get credit for their work. We wanted to make the most of the necessity to work remotely by strengthening partnerships between local academic institutions and operational partners and by establishing research teams within the implementing organisations - Oxfam and Action Against Hunger (AAH). The research teams within the NGOs were trained not just on the methods we would be using for this research, but more broadly on research principles, the rationale for different types of research methods and analysis techniques. We budgeted for analysis software for these staff so that they could play a lead role in the data analysis. This approach was time-intensive and relied on aligned collaboration and communication platforms, the establishment of clear data sharing protocols, iterative rounds of feedback and support, and simple practicalities such as having good internet connections.

There were also challenges associated with having research teams embedded within the organisations responsible for implementing the intervention that was being evaluated. For example, it has sometimes been difficult for the research staff to maintain the boundaries necessary for them to act as evaluators. However, the benefit of this approach has been that they have gained a unique vantage point into the day-to-day practicalities of trying to roll out a novel approach to community engagement like the CPT. Relationships between the research team and other organisational staff helped us to address broader issues that arose during data collection. For example, if a participant mentioned something that raised protection concerns, the research staff could easily connect with protection colleagues within the organisation to resolve the issue in an appropriate way.

While many of the adaptations we describe here were done to due the ‘unprecedented’ nature of the pandemic, we feel that many of the principles could be used to strengthen operational research in general and to make research more feasible and sustainable in other types of humanitarian crises.