The importance of piloting in the development of technology-assisted psychological interventions

In our last blog, we introduced the SH+ project with South Sudanese refugees in Rhino Camp, Northern Uganda. Here we will discuss the importance of piloting when developing a technology-assisted intervention. Our experience with SH+ showed us the need to consider adaptation not only for culture and context, but also to ensure the technology can be feasibly used and potentially scaled in humanitarian settings.

To recap, SH+ differs from conventional group or face-to-face psychological interventions in two ways. Firstly, it uses a large group format where groups of up to 30 people attend each session, secondly it uses a pre-recorded audio script instead of information provided by a facilitator or therapist. This format was inspired by similar courses such as the evidence-based “Coping with Depression” course which uses a psycho-education format. The use of pre-recorded audio is also found in some meditation and relaxation courses, which are widely available. This approach increases fidelity by using a recording, and aims to reach a greater number of people through a package that can be disseminated more easily in hard-to-reach humanitarian settings.

In each session, participants sit as a big group and listen to the course, completing experiential exercises (e.g. breathing and grounding exercises, noticing strong feelings) and participating in small group discussions. The role of the facilitator – who can be a lay person, such as a community mobilizer – is to ensure the safety of the group, answer any questions that arise, and generally support participation in exercises and discussions. The audio is central to the delivery of the intervention.

We began work on SH+ in Uganda in 2015 translating and adapting the generic English SH+ package for use with South Sudanese refugees. This included adapting all illustrations for the context and recording a high quality version of the SH+ audio. There were many discussions about how to get the tone and speed of the audio correct to try and create the same warmth and connection one might feel if listening to a therapist running a group. A substantial further part of this early work was learning how SH+ could be feasibly delivered in humanitarian contexts where there is often limited electricity and little or no familiarity with technology-assisted interventions (e.g. listening to audio instruction).

We realised prior to recording SH+ that it was difficult for people to maintain attention for so long with no visual cues (e.g. video). We also thought it would be difficult for facilitators to know when to run discussions or demonstrate an exercise without a system where they could easily track where they are in the session. We considered using video or a PowerPoint presentation, but decided that a design principle for SH+ should be keeping it as technically simple as possible to increase potential scalability (e.g. less reliance on equipment and electricity). We addressed these challenges by adding titles to each section of the course (e.g. “Introduction”, “Emotional storms”, “Grounding”), along with a bell sound to signal to the facilitator the start of a new section. This matched the sections in a facilitator manual, so if the facilitator lost track of the course, they could find their place again when the bell for the new section sounded.

Four facilitators with experience of working with South Sudanese refugees were recruited and participated in training on SH+. While all facilitators gave positive feedback about the script and the content, they felt the idea of only listening to audio instruction felt strange and may lead to disengagement because of the limited “human element” in the intervention. Accordingly, we made the role of the facilitator more central, viewing them as essential to build cohesiveness in the group, by for example running a short icebreaker activity at the start, demonstrating exercises (e.g. “grounding”) along with the audio, and where required giving further scripted examples to explain a concept or discussion question.

The initial UN Action-funded pilot of SH+ showed that these changes greatly increased its feasibility. The changes seemed to help engagement during the pilot groups and assisted participants with understanding some of the more complex ideas used in SH+. The facilitators were able to implement the course well and participants provided positive feedback. However, this initial pilot also showed the intervention was still too complicated to deliver, which would seriously limit future scalability. For example, it was difficult for facilitators to follow the course, participants often needed questions repeated or requested extra explanations, while the high proportion of people with limited literacy meant the worksheets could not be effectively used.

A number of essential adaptations were made. For example, in response to limited literacy, we removed written worksheets from the course and reworked the exercises so that people were asked to identify and remember any responses to questions instead of writing them down. For example, instead of writing down action plans, participants were asked to formulate a plan and discuss it with the person sitting next to them if they wished. These small changes mean that SH+ can now be implemented more effectively with groups where literacy is low. Facilitators reported that some of the more complex conceptual parts of SH+ were difficult to understand for some participants. This led to a re-write of the script, where ideas were expressed with increased clarity, more scripted explanations for facilitators were developed to be delivered as required, and questions were simplified.

These initial stages have been essential in helping us to refine SH+ to improve its quality and feasibility. A key part of this was thinking about the multiple aspects required to implement a technology-assisted intervention, such as a high quality, warm voice, audio recording, working out ways to ensure smooth running of a course, and how to introduce group cohesion and trust. Our feasibility RCT, recently completed under the R2HC project, suggested that these changes improved understanding of SH+ concepts and simplified delivery. Having considered implementation at scale up during these pilot and redesign phases, we are now confident this package can be quite easily adapted and implemented in difficult humanitarian settings following initial testing. The next step of this R2HC project is a large, pragmatic RCT to test the effectiveness of SH+ with South Sudanese refugees.

Dr Ken Carswell: Technical Officer, WHO.
Dr Mark van Ommeren: Public Mental Health Adviser, WHO.
Dr Wietse Tol: Peter C. Alderman Foundation, Program Director; Assistant Professor, Johns Hopkins University.