From Retrospective Detective to Fast-Tracked Feedback

Oxfam is currently wrapping up an evaluation of a 6-month pilot launched in Za’atari refugee camp in Jordan, which uses mobile devices to responsibly capture qualitative feedback received informally during conversations between Syrian refugees and Oxfam technical teams working in the camp. Our goal was to identify ways to ‘close the feedback loop’ within camp feedback mechanisms, encouraging swift redress and course corrections, learning what we’re doing well and what we aren’t.NEW_LINENEW_LINEThe way this works on a practical level is that refugees approach staff (as they normally would) who record their complaints via written summary, audio recording, or photo into a mobile application. For each piece of feedback logged, the system automatically generates a case number and redirects the case to the appropriate Oxfam team on the backend, which then conducts specific follow-up activities; all of this is chronicled within the app.NEW_LINENEW_LINEThe entire piloting process has led to provocative discussions around institutional responsiveness, reporting, and the use of data in humanitarian programming. We’ve found ourselves asking questions like: what influences decisions for global advisors at the head office, grant writers in country offices, camp directors, team leaders, and community mobilizers? How is this effectively communicated to them? Most importantly, once lessons are communicated, how can we convert that learning into action?NEW_LINENEW_LINEWe’ve engaged in these conversations within Oxfam and sought the advice of colleagues sector-wide. We presented the pilot at the Bond Tech4Dev September meeting, in a lightning talk at the ELDIS 20th Anniversary Workshop, in a panel at the European Evaluation Society and for a break out session at MERL Tech DC. We will also bring it to the agenda at the London Feedback Summit and MERL Tech London this February. These presentations revealed a high level of interest and generated meaningful discussions about issues we are still grappling with including: inclusivity, handling protection cases, and how to systematically utilize feedback data agency-wide.NEW_LINENEW_LINEWe’ve begun to see how feedback can enrich programme service delivery. The informal collection system was not only the most popular mechanism for feedback compared to the hotline from the outset, but has been increasingly adopted the feedback mechanism of choice (see graph).NEW_LINE

Percentage of feedback instances received by collection mechanics:

NEW_LINE

NEW_LINENEW_LINEWe’d believe that the main reason for this traction warrants a brief discussion of first of the Digital Development Principles “ Design with the user.” Panthea Lee argues even though focus on “user-centered design” originated in the private sector, it is a useful concept for practitioners as long as we account for the fact that our users are heterogenous. The tools and processes interventions introduce are used by different groups of people in different ways. We discuss our different users in detail in a previous blog, but we’d like to emphasize we didn’t ask the refugees to change how they prefer to give feedback. We asked the staff to change the way they collect and report it (but gave them a tool to make it easier to do so!).NEW_LINENEW_LINEDuring the application’s next iterative testing phase, we’d like to explore how systematic feedback collection can enrich the ways in which Oxfam learns from these efforts. We will turn our attention from refugees and field staff to technical team leaders and home office administrators and MEAL teams. We want to focus next on integrating new learning at the field level - knowledge gained from listening to feedback - into the Oxfam’s overarching, existing transparency and accountability requirements.NEW_LINENEW_LINEInstead of evaluators-playing ‘detective’ by retrospectively figuring out what happened prior to a given reporting interval, granular quantitative and, more importantly, qualitative details about service delivery could be accessed from a database into a reporting format whilst the project is still on-going. In this way, voices of refugees directly substantiate staff appeals for better resource allocation, and the feedback data system could amplify this up the administrative chain. This application has the potential to flesh out not just quantitative specifics- how much of a service was received - but granular qualitative details about why service delivery did or did not meet expectations and how delivery was adapted in the face of obstacles that constantly arise in crisis contexts. Could we lighten the reporting burden further by auto-generating some of the simpler measures and negate the need for write-ups? This could save one of our most precious of staff resources: time and avoids the communication delay caused by reporting mechanismsNEW_LINENEW_LINEAs we eagerly await the evaluation findings (expected imminently), our expectation is that we can do more with the data coming in to truly get the benefits from this feedback mechanism. A six month pilot wasn’t long enough to explore this fully and our focus was more on platform build-out, feedback collection and ironing out how referrals are made using the system. If this holds true in the evaluation findings, we will be looking for ways to scale up with this very much at the forefront of our mind.