Quality control during the coding process

The core quality of the Humanitarian Genome subsists in the coding of evaluation reports. These reports, all provided by humanitarian organizations, comprise the source of information from which the Humanitarian Genome drives its insights. The coding process is a qualitative one; coders read the report and then, according to strict coding regulation, identify text segments for coding and attach appropriate codes. The Humanitarian Genome coding scheme contains approximately 240 codes that are divided in to main and sub categories. As coders attach codes, pre-programmed algorithms are calculated allowing the Humanitarian Genome to sort through the meaning and content value of all the segments, enabling the system to extract the exact information the users seeks.

An example of the coding process is given below. In figure 1, coders have selected the two segments, linked them together and dragged and dropped the appropriate codes into the segment. In figure 2 below the same text segment is presented, as it appears in the Humanitarian Genome. The text appears as one quotation, and all the chosen codes are presented below it.

Such a coding process entails detailed human effort. With a team of diverse coders, the quality of the qualitative coding is subject to many influences such as interpretation, reading comprehension, background knowledge or concentration level of the coders. Additionally, as evaluation reports vastly differ from one another in writing style, structure and topic, new concepts may emerge. Therefore, ensuring the quality of the coders’ work is a crucial matter in the development of the Humanitarian Genome. During the development of the Humanitarian Genome quality control activities are being done repeatedly both to improve the individual’s coding but also to continuously tighten the reliability of the coding scheme and ensure it matches the information it is meant to decode.

At the start of the coding process, all coders were given a thorough training over the purpose of the Humanitarian Genome, the methods and tools used during the coding process and the Humanitarian Genome codebook that contains all the particular codes, definitions and coding rules. As a part of this training, the entire team coded the same two reports, all codes were then extensively discussed, thereby comparing and understanding each other’s coding and reasoning behind their choices. Going through this process helped the coders gain a clear insight into the logic behind selecting areas for coding, and the understanding of the meaning and use of the specific codes.

Once consensus over the codebook and work process was reached among the team the training period ended and coding commenced. Periodical ‘inter coder reliability’ (ICR) tests were applied. In qualitative text analysis differences in coding are expected to exist, to a certain extent, between coders. As explained above, such differences or disagreements might be caused by the interpretation or understanding of the coders, but may also be the result of the actual codes not being clearly defined. High ICR scores and values imply high agreement between coders, justify the codes that are used, ensure replicability and are a good indication of the quality of the work while low scores and values would indicate the opposite.

We performed the ICR tests using the Coding Analysis Toolkit (CAT), which calculates a Kappa score for the reliability between coders. The values and results that were produced in this process were reviewed by the team. Following this, suggestions were made for developing some extra coding rules and definitions of certain codes.

Another way of ensuring the quality of the coding process is by performing random searches in the Humanitarian Genome on a regular basis, and to provide each other with feedback on the quotations. According to the feedback coders receive, they go back to the specific quotation and revise it.

The continuous process of quality controls both qualitative and through calculation is an important and on-going exercise in improving and refining the quality of the coding process. High quality coding will remain important throughout the rest of the process and more quality tests are planned for the coming months.

11 January 2013

Nanda Talsma