Our Approach to Data Annotation

In the past year or so, we've been moving towards creating a framework that could help us compensate the people who contribute to Uli's datasets given the knowledge we have gained about how such work is done. The creation of the dataset, a time-intensive process, also had contributors emotionally drained, with some leaving due to the strain of annotating hateful content. Our motivation was to understand how to value annotation work while being in alignment with values of care, intersectionality and reflexivity.

In pursuance of this, we began by looking to other projects similarly situated: those developing AI by involving the people who it is built for in the design and development process. 29 organizations and projects were identified, and we interviewed 5 of them. We also interviewed academic researchers who had collected annotations and experts who studied the gig economy. To understand the interviewer’s perspective on responsibly engaging with data annotators, we asked them questions about recruiting, data management, contracting, compensation, and training.

This led us to make certain considerations about how to value data work. Annotators themselves are diverse: they may be immigrants, students, refugees, single mothers; and they may take on this work for a variety of reasons as well. Annotators may be engaging in this work for career development, as a primary or supplemental source of income, or from it being aligned with their profession or area of expertise. The relationship between the annotator and the dataset may also go beyond data work to stewardship as well. The motivations and relationships are not mutually exclusive, and are often overlapped. Recognising this complexity helps us understand how data work is organized, conducted and compensated.

There are also risks associated with annotation work, and have to do with the people represented in the datasets. There is a need for risk assessments for people who engage in annotation work as well. While the risks vary across tasks, there are a few forms of risk that arise specifically in the context of social impact projects, such as when annotators are asked to contribute personal data, or annotation tasks where the subject matter relates to harms (derogatory or violent content) encountered by the contributors. The latter merits higher compensation and support to the annotators given the exposure to harm. When determining compensation, harms emanating from such work must be accounted for as well. For instance, it has been highlighted that annotators work in difficult conditions with demanding workloads and targets, and face mass rejections- these risks come from the position of the annotator in the AI supply chain, rather than the nature of the work they engage in.

Data annotation work is rarely recognised as expertise. Instead, it is seen as repetitive, mundane work that does not involve specific skills or domain knowledge. However, there has been a call to recognise the contributions of the annotators better.

Through the interviews, we saw a wide variety of annotation work and expertise; and we noted the following forms of expertise sought in annotation work:

(i) Subject matter experts: those with expertise in medicine, law, journalism to build medical/legal/misinformation datasets.

(ii) Lived-experience expertise: of which building Indian language datasets is a prime example, since native speakers of languages are recruited to engage in annotation work for these datasets. The Uli datasets also drew on the lived experience of the annotators who had knowledge and experience relating to online gender based violence in India.

There are considerations to be made when recognising this work- for instance, while annotators contributing to low resource languages may be valued the same across all languages, in practice, it may be influenced by location. While there may be no difference in the kinds of lived experiences and social isolation, we must consider how this knowledge may intersect with factors such as urban centers, low baseline incomes and so on. Compensation must take an approach that does not deepen existing fault lines.

(iii) Senior annotators: annotators with more experience engaging in annotation work play an active role in the direction of projects, verifying and ensuring quality of work, however it largely remains unrecognized given the perception towards annotation work as being low-skill.

Our attempt to understand monetary compensation did not extend beyond ‘at least minimum wage’, but various strategies to protect the interests of the data workers surfaced. Efforts have also been undertaken to engage in the creation of datasets in a responsible manner, which reflects a commitment to prevailing inequities in this process. We are interested in building a framework based on these ongoing efforts, to inform how data annotation work may be equitably recognised and valued. We will be attempting to answer this question by engaging with data workers’ perspectives themselves.