Lessons in User Research from Donald Lobo

There is a famous quote attributed to Edison: “I have not failed, not once.  I’ve discovered ten thousand ways that don’t work.” This quote was his defense for the space for experimentation even if it doesn’t get you results right away.

Tattle’s mandate- to build citizen centered tools for responding to inaccurate and harmful content- implies that nearly all of Tattle’s projects start out as experiments. Our raison-de-etre is to claim more space for civic action in a problem space dominated by platforms and governments. The problem space (of information disorder) is rapidly changing. Research on the mechanics and its effects is relatively new. Tattle’s work, grounded in practice, also hopes to contribute to the growing body of work. In theory, we should embrace Edison’s comfort with failing ten thousand times. In practice however, it is painful, and frankly not feasible to fail that many times. When you build something there are many stakeholders who join you on that journey- partner organizations, funders and your own staff. And people have only so much patience to give time and resources to something that continuously fails.

The binary of success and failure, is of course a simplification. Projects succeed in some ways and fail in others. To take the example of Uli- from the first year of development, we have a unique dataset of online gender based violence in Indian languages; and the longest list of slurs in Hindi, Tamil and Indian English. We proved that user-end moderation of social media is possible. We generated original research on developing alternate visions of machine learning. But there is one way in which Uli didn’t succeed, which is what we are also trying to address in year 2: the front end that uses these backend resources, isn’t used by as many people as we wanted. Our goal was to have this plugin be used by 100 people by the December 2022. We are close, but this goal was a modest one to begin with. So by my own assessment, this is a failure.

I have been trying to understand why we failed- from our perspective we developed this tool with the people we imagined would use it. We asked gender rights activists and researchers what they would want in a tool that reduced the effects of gender based violence (in addition to redaction that is). We built some of them (archiving, mental health resources in Indian languages). But when we launched, there seemed to be a mismatch in where, when and how people would access these features and the user-flows we had built. For one, we built a web browser plugin whereas most of our targeted users use social media on mobile. We knew this going in (hence the modest goal of 100) but in year one we wanted to minimize risks with data collection, which was easier to do on a web-browser than mobile. But I don’t think that is the only reason why we hit the usage goal.

So here we are in year 2, on our second round of experimentation to get the front-end to be more useful. And while I admire Edison’s tenacity, I would like to get it right this time around.

I was expressing my frustration/aspiration with Donald Lobo in a meeting earlier this week. Donald is the founder of Project Tech4Dev. Their product Glific- a WhatsApp chatbot builder has been used by several NGOs, enabling them to scale their communication. Lobo had pointed advice on how to build useful products:

• When you talk about your idea to NGOs, they will often express excitement about an idea and say that the solution will solve many or all their problems. But this rarely happens. That is why Tech4Dev insists on partner organizations also bringing money to a pilot. Money isn't necessarily the best way to gauge investment in an idea, but it is unfortunately the only available proxy.

• Whether partners on the project bring money to the table depends on their ability to pay, as well as their faith in you to deliver on the idea.To find invested organizations, talk to 3-7 NGOs. Aim for 7 but expect three to buy into your vision and sign up.

• Have a LOT of conversations with the organizations and ‘users’ that have signed up. Find a user and talk to them daily. Most will get irritated but about 10-30% of them will stick around. Don’t be dejected by the high attrition. This is to be expected.

• Tech4Dev in particular, focuses on building platforms and not solutions.  When building a new platform, they do research to ensure that what they are building hasn't already been built. An easy way to validate the idea is to run the idea with three orgs and check if it fits 80% of their need. No platform will ever fulfill 100% need for anyone. NGOs have to accept it. Funders are also beginning to get tuned to this idea.

• I asked if Project Tech4Dev ever had to convince their partners that something that they are facing is a problem that can be fixed. Lobo said that for smaller things convincing is easy. But when it comes to the platforms they build, they don't try to convince the partner. If they have to convince then the partner isn't a good partner for the user-study.

I believe that in the last point, Uli will vary from Tech4Dev's project since we're also trying to show new possibilities in online moderation and advocacy. So, there is some 'convincing' involved which introduces some tension in the user-research process. But for the most part, I found the pointed and specific advice something all social impact projects would find helpful.