In this blog, Jim Fruchterman—an AI innovator turned nonprofit leader—explores how Retrieval-Augmented Generation (RAG) offers a practical and safe path for using AI in the social sector. While highlighting the dangers of unchecked AI use in sensitive community contexts, he draws on recent successful examples of the use of RAG to show it as a promising way to blend the power of generative AI with domain-specific, trustworthy information.  

Social sector leaders ask me all the time for advice on using AI. As someone who started for-profit machine learning (AI) companies in the 1980s, but then pivoted to running nonprofit social enterprises, I’m often the first person from Silicon Valley that many nonprofit leaders have met. I joke that my role is often that of “anti-consultant,” talking leaders out of doing an app, a blockchain (smile) or firing half their staff because of AI. Recently, much of my role has been tamping down the excessive expectations being bandied about for the impact of AI on organizations. However, two years into the latest AI fad wave created by ChatGPT and its LLM (large language model) peers, more and more of the leaders are describing eminently sensible applications of LLMs to their programs. The most frequent of these approaches can be described as variations on “Retrieval-Augmented Generation,” also known as RAG. I am quite enthusiastic about using RAG for social impact, because it addresses a real need and supplies guardrails for using LLMs effectively.  

Background on Nonprofit AI Applications 

Because of the increasing number of conversations about AI over the last two years, I created the Nonprofit AI Treasure Map (below) to help nontechnical leaders navigate the AI field (and also wrote an accompanying article, Should I Be Using AI for This?).  

The Treasure Map divides the world into two halves: the Admin Zone, and the Program and Operations Zone. The Admin Zone is concerned with the staff of the nonprofit, and the tools they use to do their jobs. And, because the AI tools are prone to ‘making stuff up’ that isn’t correct, these tasks should always have the active engagement of humans. The Admin Zone is where most nonprofits have been using LLM-based products like Gemini and ChatGPT most actively.  

I am much more careful when it comes to deploying AI in the Program and Operations Zone. Here the people interacting with the AI are not internal staff members, but the members of the public whom the nonprofit exists to serve. Here, mistakes by the AI technology could lead to serious consequences. Since nonprofits have a central social mission, simply saving money is not a good reason to adopt AI if people in the communities being served get hurt. We already know of multiple cases where AI tools have gone horribly wrong, from giving the wrong advice to people experiencing weight-related disorders, to even encouraging suicide (to tragic results). Nonprofits may even have a harder challenge implementing AI safely than their for-profit peers, because they are more likely to be working with sensitive data about vulnerable people, whereas for-profit use cases may have lower stakes (e.g., making product recommendations).  

The central challenge of generative AI (like LLMs) is that its job is to make stuff up. And, when you ask the average of the Internet for advice, or whether a certain fact is true, you might be disappointed in what you get. It’s no surprise that a weight-counseling generative AI tool trained on the internet might give the wrong advice. Or that its answer to a math problem might just be plausible but wrong. The open-ended nature of these tools means that they are missing essential guardrails needed for confident use in social good applications.  

My recommendation to nonprofits is to wait for an AI tool to be available as a product, which will be much more likely to have safety features built in, and then test it extensively on your use case. In the cases where the AI tech isn’t up to the challenge today, I encourage leaders to just wait a year. The technology has been advancing at a very rapid rate, and something which does not work well right now might work great in a year or two.  

Enter the RAG 

A central approach to safety and efficacy in more advanced GenAI applications is to constrain the content the AI tool is allowed to generate. So, when the tool is asked a question, it doesn’t just generate an answer based on the huge amount of text (from the Internet at large) the model was trained on. These broad datasets can be full of unpleasant content, like the image generation tool that included child sexual abuse imagery in its training set. And, these general datasets can be missing (and likely are missing) the most relevant information for social sector use . Retrieval-augmented generation adds in a set of additional domain-specific knowledge which is used to improve the answer. This is where the acronym comes from: the generated answer is improved by being augmented by retrieving trusted domain specific knowledge. In short, the answers from a RAG system are far less likely to make up a false or misleading answer, because it has access to a solid set of authoritative information and the AI system is told to base its answer on that solid set of information. How does this approach work in practice?  

My first example of a deployed RAG system is IDR Answers, which was created by the open source tech project Apurva.AI in partnership with the Indian Development Review, India’s leading media platform on philanthropy and social impact. IDR Answers is based on more than 2000 articles published by Indian Development Review. In addition to answering a question, it also surfaces the articles on which the answer was based. The experience is much like chatting with an expert who knows all of the IDR content and can summarize it into a neat and tidy answer based on this knowledge base. Now, IDR Answers is not likely to be a great source of information on what’s happening in Zambia in the development field, although a Zambian expert may still get value from consulting IDR Answers for ideas on something that worked in India, and might translate into the Zambian context.  

My second example is Digital Green and their Farmer.CHAT advice chatbot. Digital Green has thousands upon thousands of advice videos for farmers in their own languages. These videos were the source material for a RAG project conducted by Digital Green, a nonprofit active in Asia and Africa. Before Digital Green unleashed their chatbot on farmers, they provided it to government agriculture extension agents, which provided an additional round of feedback and improvements. Now, when a farmer asks for advice from the Farmer.CHAT service, they are highly likely to get a short video snippet of someone speaking their language and showing them how to solve their specific issue, whether it’s how to terrace a field or treat a crop pest.  

Keep in mind that these early adopters had access to serious tech talent and funding. Digital Green spent two years with a team of more than twenty technologists developing Farmer.CHAT. And Apurva.AI’s heritage includes some of India’s top technology leaders. But, something that was leading edge a year or two ago in tech is likely to become within reach of more and more organizations. For example, I think new RAG implementations in the coming year will be increasingly doable with a single knowledgeable technologist and a high quality dataset that covers the subject area well. That’s because there will be products that support RAG and make it easy to apply in specific contexts.  

The same is already true of predictive AI tools. Google and Facebook have both open-sourced tools like TensorFlow and PyTorch, that can put creating AI solutions in the hands of a single software developer with access to an interesting dataset. Resources like Hugging Face (yes, that’s the name!) are also quite useful to AI practitioners today.  

Finally, a discussion of the use of AI for social impact applications must consider ethical data collection and use. For example, the principles outlined in the Better Deal for Data are designed to reassure communities that their data won’t be used against them.  

Conclusion

These are exciting times for expanding AI use inside nonprofit organizations. However, if an organization wants to use AI in their program work, where the AI will be interacting with or supporting the communities being served, this will depend on at least some internal technical capacity. It may not require tech developers, but there will continue to be a need inside social good organizations to be good with data to make effective use of AI. First, all of the program-related uses of AI I’ve mentioned require an interesting amount of data to kick-start them into being. That data also has to be high quality. And, it is important to point out that most of the world’s languages are not currently well-supported by today’s AI tools, which makes projects in languages other than English and similar high-resource languages more challenging.  

Lastly, your team needs to know enough about data and the social impact of your work to oversee the AI tools, to tweak them to avoid them going off-track. AI solutions always make mistakes: your team is responsible for quality control to ensure that the AI solution actually is better than the alternative. RAG solutions are still not immune to LLM hallucinations and still need to have careful testing and risk mitigation, though they reduce the risk of hallucination substantially. By addressing the number one problem of generative AI technology, which is making up the wrong answer, RAG solutions are a promising way to use this technology while supplying the needed vetted information and guardrails.