*By Emily Shenfield and Alvin Lee
As generative AI (GenAI) tools grow in availability and usability, companies are making new strides in automation, decision-making, and customer interactivity. But these tools—and the large language models (LLMs) that power them—are not infallible. These AI hallucinations can pose real risks, especially when GenAI systems are responsible for performing critical tasks or interacting with customers.
AI hallucinations are incorrect or misleading results generated by AI models. They occur when an AI model generates information that seems plausible but is actually inaccurate or completely fabricated. An example might be when you are looking up the legal details of a case and are looking for references and precedents, such as similar cases that have been tried in the past, and the AI hallucinates, inventing a non-existent case with false decisions and references.
Because GenAI applications often deliver their responses with a tone of certainty and confidence, distinguishing a delusional response from a correct one can be difficult for regular users. This can lead to the spread of misinformation.
But why do AI hallucinations occur? LLMs are trained on vast data sets, but they don’t ultimately “know” facts. Instead, they predict the next word or phrase based on patterns learned during training. When an LLM doesn’t have enough context or the right data to answer a prompt, these predictive abilities can take a wild turn.
This turnaround can happen because of an overeager LLM, ready to agree with any fact presented in a prompt. It can also happen when the training dataset contains factually incorrect information.
If your company uses GenAI for operational efficiency or customer-facing services, it’s important to be aware of how AI hallucinations can arise and how to mitigate those risks. When incorrect information from an AI hallucination infiltrates your internal data, it can lead you to make business decisions based on incorrect information.
In one example, an AI demo tasked with summarizing a financial report responded with several factual inaccuracies. What if a company used this inaccurate response to inform its own strategies? In another example, a law firm submitted AI-influenced legal research in a lawsuit and was fined for the error.
Erosion of customer trust
Your company’s brand and reputation depend heavily on customer trust. What would happen to that trust if your GenAI-powered customer service chatbot provided occasionally mind-blowing answers to customer questions? Or consider the repercussions if you published AI-generated marketing content that wasn’t fact-checked.
GenAI is still an emerging technology. And while LLM is making progress toward reducing the likelihood of AI hallucinations, the phenomenon remains a concern. Currently, completely eliminating AI hallucinations is not a reasonable goal. Instead, companies using GenAI should adopt strategies to mitigate them.
Retrieval-augmented generation (RAG) is an approach that enhances a GenAI system by searching for information—in real time—that is relevant to a user’s prompt. By retrieving additional information to supplement the LLM’s original training dataset, a RAG system provides additional knowledge and context to the AI. This decreases the opportunity for hallucinations and increases the relevance and accuracy of the response.
RAG can go a long way toward reducing AI hallucinations, although it won’t eliminate them entirely, since most models will rely on their training data. In many cases, carefully crafting your prompts can help minimize instances of AI hallucinations. Basic prompt engineering techniques include:
- Give clear and specific instructions;
- Ask direct questions;
- Provide sufficient background information or context
Applying rapid engineering techniques provides expectations and guardrails for a GenAI system, guiding it toward a correct response.
Intentional handling of highly critical data
When working with highly critical data, treat LLMs as untrusted, similar to browsers or mobile apps. This approach involves using a secure middle layer to manage API interactions, ensuring that LLMs do not have direct access to critical systems.
Just as you wouldn’t expose backend APIs to an unprotected frontend, don’t allow LLMs free rein over critical operations and don’t leave room for interpretation of critical data. For example, simply embedding critical numbers and dates as part of an unstructured text prompt can expose the system to potential data mishandling and hallucinations.
To mitigate this issue, pass critical data to your system programmatically and grant the model the minimum access necessary to complete the task. Purpose-built tools can help ensure that data is handled correctly.
In addition to being able to perform complex calculations or dynamically retrieve real-time information from external APIs, tools can act as API proxies to handle authentication, rate limiting, and data validation before reaching an external API. This layered approach to security helps prevent unauthorized activity and allows you to pass critical data programmatically within your system rather than as part of the prompt body to the model.
Human in the loop
GenAI technology is evolving rapidly, but it is not yet at a place where companies can use it without supervision. Human supervision is still a necessity. The level of human supervision over GenAI activity may depend on the criticality of the tasks being performed. For example, a healthcare provider using GenAI to develop a patient care plan should include a high level of supervision from qualified physicians and healthcare professionals.
With all that said, it’s definitely worth investing in GenAI, starting to use it, and exploring its possibilities. However, it’s still a very new technology, and companies need to invest in understanding the possibilities for errors, such as AI hallucinations. There are several methods to address these issues in order to ensure verification, analysis, and validation processes that give you, your company, and your customers confidence that GenAI is contributing to a better service.
*Emily Shenfield is a Technical Marketing Engineer at Twilio on the Emerging Technology and Innovation team.
*Alvin Lee is a full-stack developer at Twilio.
Notice: The opinion expressed in this article is the responsibility of its authors and not of ABES – Brazilian Association of Software Companies
EN 
PT 
