By Princess Dickens, Data Scientist, Data Surge
Virtual Agents, or what most of us refer to as chatbots, have become ubiquitous in customer support. Many prefer them over talking to a live agent, (hello, generations of young people who hate talking on the phone.) In fact, chats with bots have an 87.58% satisfaction rate, 2% higher than when humans take over. Today, these automated assistants now handle 65% of business-to-consumer communications, and their use is rapidly growing.1
If you’ve ever interacted with a virtual agent, or maybe you even want to build one yourself, you might like to know about the different varieties and how they work. Here’s an overview on three different types, with a bonus tidbit about the current state-of-the-art for improving Generative AI chatbots.
Option #1 – The Virtual Agent
Pros and Cons:
- Lengthy setup – requires answer curation for addressing anticipated questions.
- Best for a closed domain or for fulfilling common requests.
- Invalid input is handled easily with a series of fallbacks.
You might have interacted with this type of chatbot when checking to see if your prescription has been filled at the pharmacy. In this scenario, a virtual agent interacts with the user through a series of preplanned events called “intents”, such as greetings and requests for information.
For example:
# User clicks to open the chat window #
Virtual Agent: “Hello, how may I help you?”
User: “I want to know if my prescription is ready.”
At this point, some classification is done under the hood to label the user’s statement or question as an intent to check whether a prescription has been filled or not. Once the intent is recognized, a dialog manager provides the next turn in the conversation to prompt the user to input more information.
Virtual Agent: “Sure, I can get that information for you. What is your date of birth?”
This type of chatbot typically serves to fulfill requests in a very closed domain, so that it’s easy to anticipate what the user will want and how to satisfy any requests. For this reason, answers are prepared ahead of time, as in an FAQ.
If anticipating every possible question that a user might ask and curating those answers would be too much work or not possible, then you might be more interested in one of the next two methods.
Option #2 – Retrieval-Based Chat
Pros and Cons:
- Fairly quick setup.
- Source of the answer is clear and transparent.
- Harder to establish a human handoff.
This type of question-answer application is a little more complex, but it’s often more efficient to implement than curating a set of answers. One use case for this type of chatbot is when you need to answer specific policy questions based on a set of policy documents.
Retrieval-based chats consist of an offline “indexing” stage, and an online “retrieval” stage. A retrieval based chatbot is fairly simple to implement, and it’s often preferred for sensitive information because it’s easy to decipher how the answer was selected. Unfortunately, it doesn’t always perform the best when compared to deep-learning based models. Read on for more information about that.
Method #3 – Chatting with an LLM (like OpenAI’s ChatGPT)
Pros and Cons:
- Can perform very well and offers good results.
- Extensive training is typically required.
- Results are often difficult to evaluate.
- Runtime is more expensive than other types of chatbots.
- Answers lack transparency.
Large Language Models (LLMs) employ an entirely different strategy for answering questions. There is no corpus of documents to use as the basis for answers. Instead, LLMs are trained in an unsupervised way to predict the next word in a sequence, while paying attention to other words in a given context, much like word embeddings. This training process, when done on vast amounts of data (books, newspapers, the internet), allows the model to learn the style, syntax, and semantic reasoning needed to generate human-like responses.
When a user asks an LLM a question, it generates a response based on the inherent knowledge it acquired during training. Some models also provide answers to follow-on questions based upon previous turns in the conversation, giving the user a sense of coherence.
While charming and impressive in its human-like interface, this type of model also has its share of flaws:
- Information can quickly get out of date – limited to whatever data it was trained on.
- Sources are not cited – users don’t know if the answer is coming from a reputable source.
- Hallucinations occasionally occur – sometimes, an LLM will provide an answer that sounds right, but isn’t, making it hard for humans to identify an incorrect response.
- Adjustments are necessary – Chatbots that are meant to cover a specific domain will require fine-tuning to make sure answers are always on topic.
- Security concerns – this type of model can leak inappropriate information from the training data.
- Built-in bias – any bias in the training data will become part of the LLM’s inherent knowledge base.
Fortunately, there’s an application that can mitigate these weaknesses without adding too much to the runtime complexity or cost of training.
Retrieval Augmented Generation (RAG)
Retrieval Augmented Generation (RAG) is a process that improves the performance of LLMs by limiting the context of its responses to a closed corpus. For additional security, documents can be restricted to individual user groups in the retrieval stage.
Though far from perfect, RAG can vastly improve an LLM, as answers are traceable to their sources, and updating the source documents is easier and more cost-effective than retraining. However, even with RAG, LLM’s remaining shortcomings include imprecise reasoning on semantically ambiguous questions, and difficulties with benchmarking.
Is RAG right for you?
RAG is a simple and effective solution for question-answering in sensitive domains. One popular use case is with policy documents. At Data Surge, we used our RAG Solutions Accelerator to create a ResQAI – a chatbot that helps users understand how they can qualify and apply for humanitarian aid from the U.S. federal government.
Want to learn more about virtual agents and how we can help build and implement the right for you? Contact us.
