What Is RAG and Why Does It Matter for Business AI?
Shane: I've heard you mention RAG in some of the solutions we've built for folks, and I thought it might be beneficial for everyone to understand what that means, what the purpose of it is, and how it plays into the overall automation. So what can you tell us about RAG?
Hunter: RAG stands for Retrieval Augmented Generation. Think of it as the brain of your AI chatbot. The RAG is the context and information you are giving the chatbot. If you scope it to just the RAG, it is not able to go out to other sources. We go into the brain of the chatbot and give it specific information. It only knows that information, so it can only return answers based on what is in there. If we open up web search, that enables the chatbot to pull from additional resources outside the RAG. But the RAG itself is where we fill the context with information relating to your business processes or internal tools. It is a way to scope the chatbot to only know a specific set of information.
Shane: So using RAG, we can basically put it in a box and say, this is the only data we want you to use to produce that output. We specify what that looks like, rather than having it pull from a website or any other outside source.
Hunter: Exactly. And that is really the key difference. There are two ways to extend beyond the RAG. One is a web search feature, and the other is integrating an external LLM such as OpenAI, Claude, or similar tools. Any information the user queries can be sent to that external LLM, and it returns answers directly to the user. Depending on the automation, that returned information might be used to build something, like a coding solution. You can also scope and train those external models depending on what you need them to do.
Can You Use RAG, Web Search, and an LLM Together?
Shane: Is there a scenario where you would want to use RAG along with a web search and an LLM, or is it exclusive to one or the other?
Hunter: It is not exclusive. It comes down to what you are trying to accomplish. If it is something your team will use internally that is specific to your business processes, you probably do not want to open it up to the web or external LLMs, unless those LLMs are private models your organization has already created. That said, you can fully customize these workflows. One portion of the workflow might be scoped internally, such as pulling from your ticket data and always returning answers in a specific format. Another portion might handle generic questions about a service or your website, pulling from a private LLM that is already integrated with your site. So different question types route to different sources within the same workflow.
Shane: Right. So you could say, for example, pull our internal pricing from our library, but for compliance-related questions, go out to an external source that is constantly updated. You build the model, specify the categories, and define how each type of question gets its answer.
Hunter: Exactly. It is all configured based on how the user is interacting with the chatbot and what they send to it. If they ask something specific, like a question about phone logging, it routes into a custom workflow built for that. If it is a generic question that none of the configured workflows can answer, that is when the LLM or web search steps in to return answers from external sources.
Will Your Proprietary Data Stay Private Inside a RAG System?
Shane: If we put our proprietary data into the RAG and also tie in something like a private ChatGPT instance, is that data going to be shared with any outside source, or does it stay contained?
Hunter: It stays in your box, as long as you are truly using a private instance on both sides. That is important. It is not enough to have a private chatbot connected to an open LLM instance, and it is not enough to have a private LLM connected to an open chatbot. Both need to be private instances, so your data is not used to train other people's chatbots and cannot surface in answers returned to other users. Keeping everything in private instances is always the recommended approach.
How Does RAG Give Organizations More Control Than a Generic AI Tool?
Shane: This is the key distinction from just handing everyone a generic ChatGPT account. With RAG, the organization controls where the answers come from, what type of answers are returned, and how the data is secured in the process.
Hunter: Absolutely. And beyond the answers themselves, some of the chatbots we have built include custom integrations through API access, creating actions that go beyond just returning information. If you need to look up customer data or add a customer record directly from the chatbot, we can build that integration and connect it to your existing tools. So you can have your internal library of processes and pricing, an API connection to your existing applications, a web connector for current external information, and a connected LLM, all working together in one chatbot. That gives you the most complete and controlled interaction for your team or your clients.
Shane: Really cool technology. I appreciate you helping explain what RAG is and why it matters. As this technology develops, it is going to be key for customization, control, security, and getting the best outcome.