Building a Chatbot That Answers From Your Own Documents

In this article, I will demonstrate how to use a Large Language Model (LLM) to create a chatbot that can answer questions about pool availability in Montréal.

Using LLMs in a business can be highly beneficial, but allowing an LLM to operate unchecked and make decisions can be risky, potentially leading to significant issues and financial losses. For example, you can read about how Air Canada was compelled to pay compensation due to a misleading response from an LLM: Washington Post article.

This article is intended only for educational purposes, showcasing the power of AI, generative models, and the tools that are developed very quickly.

I understand that this solution may seem overly engineered, and I agree. A simpler, less expensive solution could address this problem. However, with more advanced LLMs available, implementing a solution using an LLM can be achieved with just a few lines of code, allowing the LLM to analyze all the data.

I truly felt this while tackling this challenge:

Software ate the world. Now AI is eating software.

I — The problem

I started swimming a couple of weeks ago, and fortunately, Montréal has many indoor swimming pools. The city’s website lists all the indoor pools along with their availability: Indoor swimming pools.

The downside is that you cannot search for an available pool at a specific time, as each pool has its own schedule, which varies significantly. Some pools have their schedules embedded as a table on the website, like this one: Bain Émard.

However, for other pools, you have to leave the city website and visit a Facebook page to see the pool’s schedule: Complexe Sportif de Saint-Laurent.

Since I didn’t have a fixed time to go to the pool, I found myself constantly navigating through pool websites to find the next available slot, which was time-consuming and very manual.

II — Large language models

I decided to use a Large Language Model (LLM) like ChatGPT to address this problem. LLMs excel at understanding complex text — such as pool schedules — and grasping concepts related to time and space, like pool opening and closing times, or pools nearest to a particular address.

One challenge is that pool schedules change frequently, and recent schedules were not included in the LLMs’ training data. So, how can we solve this problem?

Here are three possible solutions:

• Fine-tune the model using recent data. Fine-tuning involves updating the model’s weights by training it with a new dataset. This is effective if you have a large, “static” dataset that was not available during the initial training of the LLM — by static, I mean data that does not change over time. However, this approach can be costly and resource-intensive.
• Provide the entire database alongside the user’s query. This technique involves including the entire database with the user’s query as a prompt, rather than retraining the model. It works well if the dataset is relatively small. Current LLMs support a large context window, but if the dataset is too large, this approach becomes impractical because the LLM can only process a limited portion of the prompt and will only consider the most recent part.
• Use retrieval-augmented generation. RAG enhances the capabilities of LLMs by integrating them with specific knowledge databases, without the need for retraining the model. This approach is cost-effective and helps maintain the relevance, accuracy, and usefulness of the LLM’s output in various contexts. RAG works by first searching through a large database to find relevant information related to a query. Then, it uses this information to generate a more accurate and informative response.

III — The design and implementation

The pipeline has two halves that run at different times. Everything on the left happens once, offline, when the schedules are collected. Everything on the right happens per question:

A. Prepare the dataset

We need to collect data on all the indoor pools. I tried to automate this process as much as possible, but there is room for improvement to handle more complex websites.

Using some Python packages, I was able to successfully create a text that contains the pool schedule for each pool:

To achieve this, we need to extract all tables and relevant information from each pool’s HTML page, such as the pool’s address and any exceptional closures.

There are tools that can help automate this process, such as LlamaIndex’s web readers — see data connectors and the web reader package.

Each pool has its own file with a schedule that looks something like this:

B. Create the embeddings

When a user asks a question about a specific pool, we need to search the database for relevant information, combine this information — also known as context — with the user’s query, and then use the LLM to generate a response.

To find relevant information in the database, we can use various search methods, such as semantic search or keyword search.

Semantic search finds data based on the intent and contextual meaning of a query, rather than an exact match on query words.

So, once the dataset of text files is ready, we need to represent each word or phrase with a special vector in a high-dimensional vector space, where words with similar meanings will be close to each other.

LlamaIndex was used to handle data preprocessing.

To create an embedding, we need an embedding model, which is an LLM used to convert text into vectors. This same model will later be used to embed the user’s query and search the database. It’s important to use the same model for both the user query and the database embeddings.

C. Prepare the prompt

With the embeddings ready, we need to obtain the user query, create an embedding for it, search the database for relevant documents, and prepare the prompt for the LLM. Tools like ChainLit can be used to create the chatbot, handling server creation, message history, themes, parsing, and more.

IV — Demo

I will not say anything here, and let you see this beauty of an LLM alive:

V — The lesson

• As a software developer, I always aim to find exact solutions that are deterministic and proven through mathematics, physics, and science. Working with LLMs made me uncomfortable because I couldn’t fully trust them, even though the responses were accurate in my case. There is no guarantee — that I am aware of — that the model will consistently perform well and continue to provide reliable results. This is why in my demo I provide the URL to the pool website as a source.
• Due to the AI boom, there are countless tools available, with many attempting to achieve similar objectives. The internet is flooded with numerous tools that often overlap in functionality, making it challenging to navigate all the technologies. Standards and interfaces are not yet well defined, and code that works today may become obsolete in a few months. However, this rapid evolution might be a natural part of progress.
• The whole thing took an afternoon, with an LLM as a coding partner, to build an app that answers questions with an LLM.
• I was occasionally banned from accessing pool websites due to intensive data collection for the database. My apologies to Ville de Montréal for the load. :)

Sources

• Washington Post — Air Canada must honor refund policy invented by its chatbot
• Ville de Montréal — indoor swimming pools
• LlamaIndex — data connectors and the web reader (the original WebPageDemo URL has since 404’d)
• ChainLit
• DeepLearning.AI — Building and Evaluating Advanced RAG