Hello Hackers!!! Ever wondered how AI chatbots with API and database access get compromised? Curious about what really happens under the hood? Let’s uncover the vulnerabilities that put these systems at risk — and learn how to defend against them.

But that’s not all — let’s explore this from both the developer’s perspective (how to build it) and the attacker’s perspective (how to break it).

The Chatbot Design: Features & Functionality

In this blog post, we’ll build a chatbot that translates human language queries into SQL commands to fetch product details from a database. The chatbot supports conversational queries like:

• “Find smartphones under $800”
• “List all laptops in stock.”

This bot allows users to query databases conversationally, making complex data retrieval as easy as chatting with a friend. But as we’ll see, this design introduces vulnerabilities that attackers can exploit, and we’ll walk through how to identify and fix them.

Tech Stack

Here’s the tech stack we’ll be working with to bring this bot to life:

• Anthropic’s Claude Haiku model — for its speed and conversational accuracy. (but you can use whichever model you want)
• SQLite — for lightweight database management.
• Chainlit — for quick prototyping of chatbot interfaces.(UI)

Architecture

Let’s begin by making the necessary imports, then create a class, and initialize the required variables.

Filename: bot.py

Line 1–18: Imports and class creation

Line 19–20: Getting the API key from the .env file and passing it to the Anthropic client

Line 21–22: Setting the database name to “product_store.db” and Establishing a connection to the SQLite database.

Line 23–24: Creating a cursor object to execute SQL commands and Calling the get_db_schema() method to fetch and store the database schema.

Next, we will create a function that will get the user input and convert it into a DB query.

In this method, generate_sql_query(), we take a user's natural language query and convert it into an SQL query. Here's how it works:

1. Define the System Prompt: First, we create a prompt that describes the task. We explain that the AI will convert natural language into SQL queries for a product database. We also include the database schema and examples to guide the AI.
2. Prepare the User Query: The user’s query (like “Show me all laptops under $1000”) is passed as a message to the AI, asking it to generate the corresponding SQL query.
3. Send the Query to the AI: We send the user’s query along with the system prompt to the AI model (in this case, claude-3-5-haiku-20241022).
4. Extract the SQL Query: Once the AI processes the request, it returns a response. We extract the SQL query from the response.

Next we will create a function to execute the SQL query generated by AI.

We will pass the output generated by the previous function to this one. The execute_db_query() method runs the SQL query on the database using the cursor, fetches the results, and returns them. If an error occurs, it catches the exception and returns the error message.

Now that we have the results from the database, we can’t show the raw data directly to the user. Since this is a chatbot, we need to present the output in a conversational manner. To achieve this, we’ll create a function called generate_response().

The generate_response() method takes the user query and database results as input, then uses an AI model to generate a friendly, conversational summary of the results. It summarizes product names, categories, and prices. If no results are found, it suggests related options. The final response is returned in natural language.

Thats it!!! we have completed the bot functionality. Now we need one more function to execute all these functions in a orderly manner. For that we will create function named process_query().

The process_query() function will combine all the operations.

The function is marked as async because database queries and external API calls (like generating responses) can take time. Using async allows these tasks to run concurrently without blocking the rest of the application, improving performance and responsiveness.

Next, we need to create our SQLite database. For learning purposes, we will create three tables and insert some dummy data. Then, we’ll simulate an attack by attempting to extract that data from the bot.

create a file named db.py

Like the previous file(bot.py) , here also Import the required libraries, create a class and initialize the required values.

Next, we will create the product_info, user_info, and purchase_history tables. Although we won't use the purchase_history table in this example I am not going to show the implementation, but if you're interested in implementing it, I'll share the GitHub repository for this code.

The product_info_table() method creates a table named products if it doesn't already exist. The table includes columns for product details such as product_name, url, price, category, and description.

The user_info_table() method creates a users table if it doesn't already exist. This table stores user information, including username, password, email, and registration_date.

Next we will insert some dummy data to the db.

The above function inserts data into the users and product_info tables. Now, execute this Python file, and it will create a database file named product_store.db. please check if the data has been properly inserted.

We have now completed the backend and database parts. The remaining task is to build the frontend. For the frontend, I used Chainlit, an open-source async Python framework that enables developers to create scalable Conversational AI or agentic applications. If you’re interested, please check the references section for the link.

Now go back to our bot.py file and add these import statements and main method.

Here, we create an instance of the UnsafeAI class using ai = UnsafeAI(). The @cl.on_message decorator listens for incoming messages. When a message is received, the main function processes the user's query asynchronously using ai.process_query(). The response is then sent back to the user through Chainlit.

Now, if you execute the following command in your terminal, the application will begin running.

chainlit run your_filename.py

When the application starts running, this is how the UI looks like.

The Attacker’s Approach

Now lets get into the exploitation part. Now as an attacker,

Looking at the previous image, the response mentions “database results,” which strongly suggests that the chatbot queries the database directly and returns the results.

This behavior raises a crucial question — how secure is the interaction between the chatbot and the database? Could an attacker manipulate the input to exploit vulnerabilities like SQL injection?

Let’s test this assumption by crafting malicious queries and analyzing the chatbot’s behavior.

First lets test it by sending a simple SQL query.

Great! It works! This confirms that the bot is accepting SQL queries.

Now, let’s analyze what’s happening here from an attacker’s perspective:

1. First, the model is processing both SQL queries and natural language inputs. This indicates that no restrictions or input sanitization have been applied.
2. Second, the response is presented in a conversational format rather than displaying direct database results.
3. This suggests that some preprocessing is being performed by the developer before showing the output to the end user.

Lets test it with some malicious input.

Unfortunately this didn’t worked. Its because of the built-in safety mechanism of claude prevents this input from executing.

Let’s try something different this time. We’ll attempt to list all the tables in the database.

Cool! It successfully listed all the tables from the database. Among these, the “users” table seems quite interesting, doesn’t it? Let’s try extracting information from this table.

From the above screenshot, we successfully extracted details about the admin user and other customers.

How to Mitigate

1. Use parameterized queries to avoid SQL injection attacks instead of directly concatenating user inputs.
2. Validate user inputs before processing them. Allow only predefined formats for queries instead of arbitrary strings.
3. Sanitize AI-generated SQL queries before execution.
   Implement logic to reject unsafe queries containing operations like DROP, ALTER, or DELETE.
4. Restrict database permissions by assigning a read-only role to the chatbot, preventing destructive queries.
5. Avoid displaying database error messages to users, as these may leak implementation details.

References

GitHub - Chainlit/chainlit: Build Conversational AI in minutes ⚡ Build Conversational AI in minutes ⚡. (opens in a new tab)
Contribute to Chainlit/chainlit development by creating an account on GitHub.

---

GitHub - UVvirus/unsafe_ai_blog Contribute to UVvirus/unsafe_ai_blog development (opens in a new tab)
Create an account on GitHub to collaborate.