Image of the Basejump AI metadata editing feature
When working with AI applications, the AI can always benefit from better context. That’s really what RAG emphasizes, providing the AI with relevant context to provide a more informed answer.
When using an AI in a text-to-SQL application, it’s crucial to the AI to have more than just the table names and columns. Here are a few metadata features that are important to also include:
- Table and column descriptions
- Primary keys
- Foreign keys
- Distinct values
- Ability to ignore information
I’ll go through why we added these features and how they can help
Table and column descriptions
Descriptions first and foremost help with retrieving relevant context. When the AI is going through the retrieval step to get relevant tables, the embeddings will help with the vector search to retrieve the correct information.
Primary and Foreign Keys
The foreign keys help the AI to understand how to join tables together. If they aren’t provided, the AI is typically going to use matching names to join tables. Or in the case of an AI agent such as ours, it might do some experimentation as well.
The primary keys does not only refer to the primary keys set in the database. It refers to the distinct set of columns that can be used to retrieve a single record from the table. This information helps the AI to ensure there are no duplicates as a result of the query. It also helps us to check the generated AI query for accuracy.
Distinct values
There are some fields in a table that only consist of a distinct set of values. These are referred to as ‘low cardinality’ fields or enums (some databases such as Postgres support an Enum type). An example of this type of field would be a demographic table with a person’s sex: Male or Female.
Ability to ignore information
There could be information provided that is not relevant or deprecated. We added this feature so that any columns that the user doesn’t want the AI to know about or use is hidden from the output. An important item to note however, is that this is not a replacement for good security practices. We recommend setting database role permissions to control access. All of these settings are simply to control what the AI can see for relevant context.
Our latest release
We’re excited to announce our latest release v0.26.0. In this release, we have many great new features, but the primary one to highlight are our new metadata features. See this video for a quick overview.
Other items included in this release
Besides the new metadata features, we also updated our UI. The UI menu is now all the way on the left when in full screen and the menu items don’t show unless they are hovered over.
Screenshot of the new menu look
Another feature that is not seen, but that is included, are improved query results by using query expansion. This breaks complex prompts into smaller sub-prompts and retrieves relevant context for each. This increases the AI’s ability to handle difficult or multi-part questions.
That is it for this release! If you haven’t tried Basejump yet or are curious to learn more, schedule a demo with us. Thanks!