How we test an AI data analyst: 115 questions, independently computed
By SchemaChat (Data BI LLC) · Published 30 October 2026 · Last reviewed 30 October 2026 · 8 min read
Before every release we ask SchemaChat 115 questions whose correct answers were computed independently from the data, in plain code, never typed in. A release that scores below a fixed floor does not ship. The suite has caught real defects — and it cannot prove the product is right on your data, only that it is not getting worse.
A number from an AI tool is not evidence. If you cannot check how it was produced and how often tools like it are right, you are trusting a tone of voice. This page describes how we test our own product so you can judge the method — and borrow it.
The test set
Four small datasets are generated by code with a fixed random seed, so they are identical on every run:
- Orders (800 rows) — an e-commerce table.
- Subscriptions (600 rows) — a SaaS table.
- Support tickets (700 rows) — a support table.
- A stacked file (600 sales rows, 60 pasted list rows, then a second header in a different column order followed by 500 repeated rows) — shaped like a messy real-world export.
115 questions are asked of them, in 16 categories:
| Category | Questions | Category | Questions |
|---|---|---|---|
| Aggregate (sums, totals) | 9 | Ranking (top / bottom) | 15 |
| Average | 8 | Ratio / percentage | 6 |
| Count | 14 | Missing values (null handling) | 5 |
| Filter | 15 | Distinct values | 6 |
| Date range | 6 | Date arithmetic | 1 |
| Extremes (max / min) | 3 | Follow-up questions | 3 |
| Having (group filters) | 1 | Should refuse (no such data) | 7 |
| Stacked / messy file | 13 | Should refuse to change data | 3 |
Where the right answers come from
Every expected answer is computed in ordinary code from the generated rows — a sum, a group, a sort — and never typed in, so the answers cannot drift from the data. Questions are phrased unambiguously on purpose: this is a regression test (“did a change make the product worse?”), not a test of how well the product guesses what a vague question means.
How an answer is scored
- Numbers must match the expected value within 0.2% (abbreviations such as “118.2K” are expanded first).
- Text answers (a top customer, a team) must name the expected value.
- “Should refuse” questions pass if the product declines, or plainly states that the data has no such field, and does not invent a number.
- “Should refuse to change data” questions pass only if it refuses and nothing changes.
The gate
The suite runs automatically before every release, together with the type checker, unit tests, a production build and a security regression suite. Three conditions stop a release: accuracy below the floor (currently 94%), any query answer that arrives without its “How this was computed” receipt, and more than three “no such field” notes under answers that were perfectly answerable.
Results
Recent runs (29–30 October 2026): 114 of 115 in most runs, 113 of 115 in one. The question that most often fails is a running total by month on the stacked file; it passes on its own and fails in some full runs, which is a sign that the language model's choice of query varies. We report that rather than hiding it.
These figures are from our own test on our own synthetic data. They are not a claim that SchemaChat is 99% accurate on your database.
Real defects it caught
- A race when two first questions hit a new data source at once (a server error).
- A question like “how many … (it is not empty)” was hijacked by a missing-values table instead of being counted.
- A model-added “is not empty” filter that changed a count — now removed from simple queries and disclosed on others.
- “No such field” notes appearing under correct answers to ordinary phrasing (8 of 21 answers in one replay) — fixed by judging only words in entity positions.
- A running total over text month labels that returned the grand total on every row, because SQLite cannot parse “Jun-21” as a date.
- Integer division that made an average 786.98 instead of 787.23.
- A pasted list inside a sales table inflating a missing-value rate from 7.4% to 12.2% on a synthetic file.
What this test cannot prove
- It is our own test, written by the people who build the product.
- The data is synthetic and small (hundreds of rows); your schema is different and larger.
- It runs on uploaded-file data (SQLite). It does not exercise every database connector.
- It is in English and tests unambiguous questions.
- Independent benchmarks show how much harder real enterprise schemas are: the Spider 2.0 paper reports 21.3% of 632 real-world workflow tasks solved by its strongest agent framework (built on o1-preview) against 91.2% on Spider 1.0 and 73.0% on BIRD. Those are different benchmarks with different metrics, and newer models score higher, but the gap is why you should test on your own data.
Run a small version on your own data
You do not need our suite. Pick ten questions whose answers you can compute yourself — a total, an average, a count with a filter, a top-5, a ratio, a question about missing values, one that should be refused, and one on a messy part of your data. Compute the answers independently, then ask the tool. Our accuracy-test kit gives the template.