The 10-question accuracy test for any AI analytics tool
By SchemaChat (Data BI LLC) · Published 30 October 2026 · Last reviewed 30 October 2026 · 6 min read
You can test an AI analytics tool in an afternoon: write ten questions whose answers you compute yourself, ask the tool, and score the results. This kit lists the ten question types, how to work out the right answer independently, and how to score honestly. It works on any tool, including ours.
A vendor's demo runs on data the vendor chose. The only test that matters runs on yours. This kit is the same method we use on ourselves (described in how we test an AI data analyst), reduced to something one person can do with a spreadsheet.
Before you start
- Pick one table you know well, with a few hundred to a few thousand rows.
- Make sure you can compute answers independently: a spreadsheet pivot table, a SQL client, or a few lines of Python.
- Write the questions and compute the answers before you ask the tool, and keep both.
- Ask each question in a fresh conversation so earlier answers cannot influence later ones.
The ten questions
| # | Type | Example question | How to compute the right answer | What it tests |
|---|---|---|---|---|
| 1 | Total | What is the total revenue? | Sum the revenue column in a spreadsheet | Basic aggregation; does it include refunds or cancelled rows? |
| 2 | Average | What is the average order value? | Mean of order totals (decide how you treat blanks first) | Missing values; averaging the right thing |
| 3 | Count with a filter | How many orders were placed in March? | Filter by date, count rows | Date handling and boundaries |
| 4 | Ranking | Who are the top 5 customers by revenue? | Group by customer, sort descending | Grouping; ties; name variants |
| 5 | Ratio / share | What share of revenue comes from the top customer? | Top customer total ÷ overall total | Denominators; double counting |
| 6 | Period comparison | How did revenue change from Q1 to Q2, in percent? | (Q2 − Q1) ÷ Q1 | Date logic; arithmetic |
| 7 | Missing values | How many rows have no email address? | Count blank cells (and placeholders like N/A) | Does it count blanks and placeholders correctly? |
| 8 | A question that should be refused | What was revenue by sales rep? (when no rep column exists) | There is no answer | Does it say so, or invent one from the nearest column? |
| 9 | A follow-up | Now only for the UK | Apply the filter to question 4's result | Does context carry over correctly (and only when it should)? |
| 10 | A messy-data check | How many rows are in the file, and is it one table? | Look for repeated headers, pasted lists, subtotal rows | Does it notice structure problems? |
How to score
| Result | Score | Meaning |
|---|---|---|
| Correct, and it showed how it got there | 2 | What you want |
| Correct, but no way to see how | 1 | Right this time, but you could not have known |
| Wrong, but it flagged an assumption or limit that explains it | 1 | A checkable failure |
| Wrong with confidence, or invented an answer | 0 | The dangerous kind |
Out of 20: below 14, do not rely on it without review; 14–17, usable with checking; 18–20, promising — then repeat with 10 new questions to be sure it was not luck. These thresholds are our judgement, not an industry standard.
What to watch beyond the score
- Ask the same question twice. Do you get the same answer?
- Does it show the query, filters and row counts, or only a sentence?
- What does it do when it cannot answer?
- Does it tell you what it did not cover?
How SchemaChat scores on its own version
We run a larger version of this against every release (115 questions, 16 categories). Our recent results and the defects the test caught are in how we test an AI data analyst, with the caveat that it is our own test. The honest way to find out how it does on your data is to run this kit during the 14-day trial.