Text-to-SQL Accuracy in 2026: Benchmarks and What They Hide

A vendor tells you their Text-to-SQL system is "95% accurate." The benchmark paper behind that number says the same. Then the system ships, a finance manager asks about "net sales excluding intercompany," and the SQL is confidently wrong. Both the vendor and the paper were telling the truth; the truth was about a dataset that looks nothing like your warehouse.

What the public benchmarks actually measure

Two benchmarks dominate every vendor deck. Understanding what they measure — and what they don't — is the fastest way to read Text-to-SQL claims correctly.

Spider (2018, Yale) is the most-cited. Roughly 10,000 questions over 200 database tables across 138 domains, with queries written to be answerable from the schema. Its successor, Spider 2.0 (2024), was explicitly built because the original had been "solved": models exceed 90% on Spider 1.0, while Spider 2.0 — featuring real enterprise dialects, long contexts, and multi-step analytics — initially scored below 25% for the best systems. That collapse between versions is the single most instructive data point in the field.

BIRD (2023) was constructed precisely to close the gap between academic queries and messy reality: larger databases, real data values, and questions requiring external knowledge or domain reasoning, built by an academic team in Hong Kong. It reset the leaderboard to the 50–60% range in 2023, and top systems have since pushed execution accuracy into the 70–80%+ range through 2025–2026, with prompt-engineering pipelines and fine-tuned open-weight models competing alongside frontier APIs.

Both are legitimate, carefully built datasets. The honest summary of what they establish:

The table's last row is the point. There is no public benchmark for your warehouse, and the transferability of public scores decays with every step toward enterprise reality.

Two additional caveats when reading leaderboard movement. First, contamination: frontier models are trained on web-scale data, and public benchmarks and their solutions circulate widely, so some share of any reported gain may reflect memorization rather than generalizable skill — a concern the benchmark maintainers themselves acknowledge and mitigate with held-out variants. Second, recipe specialization: much of the recent climb comes from pipelines engineered around each benchmark's known structure — schema linking tricks, candidate-selection ensembles, execution-feedback loops — which is legitimate engineering but transfers imperfectly to a warehouse whose structure the pipeline has never seen. Read the trajectory, not the point estimate: the honest takeaway from 2023–2026 is that the direction of travel is excellent, while any single number is fragile.

Five things the benchmarks hide

1. Schema messiness

Benchmarks ship tidy schemas with helpful names. Your warehouse has tbl_sal_sum_v3_final and a column named amt whose meaning changed in 2021. It has the same customer existing under four keys because of a CRM migration, revenue booked in three currencies without a consistent conversion date, and slowly changing dimensions that make "as of last March" a genuinely hard question. Real-world Text-to-SQL accuracy is not a function of the model's SQL skill; it is a function of how much of this mess the system can be *insulated from*. Every serious deployment invests more in the layers around the model — documented and curated table sets, tested metric definitions — than in the model itself.

2. Business jargon

Benchmark questions are written by people who can see the schema. Your business users ask about "GMV excluding cancelled-but-unshipped orders," "the South China region" (which, in your org tree, is a different set of stores every year since the 2023 reorganization), or "active dealers" — a term with three competing definitions that were never reconciled. The model cannot map this vocabulary to columns from schema documentation, because the mapping does not exist in the schema. It exists in institutional knowledge — usually in the heads of three analysts. An NL-to-SQL system that has not been taught your jargon is not slightly degraded; it is answering a different question than the one was asked.

3. Permissions and visibility

Benchmarks have one implicit user who can see everything. Enterprises have a shop manager who must not see other stores, a finance analyst cleared for the GL but not for HR tables, and a regional director whose scope changed last quarter. Query generation in production is not "write SQL against the warehouse"; it is "write SQL against the slice of the warehouse this identity may see, at this moment." That is a fundamentally different problem — part SQL generation, part access-control enforcement at execution time — and public benchmarks do not test it at all. A system that scores 85% while ignoring permissions is not 85% accurate in an enterprise; it is a data-leak vector with good marketing.

4. The cost asymmetry of errors

On a benchmark, a wrong query costs one leaderboard point. In a board meeting, a wrong number costs credibility — and, per the way executives actually behave, the whole analytics initiative. Errors also aren't uniform: a hallucinated JOIN across fact tables that silently double-counts revenue is worse than a graceful "I can't answer that," but execution-accuracy scoring treats both as the same failure. Deployed systems therefore need calibrated confidence, abstention behavior, and citation trails back to source rows — none of which appear in benchmark scoring.

5. Semantic drift

The warehouse changes under the system: tables are renamed, a metric definition is revised, a source system migrates. A benchmark is frozen; your schema is not. Systems need regression evaluation that runs continuously — not a one-time certification. This is the least discussed and most operationally important gap: accuracy is not a property of a system, it is a property of a system *at a point in time under a specific schema*.

Why raw LLM SQL is the wrong architecture

The failure pattern above explains why the "just point a frontier model at the schema" approach, which demoed brilliantly in 2023, stopped being the recommended architecture by 2025. Raw LLM SQL asks the model to do four jobs at once — interpret business language, explore schema, remember metric definitions, and write dialect-correct SQL — and it does all four with out-of-proportion confidence.

The architecture that production systems converged on instead splits the problem:

• Semantic layer. A governed mapping between business terms and physical schema — metric definitions, dimension conventions, jargon aliases — narrows the model's task from "discover the schema" to "compose governed queries against known definitions." In our deployments, this is the single largest accuracy lever, typically worth 15–25 percentage points of usable-answer rate on enterprise schemas (our own measurements, consistent with directional practitioner reports).
• Retrieval of relevant schema context. Rather than pasting 400 tables into context, systems retrieve the 10–20 relevant tables and their documentation per question. Smaller context, fewer hallucinated joins.
• Query governance. Identity-bound row and column security enforced at execution, query logging, cost controls, and read-only enforcement. This layer doesn't improve accuracy; it makes accuracy *safe to use*.
• Evaluation harness. A private eval set — 100–300 real business questions with verified answers, spanning your jargon and your schema quirks — run on every model change, prompt change, and schema change. This is the benchmark that actually matters, and nobody can build it for you.

The economic argument is as strong as the accuracy argument. Frontier-model token costs have fallen steeply (Stanford AI Index, 2025), and open-weight models have reached parity for most SQL-generation workloads — so the marginal cost of a query keeps dropping. What does *not* drop is the cost of a wrong number trusted by a decision-maker. Architecture should therefore spend its complexity budget on the layers that catch and prevent errors, not on squeezing the last few points out of the model.

The accuracy numbers to demand from vendors

When a vendor quotes a number, the correct response is not to argue about the number but to ask five questions that convert it into information:

1. On what dataset? Spider 1.0 numbers are effectively meaningless in 2026 — a 90%+ score there is table stakes, not differentiation. BIRD or Spider 2.0 scores say more; ask for the version and date.
2. Execution accuracy or exact match, and against what ground truth? Execution accuracy can pass with wrong-but-coincidentally-matching results; exact match fails correct queries with different formatting. Ask how ground truth was validated.
3. What does "accurate" mean for an ambiguous question? If the model resolves "销售额" to GMV and your analyst meant net revenue, is that a failure? Mature vendors count abstentions and clarifications separately from errors, and report all three.
4. How are permissions enforced — and tested? Ask whether row-level security is enforced at execution for every query, and whether their evaluation includes permission-violation tests. If evaluation only measures SQL correctness, you are buying an unaudited access path to your warehouse.
5. Will you score on our data? The only benchmark that predicts your deployment is an eval set built from your questions and your schema. Vendors confident in their semantic-layer approach will agree to a structured pilot with a fixed question set and published scores — this is exactly what a paid pilot (ours: two weeks, HKD 25k / RMB 20k) is for. Vendors who resist scoring on your data are telling you something.

One further negotiation point that separates serious vendors from demo vendors: accuracy commitments over time, not just at signing. Ask what happens to measured accuracy when a model version is retired, when a table is migrated, or when a metric definition changes — and whether re-scoring on your eval set is included in the contract. A vendor who treats your evaluation set as the ongoing contract instrument, with agreed thresholds and re-run cadence, is offering an accountability structure. A vendor who treats evaluation as a one-time sales exercise is offering a screenshot.

A realistic accuracy expectation for a well-governed deployment on a messy enterprise schema, based on our deployments and corroborating practitioner reports: 85–95% usable answers on questions within the curated semantic scope, with abstention on out-of-scope questions rather than hallucination — and that abstention behavior, counterintuitively, is the mark of a mature system. Anyone promising 99% on an unconstrained warehouse is promising you a demo.

What actually determines deployed accuracy

Aggregating what we and others have seen across deployments, the ranking of accuracy levers is stable and, to outsiders, almost boring:

Note what ranks below the top four: everything having to do with the model. This is the consistent finding across 2025–2026 deployments and it matches the benchmark literature's own diagnosis — Spider 2.0's authors attribute low scores less to SQL inability than to the *exploration and reasoning over unfamiliar, realistic environments*. The model is no longer the bottleneck; the context and the governance are.

Two organizational observations complete the picture. First, every lever above needs a named owner: a metric owner for the semantic layer, an analyst who curates the jargon dictionary from query logs, a BI engineer who maintains schema documentation. Text-to-SQL accuracy decays not because systems break but because nobody owns the definitions they encode. Second, the eval set changes the internal politics of analytics for the better: when accuracy is scored weekly on real questions, arguments about "can we trust the AI answers" become arguments about specific failed questions with specific fixes — which is a healthy engineering conversation rather than a vague anxiety. Enterprises that publish the score get trust; enterprises that hide it get whisper networks.

Implications for conversational BI in the IM era

The last piece of context: Text-to-SQL accuracy matters more now, not less, because the delivery surface has changed. When analytics lives inside 企业微信, DingTalk, Feishu, WhatsApp, or Teams, a wrong answer is not seen by one analyst who can sanity-check it — it is seen by an entire group chat, instantly, with your company's name on it. IM-native delivery raises the required accuracy bar and the required failure behavior (abstain, cite, escalate to a human) simultaneously.

This is why conversational BI vendors who treat accuracy as a model problem will keep disappointing, and why the ones who treat it as a governance problem — semantic layer, permissions, evaluation, citations — are the ones whose systems survive contact with an enterprise. When you evaluate an IM-native analytics platform, the Text-to-SQL accuracy question is real, but the correct form of the question is: "What is your measured accuracy on data like ours, with our permissions enforced, and what does the system do with the questions it cannot answer?"

The IM channel also changes what "accuracy" means at the margin. In a portal, a confusing answer costs a second attempt; in a group chat, a wrong answer travels with the context it appeared in, gets screenshotted, and acquires a life of its own. The practical consequence is that IM-native systems need stricter abstention thresholds than web-app systems — answering 92% of questions with 2% embarrassing errors is worse in a chat surface than answering 85% with abstention on the hardest 15%. We tune conversational deployments toward fewer, more reliable answers, and let the semantic layer expand until the abstention rate falls naturally. Accuracy strategy in the IM era is scope strategy.

The healthy answers to that question are measurable, boring, and architecture-backed. The unhealthy answer is a leaderboard screenshot.

A practical evaluation checklist

For a CIO or Head of Data evaluating Text-to-SQL claims in 2026, condensed to one working checklist:

• Discount Spider 1.0 scores entirely. Ask for BIRD or Spider 2.0 results with dates, or scores on a schema like yours.
• Require abstention behavior. The system must say "I can't answer that from governed data" on out-of-scope questions — test this explicitly with trick questions.
• Test permissions adversarially. Have a low-privilege user try to ask their way into restricted data during the pilot. Treat any success as disqualifying.
• Inspect the citation trail. Every number should link back to source rows you can verify against the warehouse.
• Build the eval set before the pilot, not after. 100–200 real questions with verified answers, weighted toward your jargon. Score weekly, publish internally.
• Check schema-change handling. Rename a table mid-pilot. Watch what happens — the answer tells you whether the system has a governance layer or just a prompt.
• Budget for the boring layers. If the proposal's line items are all model-related and none are semantic-layer-and-evaluation-related, the 90-day accuracy trajectory will disappoint.

The mature position in 2026 is neither skepticism nor hype about Text-to-SQL. The capability is real and improving — the benchmark literature documents genuine, rapid progress. But deployed accuracy is manufactured, not bought: it comes from a semantic layer that encodes what your terms mean, permissions that decide what each identity may see, an eval set that tells you the truth weekly, and a delivery surface where abstention is engineered, not embarrassing. Vendors who can show you those four things deserve your pilot. The ones showing leaderboard screenshots deserve your skepticism.