Why ERP is the hard case for text-to-SQL
Text-to-SQL, sometimes called NL2SQL, turns a plain-language question into a database query. Demonstrations usually run on tidy schemas with a few well-named tables. ERP systems are the opposite: hundreds or thousands of tables, cryptic column names, custom fields added over years of configuration, multiple subsidiaries, currencies and accounting periods that open and close.
In that environment, a query can run without errors and still be wrong. It might count cancelled orders, mix functional and transaction currency, or ignore period-close adjustments. The challenge is not producing valid SQL. It is producing SQL that matches what the business means.
What benchmarks tell you, and what they do not
Public benchmarks are useful for comparing approaches. Spider measures cross-domain text-to-SQL across many databases. BIRD raises the difficulty with larger databases, messy values and questions that require external knowledge, which is closer to enterprise reality. Leaderboard scores on these benchmarks have improved substantially as language models have advanced.
Neither benchmark contains your chart of accounts, your custom fields or your revenue recognition rules. A high benchmark score indicates general capability, not readiness for your ledger. Treat published accuracy figures as a reason to evaluate, not as a forecast of your results.
Related: See how Nexatron approaches governed natural-language analytics
Pattern one: put a semantic layer between question and schema
The most reliable systems do not ask the model to discover business meaning from raw tables. A semantic layer defines metrics, dimensions, joins and filters once, with names people recognize. The model maps the question to those definitions, and the layer generates or constrains the SQL. Revenue then means the same thing whether a controller or a sales leader asks.
Building the layer is business work as much as technical work. Finance, sales operations and supply chain owners need to agree on definitions and review them when processes change. Start with the metrics people ask about most rather than modeling the entire ERP.
Custom fields deserve special attention. Years of configuration leave columns with names only their creator understood, and some of them hold the attributes executives care about most, such as a region code or a contract type. Document what each relevant custom field means and which values are valid. A short description attached to a field often improves query accuracy more than a larger model does, because the model no longer has to guess.
Pattern two: constrain the query surface
Give the model a narrow, safe place to work. Expose curated views instead of base tables, use read-only credentials, enforce row-level security in the database rather than in the prompt, and set limits on query time and result size. These controls address several risks in the OWASP Top 10 for LLM applications, including excessive agency and sensitive information disclosure.
When questions span systems, such as ERP revenue against CRM pipeline, federated query engines like Trino or embedded engines like DuckDB can join sources without copying everything into one warehouse. Federation adds its own definition work, because customer identifiers and fiscal calendars rarely match across systems.
Pattern three: show the work
Users trust answers they can inspect. Return the generated query, the definitions it used, the sources it touched and the time range applied. When a question is ambiguous, such as whether last quarter means fiscal or calendar, the system should ask rather than guess.
Visible lineage also speeds up correction. A reviewer can spot the wrong filter in seconds, and the fix goes into the semantic layer so the same mistake does not recur. Over time, the record of corrections becomes a map of where definitions were weakest, which tells data owners where to invest next.
How to evaluate a text-to-SQL system
Build an evaluation set from real questions with reference answers confirmed by business owners. Measure correctness, coverage of the questions people actually ask, refusal behavior on out-of-scope or unauthorized requests, and consistency when the same question is phrased differently. Rerun the set whenever definitions, models or source systems change.
Text-to-SQL works on ERP data when it is treated as a governed analytics capability rather than a clever prompt. The model translates language; the semantic layer, security model and evaluation discipline make the answers dependable.