A significant portion of a Data Analyst or Business Intelligence Analyst’s time is spent translating business questions into SQL queries. These analysts often serve as an interface between humans asking questions and the computers processing the data. In business settings, Software Engineers essentially act as translators, converting business requirements into executable code. With advancements in Natural Language Processing (NLP) and Large Language Models (LLMs), the role of analysts could potentially be replaced by language models, reducing the need for manpower and enabling access to databases without expert SQL knowledge.

The state-of-the-art Text-to-SQL model has achieved a remarkable 79.1% execution accuracy, measured by the percentage of generated SQL queries that return correct results when executed on the Spider development set, and a 97.8% valid SQL ratio[^1]. In comparison, OpenAI’s Codex davinci, without any fine-tuning, reached 67.0% execution accuracy and a 91.6% valid SQL ratio. If the model generates accurate results without any corrections, its performance may even surpass that of a human. Consider how often we can write a valid SQL query on the first try and obtain the desired statistics immediately.

After spending several hours on a Saturday morning reviewing literature in this area, I’ve found that current research can be divided into a few key parts:

  • Evaluation methodology: addressing limitations and improvements in existing text-to-SQL datasets and evaluation metrics
  • Multilingual Text-to-SQL: expanding research to encompass more languages
  • Cross-Database Text-to-SQL: developing models that generalize across domains and databases
  • Pre-training Text-Table Data: training LLMs on text-table data
  • Structure Grounding: mapping natural language phrases or words to database elements, such as tables, columns, and values, while determining relationships between them.

Text-to-SQL is poised to make a significant impact on businesses. By reducing costs and increasing efficiency when generating statistics from databases, this technology has garnered attention from major companies like Microsoft, Google, and Baidu, which are actively developing their own tools.

My Two Cents:
One challenge in the field of text-to-SQL is the inherent imprecision of human language. This task is not a direct translation, but rather requires contextual information and clarification of definitions and requirements. Ideally, the model should treat SQL query generation as an interactive process, with performance evaluated based on the satisfaction of the final query compared to the cost of executions. Additionally, techniques used by human analysts could be incorporated into the language model, such as examining a few rows of data before writing the query, writing simple code first, and then adding complexity. By “teaching” language models the definitions of metrics and the correct results to check against, complexity can be added primarily in terms of dimensions.

In conclusion, text-to-SQL is an exciting field with significant potential for both human-computer interaction research and practical business applications.


[^1]: Rajkumar, Nitarshan, Raymond Li, and Dzmitry Bahdanau. ‘Evaluating the Text-to-SQL Capabilities of Large Language Models’. arXiv, 15 March 2022. https://doi.org/10.48550/arXiv.2204.00498.

Leave a Reply