---
title: "Learned and Neural Semantic Parsing"
module: Linguistic Structure
moduleNumber: 6
lessonNumber: 14
order: 614
summary: >
  Hand-writing a lexicon of lambda terms does not scale, so this lesson learns the
  parser instead. We cover the two supervision regimes (from logical forms and from
  denotations), Abstract Meaning Representation as a rooted concept graph, neural
  sequence-to-sequence parsing with constrained decoding and copy mechanisms,
  executable text-to-SQL and knowledge-based question answering, the practical
  systems that made learned parsers accurate, and how the task is evaluated.
topics: [Structure]
sources:
  - book: Jurafsky
    ref: "Ch. 16 — Computational Semantics and Semantic Parsing (AMR, seq2seq)"
  - book: Jurafsky
    ref: "Ch. 23 — Question Answering; §23.4 Knowledge-based Question Answering; §23.4.2 QA by Semantic Parsing"
---

This builds on [semantic parsing](/natural-language-processing/linguistic-structure/semantic-parsing),
which produced meaning representations by construction — a compositional walk of the
syntax tree, then Combinatory Categorial Grammar with syntax and semantics in
lockstep. Both routes need a hand-built grammar and a hand-built lambda term for
every word. Here we replace that hand labor with learning: induce the parser from
data, drop the grammar for a neural decoder, and take on the representations and
executable targets that come with the learned approach.

## Learning a semantic parser

Hand-writing a lexicon of lambda terms does not scale. A learned semantic parser
induces the lexicon and a scoring model from data. Two supervision regimes matter,
distinguished by how much the training signal reveals.[^jm-learn]

**Supervision from logical forms.** The training set is pairs
$(x_i, y_i)$ of a sentence and its correct logical form. This is the easier
signal: the target is fully specified, so learning is (roughly) structured
prediction of $y$ from $x$. Datasets such as GeoQuery (questions about U.S.
geography paired with logical queries) and ATIS (flight queries paired with SQL)
provide exactly these pairs. The cost is annotation: someone must write a correct
formal query for every training sentence, which demands expertise in the query
language.

**Supervision from denotations (weak supervision).** Often we only have the
_answer_, not the query — the sentence $x_i$ paired with the denotation $d_i$
obtained by executing the correct logical form against the knowledge base $\mathcal{K}$.
The logical form $y$ is now a **latent variable**: the parser proposes candidates,
executes each against $\mathcal{K}$, and is rewarded when a candidate's denotation
matches the gold answer. Training marginalizes over forms that execute correctly,
maximizing

$$
\mathcal{L} = \sum_i \log \sum_{y \,:\, \llbracket y \rrbracket_{\mathcal{K}} = d_i} p_\theta(y \mid x_i),
$$

where $\llbracket y \rrbracket_{\mathcal{K}}$ denotes execution of $y$ against
$\mathcal{K}$.

$$
% caption: Two supervision signals. Strong (left): the gold logical form is given,
% so the parser is trained directly against it. Weak (right): only the answer is
% given, so the logical form is latent — the parser must find a form that executes
% to the correct denotation.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=30mm, minimum height=9mm, align=center, inner sep=2pt}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % --- strong ---
  \node[box] (sx)  at (0,0)    {sentence x};
  \node[box, draw=acc, text=acc] (sy) at (0,-1.7)  {gold logical form y};
  \draw[->, acc, thick] (sx) -- (sy) node[midway, right, font=\scriptsize] {train directly};
  \node[font=\scriptsize, anchor=north] at (0,-2.6) {strong supervision};
  % --- weak ---
  \begin{scope}[xshift=6.6cm]
    \node[box] (wx) at (0,0)   {sentence x};
    \node[box] (wy) at (0,-1.7) {logical form (latent)};
    \node[box, draw=acc, text=acc] (wa) at (0,-3.4) {gold answer};
    \draw[->, thick] (wx) -- (wy) node[midway, right, font=\scriptsize] {propose};
    \draw[->, thick] (wy) -- (wa) node[midway, right, font=\scriptsize] {execute};
    \draw[->, red, thick] (wa.west) .. controls (-2.4,-2.2) and (-2.4,-1.1) .. (wy.west)
      node[midway, left, font=\scriptsize, text=red] {reward if match};
    \node[font=\scriptsize, anchor=north] at (0,-4.3) {weak supervision};
  \end{scope}
\end{tikzpicture}
$$

Weak supervision is far cheaper to collect — a non-expert can supply the answer to
_"What states border Texas?"_ without knowing any query language — but it is
harder to learn from. The reward is sparse (most proposed forms execute to the
wrong answer or crash), and _spurious_ logical forms that happen to return the
right answer for the wrong reason pollute the signal. This is the trade that
recurs whenever the annotation is the bottleneck: cheaper labels, noisier
learning.

## Abstract Meaning Representation

First-order logic and lambda calculus are one way to write meaning; they are also
verbose, and they entangle predicate-argument structure with quantifier scope,
tense, and other logical scaffolding. **Abstract Meaning Representation (AMR)**
strips that back to a single, simpler object: a **rooted, directed, labelled
graph** over concepts and the relations between them.[^jm-amr]

Nodes are **concepts**: word senses drawn from a frame inventory (PropBank-style
predicates like $\texttt{have-01}$) or plain entities. Edges are **relations**,
including the numbered semantic-role arguments $\texttt{arg0}$, $\texttt{arg1}$,
and so on. The graph is rooted (one node is the focus), and a single
node can be pointed at by several edges, so **reentrancy** (one entity filling
two roles) is expressed directly, which a tree cannot do.

$$
% caption: The AMR graph for "The boy wants to go": want-01 is the root, its arg0
% (the wanter) and arg1 (the wanted event) both drawn as edges. The boy node is
% reentrant — it is the arg0 of want-01 and also the arg0 of go-01 — so one entity
% fills a role in both events.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  concept/.style={draw, ellipse, inner sep=2.2pt, minimum height=7mm}]
  \definecolor{acc}{HTML}{2348F2}
  \node[concept, draw=acc, text=acc] (w) at (0,0)      {w / want-01};
  \node[concept] (b) at (-2.6,-2.2) {b / boy};
  \node[concept] (g) at (2.6,-2.2)  {g / go-01};
  \draw[->, thick] (w) -- (b) node[midway, above left, font=\scriptsize] {arg0};
  \draw[->, thick] (w) -- (g) node[midway, above right, font=\scriptsize] {arg1};
  \draw[->, acc, thick] (g) -- (b) node[midway, below, font=\scriptsize, text=acc] {arg0};
\end{tikzpicture}
$$

The same graph has an equivalent **textual (PENMAN) form**, which is what
annotators actually write and parsers actually emit. Each node is
`(variable / concept)` and each relation is a colon-prefixed label; reentrancy is
a repeated variable:

```
(w / want-01
   :arg0 (b / boy)
   :arg1 (g / go-01
            :arg0 b))
```

AMR is deliberately **abstract**: it drops tense, number, and articles, and it
abstracts away from surface syntax so that _the boy wants to go_, _the boy desires
to go_, and passive paraphrases all map to nearby graphs. That is the doctrine of
canonical form — inputs that mean the same thing should get the same
representation — pushed to the level of a whole-sentence graph. The cost is that
AMR is not directly executable the way a database query is; it is a normalized
meaning graph, not a program.

### AMR parsing

**AMR parsing** maps a sentence to its graph. Because the output is a graph rather
than a tree, it does not fit the compositional walk-the-syntax-tree recipe
cleanly, and two broad strategies dominate.

- **Graph-based / transition-based.** Build the graph incrementally, in the spirit
  of [transition-based dependency parsing](/natural-language-processing/linguistic-structure/dependency-parsing):
  a sequence of actions adds concept nodes and draws labelled relation edges,
  often seeded by an alignment between words and concepts. Reentrancy is handled
  by allowing an action to point a new edge at an already-created node.
- **Sequence-based (seq2seq).** Linearize the PENMAN form into a token string and
  treat parsing as translation — encode the sentence, decode the linearized graph.
  This reuses the neural machinery of the next section, at the price of having to
  guarantee the output string is a well-formed, connected graph.

Because AMR abstracts away function words and inflection, alignment between input
tokens and graph nodes is looser than in syntactic parsing, which is much of what
makes AMR parsing hard. Evaluation uses **Smatch**: over all variable alignments $M$ between predicted graph
$G$ and gold $G^\ast$, maximize the shared `(relation, node, node)` triples, then report
$F_1$ on that best match,

$$
\text{Smatch} = \max_M F_1\bigl(\text{triples}(G) \cap_M \text{triples}(G^\ast)\bigr).
$$

## Neural sequence-to-sequence parsing

The compositional and CCG parsers guarantee a well-formed logical form because
they build it by grammar rules. The neural approach gives up that guarantee for
generality: treat the logical form as **just a sequence of tokens**, and learn to
generate it with the same encoder-decoder architecture used for
[machine translation](/natural-language-processing/applications/machine-translation).
Semantic parsing becomes translation from English into a formal language.[^jm-seq2seq]

The encoder (a
[transformer](/natural-language-processing/transformers/transformers-and-attention)
or, in earlier systems, a biLSTM, frequently with a pretrained
[BERT](/natural-language-processing/transformers/large-language-models) front end)
reads the question tokens. The decoder emits the logical form token by token:
predicates, variables, parentheses, and keywords, each conditioned on the encoder
and on the tokens produced so far, exactly as a translation decoder emits target
words.

$$
% caption: An encoder-decoder semantic parser. A BERT/Transformer encoder reads
% "What states border Texas?"; the decoder autoregressively emits the logical form
% as a token sequence: lambda, x, state, (, x, ), and, borders, (, x, texas, ).
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=1pt, font=\scriptsize},
  big/.style={draw, minimum height=9mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % input tokens
  \foreach \w/\i in {What/0, states/1, border/2, Texas/3, ?/4}
    \node[tok] (in\i) at (\i*1.15,0) {\w};
  % encoder
  \node[big, minimum width=58mm] (enc) at (2.3,-1.3) {encoder (BERT / Transformer)};
  \foreach \i in {0,...,4} \draw[->, black] (in\i) -- (enc.north -| in\i);
  % decoder
  \node[big, draw=acc, text=acc, minimum width=58mm] (dec) at (2.3,-3.0) {decoder (autoregressive)};
  \draw[->, acc, thick] (enc) -- (dec) node[midway, right, font=\scriptsize] {context};
  % output tokens
  \foreach \w/\i in {lambda/0, x/1, state(x)/2, and/3, borders(x/4, texas)/5}
    \node[tok, draw=acc, text=acc] (out\i) at (\i*1.5 - 0.5,-4.5) {\w};
  \foreach \i in {0,...,5} \draw[->, acc] (dec.south -| out\i) -- (out\i);
  % autoregressive feedback
  \draw[->, black] (out2.north) .. controls (3.2,-3.7) and (4.0,-3.7) .. (out3.north);
\end{tikzpicture}
$$

The freedom to emit any token sequence is also the failure mode: nothing stops the
decoder from producing a syntactically illegal form — unbalanced parentheses, an
undefined predicate, an argument count that no relation accepts. **Constrained
decoding** addresses this: restrict the decoder, at each step, to tokens that keep
the output a valid prefix of a well-formed logical form. In practice this means
tracking the grammar of the target language while decoding and masking out any
continuation that violates it — closing every bracket that was opened, only
emitting predicate names that exist, respecting arities. Constrained decoding
folds the guarantee of the grammar-based parsers back into the neural one: the
model still learns from data, but it can only output forms the query engine can
run.

One recurring difficulty is that a logical form is full of tokens copied straight
from the question. _"What states border Texas?"_ becomes a form containing the
literal $\texttt{texas}$, and a fixed output vocabulary either has to enumerate
every entity in the knowledge base or fails on any name it never saw in training.
The standard solution is a **copy mechanism**: at each decoding step $t$ a learned gate
$p_{\text{gen}} \in [0,1]$ mixes a generation distribution over the fixed vocabulary
with a copy distribution over the input positions,

$$
p(y_t) = p_{\text{gen}} \, p_{\text{vocab}}(y_t) + (1 - p_{\text{gen}}) \sum_{i \,:\, x_i = y_t} a_{t,i},
$$

where $a_{t,i}$ is the decoder's attention on input token $x_i$. Rare entity names
flow through untouched; copying also handles SQL literals — table values, string
constants — that could never fit a closed vocabulary.

$$
% caption: A copy mechanism in the decoder. At each step the model mixes a
% generation distribution over the fixed vocabulary with a copy distribution over
% the input tokens, choosing (via a learned gate) to emit a predicate from the
% vocabulary or copy an entity name like Texas straight from the question.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=32mm, minimum height=8mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (gen) at (0,0)   {generate from vocab\\(borders, state, ...)};
  \node[box] (cop) at (5.2,0) {copy from input\\(What, states, Texas)};
  \node[box, draw=acc, text=acc] (gate) at (2.6,-1.6) {gate: mix by p-gen};
  \draw[->, black] (gen.south) -- (gate.north);
  \draw[->, black] (cop.south) -- (gate.north);
  \node[box, draw=acc, text=acc] (out) at (2.6,-3.0) {next token};
  \draw[->, acc] (gate.south) -- (out.north);
\end{tikzpicture}
$$

The trade against compositional parsing is stark. The neural parser needs no
hand-built grammar or lexicon and degrades gracefully on constructions it has
never seen, but it offers no compositional account of _why_ a form was produced
and can hallucinate structure absent constrained decoding. The two approaches are
increasingly combined: neural scoring inside a grammar-constrained search.

## Executable parsing: text-to-SQL and QA

The most consequential setting for semantic parsing is
[question answering](/natural-language-processing/applications/question-answering)
over a structured database — **knowledge-based QA**. Here the "logical form" is an
outright executable program: predicate calculus, a query language like **SQL** or
SPARQL, or a small program over knowledge-base relations. Parse the question,
execute the query, return the rows.[^jm-qa]

$$
% caption: Executable semantic parsing across target languages. The same question
% type maps to predicate calculus, to SQL over a relational database, or to a
% program over knowledge-base relations; each is run to produce the answer.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  q/.style={draw, minimum width=40mm, minimum height=9mm, align=center},
  t/.style={draw, minimum width=42mm, minimum height=11mm, align=left, inner sep=4pt, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[q] (ql) at (0,0) {What states border Texas?};
  \node[t] (pc) at (5.6,1.6)  {lambda x. state(x)\\ and borders(x, texas)};
  \node[t] (sql) at (5.6,0)   {SELECT name FROM state\\WHERE border = 'texas'};
  \node[t] (kb) at (5.6,-1.6) {(and (state ?x)\\   (borders ?x texas))};
  \draw[->, thick] (ql) -- (pc)  node[midway, above left=-1pt, font=\scriptsize] {pred. calc.};
  \draw[->, thick] (ql) -- (sql) node[midway, above, font=\scriptsize] {SQL};
  \draw[->, thick] (ql) -- (kb)  node[midway, below left=-1pt, font=\scriptsize] {KB program};
\end{tikzpicture}
$$

Predicate calculus can be mechanically converted to SQL, so the choice of target
language is partly cosmetic and partly a matter of what the backend runs. The two
supervision regimes from earlier map straight onto QA datasets: **fully
supervised** parsers learn from questions paired with hand-written queries
(GeoQuery, ATIS with SQL, WebQuestionsSP with SPARQL), while **weakly supervised**
parsers learn from questions paired only with the answer, treating the query as
latent. A standard, strong baseline for text-to-SQL is precisely the
encoder-decoder of the previous section: BERT-encode the question, decode the SQL
string, optionally constrained to the database schema so the query references only
real tables and columns.

This closes the loop opened by the first figure. _"What states border Texas?"_
enters as text; a learned encoder-decoder, constrained by the target grammar,
emits a query; the query executes against the database; the rows are the answer.
Every earlier idea is a way of making that middle arrow reliable — compositionality
for a guaranteed-valid form, CCG for a syntax-and-semantics-in-one lexicon, weak
supervision for cheap labels, constrained decoding for validity in a neural model.

## Neural semantic parsing in practice

The encoder-decoder view opened a decade of work on making learned semantic
parsers accurate and general, past what the textbook sketches. Four threads stand out.

**Sequence-to-tree decoding.** A logical form is a tree, and forcing a plain
left-to-right decoder to spell out its brackets wastes structure it already has.
Dong and Lapata (2016), _"Language to Logical Form with Neural Attention"_ (ACL),
introduced **Seq2Tree**: a decoder that generates the logical form top-down as a
tree, emitting a nonterminal and then recursively expanding its children, so
bracket structure is guaranteed by construction rather than learned as a string
convention. On GeoQuery and ATIS their attention-based model matched or beat the
hand-engineered parsers that preceded it, with no grammar written by hand.[^seq2tree]

**Data recombination for compositional generalization.** Neural parsers are
data-hungry and generalize poorly to novel combinations of familiar pieces. Jia and
Liang (2016), _"Data Recombination for Neural Semantic Parsing"_ (ACL), induced a
synchronous grammar from the training pairs and sampled new (sentence, logical
form) pairs from it — recombining fragments the model had seen into constructions it
had not — then trained a sequence-to-sequence model with attention and copying on
the augmented data. The recombined data alone raised accuracy on GeoQuery and ATIS,
a cheap way to inject compositional bias into a model that has none by default.[^jialiang]

$$
% caption: Data recombination (Jia and Liang 2016). An induced grammar over the
% training pairs generates fresh sentence / logical-form pairs by swapping in
% fragments the model has seen elsewhere, so a seq2seq parser trained on the
% augmented set generalizes to unseen combinations of known pieces.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=30mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (tr) at (0,0) {training pairs};
  \node[box] (gr) at (4.4,0) {induced grammar};
  \node[box] (aug) at (8.8,0) {recombined pairs};
  \node[box, draw=acc, text=acc] (m) at (4.4,-1.7) {seq2seq + copy};
  \draw[->, black] (tr) -- (gr);
  \draw[->, black] (gr) -- (aug);
  \draw[->, acc] (tr.south) .. controls (1.0,-1.2) .. (m.west);
  \draw[->, acc] (aug.south) .. controls (7.8,-1.2) .. (m.east);
\end{tikzpicture}
$$

**Text-to-SQL at scale.** Early text-to-SQL datasets (ATIS, GeoQuery) each targeted
a single database, so a parser could quietly memorize its schema. Zhong et al.
(2017) released **WikiSQL** with **Seq2SQL**, framing the task over thousands of
Wikipedia tables and using the executed answer as a reinforcement-learning reward
so that queries with different text but the same result are treated as equally
correct.[^wikisql] Yu et al. (2018), _"Spider"_ (EMNLP), pushed harder: a large,
**cross-database** benchmark of complex multi-table queries (joins, nesting,
aggregation) with a strict split so that the databases at test time are ones the
parser has never seen. Spider made **schema encoding** — reading the table and
column names alongside the question and grounding the query in them — the central
problem, and it remains the standard yardstick for text-to-SQL.[^spider]

$$
% caption: The shift from single-database to cross-database text-to-SQL. Early sets
% (ATIS, GeoQuery) train and test on one schema, so the parser can memorize it.
% Spider (Yu 2018) tests on databases held out from training, forcing the parser to
% read the schema and generalize to unseen tables and columns.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  box/.style={draw, minimum width=34mm, minimum height=11mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box] (single) at (0,0) {single-DB (ATIS)\\train and test\\same schema};
  \node[box, draw=acc, text=acc] (cross) at (6.0,0) {cross-DB (Spider)\\test schema unseen\\must read schema};
  \draw[->, acc, thick] (single) -- (cross) node[midway, above, font=\scriptsize] {harder};
\end{tikzpicture}
$$

**Pretrained encoder-decoders as parsers.** The same large pretrained models that
carry the rest of modern NLP fold semantic parsing back into plain text-to-text
generation: fine-tune a model like T5 or BART to emit the logical form as a string,
and it inherits broad linguistic competence for free. Combined with constrained or
grammar-guided decoding, these models now hold the top of the Spider and GeoQuery
leaderboards — a neural model that writes the query, constrained by the grammar of
the target language.

## Evaluation

How a semantic parser is scored depends on what supervision is available, and the
choice shapes what "correct" means.

| Metric | Compares | Used when | Strictness |
| --- | --- | --- | --- |
| Logical-form (exact-match) accuracy | predicted form vs. gold form | gold logical forms exist | strict — penalizes valid paraphrases of the query |
| Denotation (answer) accuracy | answer of predicted form vs. gold answer | only answers exist, or forms vary | lenient — credits any form that returns the right answer |
| Smatch $F_1$ | predicted vs. gold graph triples | AMR parsing | partial credit for overlapping subgraphs |

**Exact-match on the logical form** is the most stringent: the predicted query
must match the reference string (up to normalization). It is unforgiving, because
two different-looking queries can be semantically equivalent — the same reason
exact-match undersells a
[dependency parser](/natural-language-processing/linguistic-structure/dependency-parsing).

**Denotation accuracy** sidesteps that by running both forms and comparing
_answers_: any query that returns the correct set counts as correct, regardless of
its syntax. This is the natural metric under weak supervision, where gold logical
forms do not even exist, but it credits spurious forms that get the right answer by
accident.

**Smatch** is the graph analogue for AMR: after aligning the two graphs' variables
to maximize overlap, it computes precision, recall, and $F_1$ over shared
`(relation, node, node)` triples, awarding partial credit for a mostly-correct
graph rather than an all-or-nothing verdict.

The through-line across both semantic-parsing lessons is a single arrow: from the
words of a sentence to a structured, executable meaning that a machine can run
against the world.
[Compositional parsing and CCG](/natural-language-processing/linguistic-structure/semantic-parsing)
build that meaning from the parts by grammar rules; here, learned parsers induce
the lexicon and scoring model from data, AMR captures meaning as a normalized graph,
and neural seq2seq learns to write the logical form out directly, constrained to
well-formed outputs. All four routes end at the same target: a sentence turned into
something with a denotation.

[^jm-learn]: **Jurafsky & Martin**, Ch. 16; §23.4.2 — QA by Semantic Parsing: learning semantic parsers with full supervision from question–logical-form pairs (GeoQuery, ATIS) versus weak supervision from question–denotation pairs, with the logical form as a latent variable.
[^jm-amr]: **Jurafsky & Martin**, Ch. 16 — Computational Semantics (Abstract Meaning Representation): AMR as a rooted, directed, labelled graph over PropBank-style concepts and numbered-argument relations, its equivalent PENMAN textual form, reentrancy for shared arguments, and Smatch for evaluation.
[^jm-seq2seq]: **Jurafsky & Martin**, §23.4.2; Fig. 23.15 — an encoder-decoder semantic parser with a BERT pre-encoder that translates a question to its logical form as a token sequence, and the role of constraining the decoder to well-formed outputs.
[^jm-qa]: **Jurafsky & Martin**, §23.4 — Knowledge-based Question Answering; Fig. 23.14: executable semantic parsing to predicate calculus, SQL/SPARQL, or knowledge-base programs, run against a structured database to return the denotation, under full or weak supervision.
[^seq2tree]: **Dong and Lapata (2016)**, "Language to Logical Form with Neural Attention," _ACL 2016_ — a Seq2Tree attention decoder that generates the logical form top-down as a tree so bracket structure is guaranteed, matching or beating hand-engineered parsers on GeoQuery and ATIS.
[^jialiang]: **Jia and Liang (2016)**, "Data Recombination for Neural Semantic Parsing," _ACL 2016_ — inducing a synchronous grammar over the training pairs to sample recombined (sentence, logical form) examples that inject compositional generalization into an attention-plus-copy sequence-to-sequence parser.
[^wikisql]: **Zhong, Xiong, and Socher (2017)**, "Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning" — the WikiSQL benchmark over thousands of Wikipedia tables, using execution accuracy as a reward so answer-equivalent queries are credited equally.
[^spider]: **Yu et al. (2018)**, "Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL," _EMNLP 2018_ — a cross-database benchmark of complex multi-table SQL with unseen test schemas, making schema encoding the central problem and setting the standard text-to-SQL evaluation.