Abstract
Citation-grounded question answering promises answers a lawyer can check: every sentence tied to a provision, every quote drawn from the text in force. A grounding claim on its own says little, because a system that refuses often, and one that answers everything while quoting loosely, can both post a high grounding score. We report answer quality for Licentium Compass, a citation-grounded system over the consolidated text of European Union financial, crypto, and AI regulation, on two question sets that two people wrote and source-checked apart. Each set marks, per question, the articles a correct answer must cite (core) and those that add credit (supporting), and each holds out-of-scope questions the system should decline. Grading runs by rule, with no person or model judging at scoring time. Core recall reached 0.829 and 0.843 across the two sets, which were built independently and agree within one and a half points. Every displayed citation, 592 in total, matched its source word for word. We report the three numbers together, on two sets built apart, and set out where the system still falls short.
Introduction
Retrieval-based question answering over primary law places a strict demand on every answer: a reader must trace each statement to a specific article of a specific act, and check the quoted words against the source. A general language model can produce a fluent regulatory answer that is wrong, and a wrong citation is the costly kind of error. Licentium Compass answers questions on European Union financial, crypto, and AI regulation from the consolidated text of the law, shows the exact article behind each sentence, and declines when its corpus does not hold the answer.
Three earlier reports set out how the retrieval and answer path was built, each measuring one stage.
The first compared five text-embedding models for first-stage retrieval over consolidated EU legislation, across 159 questions in four rounds of rising difficulty. A general-purpose model led, the legal-specialist models trailed, and the strongest retrievers proved the weakest at separating answerable from unanswerable questions by similarity alone. Retrieval strength and the ability to decline are different properties.
The second report added a reranker that re-reads each shortlisted article against the question; on hard, colloquial questions it raised mean reciprocal rank from 0.55 to 0.70 and the top-five hit rate from 76 to 87 percent.
The third described the answer step: Compass binds each sentence to a retrieved provision and checks every quoted span character by character against the source, drops any span that is not an exact substring of the cited article, and assembles the citation from the stored record so the model never writes an article number. That report measured a 100 percent word-for-word rate on a hard set, and named recall, whether the answer includes the provisions it should, as the harder number still to be reported.
This report supplies that number, and two more beside it. We evaluate Licentium Compass end to end on two question sets that two people built and source-checked apart, and we report three quantities together: how much of the required law each answer cited (recall), how often each displayed citation matched its source word for word (grounding), and how often the system declined a question with no answer in the corpus (abstention). Reporting the three together, on two independent sets, is what the earlier work argued a grounding claim needs.
The corpus
Compass reads statutes, not summaries of them. The corpus holds the consolidated text of the acts that govern this area: MiCA, EMIR, DORA, the AML Regulation, the sixth AML Directive, the Transfer of Funds Regulation, the AI Act, the Digital Services Act, the Data Act, MiFID II, PSD2, the e-money rules, and others. That comes to 54 acts and more than 4,400 article-level provisions in English. Each provision is stored with a hash of its source text, so a change in the official version is detectable. Compass records which consolidated version of each act it holds and the date of that version, and it marks an act for refresh when the EU publishes a newer consolidation or an amending act dated later than the copy on file. Articles point at other articles, and the corpus stores 20,138 such internal references as links, so a reader who opens a cited article sees the provisions it invokes.
The path from question to answer has four stages, set out in full in the earlier reports and repeated here so the results stand on their own. A dense embedding model retrieves a pool of candidate provisions by meaning. A reranker reorders that pool so the most relevant article sits at the top. An answer model writes a reply bound to the retrieved provisions and quotes them. A separate check reads each quoted span against the primary source and drops any span that is not an exact substring of the cited article. The four stages are independent, and the grounding check runs the same way every time, outside the writing step. The measurement here covers the output of the whole path, not any stage alone.
How we measured
Two gold sets, built apart
Two people each wrote their own questions on EU financial, crypto, and AI regulation, and each looked up the governing articles in the official EU sources and recorded them. Neither saw the other's set. The first set holds 41 answerable questions and 13 out-of-scope questions. The second holds 42 and 12. Together that is 83 answerable questions and 25 with no answer in the corpus. The answerable questions span the acts the system serves, from MiCA and DORA through the AML instruments, the AI Act, the Digital Services Act, the Data Act, MiFID II, EMIR, PSD2, and the e-money rules.
Core and supporting provisions
For each answerable question, the author split the required articles into two groups. Core articles must appear for the answer to count as correct. Supporting articles add credit but are not required. The headline figure is core recall: of the articles a correct answer must cite, the share the system actually cited. Splitting the two keeps a central-article hit from being diluted by a long tail of related but optional provisions, and it keeps the core bar strict.
Out-of-scope questions
The 25 out-of-scope questions have no answer in the corpus. They test one behaviour: does the system stay silent, or does it reach for a provision anyway. A correct response cites nothing.
Scoring
Grading runs by rule, with no person and no second model judging at the moment of scoring. The scorer compares each cited article to the answer key. It treats the base and the amended consolidated version of an act as the same act, so a citation to either form counts. It expands article ranges, so a gold label of Articles 16 to 18 is met by 16, 17, and 18 in any order. Annex references without an article number leave the denominator rather than count as permanent misses. A quoted span counts as grounded only when it appears verbatim in the cited provision.
The seven measures
Core recall is the mean over questions of the share of core articles cited. Core hit rate is the share of questions that cite at least one core article. Core full rate is the share that cite every core article. Answered rate is the share of answerable questions that produce at least one verified citation. Supporting recall is the mean share of supporting articles cited, over questions that have any. Grounding rate is the share of displayed citations that pass the verbatim check. Correct abstention is the share of out-of-scope questions answered with no citation.
Findings
Table 1 gives the seven measures for both sets. Table 2 breaks core recall down by act.
Recall
Core recall came out at 0.829 on the first set and 0.843 on the second. On a typical question, Compass cited about 83 to 84 percent of the articles a correct answer required. The two sets were written and checked by different people, and the two figures land within one and a half points. Two supporting figures track it. On the second set every answerable question cited at least one core article; on the first, about nine in ten did. The system answered every in-scope question, and refused none that had an answer.
Recall by act
Table 2 splits core recall across ten act groups. The acts whose questions turn on one clear article sit near the top: Data Act and MiFID II at or above 0.88 on both sets, PSD2 and EMIR high on both. The lower cells fall into two kinds. Some are a single hard question standing for a whole group, so one miss reads as a large drop; the e-money row is one question on each side, missed on the first set and hit on the second, and carries no weight on its own. Others reflect the compound-question pattern below, where a correct answer needs several articles and the system supplies the lead one. Read across, no act group collapses, and the two independent sets rank the acts in a broadly similar order.
Grounding
Every citation Compass displayed, across both sets, matched its source word for word. The two sets together produced 592 citation instances, and not one failed the character-level check. The 100 percent rate first reported on a smaller, single set held on 108 new questions written by two other people. One point sharpens what this means. On the out-of-scope questions the system got wrong, the quotes were still real: when it should have stayed silent and instead cited an article, that article existed and the quoted text matched. Grounding and relevance are separate axes, and the system is now perfect on the first and imperfect on the second.
Abstention
On the out-of-scope questions, Compass stayed silent 62 percent of the time on the first set and 67 percent on the second. The rest of the time it cited a real but off-point article rather than declining. The first report in this series anticipated the shape of this: a retriever cannot tell an answerable question from an unanswerable one by similarity alone, so the decision to refuse sits in a later component. These two figures measure that component from end to end. With 13 and 12 out-of-scope questions, the interval around each figure is wide, and both read as an early bound rather than a settled rate.
Compound questions
The clearest recall gap is on questions whose correct answer needs three or four articles at once. There the system reliably supplies the main article and misses one or more of the rest. Single-article questions come back near-perfect on both sets. The gap between core hit rate and core full rate, 0.90 against 0.76 on the first set and 1.00 against 0.71 on the second, is almost entirely these compound questions.
What the numbers mean
A grounding rate on its own can mislead. A system that answers one question in five, and declines the rest, can show a perfect grounding rate on the little it says. A system that answers everything can pair a high grounding rate with loose or incomplete citation. The reading that means something needs three numbers at once: how often the citation is real (grounding), how much of the required law the answer caught (recall), and how often the system declines when it should (abstention). Compass posts 1.00, about 0.84, and about 0.65 on questions two independent authors wrote. Naming the weak one next to the strong ones is the point of the exercise.
For the lawyer who has to rely on the output, the test reduces to two habits the system already supports and one it does not yet. Every sentence carries a link to the provision under it, and the quoted text is the real text of that provision, so a claim can be checked in a click. Recall says how often the set of links is complete, and here it is complete about five times in six for a single-article answer, less for a compound one. Abstention says how often the system knows to stop, and here it stops about two times in three. A user can lean on the first two today, and should treat a confident answer to an edge-case question with care.
We are not aware of another citation-grounded answerer over consolidated EU financial, crypto, and AI regulation that reports recall, grounding, and abstention together on two question sets built and source-checked apart. The combination is what makes the result hard to overstate: the grounding rate is bounded by a mechanical check, the recall figure is reproduced across two independent sets, and the abstention figure is named rather than hidden.
Limitations
- Gold-label authorship. Two domain authors wrote the questions and looked up the articles, working apart and checking each label against the official source. No independent panel adjudicated them, so a labelling error would move a question's score.
- Scoring leniency. Base and consolidated forms of an act count as one, article ranges expand, and annex-only references leave the denominator. Each reflects a citation a lawyer would accept, and each can only raise recall. A stricter scorer would report a lower number.
- Single language, point-in-time corpus. Grading covers the English consolidated text; cross-lingual answering is not measured here. The corpus is a fixed copy, so the figures pertain to the versions held at the run and the models then in the path.
- Small out-of-scope sets, single run. With 13 and 12 out-of-scope questions, the abstention figures carry a wide interval, and the single pass shows no run-to-run variance. Read the abstention rate as an early bound.
- Article-level scope. A citation to the right article with a wrong emphasis in the surrounding sentence still counts as a recall hit. Faithfulness of the prose to the cited article is a separate question this evaluation does not score.
Outlook
The refusal component is the near-term focus: raising the share of out-of-scope questions the system declines, without costing the in-scope answers it already gets. Compound-question recall is the second: closing the gap between citing the lead article and citing the full set a correct answer needs. Both will be reported the same way, the miss rate and the refusal rate beside the grounding rate, on question sets graded by rule rather than by hand. The measure that matters is not a single high number but the three read together, and held steady as the corpus grows.
