The corpus, and which version of the law is in force
Compass reads 46 EU legislative acts holding 43,916 citable provisions in English, with 20,215 cross-references between provisions recorded as explicit links and 495 amendment relationships tracked between acts.
The count matters less than the version question behind it. EU legislation is amended continuously, and the EU publishes a consolidated text after each amendment takes effect: the original act with every amendment already written into it. A system that cites the original 2014 text of an act gives answers that were correct in 2014. That failure is obvious.
The opposite failure is not obvious. The EU also publishes consolidations dated in the future, prepared for amendments that have been adopted but do not yet apply. Loading the newest available consolidation therefore risks quoting a rule that binds nobody today, in text that looks entirely authentic and carries an official reference.
We found exactly this in our own corpus on 11 August 2026. The Solvency II consolidation we were serving carried an application date in 2027. Any answer drawn from it would have stated a future rule as current law. We rewrote the version-selection rule so the ingestion job refuses any consolidation whose application date falls after the date of the run, then re-ingested the affected act. The check now runs on every ingestion, so a future-dated text cannot re-enter without failing the build.
A related product decision followed. Some provisions in force today apply only from a later date. Compass now marks those provisions with the date from which they apply, rather than presenting them as immediately binding.
What character-level verification proves, and what it does not
Across every evaluation run since June 2026, the proportion of citations matching the stored text exactly has been 1.000, so the verifier has had nothing to reject. That figure means less than it appears to.
Exact matching proves that Compass did not invent a quotation, did not paraphrase a provision into something the law does not say, and did not attribute text to the wrong article. It proves nothing about completeness. An answer that quotes one correct article perfectly, while omitting three others that also govern the question, passes verification and still misleads a reader who assumes the answer is whole.
Completeness is therefore the harder problem, and it is where our measurement effort went.
How accuracy is measured
Two legal researchers, working separately, wrote question sets against the corpus. For each question they recorded which provisions a correct answer must cite and which provisions deserve credit without being required. A third category holds questions the corpus cannot answer at all, where the correct behaviour is refusal.
Four properties of this design matter.
Required and optional citations are scored separately. Averaging them hides the failure that costs a reader most: a missing mandatory article.
Scores are reported two ways. Per-question averaging treats a question with two required articles as equal in weight to one with six. Per-article averaging counts every required article once. The two numbers can move in opposite directions, and reporting only the flattering one is a way to fool yourself. We report both.
Refusal questions carry their own score. A system that answers everything scores well on recall and is dangerous in a compliance setting.
Part of the question bank is sealed. It has never been run against the engine and will be scored once, on a configuration fixed in advance. Every other set is available for tuning, which makes those sets progressively less honest as estimates over time. Scoring the sealed set once, against settings frozen beforehand, produces the only estimate in the programme that no tuning has touched.
The current bank holds 80 development questions carrying 269 required articles, 90 refusal questions, and 190 sealed questions.
Provisions that were present but unreachable
The first failure we found produced no error message anywhere.
582 provisions across nine acts had been loaded into the database correctly, with accurate text and correct article numbers, and had never been prepared for search. They existed. No answer could ever cite them. Nothing in the product indicated a gap, because from the outside a provision that is never retrieved is indistinguishable from a provision the question did not call for.
We now run a check after every ingestion that fails the build when any citable provision lacks a search representation, and lists the affected acts. The check reads only the database and needs no external service, so there is no reason to skip it. Coverage gaps have been zero on every run since.
Answers that stopped before they finished
The most instructive failure looked like a reasoning problem and was a configuration problem.
Compound questions, meaning questions with two or more distinct legal parts, scored far worse than single-part questions. Required-article recall sat near 0.60 across successive runs. We tested the plausible explanations in turn: rewriting the instructions given to the answer model changed nothing measurable, and enlarging the candidate list changed nothing either. Recall appeared to be capped at roughly two correct articles per answer, whether the question required two or six.
The cause, found on 11 August 2026, was a limit on how long an answer could be. Every one of 22 compound answers examined had stopped mid-generation at that limit. The later parts of each answer, and every citation those parts would have carried, were never produced. Nothing in the output said so. A truncated answer is visually indistinguishable from a model that chose not to cite anything further.
Raising the limit changed recall on the same questions, with the same evidence, from 0.670 to 0.977. The system gained 27 required articles and lost none. An exact paired test put the probability of that pattern arising by chance below one in a million.
Two consequences deserve stating. The defect was live, not confined to evaluation: user queries ran through the same limit. And the whole preceding investigation had been measuring a ceiling imposed by our own configuration, while attributing it to the model's legal reasoning. Compass now records why generation stopped on every single answer, so the same masking cannot recur.
Separating the search stage from the drafting stage
Once truncation was gone, a cheaper measurement became possible.
Compass works in two stages: a search stage that assembles candidate provisions, and a drafting stage that writes the answer from those candidates. A missing article is either a search failure or a drafting failure, and the remedies are unrelated.
We built a measurement that checks only whether each required article reaches the drafting stage, without asking any model to write. It costs a fraction of a full evaluation, and one ranked list yields the answer for every candidate-list length at once.
The result reset our priorities. The drafting stage cites very nearly everything it is shown. Measured end-to-end recall now tracks the search-stage ceiling to within two percentage points, which leaves prompt work and answer-model changes almost no room to move the number. About 18 of every 100 required articles never enter the candidate list at all, at any list length we tested.
Three measurement errors we made
Each of these errors nearly produced a published number that was wrong.
We compared a per-question average against a per-article average and briefly concluded that a change had caused a regression. The configuration had in fact reproduced its baseline exactly. Statistics computed different ways are not comparable, and labelling them clearly is the only defence.
We read a report generated before a fix as though it came after, and drew a conclusion from stale data. Two evaluation reports are indistinguishable from their contents alone. Every report now carries the engine settings that produced it, stamped in at generation time.
We counted citation spans and reported them as distinct provisions, overstating how many separate articles an answer cited by roughly half. Answers now record both counts separately.
Where accuracy stands
Two question sets are in play, and their numbers must not be mixed.
On the original researcher sets, required-article recall moved from 0.605 to 0.805 between June and August 2026, and the proportion of questions where every required article was cited moved from 0.188 to 0.487. Exact-match verification stayed at 1.000 throughout.
The second set is harder by design: 80 questions built to include compound and multi-part reasoning, carrying 269 required articles. Measured on 14 August 2026, the configuration serving users today cites 207 of those 269, or 0.770. A candidate configuration tested the same day cites 217, or 0.807, and answers 52.5% of questions with every required article present, against 45.0% today.
Three qualifications belong with those figures.
The gain of ten articles was predicted in advance. Before any answer model ran, the search-stage measurement forecast a gain of eleven. Predicting a result and then observing it is stronger evidence than any statistic computed afterwards.
The paired significance tests do not clear the conventional threshold. The candidate configuration was better on 14 questions and worse on 5, giving a two-sided probability of 0.064 under chance; on the all-articles-present measure the corresponding probability is 0.109. At 80 questions the direction is consistent and the size is not yet certain.
None of these figures is a certified result. The sealed question set has not been run. Every number above comes from sets the engine has been tuned against, which is the correct use of those sets and also the reason they overstate. The certification run is the next measurement scheduled.
What Compass declines to answer
Refusal is a designed behaviour with a measured setting, not a fallback.
Compass scores its own confidence in an answer from how much of the answer is verifiably grounded, how completely the cited provisions cover the question, and how many separate provisions carry the answer. A configurable strictness level then decides how much confidence is required before the system will answer at all. Ten levels exist, and the behaviour of each was measured across the full question bank rather than assumed. The deployed level was chosen from that measurement.
The direction of the trade is worth stating plainly. Higher strictness means more questions declined, including some the system would have answered correctly. We accept that cost, because a confident wrong answer about a regulatory obligation is more expensive than a refusal.
Compass carries no human legal review stage. It is a screening tool that shows a reader the primary law relevant to a question, with citations they can check themselves in the Official Journal. It does not give legal advice and does not replace a qualified adviser.
