EU AI Act compliance assistant
A chat assistant that answers questions about the EU AI Act in a few sentences, each one citing the provision it comes from. When the Act does not support an answer, it says so plainly and stops.
portfolio projectThis is a portfolio project. I designed and built it myself, to the standard I would hold client work to. No client commissioned it, and the code is public.
- What it is
- Citation-checked question answering over one document set, the EU AI Act (Regulation (EU) 2024/1689). Every sentence carries a citation, or the answer becomes a refusal.
- Type
- Portfolio project, built by Filip. Not a client engagement.
- Stack
- Python 3.12, LangGraph, Chroma, BM25, OpenAI models, Streamlit.
- Status
- The code is public on GitHub. It runs locally with your own OpenAI key. There is no hosted demo.
- What to look at
- The two result tables on this page, then the repository’s DECISIONS.md and evals folder.
0.86recall@5, vector only
A correct chunk in the top five, on 36 hand-labelled questions.
source: DECISIONS.md
0.58recall@5, hybrid as shipped
Worse than vector alone on this corpus. The page shows why.
source: DECISIONS.md
The problem
A chatbot over your documents answers fluently whether or not the documents support the answer. For a compliance officer or a product team that is the wrong failure. They do not need a paraphrase of what the law probably says. They need to know where it says something, so they can read it and act on it.
The same holds for any business’s own policies, contracts and manuals. An answer you cannot trace to a passage is not one you can use, and a confident answer with no source is worse than none.
Most retrieval chatbots ask the model to cite its sources and trust that it did. I wanted the opposite: let the model write, then check its work in code, and treat a refusal as a normal, logged result.
What I built
You ask a question. You get an answer of four sentences at most. Every sentence ends with a citation, such as Article 50(1) or Annex III, point 4, that links to that provision on EUR-Lex. Or you get a plain refusal.
The difference is the check that runs after the model answers. Every citation has to trace to a provision that was actually retrieved, every sentence has to carry one, and the answer has to fit in four sentences. If any check fails, the answer is replaced by a refusal, and the log records which check it was.
How it works
The drawing shows the route one question takes. Grey is the system working, orange is a person asking and green is a state reached. There are two ways out that are not an answer, and both are refusals.
- The corpus. Regulation (EU) 2024/1689 from EUR-Lex. Articles and annexes come from the consolidated text in force on 2026-07-27. Recitals come from the Official Journal version, because the consolidated version drops them (I counted 180 against none). The sources and their sha256 hashes are recorded in the repository.
- Chunking on legal structure. 901 chunks: 675 article paragraphs or points, 46 annex sections and 180 recitals. Each carries its exact citation label, title, chapter and source URL.
- Hybrid retrieval. OpenAI
text-embedding-3-smallvectors in Chroma, plus BM25 for exact legal terms such as “Annex III”, merged by Reciprocal Rank Fusion. Its measured cost is in the results below. - A three-step LangGraph pipeline. Rewrite (follow-ups only), retrieve, generate. There is no agent loop and no checkpointer, so a turn makes a known number of model calls: at most two, one rewrite on a follow-up and one answer, and none when the question is refused before the call.
A person actsThe system worksState reached
What an answer looks like
There are no screenshots of this project, so here are the two outcomes set as plates. Both questions are among the three suggested in the app. The provision text is real, copied from the chunk the assistant retrieves. The wording of the first answer is mine, written in the format the prompt enforces. It is not a captured session.
Do I have to tell customers they are talking to a chatbot?
The obligation in Article 50(1) falls on providers: they must design AI systems that interact directly with people so that the people are told they are dealing with an AI system, unless that is obvious to a reasonably well-informed, observant person in the circumstances [Article 50(1)].
Sources
Article 50(1): Transparency obligations for providers and deployers of certain AI systems
Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that the natural persons concerned are informed that they are interacting with an AI system, unless this is obvious from the point of view of a natural person who is reasonably well-informed, observant and circumspect, taking into account the circumstances and the context of use. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate or prosecute criminal offences, subject to appropriate safeguards for the rights and freedoms of third parties, unless those systems are available for the public to report a criminal offence.
The answer’s wording is mine, written to show the format. The provision text is real.
What the answer leaves out. It states the obligation and the exception for what is obvious. It omits the carve-out for AI systems authorised by law to detect or prosecute criminal offences. The four-sentence limit makes an answer choose what to leave out, and the link lets a reader check the rest.
In the app, each citation links to its provision on EUR-Lex in the same way.
What does the GDPR say about consent?
The retrieved provisions of the AI Act do not support an answer to that question. Rather than guess, I am not answering it.
The refusal text is copied from the code.
No sources under a refusal. A refusal ships none. A refusal printed above five retrieved provisions reads like a cited answer to anyone skimming.
The GDPR is outside this corpus on purpose. Questions about neighbouring law score 0.47 to 0.54 against the Act, which is above the free filter, so they reach the model, which declines. That costs one call, and it is a limitation listed below.
The six reasons for a refusal
Every refusal is logged with the check it failed, so a model declining can be told apart from a caught fabrication.
| Reason | What happened | Model called |
|---|---|---|
no_hits | Retrieval returned nothing | No |
below_floor | The best match scored under 0.18 cosine similarity, so the question is off-topic | No |
unsupported_citation | The answer cites a provision that was not retrieved, or a point that is not in the retrieved text | Yes |
no_citation | No citation at all, which includes the model declining in its own words | Yes |
uncited_sentence | At least one sentence has no citation | Yes |
too_long | More than four sentences | Yes |
How I planned the build
I planned the build as two days: a thin path that works from end to end first, then measurement and improvement. It took longer than that. The repository’s commits run from 19 to 24 September, so read the plan as an order of work, not as a record of time.
Day one: a thin path, end to end
- Skeleton. Config, pinned dependencies, the corpus.
- Chunker. Split by article and add metadata.
- Index and command-line retrieval. Embed, persist, print the top five.
- Generation. Cite the article, or refuse.
- Streamlit shell. Chat, citations, disclaimer.
- Gate. A question goes in and a cited answer comes out.
Day two: measure, then improve
- Eval set. Labelled questions, recall@5.
- Hybrid search. BM25 plus vectors, then re-run the eval.
- Query rewriting. Standalone follow-ups, then re-run the eval.
- Hardening. Injection, off-topic questions, logging.
- Ship. Secrets, deploy, README.
The point of the order is that nothing is tuned until something works, and every change after that has a number beside it.
Six decisions, and the evidence for each
Each decision answers a way document assistants usually go wrong. The note beside it is the evidence, and every figure is in the repository’s DECISIONS.md.
Chunks follow the law’s structure, not a character count
An article is split at its numbered paragraphs, an annex at its sections or numbered areas, and a recital stays whole. The lead-in of a list, such as “The following AI practices shall be prohibited:”, rides along on every piece of it. Without it a chunk can read as the opposite of what the law says.
Citations are checked in code, not requested in a prompt
After the model answers, every citation must trace to a retrieved provision, every sentence must carry one, and there can be at most four sentences. A fabricated or misremembered article turns the whole answer into a refusal. The matching is strict on boundaries: Article 5(1)(a) is accepted from a retrieved Article 5(1) only if point (a) appears in its text, and “Article 6” is not satisfied by “Article 60(1)”.
Refusal is a normal result, and the log says why
An off-topic question should cost nothing and answer nothing. A relevance floor refuses it before any paid call. The floor is 0.18, the best cosine similarity among the retrieved provisions, and I chose it from a sweep, not by taste.
Retrieved text and visitor text are data, never instructions
Provisions reach the model only in the user message. The system prompt is a constant with nothing interpolated into it. Every line a visitor types is quoted with a leading “> ”, so a question cannot pose as a provision of the Act.
Spend is bounded where the call is made
A turn costs about $0.01 with the default model and five retrieved provisions. A public URL without limits would invite someone to run up the bill, so each session is capped at 10 questions and each client address at 30.
Every query is logged with its outcome
Each log line holds the question, the rewritten question, the retrieved chunk ids with their scores, and the outcome including the refusal reason. The visitor’s address is stored only as a salted hash, and only when a salt is set. Logs are pruned after 30 days by default.
What I measured
Retrieval is scored against 36 hand-labelled questions. 28 are phrased in the Act’s own vocabulary and 8 as a layperson would ask. The first table is the share of questions with a correct chunk in the top five results.
| Measure | Vector search only | Hybrid, as shipped |
|---|---|---|
| A correct chunk in the top 5 | 0.86 | 0.58 |
| Every expected chunk in the top 5 | 0.64 | 0.39 |
| A correct chunk in the top 20 | 0.92 | 0.92 |
| MRR@5 | 0.678 | 0.490 |
| Top 5, the 28 questions in the Act’s vocabulary | 1.00 | 0.68 |
| Top 5, the 8 lay-phrased questions | 0.38 | 0.25 |
Adding BM25 made the top five worse on this corpus and left recall at depth unchanged. The right provisions are still found, just lower down. I recorded that cost and I did not tune the shipped retriever to look better.
The cause is the equal vote. Reciprocal Rank Fusion, unweighted, treats the two rankings as equally trustworthy, and on this corpus they are not. A chunk the vector search ranks first, which BM25 never returned, scores 1/61, or 0.0164. A chunk ranked second by vectors and third by BM25 scores 1/62 + 1/63, or 0.0320.
I swept the alternatives. The settings that reach parity with vector search do it by turning BM25 down until it does nothing. Removing stopwords lifted 0.58 to 0.64, and I did not take it, because that is tuning toward parity. No setting improved the lay-phrased group, which stays at 0.38.
The repository does not record a reason for keeping hybrid over vector-only, so I will not invent one here. On your documents I would run this comparison on your own questions before choosing.
Follow-up questions
A follow-up such as “and for small companies?” means nothing alone, so the assistant rewrites it as a standalone question before searching. This has its own eval, on 11 follow-ups with the earlier turns fixed. Each is retrieved three ways: as typed, as rewritten, and as a standalone question I wrote by hand, which is the ceiling.
| Measure | As typed | Rewritten | Standalone, the ceiling |
|---|---|---|---|
| A correct chunk in the top 5 | 0.55 | 0.73 | 1.00 |
| Every expected chunk in the top 5 | 0.45 | 0.73 | 0.82 |
| MRR@5 | 0.417 | 0.667 and 0.621 in two runs | 0.705 |
The result was stable across two runs at temperature 1.0: the same eight items hit and the same three missed, and only MRR moved. The rewrite lost no hit, and it rescued one the control missed.
The three misses are instructive. Two are vocabulary: “small companies” was carried over where the Act says “SMEs”, and a lay question kept the user’s register. The third is a question that retrieves differently depending on its exact wording. The prompt resolves pronouns and ellipsis, but it does not translate lay vocabulary into the Act’s, and I have not changed it. The control is also weaker than I intended: only five of the eleven items separate a rewrite from no rewrite.
Tests
There are 245 tests, and they run offline with no API key. They use real EUR-Lex HTML fixtures with a fake embedder and a fake generator, and I checked them with outbound sockets blocked and the key unset. Writing the chunker tests first found a real bug: a citation reading “Article 105(‘5)” that borrowed another directive’s numbering.
What it does not do
- It checks that a citation was retrieved, not that the sentence is true to it. A real citation attached to invented prose passes. “Roses are red [Article 5(1)].” passes. Catching that needs a second model call as a judge, which I declined on cost.
- Neighbouring law gets past the free filter. GDPR and Digital Services Act questions score 0.47 to 0.54, so they reach the model, which refuses. Each costs one call.
- Lay phrasing retrieves worse. It is the weakest group in the eval, and the rewrite misses 3 of the 11 follow-ups.
- The eval sets are small. 36 questions, of which only 8 are lay-phrased, and 11 follow-ups. They show direction, not precision.
- Scope is narrow. English only, one regulation and no user accounts.
- The per-address cap is not a security control. It is easy to spoof and a restart clears it.
- It is not legal advice. Verify anything you act on against the Official Journal.
What this means for your business
If you have documents your people keep asking questions about, such as policies, contracts or manuals, this is the shape of what I would build. I would chunk them on their own structure, cite and check the citation in code, and have the assistant refuse when the documents do not support an answer. Before and after every change I would measure it on questions your people really ask. To start I would need your documents and a sample of those questions.
- RAG development
- A RAG assistant starts from €4,000. It cites its sources, refuses when they do not support an answer, and is measured on your real questions.
- LLM hardening audit
- Spend caps, quoting of untrusted text, logging and evals are what I look for in an existing LLM feature. Audits start from €1,500.
- AI agents and workflow automation
- For an agent that waits for a person’s approval before it writes anywhere, see the coordinator, my other portfolio project.