The usual advice for RAG retrieval is to run keyword search (BM25) next to vector search and fuse the two rankings. It is called hybrid search, and the claim is that it beats either method alone. I built it into the EU AI Act assistant, a portfolio project of mine and not a client engagement, and measured it against 36 questions I had labelled by hand. Recall@5 fell from 0.86 to 0.58. Recall@20 stayed at 0.92.
This article gives the numbers, what I tried to repair it, what I shipped and why, and a procedure to run on your own documents before you believe any best practice, including this one.
How I measure retrieval
The eval is a file of 36 questions, each paired with the ids of the chunks that genuinely answer it. Twenty-eight are phrased in the Act’s own vocabulary and eight as a layperson would ask. Recall@k is the share of questions with a correct chunk in the top k results. I report recall@5 because the assistant gives the model five passages, so what sits outside the top five might as well not exist. Mean reciprocal rank (MRR) is the second number: how high the first correct chunk sits.
A few details earned their place:
- Gold is keyed on chunk id, not on the citation label. “Article 6” is a prefix of “Article 60”, and matching labels would need prefix logic.
- The runner checks that every gold id exists before it spends anything. A typo in the expected list is indistinguishable from a retrieval failure once the numbers print. It just looks like a miss.
- There is a strict score beside the loose one. The loose score counts a question as found if any expected chunk is in the top five. Twenty of the 36 questions expect more than one chunk, so I also report the share where all of them appear.
- The set is split into two groups. The 28 statutory questions already scored 1.00 at recall@5 with vector search alone, which leaves nothing for a later change to move. I added the eight lay questions so the eval could tell two retrievers apart.
- No threshold fails the build. A quality bar encoded as a failing test is a bar people learn to route around. The obligation is to report the number, including when it gets worse.
Thirty-six questions is small. One question is 2.8 points of recall, and one lay question is 12.5 points of that group. Read everything below with that in mind.
The baseline was moved by chunking, not by search
Before BM25, vector search alone scored 0.83 at recall@5. Looking at the misses showed a chunking fault, not a search one. Annex III, the list of high-risk uses, was a single 7,291-character chunk covering eight unrelated domains. One vector averaged over eight domains matched none of them: on “What makes an AI system high-risk?” it scored a cosine similarity of 0.528 and sat outside the top 50, against 0.679 for the top hit.
Splitting the annex at its numbered areas took the corpus from 886 to 901 chunks, recall@5 from 0.83 to 0.86, and the lay group from 0.25 to 0.38. That was the cheapest gain in the project, and it came from reading failures rather than adding a component. The 0.86 is the baseline hybrid had to beat.
What hybrid did to the numbers
I had put BM25 off until a recall number existed to move, on the rule that a fusion weight chosen before there is a number is chosen by taste. The reasons to want it were real. Legal text is full of exact tokens like “Annex III” that embeddings blur together. Measured on its own over the eight lay questions, BM25 put a correct chunk in the top five once, against three for vector search. On its own it won one question outright: Annex III point 4 contains “analyse and filter job applications” verbatim, so lexical matching ranks it first for the question about sifting job applications.
I fused the two with Reciprocal Rank Fusion, unweighted. Each retriever contributes 1 / (60 + rank) for every chunk it returns, 50 candidates deep, and the sums decide the order. Rank-based fusion needs no normalising of a bounded cosine score against an unbounded BM25 score, and a chunk one retriever buried at rank 724 of 901 contributes almost nothing instead of poisoning the list.
| At k = 5 | Vector only | Hybrid |
|---|---|---|
| Any expected chunk found | 0.86 | 0.58 |
| All expected chunks found | 0.64 | 0.39 |
| MRR@5 | 0.678 | 0.490 |
| Statutory questions (28), any | 1.00 | 0.68 |
| Lay questions (8), any | 0.38 | 0.25 |
Across depths, the damage sits at the top of the list:
| Recall, any expected chunk | k = 1 | k = 3 | k = 5 | k = 10 | k = 20 |
|---|---|---|---|---|---|
| Vector only | 0.56 | 0.78 | 0.86 | 0.92 | 0.92 |
| Hybrid | 0.44 | 0.50 | 0.58 | 0.75 | 0.92 |
Why fusion made it worse
I checked first that this was not a bug. With BM25 returning nothing, my fusion function reproduces the vector ordering exactly. The mechanism is the equal vote.
Equal weighting asserts that both rankings are worth trusting. On this corpus that assertion is false. A chunk that both lists return, even at rank 50 in each, outscores a chunk that only vector search found, even at rank 1. A correct chunk that BM25 never returned is therefore easy to push out of the top five.
The question “Which AI practices are banned outright under the AI Act?” shows it. The correct chunk, Article 5(1), was vector rank 1. After fusion it was absent from the top five, displaced by Article 5(8) (vector rank 2, BM25 rank 3), recital 29 (vector 5, BM25 1) and Article 5(1b) (vector 3, BM25 7). All three are on topic. They are just not the answer.
Recall at depth is unchanged on the loose measure: 0.92 at k = 20 for both methods. The correct chunks are still retrieved, so fusion damaged their position inside the cut, which makes this a ranking result rather than a retrieval one. Since the model reads only five passages, position inside the cut is what the user experiences.
What I tried to fix it
I swept the fusion rule offline over a single pass of embeddings, changing the BM25 weight, its depth and its tokeniser.
| Variant | Any @5 | All @5 | Statutory | Lay |
|---|---|---|---|---|
| Vector only | 0.86 | 0.64 | 1.00 | 0.38 |
| Plain BM25, weight 1.0, depth 50 (shipped) | 0.58 | 0.39 | 0.68 | 0.25 |
| Plain, weight 0.1 | 0.83 | 0.64 | 0.96 | 0.38 |
| Stopwords removed, weight 1.0 | 0.64 | 0.42 | 0.71 | 0.38 |
| Stopwords removed, weight 0.25 | 0.78 | 0.61 | 0.89 | 0.38 |
| Stopwords removed, weight 1.0, depth 3 | 0.86 | 0.61 | 1.00 | 0.38 |
| Body text only, weight 1.0 | 0.58 | 0.36 | 0.68 | 0.25 |
Two patterns stand out. No configuration improved the lay group, which is the group BM25 was built to help: it holds at 0.38 everywhere except the shipped version and the one that indexes only the body text, which fall to 0.25. And every row that reached parity with vector search did it by shrinking BM25’s influence, through a weight of 0.1 or a depth of 3, so that it barely voted.
What I shipped, and why
I shipped the plain, unweighted version, at 0.58. That needs explaining, because the table says vector-only retrieves better on this eval at every depth up to 10 and ties at 20. The reasoning in my decision log is about what I was willing to do to the number. It does not claim hybrid is better.
- The case for hybrid was a prediction, and the eval did not confirm it. My reasoning in the code was that legal text needs both retrievers: embeddings blur exact tokens like “Annex III”, and a word counter cannot cope with a question that avoids the Act’s vocabulary. The one case BM25 won on its own was the job-applications question, and it won because one annex point happens to contain the question’s key words verbatim. That is a thin reason, and I should say so: it is one question out of eight lay-phrased ones, and a quirk of that wording tells me little about how the two retrievers compare everywhere else.
- No weight chosen to hide a regression. The rule that deferred BM25 says a weight picked before there is a number is picked by taste. It applies equally to a weight picked afterwards to make the number recover.
- The one real gain was not taken. Removing stopwords lifted 0.58 to 0.64. Shipping it would put a stopword list in the retrieval path to improve a figure that still loses to vector-only. That is tuning toward parity rather than toward a result.
- Turning BM25 down is the same as not having it. The rows that match vector search do it by silencing the lexical side, which leaves a second index to maintain for no effect.
I am not arguing that hybrid is the right call for this corpus. The numbers do not support it, and the README reports the lower top-5 ranking beside the table. What the decision log does is record the cost in plain words, “it ships, and it costs recall”, so that nobody reading the 0.58 later has to wonder whether I noticed.
Test your own retrieval first
The lesson is not that hybrid search is bad. On a corpus where users search for product codes, error strings or names, a lexical retriever may earn its place. The lesson is that I could not have known which case I was in without measuring, and neither can you. Before adding a retrieval component:
- Write 30 to 40 real questions. If your users do not talk like your documents, include both phrasings.
- Label each with the ids of the chunks that answer it. Check the ids exist before you run anything.
- Score vector search alone first. Record recall at 1, 3, 5, 10 and 20, MRR, and a strict score if answers need several chunks.
- Change one thing, run the same eval, and write down before and after, including when it is worse.
- Read the misses. When a question misses, my runner prints what came back instead and where each retriever ranked it, such as
[d2 b3]for vector rank 2 and BM25 rank 3. A correct chunk just outside the cut is a ranking problem. One absent entirely is an embedding problem. An aggregate number cannot tell them apart. - Keep a group of questions that can fail. If everything scores 1.00, the eval cannot discriminate.
One limit on all of this: these results come from 36 questions, one regulation and one embedding model. I trust the method more than the specific numbers.
If you are choosing how a document assistant should retrieve, my RAG development service starts with an eval like this one on your own questions, and you can send me a brief to begin. The measurements for this project are in the EU AI Act assistant case study, and the decision log is in the public repository.