---
title: "What Retrieval Still Hasn't Decided"
date: "2026-09-20"
excerpt: "Split post-retrieval into rerank, filter, and compress, three different judgments measured on live runs. Dedup cut 0.2% of characters, so I didn't ship it."
ogImage: "/images/blog/what-retrieval-still-hasnt-decided.jpg"
tags: ["Reranking", "RAG", "Evidence Filtering", "Semantic Deduplication", "CLI"]
---

Started building a reranker CLI on top of Jev for the obvious reason: get the relevant documents to the top. Ended up with three modes in [jev-reranker](https://github.com/shinpr/jev-reranker), and one I decided not to ship.

The split appeared while I was building. There are several different things I want decided about a set of search results. Is this document related to the question? Does it contain evidence I can answer from? Which part of it should I pass on? Given what I have already selected, does this add anything?

All four happen after retrieval, and none of them is the same judgment. So alongside reranking I tried evidence filtering, sentence-level compression and semantic deduplication against the live API. The CLI ships the first three. Dedup did not make it.

This article covers what each mode judges and when it is worth using. The numbers are from exploratory runs in September 2026. The selection and compression studies were pinned to `jev-1.13.0`; the retrieval comparison in section 3 ran against `jev-latest`, so its numbers and theirs are not directly comparable.

## 1. What Is Left to Decide After Retrieval

[The previous article](/blog/local-rag-agentic-coding) was about chunking and hybrid search in mcp-local-rag. This one is about the stage after that: turning the candidates you retrieved into the context an LLM actually receives.

```text
Question → Search → Candidate set → Select / transform → Answering LLM
```

Say you ask "how long are logs retained?" and get these back. The wording is made up to show the differences:

```text
A: This section explains the log retention period.
B: Logs are retained for 30 days.
C: Logs are retained for 30 days. Audited logs are kept for one year.
D: The standard retention period for logs is 30 days.
```

A is on topic and contains no number. B has the answer. C has the answer plus the exception. D is close to a paraphrase of B.

| Mode | What it judges | What changes |
| --- | --- | --- |
| rerank | how related the document is to the question | the order of the candidates |
| filter | whether the document holds concrete evidence to answer from | which candidates survive |
| compress | which sentences have to stay for the answer | how much of the document you pass |
| dedup | whether a candidate adds evidence the selected ones lack | redundancy in the candidate set |

Rank matters. Rank alone cannot express "none of these answers the question," because every candidate set has a first place.

## 2. Jev Takes the Condition You Want Judged

The Jev API takes the material to evaluate as `state` and the question as `questions`, and returns a structured judgment. I use the `noul` format, which answers a yes/no question with the probability of yes. See the [TypeSafe API reference](https://docs.typesafe.ai/api).

The CLI puts the question and the document into state and writes the condition to be judged into instructions: for rerank, whether the document is related to the question; for filter, whether it holds concrete evidence usable in an answer. Sorting and extraction happen in the CLI.

What matters is knowing what the number between 0 and 1 is a judgment about. A relevance score of 0.9 does not mean the answer is 90% likely to be correct. Change the question being asked and the same body of text produces a value that means something else.

Any Jev path sends the question and the selected text and context to an external API. If your reason for running a local RAG is that nothing leaves the machine, this post-processing is a separate decision from the retrieval one.

## 3. What a Distance Ranking Actually Returns

Before splitting rerank off from filter I wanted to know what a plain rerank was actually fixing. So I ran mcp-local-rag over 59 arXiv papers (27,563 chunks, `all-mpnet-base-v2`, keyword boost off) with 36 queries, pulled 20 candidates each and compared the top five under three configurations: the retriever alone, a Jev rerank, and a fusion of Jev's probability with the retriever's distance (`score / (1 + p * weight)`, weight 1.0). All three read the same candidates, so the reranker was the only variable.

| Configuration | Top result differs from the retriever | Items replaced in the top five, mean |
| --- | ---: | ---: |
| retriever only | baseline | baseline |
| Jev rerank | 31 of 36 | 2.92 |
| fusion with distance | 8 of 36 | 1.08 |

The order moving is not the same as the results getting better, so two models, Claude and Codex, judged the result sets independently and counted, per query, how many of the five could support an answer. They read different subsets, so these samples are smaller than the counts above.

| Evaluator | Queries read | retriever only | Jev rerank | fusion |
| --- | ---: | ---: | ---: | ---: |
| Codex | 18 | 2.39 | 3.33 | 2.83 |
| Claude | 6 | 2.00 | 3.17 | 2.67 |

Both arrived at the same mechanism. The retriever's distance rewards paper titles, section headings, figure captions and bibliography lines, because those are lexically almost pure query terms while carrying no information. Rejecting them is most of what the reranker does.

The clearest evidence is that the promoted chunks carry worse distances than the ones they displace. On a question about chunk size, the results the reranker moved up scored 0.25 to 0.32, while the paper's own title and a truncated citation it pushed out scored 0.15 to 0.26. No closer chunk carried an answer, so what the reranker did was throw out text that was close and empty.

Fusion barely did anything at weight 1.0. Jev returned at least 0.7 for 89% of the reranked results, with a median of 0.92, because the 20 candidates had already been filtered by the retriever and were mostly plausible. That puts the divisor `1 + p` between roughly 1.70 and 1.97, which is small against the spread in the distances, so distance keeps deciding the order.

Whatever the mode, the gain is conditional on the evidence being in the candidate set. On queries whose candidates were all bibliography fragments and clipped lines, none of the three configurations returned anything usable. And on a question the corpus does not cover at all, passport renewal in a corpus of retrieval papers, every configuration still returned five chunks, even though Jev scored them 0.01 to 0.03 against 0.92 to 0.97 on a question the corpus does answer. Reranking earns the call where a small top-k reaches a model and the candidates plausibly hold an answer.

These are LLM judgments, not human annotation, and the useful counts are each evaluator's own. The aggregates in the first table were produced independently and matched. The two disagreed on one query, where the rerank returned four of its five chunks from a single paper and they split on whether losing that spread of sources is a defect. Latency and money were not measured.

## 4. rerank: What to Read First

rerank scores each document for relevance and sorts descending. The distance or score from your retriever stays in the output and takes no part in the sort. Ties keep their input order.

```sh
search-command --json \
  | jev-reranker --query 'How long are logs retained?' --top 5
```

Replace `search-command` with whatever local search command emits a JSON array. If the body lives under a key other than `text`, name it with `--text-field`.

This is the mode for when the information you need is somewhere in the candidate set but not at the top of it. You change what the answering LLM reads first without running retrieval again.

It cannot add information that is not in the set. A heading or an introductory sentence on the right topic can still come out on top, the way A does above, because the relevance judgment never asked for a concrete answer.

The first implementation also blended the retriever's score with Jev's judgment, which is the fusion row above. It moved eight top results out of 36. In a general-purpose CLI the scale and the direction of that incoming score differ for every caller anyway, so before adding fusion variants I split apart what the user is actually trying to change. The default today is a Jev rerank.

## 5. filter: Is There Evidence to Answer With

filter judges whether the document contains concrete information for answering the question. It does not have to answer all of it. Part of a compound question counts, and so does a necessary condition or the text supporting an exception.

```sh
search-command --json \
  | jev-reranker --query 'How long are logs retained?' --mode filter
```

Telling A ("this section explains the retention period") apart from B ("retained for 30 days") is where this mode started. Raising a relevance threshold does not change the condition being judged.

The current implementation keeps candidates with `evidenceScore >= 0.5` in input order. The threshold is configurable, and 0.5 is a starting point to try against your own data. With nothing above it the output is an empty array, and nothing is backfilled to reach a count.

### Why selection and ordering came apart

I expected evidence filtering to be a stricter rerank. The labelling run said otherwise. I labelled 220 candidates across 11 deliberately hard queries for whether they contained evidence, then compared variants. Codex produced the labels before seeing the new Jev scores, so they are not multi-annotator ground truth. Five candidates per query at most, filter threshold 0.5.

| Prototype variant | Items returned | Candidates with clear evidence |
| --- | ---: | ---: |
| rerank by relevance | 55 | 22 |
| evidence filter, input order kept | 39 | 20 |
| sort by evidence score, then filter | 39 | 29 |

The third row is the strongest variant: the same 39 items out, 29 of them carrying evidence, against 22 out of 55 for rerank alone. It is not the one I shipped. Sorting by evidence score puts the evidence judgment in charge of the order as well as the membership, which collapses the two things I had just finished separating.

What filter does is the second row, and its evidence count sits below plain rerank. Keeping input order dropped the useless candidates and took some of the reachable evidence with them. So adding filter does not by itself get you more evidence: the ordering goal and the retention condition both move the result.

Input order is what lets filter sit behind a retriever whose ranking you want respected. If you want relevance order and then selection, pipe rerank into filter; the surviving order is still relevance order.

Watch where `--top` goes. Narrow to five in rerank and filter only ever sees those five. To get up to five after selection, sort the full set and put `--top 5` on the filter at the end.

One more caveat: when the search is for papers themselves, a title or a citation line can be the result you wanted. Demanding evidence in the body is right when you are assembling material to answer from, and wrong when you are looking for the documents themselves.

## 6. compress: Keep the Sentences That Hold the Evidence Up

In a long document only part of it is usually relevant to the question. compress splits the body into sentences or lines, judges each one for whether it has to stay, joins the survivors in original order and puts the result in `compressedText`.

Each judgment also receives the full parent document. Looking at one sentence alone makes conditions and pronoun referents hard to resolve.

Keep only "logs are retained for 30 days" from C and the audit exception is gone. Every word can be verbatim and the extraction still changes what the answer means. So the instruction asks for conditions, exceptions, definitions and referents alongside the direct answer.

```json
{
  "text": "Logs are retained for 30 days. Audited logs are kept for one year. The admin console is blue.",
  "compressedText": "Logs are retained for 30 days.\nAudited logs are kept for one year."
}
```

That is the intended output. Nothing asks Jev to rewrite prose; the CLI extracts the selected units of the original, and it can select the wrong ones.

On 40 answerable questions from [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/), the prototype took 31,440 characters down to 8,290, and the published answer span survived in 38 of them. That is roughly a 74% character reduction. It is not a token count and not an end-answer accuracy figure, and it does not establish that every surrounding condition survived.

Of the two failures, one scored a necessary sentence too low and dropped it. The other was a splitting problem: a period after a person's initial split the sentence and the full name did not survive. The current implementation keeps initials and some abbreviations intact, but these 40 results are prototype numbers.

Keeping the original text and its source, with the extract in its own field, is what makes that kind of loss visible. Downstream you have to use `compressedText`. Send the original `text` along with it and the context does not get any shorter. A document with no unit worth keeping drops out of the output.

Compression costs something too. One request judges 30 units by default, and a document spread across several batches has its full text sent again with each one, so a 100-unit document is four batches. Weigh the reduction downstream against the input and the latency of the selection itself. Chunks that were short to begin with have little to give up.

## 7. dedup: Similar Is Not the Same as Droppable

The question for dedup is whether dropping a candidate loses information needed to answer. One of B and D may be enough. C carries the audit exception, and dropping it for being on the same topic takes the exception with it.

That judgment has a direction. C can cover what B says; B cannot cover C's exception. So I tried handing Jev pairs of a candidate and an already-selected document and asking whether keeping this one loses no evidence if that one goes.

On the first synthetic set it kept the needed facts out of a pile of paraphrases and cut the volume returned. Reducing volume on data where I planted the duplicates is not enough to justify shipping a mode.

The follow-up compared plain rerank against deduplication over 23 queries, selecting up to five from twelve candidates each. At a duplication threshold of 0.9 both returned 115 items in total, and the body text moved from 60,819 characters to 60,701. A 0.2% reduction. Another candidate fills the slot a duplicate vacates, so removing duplicates does not translate into returning less.

Widening to 40 candidates barely changed the character count either. Lowering the threshold produced candidate sets that had lost conditions and exceptions. Aim at the reduction figure alone and that loss goes unnoticed.

Comparing candidates against each other also costs more judgments than scoring each one independently. All pairs is `n(n-1)/2` for n candidates, so 190 at twenty, and doubling that if you judge each direction. Comparing only against what is already selected still lets each decision change the next comparison.

On this data I could not find a gain worth the extra judgments, so the dedup mode is not implemented. Data where reposts and paraphrases dominate the top of retrieval would be worth another look. What I would want measured there is the items and characters removed, plus the necessary facts left standing and the exceptions lost.

## 8. Where the CLI Stops

jev-reranker processes a JSON array handed to it by a retriever. It does not run retrieval again and it does not produce the final answer.

```text
Search CLI → jev-reranker → you / the answering LLM
```

rerank is the default, with filter and compress as options. There is no assumption that you run all three in sequence. Order at the top is a rerank problem; candidates with no evidence are a filter problem; passing part of a long body is a compress problem.

Calling it from inside an MCP server needs a contract that allows the result to be transformed. Make "return the same count with the same identifiers" the condition for accepting output and you throw away the empty array that filter legitimately produced, along with any body text the command rewrote. So mcp-local-rag accepts output that conforms to the schema of its own search results. Guaranteeing the format and deciding what to select are two different jobs.

Evidence surviving is also not the same as the whole question being answerable. Retention evidence is no help if the question also asked where logs are stored. Whether to search again, or to answer while naming what is missing, belongs to the caller that can see the whole question and everything retrieved.

The implementation and usage are in the [jev-reranker README](https://github.com/shinpr/jev-reranker#readme). To try it on your own RAG, put the raw search results and the post-processed results side by side for the same question and look at which evidence stayed and which conditions fell out. What that comparison tells you is which decision you were trying to make after retrieval. The mode follows from the decision.

Code: https://github.com/shinpr/jev-reranker

## References

- Building a Local RAG for Agentic Coding: /blog/local-rag-agentic-coding
- jev-reranker: https://github.com/shinpr/jev-reranker
- mcp-local-rag: https://github.com/shinpr/mcp-local-rag
- TypeSafe API Reference: https://docs.typesafe.ai/api
- SQuAD 2.0: https://rajpurkar.github.io/SQuAD-explorer/