Your AI Agent May Be Reading the Wrong Pages. NVIDIA's Reranker Cut Tokens About 14%

If your AI agent gives a wrong answer or takes forever to finish, where do you look first? Most of us look at the main model, because that is the part that is easiest to swap. So we try a bigger model, or a newer one, or we rewrite the prompt again. But there is another step inside the agent that decides a lot of the bill, and that is search, the step where the agent goes off and collects the documents it needs before it can say anything useful.

NVIDIA released its Nemotron 3.5 retrieval models on October 21, 2026, and there is one number in their announcement on Hugging Face that I think deserves a slower read. In NVIDIA's own tests, adding the new reranker to an agent's search lowered its token cost per query by roughly 14% when it was paired with the 8B embedding model, and by roughly 13% when it was paired with the 1B one. Retrieval accuracy also went up in the same tests. So the agent found better evidence and spent less to reach its answer, and the change that did this was in search, while the model writing the answer stayed the same.

So why would a search step change how many tokens an agent burns through? It makes more sense once you look at what the agent is really doing every time it runs a search.

What the agent is doing every time it searches

NVIDIA describes agentic search as a loop. The agent sends a query through a search tool into a collection of documents, reads the evidence that comes back, and uses what it found to decide its next search. Each document it reads gets added to its context, and that costs tokens in two ways: input tokens to read the document, and output tokens for whatever the agent writes while working out what to do next.

Let's take a case where the evidence is weak. Say your agent is asked for a part's maximum operating temperature, and the number lives in a table on a datasheet that someone scanned into a PDF. The search tool comes back with five documents. Maybe the datasheet is not one of the five, or maybe it is there but ranked fifth, behind four pages that are only loosely related. Either way the agent reads all five, finds that none of them actually has the answer, rewrites its query and goes off to search again.

NVIDIA's article says pretty much this. With weak evidence, the agent ends up searching more times and opening more documents, and it uses more inference tokens to get through the task.

So an agent's tokens go to two things, thinking and reading, and a good share of the reading can go to pages that were never going to help. If the right document turns up near the top earlier, there is less for the agent to get through and fewer reasons to go searching again. My guess is that this is where most of the saving comes from, but NVIDIA does not split the number into how much came from fewer searches and how much came from shorter context, so I can't say that for sure.

The three models in NVIDIA's release

The release has three kinds of models, and each one is aimed at a different step in that loop.

The first is the embedding model. An embedding model takes a piece of text, or in this release even a picture of a document page, and turns it into a long list of numbers, so that two things with a similar meaning get similar numbers. It works a bit like a library where books are grouped by topic, so all the books about engines sit on one shelf even if none of their titles say engine. When you come in with a question, you walk to the right shelf first and only then start looking at individual books.

Nemotron 3.5 Embed has three versions. There is an 8B model, a 1B model distilled from that 8B model, and a second 1B model that is quantized to NVFP4, which is NVIDIA's own number format. All three handle plain text and also images of documents, like tables, charts, infographics and datasheets.

The shelf only gets you near the answer, though. You still have to pick the books up one by one and see which of them actually has what you need, and the second model, the reranker, is there to do that checking. Nemotron 3.5 Rerank 8B works as what is called a cross-encoder. That means it looks at your question and a single candidate document side by side and scores how well this one document answers this one question.

Reading every pair that carefully takes time, so a reranker is slower than an embedding search. That is why the two are usually combined. The embedding model quickly grabs a longer list of possible matches, and the reranker only has to sort that shortlist. In one of NVIDIA's comparisons, for example, the shortlist was the 50 highest-ranked results coming out of the 1B embedding model, and the rerankers put those 50 into a new order.

The third model is Nemotron 3.5 Parse, which is a document parser. NVIDIA lists it at 631M parameters, and the Hugging Face listing rounds that to 0.6B. You give it an image of a page and it gives you back text that a machine can work with, in the same order a person would read it. Tables come out as rows and columns, charts come out as structured data, and the result can come out as Markdown, HTML tables or structured blocks, depending on what you need.

Why parsing matters more than it seems

Parsing is the part of this release that sounds the least exciting, but I think it could be where a lot of real projects actually get stuck. NVIDIA's article lists what goes wrong when you simply pull the embedded text out of a PDF. The reading order can get lost, table values can end up separated from the headers that explain them, and anything that only exists inside a chart can be missed completely. And with scanned documents or PDFs that are really just images, there might not be a usable text layer at all.

Think about the datasheet again. Say the temperature table in it was scanned. A basic text extractor could hand back an empty result. Or it could hand back "85" and "-40" with no column header anywhere near them, so nothing tells you what those numbers even are.

In that case even a great embedding model can't help much, because the text it is searching over was broken before it ever got there. So the agent reads garbage, goes back for another search, and the token count keeps going up. The way I think about it, parsing decides how good the material is at the start, the reranker decides what the agent sees at the end, and the token bill depends on getting both of them right.

What NVIDIA's numbers show, and what they leave out

NVIDIA's 8B embedding model got 61.17 nDCG@10 on ViDoRe v3.1, a benchmark that tests how well a model finds visually rich documents. According to NVIDIA, that was the highest score of any dense retrieval model as of October 20, 2026.

If nDCG@10 is new to you, it basically grades the first ten results a search returns, and it rewards a system more when the right documents show up near the top of those ten instead of near the bottom.

The smaller 1B model came in at 58.38, which is about 95% of the 8B score, so you give up very little for a much smaller model. The parser scored 75.20 on ParseBench in the group of models under 1B parameters, and it handled 10.5 pages per second on a single H100.

The 14% figure came from a different test. NVIDIA gave an agent a search tool, ran it on ViDoRe v3, BRIGHT and BrowseComp-Plus, and added up the input and output tokens over all of the agent's turns. I think counting it this way is fair, because a second or third search shows up in the total, and those extra searches are exactly where wasted tokens would hide.

The article does leave a few gaps, though. It never names the LLM that ran the agent. It also doesn't make clear if the 14% and 13% are averaged over all three benchmarks or taken from a single one. And NVIDIA ran these tests on its own models, so I would treat the numbers as a reason to run your own test, and I wouldn't expect to see exactly 14% in your own system.

There is one more cost I don't think the 14% captures. The reranker is itself an 8B model, and it scores every candidate for every query, so some of the compute you save on the agent's LLM is just being spent in retrieval instead. I think whether that is a good deal depends on two things: what your main model costs per token, and how long the list is that you ask the reranker to sort.

For example, with a pricey frontier model behind the agent, trimming its tokens by that much could cover the reranker without any trouble. With a cheap model behind it, the saving might be smaller than the headline number makes it sound.

What this means for people building agents

So what do I take from all this? I think the idea is useful even if you never touch NVIDIA's models. Most of the time we judge an agent by what comes out at the end, the answer it gave and what it cost, and if either looks bad the first fix we reach for is the main model. But some of that cost is the agent reading, and retrieval is what chooses its reading material, so retrieval deserves its own measurements.

For example, you could check three numbers: in how many tasks the correct document makes it into the first few results, how many times the agent calls search before it finishes a task, and what share of its context is taken up by documents it read and then never used.

If those numbers are poor, I think the cheaper fix is probably on the search side, like parsing scanned files and tables properly, switching to a stronger embedding model, or putting a reranker between search and the agent. But if retrieval looks healthy and the answers are still wrong, then the main model is the more likely problem, and paying for a better one starts to make sense.

Check what your agent reads before you upgrade it

So my question before upgrading the model would be a simple one: what is the agent actually reading on its way to an answer? If you want to try this, NVIDIA has put the models up on Hugging Face, and NVIDIA's write-up goes through the benchmark setup in more detail. NVIDIA describes the models as open and commercially available, but the article does not name a license, so it is worth reading each model card before any of this goes into a product.

Model details and numbers are from NVIDIA's Hugging Face announcement published October 21, 2026.