NVIDIA Generative AI. Open Book: RAG Done Honestly.
Hosts: Shelley and Rob. Cloudadorn Academy.
This episode is narrated using AI voice technology. The content and script are original.

Shelley: I'm Shelley, I ask the questions. Rob builds these things for a living, which is why he gets the answers.

Rob: Morning.

Shelley: I asked one of these things about my own company's refund policy. It answered instantly and it was wrong.

Rob: Confidently wrong.

Shelley: Completely confidently. Where did it get that?

Rob: From everything it read while it was being built. Your refund policy was not in there. So it produced the shape of a refund policy.

Shelley: It made one up.

Rob: It made a plausible one. That is the whole failure. Nothing inside it separates remembering from inventing.

Shelley: Then how do I get my actual policy into it?

Rob: Two ways. One of them is the expensive one, and it is the one people reach for first.

Shelley: Give me both.

Rob: You train the facts into the model itself so they live inside it. Or you hand it the facts at the moment somebody asks.

Shelley: And the second one is cheaper?

Rob: Cheaper, faster to change, and easier to check. Today is about why it still goes wrong constantly.

Shelley: Give me a picture, please.

Rob: Two students sitting the same paper. The first one memorised the textbook. Months of it.

Shelley: And the second?

Rob: The second walks in with the book open on the desk and looks up the page.

Shelley: The first one sounds better prepared.

Rob: The first one is faster and sounds more fluent. Now the policy changes on Monday.

Shelley: The book gets a new page.

Rob: The book gets a new page, and the second student is right on Tuesday morning. The first is confidently reciting last year's rules.

Shelley: And to fix the first student?

Rob: You put them back through the memorising. Slow, expensive, and stale again the moment somebody edits the document.

Shelley: So the open book wins.

Rob: For facts, yes, and somebody measured that rather than asserting it. Training the facts in against handing them over, and handing them over won on both.

Shelley: Won on both what?

Rob: Facts the model had already seen, and facts entirely new to it. The blunter finding underneath is that models struggle to learn new facts that way at all.

Shelley: Does the open book one have a name.

Rob: Retrieval augmented generation. RAG, if you want the short version. Retrieve, then generate.

Shelley: That sounds like one thing.

Rob: It is two, and the second half gets all the credit while the first half causes all the problems.

Shelley: Split it for me.

Rob: Phase one happens before anybody asks anything. You take your documents and you prepare them. Phase two happens in the second after somebody hits enter.

Shelley: Start with phase one.

Rob: You cut every document into pieces. Each piece gets turned into a long list of numbers. The numbers go into a store built for searching lists of numbers.

Shelley: So, three steps.

Rob: Cut it up, turn it into numbers, put it somewhere findable. That is the whole of phase one, and that is where the work is.

Shelley: And phase two?

Rob: Somebody asks a question. You turn the question into a list of numbers the same way. You find the pieces whose numbers sit closest to it.

Shelley: And then what?

Rob: You paste those pieces into the prompt above the question, and let the model answer with them sitting there in front of it.

Shelley: That is it.

Rob: Nothing more. The model is not doing anything clever. It is reading a handout you assembled a moment earlier.

Shelley: So it never learned my policy.

Rob: It never learned it. It is answering with it open in front of it. Nothing about the model changed, which is the point.

Shelley: Which phase goes wrong?

Rob: Usually the one that happened weeks ago and that nobody has looked at since.

Shelley: Go back to turning text into numbers. That sounds like nonsense.

Rob: It is a long list of numbers, and what matters is what makes two lists similar. Pieces about similar things end up with similar numbers.

Shelley: Similar meaning what, exactly?

Rob: Close together. You can measure the distance between two of these lists, and a short distance means related meaning.

Shelley: Not the same words.

Rob: Not the same words, and that is the entire reason anybody bothers. Somebody asks about time off. Your document says annual leave.

Shelley: No word in common.

Rob: Not one. Matching on the words finds nothing, and the two lists of numbers land next to each other.

Shelley: Is that measured or is that a story?

Rob: Measured and published. Doing it this way beat a strong word matching system by a wide margin on how often the right passage came back at all.

Shelley: Does the conversion have a name.

Rob: The list of numbers is called an embedding, the same idea as a word becoming a location in a space, now done for a whole paragraph. The thing that produces it is an embedding model. Hold on to that second phrase.

Shelley: Why that one?

Rob: Because the nastiest trap today is about which one you used.

Shelley: Where do all these embeddings live?

Rob: In something built for exactly one question. Given this list, hand me the closest ones you have. At scale.

Shelley: It has a name too.

Rob: A vector database. Vector is just their word for the list of numbers.

Shelley: And it checks everything I own?

Rob: It does not. Comparing against every piece you have is too slow, so it uses a structure that finds the closest ones approximately.

Shelley: Approximately, you said.

Rob: Approximately. It trades a little exactness for a great deal of speed, deliberately, and how far it trades is a setting somebody chose.

Shelley: So it can miss the right piece.

Rob: It can. That makes it a real failure mode with a dial attached, rather than a mystery.

Shelley: You said it hands back the closest pieces. How many?

Rob: More than you would want to paste in. Then a second thing goes through that shortlist and puts it into proper order.

Shelley: Why two passes?

Rob: Because the careful comparison is far too slow to run against everything you own. So you run something fast and wide, then something slow and good over whatever survives.

Shelley: A sift and then a sort.

Rob: Exactly that. The second one reads the question and each candidate together and scores how well they actually match.

Shelley: And that's a different piece of software.

Rob: Different model, different job. It is called reranking, and it has been the standard shape of serious search for years. These systems inherited it.

Shelley: Back to cutting documents up. That sounded like the boring step.

Rob: It is the boring step, and it is a common real cause of one of these systems being bad. Not the model. The scissors.

Shelley: The cutting has a name as well, presumably.

Rob: Chunking. The pieces are chunks. It sounds like nothing and it decides everything.

Shelley: How does cutting go wrong?

Rob: Cut too small and the piece loses its context. A sentence reading "this does not apply to contractors", with no paragraph attached saying what "this" is.

Shelley: That's actively dangerous.

Rob: That is a correct retrieval producing a wrong answer, which is the worst combination available. Now cut too large.

Shelley: Then what breaks.

Rob: Two things. One embedding now has to stand for a page covering four different subjects, so it stands for none of them sharply.

Shelley: It goes vague.

Rob: Vague, and expensive. You are pasting a whole page into the prompt to deliver one useful sentence, and that space is finite and you are paying for it.

Shelley: So what's the right size?

Rob: There is not one, and anybody who hands you a number is guessing. That has been studied across a spread of different collections.

Shelley: What did they find?

Rob: Small pieces win where the answers are short and factual. Large pieces win where the question needs a broader view. And it shifts again depending on which embedding model you are using.

Shelley: Three things interacting.

Rob: Piece size, embedding model, and the shape of the questions people actually ask. Which is why this gets measured per system instead of decided in advance.

Shelley: Give me the trap that bites hardest.

Rob: The question and the documents have to be converted by the same embedding model. Mix two of them and retrieval quality is destroyed.

Shelley: Destroyed sounds dramatic.

Rob: It is the right word. Two different models put meaning in different places. The distances between them mean nothing whatsoever.

Shelley: And I'd see an error?

Rob: You would see nothing. That is why this is the one I would warn you about first. Nothing crashes.

Shelley: Nothing at all.

Rob: The lists are still the right length. The search still runs. It still returns the closest things it can find. They are simply the wrong pieces.

Shelley: So it looks like it's working.

Rob: It looks like it is working, and the answers are quietly mediocre. And people spend weeks blaming the model.

Shelley: Who swaps it by accident, though?

Rob: Somebody upgrades the embedding model and re indexes half the documents. Somebody adds a second source that came with its own default.

Shelley: Is "the same one" literally the same.

Rob: Some of them have one path for questions and a separate path for documents.

Shelley: That contradicts what you just told me.

Rob: It refines it. Those two paths are halves of one trained system, built to land in the same place. Running the question path over a document is still wrong.

Shelley: So the rule is matched, not identical.

Rob: Matched halves of one system. And the people who write these things say plainly that picking the right one for your kind of search is critical.

Shelley: Meaning what in practice.

Rob: A short question against a long paragraph is a different problem from comparing two sentences of similar length. Different job, different model.

Shelley: Right. With all of that done properly, the made up answers stop.

Rob: They reduce. They do not stop. There is a word for the made up ones, hallucination, and anybody telling you this prevents them is wrong.

Shelley: That's strongly put.

Rob: They measured the drop. That it is the most oversold claim in this field is me.

Shelley: Noted. But the facts are right there in front of it.

Rob: They are. It can still misread them. It can still take what is there and extend past it into something the page never said.

Shelley: It over extrapolates, then.

Rob: Over extrapolate. And there is a worse case, which is retrieval doing its job perfectly and the answer still coming out wrong.

Shelley: How does that happen?

Rob: Somebody put the right passage in the middle of a long handout. There is a well known result here. These things use the beginning and the end of a long input well, and go soft in the middle.

Shelley: With the answer sitting right there.

Rob: Sitting right there, retrieved correctly, and missed. Where it landed in the handout changed the answer, which ought to disturb you.

Shelley: It does disturb me.

Rob: Pasting in more is not safer. More is a cost, not an insurance policy.

Shelley: Has anybody put a proper harness around this.

Rob: Yes, and the hard parts turned out to be elsewhere. Saying "it is not in here" when it is not in here. Combining two pieces that each hold half the answer.

Shelley: And when the page it found is simply wrong.

Rob: It tends to believe the page. The people who measured this said there is a considerable journey ahead before this is applied to these models effectively.

Shelley: Then how do I know whether mine is any good?

Rob: You stop asking whether it is good. It is two systems, so the question has to be asked twice.

Shelley: Asked twice how?

Rob: First. Is the answer actually supported by the pieces it was handed. Second. Were those the right pieces to hand it.

Shelley: Those feel like the same question.

Rob: They are opposites, and this is the most useful split today. The first one has a name. Faithfulness, or groundedness.

Shelley: Define it for me.

Rob: Take the answer, break it into individual claims, and check each claim against the pieces that were handed over. The fraction that holds up is the score.

Shelley: Not whether the claim is true in the world.

Rob: Not whether it is true in the world. Only whether the handout supports it. A faithful answer built on a wrong page is faithful and wrong.

Shelley: That's a strange thing to want to measure.

Rob: It is exactly what you want, because it isolates one half of the machine and says nothing about the other.

Shelley: And the second name?

Rob: Context relevance. Did the search surface what was needed at all, and did the useful things come back near the top rather than buried far down.

Shelley: So one grades the search and one grades the writing.

Rob: One grades the search and one grades the writing. Keep them apart and you can repair things.

Shelley: Show me that in practice. Mine gives bad answers. Go.

Rob: Look at the retrieval side first. Always first.

Shelley: Say it's bad.

Rob: Then whatever writes the answers is innocent and you are wasting your time on it. Go and look at how you cut the documents, and at what you converted them with.

Shelley: Scissors or embedding model.

Rob: One of those two. They decide what can ever be found in the first place.

Shelley: Now say retrieval looks fine.

Rob: The right pieces came back, near the top, and the answers are still wrong. Now it is the second half. Your prompt, or the model doing the writing.

Shelley: And that's a completely different afternoon?

Rob: Completely different afternoon, often different people, and a different fix. That is what measuring the halves separately buys you.

Shelley: And if I only had one overall score?

Rob: You would know it is bad and have no idea which end to walk to.

Shelley: One thing I keep skipping past. My documents aren't neat text. They're reports.

Rob: Then you have found the step before the step. Something has to get the content out of the file before anything can cut it up.

Shelley: Half of what I need is a table.

Rob: Tables, charts, diagrams, scanned pages. If whatever reads the file flattens your table into a row of loose numbers, the chunk is wrong before anything else runs.

Shelley: And no amount of good searching fixes that.

Rob: None at all. You will retrieve the corrupted chunk perfectly. Which is why the tooling here has grown an extraction stage aimed at tables and charts specifically.

Shelley: Rather than treating a document as a wall of words.

Rob: Rather than that. The packaged versions hand you three services. Something that pulls content out of files, something that makes embeddings, something that reorders results.

Shelley: Extract, embed, rerank.

Rob: Extract, embed, rerank, behind one interface for putting things in and asking things. And extraction is where the recent effort has gone.

Shelley: Who's this genuinely for?

Rob: Anyone whose real answer lives in documents that change and that the model was never shown. An assistant over your own internal material.

Shelley: Which is my refund policy?

Rob: Which is your refund policy. Legal work over case law. Clinical work over treatment guidance that gets revised.

Shelley: And the revising is the point?

Rob: The revising is the whole point. Guidance changes, you replace the document, and the next question gets the new answer. Nobody retrains anything.

Shelley: Give me the honest summary. Not the brochure.

Rob: It is two systems wearing one name, and you have to hold them apart or you cannot fix either.

Shelley: And the cutting matters more than I'd have guessed.

Rob: The cutting matters more than the model does, a lot of the time. So does the embedding model, and that one fails without telling you.

Shelley: And it doesn't stop invented answers?

Rob: It reduces them. It gives you something to check the answer against, which is genuinely new, and it moves the failure to somewhere you can see it.

Shelley: That's still worth having.

Rob: It is worth a great deal. Just do not buy it as a cure, because the people who measured it are clear that it is not one.

Shelley: So what do I do on Monday?

Rob: Measure the two halves separately before you change anything. That is the one habit worth taking out of today.

Shelley: And what's next?

Rob: The numbers themselves. I have said measure all the way through as though that were simple, and it is not. Which score, what it quietly rewards, and the ones that run backwards.