NVIDIA Generative AI. Teaching an Old Model New Tricks: Transfer Learning, LoRA, and RLHF.
Hosts: Shelley and Rob. Cloudadorn Academy.
This episode is narrated using AI voice technology. The content and script are original.

Shelley: I'm Shelley, I ask the questions. Rob builds these things for a living, which is why he gets the answers.

Rob: Morning.

Shelley: I want a model that knows my business.

Rob: You have two options, and only one is real.

Shelley: Give me the unreal one first.

Rob: You build your own from nothing. Almost nobody is in a position to, and most who could shouldn't.

Shelley: That's two different claims.

Rob: It is. Can't is the true one. Shouldn't is me handing you advice.

Shelley: Then what's the real option?

Rob: You take one that already exists and reuse what it already learned. Everything today is a version of that.

Shelley: Reuse what, though. It learned somebody else's job.

Rob: It learned a lot on the way there, and that's the part that transfers. People have taken a trained network apart layer by layer to check. The early layers aren't specific to the job at all.

Shelley: General meaning what?

Rob: The same early machinery is useful on datasets and tasks it never saw. The specific part sits at the far end, near the answer.

Shelley: And the far end is what you change.

Rob: The far end is what you change. And starting from a trained model beats starting from nothing even when the two jobs are far apart. The advantage doesn't wash out once you have trained on your own job either.

Shelley: Does this have a name?

Rob: Transfer learning. Everything else today is how much of the model you're allowed to disturb.

Shelley: Give me a picture.

Rob: You're hiring for a café. One candidate has pulled espresso for years. The other has never seen the machine.

Shelley: Obviously I take the first one.

Rob: You take the first one and teach them your menu in an afternoon. The second takes months, and you're teaching them to steam milk before you get near your menu.

Shelley: So the first training is the espresso.

Rob: Somebody else paid for those years. You're paying for the menu, and next to the years that's nothing. The gap is the economics of this field.

Shelley: All right. How do I teach it my menu?

Rob: Five ways. Four of them are stages of one pipeline, and the fifth is a cheaper way of doing the later stages, rather than five separate products.

Shelley: Five sounds like a list I'll forget.

Rob: Don't sort them by price. Sort them by two questions. How much of your own material does each need, and who actually does it.

Shelley: Who does it's a strange axis?

Rob: It's the honest one. Some of these are done by a handful of companies on earth. Some are done on a Tuesday by one person with a laptop.

Shelley: Start me at the top.

Rob: The original training. You show it a web-scale pile of text and it predicts the next word, over and over.

Shelley: Who labels all that?

Rob: Nobody, and that's the trick. The text is its own answer key, because the next word is already sitting there. The word for it's self supervised.

Shelley: And who does it?

Rob: Model labs. That's the whole column. If you aren't a model lab, this stage is something that happened to you, not something you do.

Shelley: Then stage two.

Rob: The same training again, on one field. Legal text. Medical text. Code.

Shelley: Why do the same thing twice?

Rob: Because the language of a field isn't the language of the internet. You want it fluent in the dialect before you teach it tasks.

Shelley: How much material?

Rob: Billions of words of it. Which is why the column says large enterprises and means it.

Shelley: Does that work, or is it just expensive?

Rob: It works, and somebody measured it. A second pass in-domain across four separate specialist fields gave gains in every one.

Shelley: With mountains of material.

Rob: With mountains, and with very little. Both settings. That's the part that surprised people.

Shelley: Take me to stage three.

Rob: Humans arrive. You write down pairs. Here's a request, here's the answer we wanted. Thousands rather than billions, and it's the first stage here a normal organisation actually runs.

Shelley: Does it have a name?

Rob: Supervised fine tuning. Supervised because a person wrote the answer down.

Shelley: And stage four?

Rob: Stage four is the interesting one. You stop showing it the right answer, and start showing it which of two answers people preferred.

Shelley: Why would I do that? I know the right answer.

Rob: Do you. Write me down the right answer to "explain this refund policy kindly".

Shelley: I can't do that.

Rob: No. But hand you two attempts and you'll tell me which is better in a second. That asymmetry is the reason this stage exists.

Shelley: So the humans rank instead of write.

Rob: The humans rank. The founding work here was explicitly about goals nobody can write a score for. People compared pairs.

Shelley: And then what happens to the rankings.

Rob: A second model learns to predict them. It learns what people tend to prefer. Then the first is trained to score well against the second.

Shelley: So there are two models.

Rob: Two models and a loop. That arrangement has a name you'll hear in rooms. Reinforcement learning from human feedback. RLHF.

Shelley: What does this stage buy me?

Rob: Manners. The first training taught it everything it knows. This teaches it how to behave, what not to say, and how to be useful rather than merely correct.

Shelley: Finishing school, then.

Rob: Finishing school. And the result that made everybody sit up is that a much smaller model put through it was preferred to a far bigger one that wasn't.

Shelley: How does the training part work?

Rob: An algorithm called proximal policy optimization. Its entire personality is caution. Improve, but don't lurch away from what you already were.

Shelley: And that's the standard?

Rob: It was. And it's complex and often unstable, which isn't me being rude. That's how the people who built the replacement describe what they replaced.

Shelley: So what's the replacement?

Rob: Direct preference optimization. It throws away the second model and trains straight on the preferences, with an ordinary classification loss.

Shelley: And that's cheaper?

Rob: Much. The older method keeps stopping to make the model generate fresh samples while it trains. The newer one never does. Most of the bill, gone.

Shelley: Is that the end of the story?

Rob: The middle. There's a newer one again, group relative policy optimization, a variant of the older algorithm rebuilt to use less memory.

Shelley: How do I know any of this is current?

Rob: Fair. There's an open library for this stage of work, and its list of what it runs carries the old algorithm and both newer ones. Not a history lesson.

Shelley: You said five stages. That was four.

Rob: It was, and the fifth is where most people live. You freeze everything the model already is, and train something tiny alongside it.

Shelley: Freeze meaning it can't change.

Rob: Not one of the original numbers moves. They're read only for the whole of training.

Shelley: Then what's learning?

Rob: Small new matrices, injected next to the existing ones, layer by layer. They start empty. They're the only thing updated.

Shelley: How small is small?

Rob: A tiny fraction. On the biggest model in the paper that introduced it, against the standard way of fine tuning that model, ten thousand times fewer parameters being trained.

Shelley: Ten thousand times.

Rob: On that model, against that baseline. And three times less memory while you do it. Their number with their conditions attached, because both halves matter.

Shelley: Does the family have a name.

Rob: The family is PEFT. Parameter efficient fine tuning. The famous member is LoRA. Low rank adaptation.

Shelley: And what do I end up holding?

Rob: A file. Megabytes rather than gigabytes. Which means you can keep a pile of them. One shared model underneath, many small files on top, and you pick whichever the request needs.

Shelley: Surely it's worse than doing it properly.

Rob: That's the thing. It isn't. The published finding is that it lands on par with or better than fine tuning the whole model.

Shelley: On par with the real thing.

Rob: In model quality. And a page written by people who aren't the authors says the same in its own words. Two sources, not one.

Shelley: Why did anyone need this?

Rob: Because of what a full fine tune holds in memory at once. Three things, not one.

Shelley: Name the three.

Rob: The weights. The gradients. And the state the training process keeps for itself while it runs.

Shelley: And together that comes to?

Rob: Often several times the size of the model. So one that sits comfortably on your hardware for answering questions doesn't fit at all for changing it.

Shelley: So running one and training one.

Rob: Aren't the same size problem, and that gap is where the pain lives. It prices most teams out entirely.

Shelley: And there's a second cost?

Rob: Quieter and often worse. You get a whole new copy of the model per use. Five departments want five behaviours, you serve five full models.

Shelley: And with the small file?

Rob: One model, five files. Take a bank.

Shelley: Walk it for me.

Rob: One base model on shared hardware. One small file for mortgage questions. One for card disputes. One for internal compliance.

Shelley: All on the same machine.

Rob: Same machine, same base, three behaviours, and switching between them takes minutes rather than a project.

Shelley: Is that real, or a brochure?

Rob: Fair question, and I checked. The serving software keeps these files in a store, each with a name, and a request says which one it wants.

Shelley: And it loads it up?

Rob: It pulls it into a cache, and runs several at once, through the same base, for different people at the same moment.

Shelley: And when a department wants a new one.

Rob: They can be added and taken away while the service is running. Nobody restarts anything.

Shelley: Has anyone improved on it?

Rob: Twice, and both are a sentence each. First, QLoRA. Squash the frozen copy down to something coarser first, so it takes less room, then train the small piece against the squashed version.

Shelley: Doesn't that damage it?

Rob: You'd think so. The published claim is that it holds the quality of the full precision version of the same job.

Shelley: And the second?

Rob: DoRA. It splits each original weight into two ideas, how big it's and which way it points, then lets the small piece move the direction.

Shelley: Is that a model I can use?

Rob: No, and that confusion is why it's worth naming. It's a way of training. There's no chatbot called that.

Shelley: Does that slow it down?

Rob: No. No extra time to answer, stated on the page.

Shelley: New confusion, please.

Rob: Four words people use interchangeably that aren't. Fine tuning. Distillation. Quantization. Pruning.

Shelley: They all sound like making it smaller.

Rob: Only three of them are. That's the first thing to fix.

Shelley: So which one isn't?

Rob: Fine tuning. It changes behaviour. The model comes out exactly the same size it went in.

Shelley: Then what changes.

Rob: What it does. Better at your job, and here's the honest half, possibly worse at things it used to be fine at.

Shelley: Hold that thought. Second one.

Rob: Distillation. A big model teaches a small one to imitate it. You end up with something much smaller and slightly worse.

Shelley: Give me the third.

Rob: Quantization. Same model, same shape, except the numbers inside are stored less precisely. Fewer distinctions. The ways of being less precise are a subject with real engineering in them, so that gets its own conversation, later.

Shelley: What does it cost me?

Rob: Smaller, and a small loss if it's done well. That last clause is doing work. Done badly, it isn't a small loss.

Shelley: And the fourth?

Rob: Pruning. You remove what isn't pulling its weight. Individual connections, sometimes whole structures.

Shelley: And the damage?

Rob: Depends entirely on how hard you cut. Classic work took a network down to a fraction of its size with no loss at all. Plenty of people have cut too far.

Shelley: You told me to hold a thought.

Rob: The thing fine tuning can quietly break has a name, and it's the most dramatic name in the field. Catastrophic forgetting.

Shelley: That isn't subtle.

Rob: Nor is it subtle in practice. Push the training hard on your task and the model gets measurably worse at general ability. Things it could do before, gone.

Shelley: Because you overwrote it.

Rob: Because you moved the numbers that were holding the old skill. There's nowhere else for that skill to live.

Shelley: Is that avoidable?

Rob: Reducible. And this is where the small frozen piece stops being a cost trick and becomes a safety one.

Shelley: Because nothing gets overwritten.

Rob: Because the originals are never touched.

Shelley: Reduced, though. Not gone.

Rob: Reduced. I'm using the careful word on purpose.

Shelley: Give me the second trap.

Rob: This one catches everybody. The small piece changes how many parameters you train. Not how many the model has.

Shelley: Meaning at the end of it.

Rob: What you serve is still a full sized model with a small file riding along. Your training bill fell through the floor. Your serving bill didn't move.

Shelley: That feels like a swindle.

Rob: Only if somebody sold it as both. Cheaper to make isn't cheaper to run. If you want cheaper to run you want the coarser numbers, and that's a different lever.

Shelley: And the third trap?

Rob: If somebody describes making a model smaller and there's no second model anywhere in the story, that isn't distillation.

Shelley: Because distillation needs a teacher.

Rob: No teacher, no distillation. What they're describing is coarser numbers or fewer connections, and it matters, because those three fail in different ways.

Shelley: There's a fourth, isn't there.

Rob: There's a fourth, and it's the one that wastes the most money.

Shelley: Then say it.

Rob: Fine tuning teaches skill. Style. Format. The shape of how it works. It's a poor way to put facts in.

Shelley: But that's why everybody does it.

Rob: I know. The answers are wrong about the company, so the plan becomes train it on the company. And it half works, which is the worst outcome available.

Shelley: Why only half?

Rob: Because somebody ran that comparison properly. Training the facts in, against handing them over at question time.

Shelley: And which won?

Rob: Handing them over, consistently. On knowledge it had seen and on knowledge that was new. The finding underneath is blunter. Models struggle to learn new facts that way at all.

Shelley: So what do I do instead?

Rob: You hand it the facts at the moment you ask. That's called RAG, it's next, and I'm not compressing it into thirty seconds now.

Shelley: You're keeping something back.

Rob: I'm keeping it whole. Today's boundary is behaviour, voice and format. Facts, next.

Shelley: Give me the habit.

Rob: Before you touch anything, say out loud what you're trying to change. Three answers, and they don't overlap.

Shelley: Which three answers?

Rob: Behaviour. Size. Facts. Behaviour is training. Size is coarser numbers, fewer connections, or a smaller student. Facts are handed over, not trained in.

Shelley: And people get that wrong?

Rob: Constantly. I'd say most of the money wasted here's somebody solving one of those three by reaching for another. That's my judgement, not a measurement.

Shelley: Noted as your opinion.

Rob: Noted as my opinion. It's the pattern I watch.

Shelley: And the freezing?

Rob: Freeze what already works, train something tiny next to it, keep the file. That covers more of real life than anybody expects, and it's the cheapest thing here to undo when you're wrong.

Shelley: Undo it how?

Rob: You delete the file. The model underneath never changed. Now try undoing a full fine tune that went badly.

Shelley: I'm guessing you can't.

Rob: You retrain, or you go back to a copy you hopefully kept. That asymmetry is worth more than any arithmetic today.

Shelley: So I've spent all this time learning how not to break something expensive.

Rob: You have spent it learning that the expensive thing doesn't have to be touched at all. That's the trade.

Shelley: And what's next?

Rob: Facts. Where they live, how they get fetched, and why the thing everybody treats as solved is the one that quietly goes wrong most.