NVIDIA Generative AI. The Kitchen: NeMo from Raw Data to a Trained Model.
Hosts: Shelley and Rob. Cloudadorn Academy.
This episode is narrated using AI voice technology. The content and script are original.

Shelley: I'm Shelley, I ask the questions. Rob builds these things for a living, which is why he gets the answers.

Rob: Morning.

Shelley: Every time you say finished model, I hear the word finished, and nobody ever says who finished it.

Rob: Guilty. I say finished model constantly and never once say who did the finishing.

Shelley: So who did?

Rob: Somebody in a kitchen. And I mean that almost literally, because the room where this happens is organised exactly like one.

Shelley: I like the parcel. The sealed meal that turns up warm and works wherever you set it down.

Rob: Everybody likes the parcel. The container with the model and everything it needs already inside.

Shelley: And today we do what?

Rob: We go through the door behind it. Into the room where somebody sourced the ingredients, threw most of them away, and then stood over the stove for weeks.

Shelley: Weeks over a stove.

Rob: Weeks. And the thing nobody tells you about that room is how little of the time is cooking.

Shelley: What's it then?

Rob: Shopping. Washing. Throwing away. The stove is the short part, and it is the part everybody talks about.

Shelley: Give me the name.

Rob: NeMo. You have heard it once, in a list, one line long, and I said I would come back to it.

Shelley: Give me one sentence.

Rob: A framework for building these models. Preparing the data, doing the training, adjusting the result, measuring it. Language models, picture models, speech models, all in the same room.

Shelley: Framework is a word that hides things.

Rob: It is, so take the two things it is not. Both are mistakes I watch people make.

Shelley: Give me the first.

Rob: It is not a serving layer. It does not stand at the door and answer requests. That is the front desk, and it is a different trade with different people doing it.

Shelley: And the second?

Rob: It is not a model. You cannot fetch it and ask it a question. It is the room, not the dish.

Shelley: That sounds like a small distinction.

Rob: It is the most common mix up in this entire stack. Somebody asks which model to use and gets handed the name of a kitchen.

Shelley: Has it always been one thing?

Rob: No, and this is the part worth knowing if you looked at it a few years ago. It began as a library. One thing you installed, on top of the toolkit most researchers already write in, and then you ran it yourself.

Shelley: And what is it now?

Rob: Now it is also a set of separate services you stand up on your own cluster, one for each stage, and your software calls them over the network like anything else.

Shelley: Why does that matter to me?

Rob: Because a library is something a researcher runs on a Tuesday. A set of services is something a company operates on every day of the year. That is the whole distance between an experiment and a business.

Shelley: Start at the front.

Rob: The front is raw material, and it is worse than you are imagining. Web crawls. Documents. Pictures. Video. Audio. Whatever could be got.

Shelley: Worse in what way.

Rob: The same page turns up again and again under different addresses. Some of it is a home address belonging to a real person who never agreed to any of this.

Shelley: So somebody cleans it.

Rob: There is a station whose entire job is that, and it is first on the line. It is called Curator.

Shelley: What does it actually do?

Rob: Four things worth naming. It takes out duplicates. It scores what is left and drops the bottom. It hunts for personal details and takes them out. And it works out what language each piece is in and labels it.

Shelley: Duplicates sound like the easy one.

Rob: The easy kind is easy. The same page twice, character for character, gone. The hard kind is the same page with a different advert down the side, or one story rewritten by three sites, or two documents that say the same thing in different words.

Shelley: And it catches those?

Rob: Three separate techniques, roughly one per case, and the last of them compares meaning rather than characters.

Shelley: Why does repetition matter so much?

Rob: Because everything downstream is statistics. If a paragraph appears in the pile far more often than it appears in the world, you have not taught the model about the world. You have taught it about your pile.

Shelley: And the personal details?

Rob: Two reasons and they are both plain. One, a model trained on private details about real people can be persuaded to hand them back. Two, the rules about personal information do not stop applying because the material went into a training pile.

Shelley: Is this one machine doing all that?

Rob: It is a pipeline across a cluster, because at this size the cleaning is itself a heavy job. And it is not only text any more. It does pictures, video and audio too.

Shelley: Give me the sentence people say about this stage.

Rob: Rubbish in, rubbish out. It is old, it is unfashionable, and it is the whole argument. What comes out of the kitchen is decided here, before anybody switches on a stove.

Shelley: What's the next station?

Rob: Next station makes data that does not exist.

Shelley: That sounds like cheating.

Rob: It sounds like cheating and it is now ordinary practice. You describe the shape of the samples you need, the machine writes them, and you check them.

Shelley: Why would I want invented samples?

Rob: Three problems, and every one of them is a real problem somebody has this week.

Shelley: Give me the first.

Rob: Cost. Real samples usually have to be labelled by people, one at a time, and people are expensive and slow and get bored.

Shelley: And the second?

Rob: Privacy. If your real material is medical or financial you may not be allowed to train on it.

Shelley: And the third?

Rob: Rarity. The case you most need it to handle is the case you almost never see. You cannot go out and collect a thousand samples of the thing that happens twice a year.

Shelley: So you write them.

Rob: So you generate them. And the station has a name, Data Designer, and the useful part of it is that you look at a handful first.

Shelley: Meaning what, in practice.

Rob: You get a small preview, you read it, and if it is rubbish you change the description and try again. Then you make the large pile. Nobody wants to find out after the run that every invented sample started with the same three words.

Shelley: Is there a catch?

Rob: Of course. A model writing training material for a model is a circle, and circles can go stale. Which is why the checking is the point and not an afterthought.

Shelley: Now the cooking.

Rob: At last. There is a library underneath whose entire job is running a very large training run across a great many machines at once. It is called Megatron Core, and I have mentioned it once before and refused to open it.

Shelley: Open it now, then.

Rob: The problem is simple to say. The model is too big for one machine, the data is too big for one machine, and the arithmetic is too slow on one machine. Those are three different problems and they have three different answers.

Shelley: Give me the first.

Rob: Every machine gets the same complete model and a different basket of the data. They all work, then they compare notes and agree on the correction. That is splitting the data.

Shelley: And the second problem?

Rob: The model itself does not fit, so different machines hold different parts of it. That is splitting the model, and there are two ways to cut it.

Shelley: Cut it how?

Rob: Along the production line, or across a single job. Along the line, each machine owns a run of layers and hands its result to the next machine, like courses in a restaurant. That is the pipeline.

Shelley: And across a single job?

Rob: One operation is too large for one machine, so four machines each take a quarter of that one operation and their answers are stitched together. Four people carrying one table. That is the finest of those cuts.

Shelley: How many of these am I running at once?

Rob: All of them, usually. And there are more than four in the list now, but those are the ones that make the picture.

Shelley: What breaks, in a run like that.

Rob: Machines. That is the part people underestimate. Over a run of weeks on thousands of chips, something fails. Not might. Does.

Shelley: And then you start again?

Rob: Not any more, and that is most of what this library buys you. It writes down where it had got to, often and quickly, and it picks up from the last written point.

Shelley: So it has to notice the failure.

Rob: It notices a machine that has died. And, worse, a machine that has quietly stopped moving. Then it restarts by itself.

Shelley: Why is that worse?

Rob: Much worse. A dead machine announces itself. A hung machine sits there holding everybody else up while the meter runs.

Shelley: Is this room sealed off from everybody else?

Rob: No, and that would be fatal if it were. There is a bridge alongside it that converts models back and forth between this way of working and the format most of the open world publishes in. You are not choosing a prison.

Shelley: The dish exists.

Rob: The dish exists and it is generic. Two more stations, and both of them are services now rather than scripts.

Shelley: Give me the first of the two.

Rob: Adjusting it on your own material, so it speaks about your business rather than about everything. There is a station called Customizer whose whole job is that, and how that works had a whole conversation of its own, so here it is only a stop on the line.

Shelley: And the manners?

Rob: Also had its own conversation. Teaching it what to refuse and how to behave is its own stage with its own name, and I am not doing it twice.

Shelley: And the second station?

Rob: Measuring. It is called Evaluator, and it runs the standard benchmarks, and the custom ones you write, and the awkward modern one where you have another model mark the work.

Shelley: Why is that a station and not a spreadsheet?

Rob: Because if measuring is a thing somebody does at the end when they remember, it does not happen. If it is a service on the line with everything else, it happens every time.

Shelley: Then it leaves.

Rob: Then it leaves, and the rest is the order of the work. I will name the stages and stop, because each one of them is a conversation of its own.

Shelley: Name them then.

Rob: Prepare the data. Train or adjust the model. Measure it. Then compile it for the exact chip it will live on, which is TensorRT.

Shelley: That's the first four.

Rob: Then serve it, which is Dynamo and the general purpose sibling. Then package the whole thing as a parcel anyone can run, which is NIM. Then put rails on it at the moment it speaks, which is Guardrails.

Shelley: That's a lot of names.

Rob: It is one line each and it is the most useful list in the season. Almost every argument I sit in about this stack is two people who agree completely, standing at different stations.

Shelley: Is the kitchen the whole building.

Rob: It is not, and three more rooms are worth a line each before we go outside.

Shelley: What's the first room?

Rob: Speech. Listening to a person and writing down what they said, saying something out loud in a voice, and translating between languages, all fast enough to hold a conversation. That one is called Riva.

Shelley: And the second?

Rob: Pictures, for people who are not going to train anything from scratch. You take a model that already sees, point it at a few thousand of your own photographs, and it learns your particular defect on your particular production line.

Shelley: Does that one have a name?

Rob: That one is called TAO.

Shelley: And the last one?

Rob: Cleaning up a live picture and a live voice while they are happening. Taking the dog out of the background, taking the room echo off the sound, putting your eyes back on the camera.

Shelley: That one has a name too.

Rob: Maxine. And it is not there to generate anything. It is there to make what already exists less unpleasant.

Shelley: You said outside.

Rob: Cameras. Because this is where all of it lands in a way you can walk past.

Shelley: Start with the small version.

Rob: A doorbell camera. It notices a person, and it tells you a person is there. One camera, one question, one answer, and it is genuinely useful.

Shelley: And the large version?

Rob: A whole airport. Thousands of feeds at once, in real time, and the questions are different in kind. How is the crowd flowing. Which queue is about to fail. Has somebody walked into a place people do not walk into.

Shelley: That isn't the same job at all.

Rob: It is not, and the platform for it is called Metropolis. Underneath is the piece that takes many video streams at once and keeps up with them live, and that is called DeepStream.

Shelley: What else is underneath?

Rob: The picture toolkit I just named, so the cameras can be taught your particular site. And the sealed parcels, with detection models already inside them.

Shelley: Where's this actually running?

Rob: Factories. Warehouses. Shops. Airports. Road junctions. And a fair amount of it on a small machine bolted to a wall, because sending every frame across the country and back is a bill nobody signs.

Shelley: Do I have to build this kitchen?

Rob: No. Every station I have described has a rented equivalent on the large clouds.

Shelley: Then why would anybody run their own?

Rob: That is the right question and the answer is not on the feature list. The feature lists look much the same.

Shelley: So what's different?

Rob: Where the machines are. This kitchen will run inside your own building, on hardware you own, behind your own door. The rented ones run in a building that belongs to somebody else.

Shelley: And for most people that's fine?

Rob: For most people that is fine, and cheaper, and I would tell them to rent. And then there is the hospital.

Shelley: Go on, the hospital.

Rob: The scans are the material. Training on them means the material moves. In a great many places the rules say the material does not leave the site, and no amount of contract makes that go away.

Shelley: Who else has walls like that?

Rob: A bank with a regulator who wants to know exactly which machines held what. A government department that is not permitted to send it anywhere. A factory whose most valuable possession is a picture of how it makes the thing it makes.

Shelley: So it isn't a technical argument.

Rob: It almost never is. It is a question about walls, and it is why an on premises option is not a nostalgia feature. For those buyers it is the only feature.

Shelley: Give me three things.

Rob: One. Most of building a model is not building a model. It is deciding what goes in, and the station that throws things away matters more than the station that does the arithmetic.

Shelley: And the second?

Rob: A long training run is an operations problem wearing a research hat. Splitting the work is the clever part. Surviving three weeks of hardware failing is the expensive part.

Shelley: And the third?

Rob: The names are stations on a line, not competitors. Prepare, train, measure, compile, serve, package, rail. If you can put a name at each stop you can follow almost any conversation in this field.

Shelley: What did you skate past?

Rob: The bit where it stops being a thing you ask and starts being a thing that goes off and does something. Everything today has been one question in, one answer out.

Shelley: And that changes?

Rob: It changes completely. Give it a goal instead of a question, and some tools, and let it keep going until it is finished. That is next, and it has a price attached that nobody saw coming.