NVIDIA Generative AI. Serving It: Dynamo, Dynamo-Triton, and NIM.
Hosts: Shelley and Rob. Cloudadorn Academy.
This episode is narrated using AI voice technology. The content and script are original.

Shelley: I'm Shelley, I ask the questions. Rob builds these things for a living, which is why he gets the answers.

Rob: Morning.

Shelley: I have built the thing. It's finished. It works.

Rob: Congratulations. Where is it.

Shelley: On a machine.

Rob: And when a thousand people want it in the same second.

Shelley: They can queue.

Rob: They can. The machine will sit nearly idle while they do, and the people at the back will leave.

Shelley: Why would it sit idle?

Rob: Because what is underneath is built to do an enormous number of small things in the same instant, and you are handing it one at a time.

Shelley: Give me that as a picture.

Rob: A press that can run ten thousand sheets an hour. Printing one leaflet. Stopping. Printing one leaflet.

Shelley: So I need something in front of it.

Rob: A front desk. Which turns out to be a trade of its own, with names, arguments and a decade of engineering in it.

Shelley: All to answer the phone.

Rob: All of it. Which is today.

Shelley: Start with what it does.

Rob: It stands at the door of the building. Requests arrive, it decides what happens to each one, it hands answers back.

Shelley: That sounds thin.

Rob: It is the difference between a demonstration and a business. And before what it does well, three things it is not, because that is where people go wrong.

Shelley: One of three.

Rob: It does not own the building. It is not the machines. Somebody else provides those and the desk sits on top of whatever you have.

Shelley: And the second thing?

Rob: It does not teach anything. It is not where a model gets built, or trained, or improved. It receives one that is already finished.

Shelley: And the third thing?

Rob: It does not make your model faster. A separate step does that, taking a finished model and building a fast program for one particular chip.

Shelley: Does that step have a name?

Rob: TensorRT. It is a compiler, it had its own conversation, and it is not this one.

Shelley: Those two get muddled.

Rob: Constantly. They are not alternatives, they are in order. One makes the thing quick, the other answers the door. Choosing between them means misunderstanding both.

Shelley: You said names.

Rob: Two, and they are siblings rather than synonyms. There has been a general purpose one for years, and it serves anything. A picture classifier, a speech model, a bit of classical statistics, in whatever toolkit.

Shelley: And it has a name?

Rob: It kept one for years, then it was folded into a larger platform and renamed. It is now Dynamo Triton, and both names are still all over the documentation, which is a tax on everybody reading.

Shelley: And the sibling?

Rob: Very large language models spread across an enormous number of machines. Newer. They call it Dynamo. The people who make it call it the successor.

Shelley: Successor. So the old one is done.

Rob: No, and that is the trap the word sets. The old one is still the answer for anything that is not a language model. The new one, when one enormous conversational model runs across a fleet.

Shelley: How do I keep them apart?

Rob: Anything, anywhere, against one kind of thing, everywhere at once.

Shelley: And they're the same product?

Rob: One platform, two pieces. Not two words for one thing.

Shelley: Give me something the desk does that I couldn't do myself.

Rob: The one that pays for the machine. Requests turn up from different people at different moments, and the desk does not push them through singly. It gathers whoever arrived close together and sends them as a group.

Shelley: A group of unrelated people.

Rob: Completely unrelated. Never met, asked different things, going to get different answers. The machine does not care.

Shelley: Why does it not care?

Rob: Because the expensive part is dragging the whole model past the chip, and that costs the same for one person or forty.

Shelley: So do it once for forty.

Rob: Once for forty instead of forty times for one. And nobody has to ask. No code that anybody wrote changes. The desk builds the group on the way in.

Shelley: Does it have a name?

Rob: Dynamic batching. And dynamic is doing real work there, because the group is made out of whoever happened to turn up.

Shelley: There's a catch.

Rob: There is a lift in it. You are in the lobby, the doors are open, and the lift holds a moment because two more people are walking over.

Shelley: I wait longer.

Rob: You wait longer. The building moves more people. Those are the same sentence.

Shelley: Can I turn it off?

Rob: You can turn it down. There is a dial for how long the doors hold. Short, and everybody moves quickly in a half empty lift. Long, and the lift is full while you wait.

Shelley: So what do I set it to?

Rob: Whichever your situation is. Somebody watching a cursor blink is one problem. Ten thousand documents to get through by morning is another.

Shelley: And if I guess?

Rob: The setting that looks best on a chart is rarely the one the person in the lobby can feel.

Shelley: What else does the general one do?

Rob: It runs several models at once on one machine. Or several copies of one model side by side, so three requests are not queueing behind each other.

Shelley: Why does that help?

Rob: Most models do not fill a machine on their own, and an idle machine is the most expensive object in the building.

Shelley: Anything else it does.

Rob: You can chain them. What comes out of one goes straight into the next, and the desk carries the material between them internally instead of handing it back to you.

Shelley: Why does that matter?

Rob: Because the trip out and back is often dearer than the work. Something tidies your input, the model runs, something tidies the answer. One request, no round trips.

Shelley: What if I need the new model on without killing the old one?

Rob: More than one version of a model can sit there together, so you move traffic across gradually rather than turning one off, the other on, and praying.

Shelley: What does it need underneath?

Rob: Less than you would think. Their own chips. Accelerators from other companies. Ordinary main processors of both common kinds, with no accelerator at all. It speaks the plain web request everything speaks, and plugs into the usual machinery for adding copies.

Shelley: Then the newer one. You said it was for one kind of thing.

Rob: Language models at scale. And the whole argument is one observation the field took a while to act on. Answering you is two different jobs wearing one word.

Shelley: Give me the first.

Rob: It reads your question. All of it, at once. A long prompt, a whole document, it chews the lot in parallel, because every part of it is already sitting there.

Shelley: And the second job?

Rob: Then it writes. One word. Then the next, which depends on the one before it. Then the next. It cannot do those together, because it does not know the second word until it has written the first.

Shelley: So reading is a sprint and writing is a queue.

Rob: That is exactly it, and they strain the machine in opposite directions. Reading is a wall of arithmetic and the chip is flat out. Writing is one small sum at a time, waiting most of its life for the next load.

Shelley: What happens if you do both in the same room?

Rob: They get in the way of each other. A long document turns up and everybody mid sentence stalls, because the machine is reading.

Shelley: So separate them.

Rob: Reading on one set of machines, writing on another, each tuned for what it is actually doing. And the halves have names. Reading is called prefill. Writing is called decode.

Shelley: And that's the headline?

Rob: Disaggregated serving, which is an ugly phrase for a plain idea. Two jobs, two rooms.

Shelley: Who decides how many rooms?

Rob: Something above it, watching the traffic and moving machines from the reading side to the writing side as the day changes shape.

Shelley: Second thing the new one does.

Rob: My favourite, and the most human thing today. When one of these has been talking to you a while, it holds a pile of working notes about the conversation so far.

Shelley: Holds them in memory.

Rob: In memory, on the machine that was talking to you. Now you ask a follow up. If it lands on whichever machine is free, that one has none of your notes and works the conversation out again from the beginning.

Shelley: That seems mad.

Rob: It is mad, and it was normal. So the desk keeps a map of which machine holds which notes, and sends your follow up there.

Shelley: The clerk who still has my file on the desk.

Rob: The clerk who still has your file on the desk. And not purely that, because that clerk might have a queue out the door. So it weighs how much of your conversation the machine holds against how busy it is.

Shelley: What happens to notes nobody is using?

Rob: Down a ladder. The fast memory beside the chip is the dearest space in the building, so notes that go quiet get pushed out to ordinary memory, then a disk, then storage somewhere else.

Shelley: And if the conversation wakes up?

Rob: They come back, which is the point. Working them out again costs more than fetching them. That is the whole trade in one line.

Shelley: Do the notes have a name.

Rob: They have one. The KV cache, for the keys and values it worked out along the way. Call it the cache.

Shelley: What do I need to hold?

Rob: That it exists, that it is expensive, and that finding it beats working it out twice.

Shelley: Give me the choosing rule.

Rob: If what you serve is not a language model, or is several things at once, or runs somewhere odd, you want the general purpose one.

Shelley: And the other way?

Rob: One very large language model over a lot of machines, with conversations long enough that the notes matter. Then the newer one.

Shelley: And if it's one language model on one machine?

Rob: Then most of this is machinery you do not need yet, and you should be pleased about that.

Shelley: Last one. Does the new one replace the thing that actually runs the model.

Rob: No. Underneath are engines that do the running. The newer thing sits above them and turns a pile of separate machines into one system. It coordinates. It does not replace.

Shelley: Right. What's the second half of today?

Rob: The half where all of this stops being your problem. There is a gap nobody draws. On one side the model exists. Somebody trained it, you can download it, it works.

Shelley: And the other side?

Rob: The model answers real traffic, efficiently, at a price you can live with.

Shelley: How wide is the gap?

Rob: Months, for a lot of teams. And what is in it is unglamorous.

Shelley: Such as what?

Rob: The dependencies fight each other. It has to be built for the chip you own. Somebody has to wrap it in an interface, and make more copies appear when the queue grows.

Shelley: That's a specialist job.

Rob: It is, there are not many of those people, and here is the miserable part.

Shelley: Which is what?

Rob: Every team does it again. Same problem, same order, same month gone, in thousands of buildings.

Shelley: So somebody sells the answer.

Rob: Somebody packages the answer.

Shelley: Tell me what's in it.

Rob: A container, which is a word worth owning. A sealed parcel with the program inside and everything it needs, so it runs the same way wherever you set it down.

Shelley: And inside this particular one?

Rob: The model. The engine that runs it, already built for the sort of chip it will find. The tuning somebody who knows those chips did for you. And a door on the front, in the shape the industry already writes against.

Shelley: So I wire nothing up.

Rob: You pull it. You run it. You point your application at it like any other service on the network. It is called NIM.

Shelley: And it's genuinely faster than doing it myself.

Rob: Faster to stand up, certainly. Faster to run as well, and here I want to be careful with you.

Shelley: Careful in what way.

Rob: You will see very large multiples quoted. On the case they publish themselves, one model on one chip with a couple of hundred people asking at once, it gets through about twice as much per second as the same model stood up plainly.

Shelley: Twice. Not ten times.

Rob: Twice, on that case. Real, worth having, and not what the posters imply. It will be a different number for your model on your chip, and anybody quoting one multiple has told you nothing.

Shelley: You said the engine was already built.

Rob: This is the part that changed, and it is what I would tell anybody who last looked a year ago. It used to be one engine. One compiler underneath, one server in front.

Shelley: And now it doesn't?

Rob: Now the box looks at the model it was handed, and at the machine it finds itself on, and chooses.

Shelley: Chooses from what?

Rob: Three engines. It works out which your model and your hardware can support, and how much memory each would want, before it starts.

Shelley: Without asking me.

Rob: Without asking. You can override it when you know better, and most people do not know better and should not have to.

Shelley: Do I need the three names?

Rob: You do not. What you need is that picking the engine used to be your job and is not any more.

Shelley: Where do these come from?

Rob: A catalog. Containers already built for these chips, models already trained, the tooling, and the recipes for putting the lot on a cluster. That is NGC.

Shelley: Do I pay for the catalog?

Rob: Not for much of it. Then a paid tier, which is the same software with a contract wrapped around it.

Shelley: What does the contract buy?

Rob: Three dull things that decide whether a serious organisation can touch any of this. Somebody patches the security holes. The interfaces hold still long enough to build against. And there is somebody to ring.

Shelley: That's what enterprise means.

Rob: That is the whole of it, and it is called AI Enterprise. When somebody insists on the supported version, that is what they are insisting on. Not different software. The same software with a promise attached.

Shelley: Can I try one before any of that?

Rob: They run a public site where a pile of these are already running. You call them over the wire and see what comes back before installing anything.

Shelley: Does it have to be their cloud?

Rob: No, and that is the strategic point of the exercise. It runs in your building, on machines you own, behind your own door.

Shelley: Who needs that?

Rob: A hospital. The imaging model runs on machines the hospital owns, the pictures never leave the site, and the software team wires it up like anything else on the network.

Shelley: Anybody else need that.

Rob: Anybody with rules about where the material may live. And anything physical, where the answer has to arrive whether or not the connection is up.

Shelley: And the catch?

Rob: The tuning inside that parcel is for chips from one company. Change vendor and that work does not come with you.

Shelley: That sounds deliberate.

Rob: The most deliberate thing today. What it does to a market is a conversation of its own, later, and not this one.

Shelley: Give me three things.

Rob: One. Serving is a trade. It does not own the machines, it does not train anything, it does not make your model faster. It answers the door, and doing that badly wastes more money than anything upstream.

Shelley: And the second?

Rob: Two jobs hide inside one word. Reading your question and writing the answer are different work with different appetites, and the largest recent gain here came from putting them in separate rooms.

Shelley: And the third?

Rob: Almost nothing you are about to spend a month on is specific to you. The parcel exists. The engine picks itself now. Take the month back.

Shelley: What did you skate past?

Rob: Where the model came from. I have said finished model all the way through and never once said who finished it.

Shelley: Somebody built it.

Rob: Out of raw material, in stages, and every one of those stages has a tool with a name on it. That is next.