NVIDIA Generative AI. Sand and Statues: How Diffusion Makes a Picture.
Hosts: Shelley and Rob. Cloudadorn Academy.
This episode is narrated using AI voice technology. The content and script are original.

Shelley: I'm Shelley, I ask the questions. Rob builds these things for a living, which is why he gets the answers.

Rob: Morning.

Shelley: Somebody showed me a photograph on Sunday. A lighthouse in a storm, spray coming over the rail.

Rob: Good one.

Shelley: Very good one. And there's no such lighthouse. There was no storm and nobody stood there.

Rob: Somebody typed a sentence and a machine handed that back.

Shelley: Which I have had described to me twice and believed neither time. Where does a thing like that start?

Rob: Worse than a blank page. It starts from static.

Shelley: Static as in the snow on an old television.

Rob: That, in a rectangle. Nothing in it. Nothing hidden in it. No lighthouse buried under it.

Shelley: And then what?

Rob: Then something takes a little of the static off. And a little more. And after enough of that, the lighthouse is there.

Shelley: That isn't an explanation. That's the same trick, told slower.

Rob: Agreed. The honest version takes all of today. But the shape of it goes in one sentence.

Shelley: Then say the sentence.

Rob: Whatever is doing that work was never taught what a lighthouse looks like. It was taught what static looks like.

Shelley: Start me on the teaching, then. Not on the making.

Rob: And that order matters. There are two directions here, and almost everybody who gets this wrong gets it wrong by swapping them.

Shelley: Two directions of what?

Rob: One is training. One is making. They look like the same road run opposite ways, and only one of them is learned.

Shelley: Give me the training one.

Rob: A sculpture on a beach. Stone, life size, and a sandstorm coming in.

Shelley: Fine. Give me the sandstorm.

Rob: You set a camera on a tripod and film it going under. A little more sand each frame, until there is only a dune.

Shelley: And the sculpture is gone?

Rob: Gone. Now the part that matters. You kept every frame, and you know exactly how much sand fell between each one.

Shelley: Because I was the one throwing it.

Rob: Because you wrote the recipe. Nobody learned to bury that statue. You decided in advance how much falls each step and followed your own instructions.

Shelley: So the burying isn't the clever part.

Rob: The burying is arithmetic. It is fixed. Nothing about it is learned, and that one word is the thing people most often get backwards.

Shelley: Then why do it at all?

Rob: To manufacture homework. At the end you hold an enormous pile of pairs. A frame, and exactly how much sand is on it.

Shelley: And the half buried ones count as much as the rest.

Rob: Every one. Barely dusted, half gone, almost nothing, nothing. Each is a question with the answer already written on the back.

Shelley: And the name for the burying?

Rob: The forward direction. Adding. Fixed. Training. Hold those four together, because we are about to run the other way.

Shelley: So what's the machine actually taught to do?

Rob: One job. Not two, not a list of them. One.

Shelley: Which is what?

Rob: I show it one frame out of that film, tell it which frame it is, and ask one question.

Shelley: Ask it what?

Rob: How much sand is on this, and where.

Shelley: Not what's underneath the sand.

Rob: Never what is underneath. That is the sentence people fall over. It does not guess the statue. It guesses the sand.

Shelley: That seems like the wrong question to be asking a picture machine.

Rob: Until you notice you can mark it. You threw the sand, so you know the answer before it opens its mouth.

Shelley: So I grade its homework.

Rob: You compare its guess against what actually fell, and that gap is all it learns from. Nothing else is going on in there.

Shelley: And how a machine learns from a gap.

Rob: Is the whisper chain, and that had its own conversation. What you need here is that the gap is the one thing being measured.

Shelley: Say the strange part to me again.

Rob: It is not a picture machine. It is a sand detector. Everything you will ever watch it make falls out of that one skill.

Shelley: Then make me something.

Rob: Now I hand it a dune with nothing underneath it. Fresh static, made on the spot.

Shelley: No statue in there.

Rob: None whatsoever. And I ask it the only question it knows. How much sand is on this?

Shelley: And it answers?

Rob: It answers. It has to. Nobody gave it the option of saying there is nothing here.

Shelley: So it invents.

Rob: It reports. Then I take a portion of what it reported away. Not the whole of it. A portion.

Shelley: Why not the whole of it?

Rob: Because taking it all off at once gives you mud. Short strokes, many of them, and you ask again after each one.

Shelley: Brush a bit, ask again.

Rob: Brush, ask, brush, ask. And every time round, slightly less static and slightly more something.

Shelley: And at the end of it?

Rob: A statue is standing on that beach that never existed anywhere, settled entirely by a thing that was only ever guessing at sand.

Shelley: And the name for that direction?

Rob: The reverse direction. Removing. Learned. Making. Set against forward, adding, fixed, training.

Shelley: And if I swap those two in my head?

Rob: Then the whole subject is upside down, and you will believe the machine adds static when you ask it for a lighthouse.

Shelley: And the pair of them together have a name.

Rob: The pair together is called diffusion. Named for the way a drop of ink spreads through water and never gathers itself back up.

Shelley: How many of these strokes are we talking about?

Rob: That is the first thing on the control panel. Too few and you get a blur. Too many and you are paying for nothing.

Shelley: Paying for what?

Rob: Every stroke is the entire machine running again. Which is why this is slow, and the slowness is the honest complaint about it.

Shelley: And who decides how big a stroke is?

Rob: A separate piece of arithmetic with its own name. The sampler. It sets how far you move each time, and therefore how many moves you need.

Shelley: And there's more than one of those.

Rob: Several, and their names will not help you. What matters is that a good one gets you there in far fewer strokes.

Shelley: Nothing you have said so far involves a lighthouse.

Rob: Correct. What I have described will happily hand you a statue. It has no idea you wanted a lighthouse.

Shelley: So where do my words go in?

Rob: First they stop being words. A separate piece turns your sentence into a long list of numbers.

Shelley: And that list goes in when?

Rob: Not once at the start. That is the second thing people get wrong. It goes in at every stroke.

Shelley: Every single stroke.

Rob: Every one. Picture somebody behind the sculptor saying make it a horse, and saying it again before every stroke of the brush.

Shelley: That sounds insufferable.

Rob: It is also the only reason you get a horse. Take the voice away halfway and the rest of the statue wanders off.

Shelley: So the half made picture is listening.

Rob: At every stroke it looks across at that list and adjusts what it was about to do. That looking is called attention.

Shelley: Attention, as in the word.

Rob: And the flavour where one thing looks across at something else is cross attention. What is inside it is the group chat, and that had its own conversation. What you need is when.

Shelley: Which is every stroke?

Rob: Every stroke. Steering, not an instruction left at the door. The word for the whole arrangement is conditioning.

Shelley: Is that steering strong enough on its own?

Rob: Not as strong as people wanted. So there is a trick laid on top, and it is strange.

Shelley: Lay it out.

Rob: At every stroke you run the sand detector twice. Once having heard your sentence. Once having heard nothing at all.

Shelley: Deliberately deaf, the second time.

Rob: Deliberately deaf. Now you hold two guesses about the sand. One from a thing that heard your sentence and one that did not.

Shelley: And you compare the two?

Rob: You take the difference and push further along it. You are saying whatever my sentence changed, give me more of that.

Shelley: So you exaggerate the effect of my words.

Rob: Precisely that. And how hard you push is a dial. The name of the trick is classifier free guidance. CFG.

Shelley: And the dial is called?

Rob: The guidance scale. Turn it low and you get something creative and only loosely about what you asked for.

Shelley: And turn it up?

Rob: It obeys closely and starts to look wrong. Over saturated. Stiff. The published work says exactly that and had to invent a fix.

Shelley: So there's a middle.

Rob: There is always a middle, and hunting for it is most of what people mean when they say prompting.

Shelley: Now go back. The deaf run. What's it hearing?

Rob: By default, nothing at all. An empty sentence. But you are allowed to put something in that slot.

Shelley: Such as what?

Rob: A list of what you do not want. Blurry. Extra fingers. A watermark in the corner. That list goes in the deaf slot.

Shelley: So I'm pushing away from it.

Rob: You are pushing away from it. That is what a negative prompt is. Not a filter, not a rule. The thing you run from.

Shelley: And it only works because of the twice.

Rob: It only works because of the twice. Switch the guidance off and that list does nothing, because there is nothing to run from.

Shelley: Put the whole control panel in front of me.

Rob: Six things on it, and you have already earned four.

Shelley: Read me the four.

Rob: How many strokes? Which sampler. How hard the guidance pushes. And what goes in the deaf slot.

Shelley: Which leaves two.

Rob: The first is the burying recipe. How fast the sand went on during training.

Shelley: That isn't something I turn.

Rob: You inherit it. But somebody chose it, and a bad choice does real damage. There is published work on a recipe that destroyed the picture faster than it needed to.

Shelley: And the repair?

Rob: A gentler curve. It stopped throwing sand quite so hard at the very beginning and the very end.

Shelley: And the sixth thing?

Rob: The static you started from. Somebody has to produce that first rectangle, and a number settles which rectangle you get.

Shelley: So the randomness isn't really random.

Rob: It is bookkeeping. Same starting rectangle, same sentence, same settings, same picture, every time. That number is called the seed.

Shelley: Which is why people write them down.

Rob: It is the only handle you have on a machine that otherwise hands you something different every time.

Shelley: You said every stroke is the entire machine running again.

Rob: I did, and I have to own up to what that used to cost.

Shelley: Cost how much.

Rob: Doing this on the actual dots of a large photograph is brutal.

Shelley: And now it sits on a laptop.

Rob: Now it sits on a laptop, and the reason is one idea, and the idea is almost embarrassing.

Shelley: Say the idea.

Rob: Do not work on the picture. Shrink it first, do the whole brushing business in the small version, then blow it back up.

Shelley: Shrink it by how much.

Rob: A common setting takes each side down by eight.

Shelley: Which sounds like eight times cheaper.

Rob: It sounds like it and it is not. Shrink both sides by eight and you have sixty four times fewer places to work.

Shelley: Because a picture is flat.

Rob: Because a picture has two directions and you shrank both. That one piece of arithmetic is why any of this reached ordinary machines.

Shelley: And who does the shrinking?

Rob: A shrinker and an expander, trained together beforehand so the expander can undo what the shrinker did.

Shelley: And they have a name?

Rob: VAE. And the thing to hold is what it is not. It does not make your picture. It is the lift down and back up.

Shelley: And the small version has a name.

Rob: The latent. Running the whole business down there is called latent diffusion.

Shelley: Is anything lost on the way down.

Rob: Fine detail, and how far to shrink is a genuine trade. Go too far and you are brushing sand onto something that no longer holds the picture.

Shelley: And the expander at the end?

Rob: Is a separate part, and you can swap it. There are replacement expanders that drop straight in and change skin and texture, leaving the picture maker alone.

Shelley: What in all of that has changed lately.

Rob: Three things, and the first one is under the floor.

Shelley: Under the floor meaning.

Rob: I have been saying the sand detector as though it were one fixed thing. It is a network, it had a shape, and the shape was named after a letter.

Shelley: Which letter was it?

Rob: The letter U. Why it is drawn that way is next, and it deserves the time.

Shelley: And that shape is going?

Rob: At the front of the field it has largely gone. In its place, a stack of blocks that reads sentences, working on the small grid.

Shelley: The thing that reads sentences is now the thing that detects sand.

Rob: No. Still a sand detector. The name for it is DiT, and the argument is that it improves more predictably as it grows.

Shelley: And the second thing?

Rob: Video. You add time to the picture and brush a whole run of frames at once rather than one.

Shelley: And the hard part there is?

Rob: Keeping a thing the same thing. If the coat is blue in one frame and green in the next, the eye catches it and the run is ruined.

Shelley: So the difficulty isn't the picture.

Rob: The difficulty is the agreement between the pictures. It has a name. Temporal consistency. And the work is mostly tying frames to each other rather than making each one better.

Shelley: And the third?

Rob: The strokes. I told you the slowness was the honest complaint. That complaint is being paid off.

Shelley: Paid off how?

Rob: By training a second thing to cover the whole distance in very few strides. Dozens down to a handful, and at the strongest, one.

Shelley: One stroke, total.

Rob: One stroke, at a quality people accept. Which quietly reopens every argument that ever rested on this being slow.

Shelley: Give me places this is doing real work.

Rob: Three, and the first pays for the other two. Scenes you cannot film.

Shelley: Can't film meaning?

Rob: A child running into a road after a ball, at dusk, in heavy rain. A car needs to have seen many of those and you cannot arrange one.

Shelley: So you make them.

Rob: You make them. Rare, dangerous and specific, produced to order, in any weather you like.

Shelley: And the second?

Rob: Catalogue photography. One studio shot of a chair becomes a room, a garden and a shop window, none of which costs a day of work.

Shelley: And the third?

Rob: A medical set with too few of the rare thing in it. You generate more so the set stops leaning.

Shelley: That sounds like it ought to be against the rules.

Rob: It is less alarming than it sounds, and there is one hard limit you should carry out of here.

Shelley: Say the limit.

Rob: It cannot invent information that was never in what it was shown. It rearranges and recombines. It cannot tell you about a condition it has never seen.

Shelley: Three things I can repeat.

Rob: Two directions. Adding sand is fixed and is training. Taking sand off is learned and is making. Swap those and everything else comes out wrong.

Shelley: That is one.

Rob: It guesses the sand, not the statue. What you see is whatever is left when a very good guess about static is subtracted, over and over.

Shelley: And the third?

Rob: Your words are a voice at every stroke, not an instruction left at the door. And the trick that makes them bite runs the thing twice and pushes away from the deaf run.

Shelley: And what did you skate past?

Rob: The letter U. I named a shape and told you nothing about it.

Shelley: Next one, then.

Rob: Next, and it does not start with pictures. It starts with a doctor drawing a line around a tumour and needing to be right to the last dot.