NVIDIA Generative AI. More Than Text: Modalities, CLIP, and One Shared Space.
Hosts: Shelley and Rob. Cloudadorn Academy.
This episode is narrated using AI voice technology. The content and script are original.

Shelley: I'm Shelley, I ask the questions. Rob builds these things for a living, which is why he gets the answers.

Rob: Morning.

Shelley: Somebody said two words to me on Friday. Nice job.

Rob: Sounds pleasant.

Shelley: It wasn't pleasant. It was flat, and slow, and they were looking straight past me while they said it.

Rob: So the words were positive and the meaning was the opposite of the words.

Shelley: And I want to know how a machine is ever supposed to catch that.

Rob: It cannot, if all you hand it is the words. The words are not where that lived.

Shelley: Then where did it live?

Rob: In the gap. Between what was said, how it was said, and what the face was doing.

Shelley: And one of those on its own.

Rob: Gives you a confident wrong answer.

Shelley: So the meaning is in the mismatch.

Rob: Sarcasm only exists in the mismatch. That is not a neat line, it is the reason today exists.

Shelley: Give me the word for the kinds of thing.

Rob: A modality. It means a distinct type of data. Words. Pictures. Sound. Video.

Shelley: Four of them.

Rob: Four that matter here. And a model is multimodal when one model takes in, or produces, or reasons across more than one of them at the same time.

Shelley: At the same time is doing work in that sentence.

Rob: It is doing all of the work, and we come back to it.

Shelley: And why bother?

Rob: Because meaning frequently lives in the relationship between two of them and not inside either one. Which is your Friday.

Shelley: Has anyone measured that, or is it a nice story?

Rob: Measured. There is a study of your exact problem, sarcasm on video, and the paper lists the cues.

Shelley: Name them for me.

Rob: A change of tone. Overemphasis on one word. A drawn out syllable. A straight face. None of those survive a transcript.

Shelley: And using more than one stream helped.

Rob: It did, against using the streams one at a time.

Shelley: So how does a machine hold a sound?

Rob: Everything becomes numbers. That is the only thing a model can hold. The interesting part is what each type goes through first.

Shelley: Start with words.

Rob: Words get cut into pieces and each piece becomes a list of numbers.

Shelley: Now do pictures.

Rob: A picture arrives as a grid of dots. Something has to turn that grid into a sequence of numbers.

Shelley: And there's one way to do that.

Rob: There are two families of reader that do it. Which one you pick is a real decision, and it is next, not today.

Shelley: What about sound?

Rob: Sound is the one people find surprising. Before anything reads it, the wave gets turned into a picture of itself.

Shelley: A picture of a sound.

Rob: A chart. Time along the bottom, pitch up the side, brightness for how much of that pitch is present.

Shelley: That's still one long thing though.

Rob: Chopped into short overlapping slivers of time. One of the best known transcription systems does precisely that, and writes down the length of the slivers in its own paper.

Shelley: So the sound model is looking at an image.

Rob: Effectively.

Shelley: And what about video?

Rob: Frames, plus a clock. And they do not take one frame at a time. They take blocks that cover a bit of space and a bit of time at once.

Shelley: Give me the word for the thing doing the turning.

Rob: The encoder. One per type. That is the whole of the first table anybody draws about this subject.

Shelley: Fine. So what do these things actually do?

Rob: Put what goes in down one side and what comes out along the other, and almost every system you will meet is one cell of that grid.

Shelley: Walk me through it.

Rob: Picture in, words out. Describe the photograph. That is captioning. One of its landmark papers calls it the problem that connects seeing and language.

Shelley: And if I want to ask about the picture rather than be told about it?

Rob: Same side of the grid, harder job. You hand it an image and an open question in ordinary language, and it answers in ordinary language. VQA.

Shelley: And the other direction?

Rob: Words in, picture out. That is a generator, and how it works is two hours away.

Shelley: What about sound?

Rob: Words in, sound out is speaking. TTS. Sound in, words out is transcribing. ASR.

Shelley: Those feel old and boring next to the rest.

Rob: One of the systems was trained on six hundred and eighty thousand hours of audio pulled off the internet.

Shelley: Off the internet.

Rob: Scraped, at that scale. Nothing about that is boring. Same lesson as the rest of today.

Shelley: That leaves video.

Rob: Words in, video out is video generation. Video and a question in, words out is understanding footage, which the grid keeps as its own row rather than a picture row with extra frames.

Shelley: And the one that makes people nervous.

Rob: Several types in, movement out. A camera, an instruction, and what comes out is not a sentence. It is a robot arm going somewhere. Vision language action. VLA.

Shelley: How much of knowing this subject is knowing that grid?

Rob: More than you would like. Put a described system in a cell and you usually know what it is and what it cannot do.

Shelley: Here is one. I take a recording, I run it through a transcriber, I paste the transcript into a chatbot. I have built a multimodal system.

Rob: You have not. You have built a cascade.

Shelley: What's the difference? Sound went in, an answer came out.

Rob: Two models in a row, each doing one type. Nothing anywhere in that chain ever holds sound and words at the same time.

Shelley: And that matters because?

Rob: Because of Friday. The transcript is where the tone died. Everything that made nice job sarcastic was gone before the second model saw a single word.

Shelley: So it isn't decided by what goes in and what comes out.

Rob: What decides it is whether one model is jointly holding more than one type.

Shelley: And if it isn't?

Rob: Two models passing text between them is a cascade, and it will be blind in exactly the way the middle step is blind.

Shelley: So text is a bottleneck I chose.

Rob: Text is a bottleneck you chose. Often the right choice. Never a free one.

Shelley: Then if joining them is the point, what's hard about joining them.

Rob: Lining them up. If the sound and the picture are half a second out of step, the mismatch your model finds is an artefact of the timing and not sarcasm.

Shelley: And it can't tell the difference?

Rob: It will happily learn the artefact.

Shelley: That sounds like a small engineering problem.

Rob: It is one of five named hard problems for this whole field. Alignment.

Shelley: So somebody solved a piece of it.

Rob: One model did, for pictures and words. Start with what was wrong before it.

Shelley: Go on, what was wrong.

Rob: An ordinary picture classifier is trained on a fixed list of labels. Cat, dog, fire hydrant. It can never say anything outside that list.

Shelley: And if I want a new category?

Rob: You go and get labelled data and you train again.

Shelley: Expensive and slow.

Rob: Both. So the question somebody asked was, what if we never use labels at all. What if the supervision is the sentence that happened to be sitting next to the picture on the internet.

Shelley: Alt text. Captions.

Rob: Whatever was there. Four hundred million pairs of a picture and its words, scraped, labelled by nobody.

Shelley: And you train what on that?

Rob: Two readers at once. One reads pictures, one reads sentences.

Shelley: Two, running separately.

Rob: Side by side, and both are made to put their answer into the same space, with the same number of dimensions. Two different kinds of thing become directly comparable.

Shelley: Comparable in what sense.

Rob: By direction. You measure how closely two of these number lists point the same way. That measure is a cosine similarity, and it is the only arithmetic in the story.

Shelley: And the training itself?

Rob: Take a batch of pictures with their real captions. Score every picture in it against every caption in it. That is a square grid of scores.

Shelley: Most of which are wrong pairs.

Rob: Nearly all of them. The true pairs are the diagonal. Everything off the diagonal is a picture sitting next to somebody else's caption.

Shelley: And the training does what with that.

Rob: Pull the diagonal together. Push everything else apart.

Shelley: That is it.

Rob: Nothing else. It is called a contrastive objective, because it learns by contrast rather than by being told the answer. Say that sentence and you understand the model.

Shelley: Give me the picture.

Rob: A library where every book is shelved beside its own cover illustration. The librarian walks the aisles checking.

Shelley: Checking for what?

Rob: Is this cover next to its own book? Move them closer. Is it next to the wrong book? Push them apart.

Shelley: And it does that forever?

Rob: Millions of times. And after enough of it the shelf position itself carries the meaning. You can find a book you have never seen by walking to where its description would sit.

Shelley: And the name?

Rob: CLIP. What it produces is one space where a photograph of a dog and the string a photo of a dog land next to each other.

Shelley: With no labels anywhere.

Rob: No list of labels anywhere in the building. The proper name for that space is a shared embedding space.

Shelley: So what do I do with it?

Rob: Classify things it was never trained to classify. You write the candidate sentences yourself. A photo of a cat. A photo of a fire hydrant.

Shelley: And then what happens?

Rob: You put the picture in the space, you put your sentences in the space, you take the nearest one.

Shelley: On categories nobody trained it on.

Rob: That is the claim, and they checked it across a lot of different vision collections rather than one. Zero shot, meaning no training run of any kind for the new categories.

Shelley: Why the odd phrasing? A photo of a cat rather than cat.

Rob: Because the thing was trained on real sentences off the internet, and a bare word is not a sentence. The template drags your query back towards the kind of text it actually saw.

Shelley: That's the only reason.

Rob: There is a second. A bare word carries no context, and plenty of words mean two things.

Shelley: Where does that pay off in a building rather than a paper?

Rob: Anywhere the list of categories moves faster than you can retrain. Content policies change weekly. A fixed classifier needs a training cycle for every newly prohibited category. This needs a new sentence.

Shelley: And searching my own photographs?

Rob: Show me the red bicycle by the canal. Nobody ever tagged that. The sentence goes into the space, the pictures are already in the space, you sort by distance.

Shelley: Does it go the other way?

Rob: Both directions, and that is the point of one space rather than two. Sentence finds pictures. Picture finds sentences. Same arithmetic, run the other way round.

Shelley: And I have heard the name attached to a score.

Rob: Same space, different use. If you can measure how close a picture is to a sentence, you can measure whether a generated picture matched the prompt that asked for it.

Shelley: So it grades the generators.

Rob: That is a scoring job, and it is not for today.

Shelley: Does CLIP draw.

Rob: It draws nothing. It is a pair of readers. A scorer and a placer.

Shelley: Then why is it in every conversation about picture generators?

Rob: It made a generation of them possible by giving them a way to be pointed at a sentence. It has never produced a single image itself.

Shelley: So where does it fall over?

Rob: Counting. Ask it how many of a thing are in a photograph and it is unreliable, and the people who built it put that in their own limitations section.

Shelley: That's a strange thing to be bad at.

Rob: It is a clue about what it learned. A later group pushed harder.

Shelley: And what did they find?

Rob: Poor understanding of relations. Blunders when linking a thing to the word describing it. And almost no sensitivity to word order.

Shelley: It ignores word order.

Rob: Rearrange the sentence into nonsense and the score barely moves. Their phrase for it is that the model behaves like a bag of words.

Shelley: Which for this model is?

Rob: A serious charge. Its entire job is matching sentences to pictures.

Shelley: Anything else it's bad at.

Rob: Handwriting. On handwritten digits it loses to the simplest statistical method you can run straight on the raw picture. The paper's own word for that method is embarrassingly simple.

Shelley: Why that, of all things?

Rob: Because there was almost nothing resembling handwritten digits in four hundred million internet pictures. Which is the honest version of the whole story. It knows what the internet had a lot of.

Shelley: And what else did the internet have a lot of?

Rob: Whatever those captions already assumed about people. The pairs were unfiltered and uncurated, and the team says so plainly.

Shelley: And then let it go?

Rob: They went further. Deployed use of the released model is out of scope. Surveillance and face recognition are out of scope regardless of how well it performs.

Shelley: That's a strong thing to say about your own model.

Rob: There is a stronger one. They say the performance depends heavily on which categories you choose to include and leave out.

Shelley: That isn't a small footnote.

Rob: It is not a footnote at all.

Shelley: Is this still how it's done?

Rob: The idea is. The arithmetic moved. The original CLIP has to look at the whole grid of scores at once, because it turns each row into a competition between all the candidates.

Shelley: Which means the batch has to be huge.

Rob: Very large, purely to supply enough wrong pairs. The successor scores each pair on its own.

Shelley: Meaning what exactly.

Rob: Is this caption right for this picture, yes or no, with no view of the rest of the grid required.

Shelley: And that helps how?

Rob: It performs better at small batch sizes and it also scales further up. Both ends. That one is called SigLIP, and there is already a second generation of it.

Shelley: So the trick survived and the arithmetic underneath got replaced.

Rob: Which is most of the progress in this field, honestly.

Shelley: Give me the summary I can repeat.

Rob: Three things. A modality is a type of data.

Shelley: That is one.

Rob: A system is only multimodal if a single model is holding more than one of them at once. Which rules out the transcribe then ask pipeline.

Shelley: And the third?

Rob: Put two different kinds of thing into one space and you can compare them with arithmetic. Everything else today is a consequence of that one move.

Shelley: And the thing you skated past?

Rob: Twice. I said something turns the picture into a sequence and I did not say what.

Shelley: And next is that something?

Rob: It is, and it is stranger than it sounds.

Shelley: Stranger in what way.

Rob: You stop treating a picture as a picture. You cut it into small squares, put them in order, and hand them to the same machinery that reads sentences.