Before AI could draw, it had to learn how to see

Introduction

Think about the last time you asked ChatGPT to generate an image. Maybe your prompt was as simple as: “Generate an image of a dog.” A few seconds later, it produced a convincing image of exactly what you asked for. But have you ever stopped to wonder how it knew what a dog looks like?

After all, it doesn’t see the world the way we do. To it, an image is nothing more than a bunch of pixel values, and the word “dog” is just another sequence of text.

The answer lies in one of the most influential models in modern AI: CLIP (Contrastive Language–Image Pretraining). Before AI could generate images from language, it first had to learn how to connect language with vision. This article breaks down exactly how that happened.

Now CLIP wasn’t the first model which tried to connect text and images, but its approach is what made it different. Instead of giving the model an image and asking what it contains, CLIP gives the model an image-text pair and asks whether they match. I’ll tell you how it does that and how this one small change helped pave the way for modern image genertion models.

How CLIP actually works

CLIP is actually a combination of two models: one responsible for converting text into a 512 dimensional vector embedding, and another responsible for converting images into a 512 dimensional vector embedding. These vectos share the same embedding space. Below is an illustrative 3D projection of the 512D embedding space that displays the idea in a simpler manner.

Notice how the text and images that match are close to each other, we can predict whether they match by measuring the cosine similarity between their image and text vectors. The goal is to maximise the cosine similarity between matching text-image vector pairs.

But that’s not all. CLIP also minimises the similarity between non-matching pairs. The entire idea is to take images and text, embed them into a 512-dimensional vector space, while maximising the cosine similarity of matching image-text pairs and minimising it for non-matching pairs. This method is called contrastive pre training.

This image is a part of the actual paper that reasearchers at openai published and demonstrates the concept of contrastive pre training, the vectors T1-I1, T2-I2….Tn-In are the matching pairs, as seen inside the blue boxes. But the goal isn’t just to maximise the cosine similarity between them, its also to minimise it between the non matching pairs within the batch.

Here is that same matrix, but running. Press train and watch the diagonal pull away from everything around it.

Untrained: every pair looks alike

Train

“a photo of a dog”“a photo of a cat”“a photo of a bicycle”“a photo of a building”“a photo of a mountain”“a photo of a sunset”

← rows are imagescolumns are captions ↑

Tap a square to see the pair it scores.

Now, if you remember that text and image embeddings coexist in a shared space, this actually makes the model capable of much more. We had already seen something similar with word embeddings, where relationships between words could emerge as geometric relationships in the embedding space. One of the most famous examples is king − man + woman ≈ queen. This means that if you take the vector representing king, subtract the vector representing man, and add the vector representing woman, you get a vector that points surprisingly close to queen in the embedding space.

Here is a live example that demonstrates a similar idea, there is an image of “a person wearing a hat” and you have a few images as options, you can pick one and see the results of the vector arithmetic. This time, instead of operating on word embeddings, we subtract the image embeddings and then compare the resulting vector with the embeddings of several text prompts and you can see the results yourself.

Choose what to subtract

person with hatperson with glassesperson

Î_{person with hat and glasses} − I_{person with glasses}

person with hat and glasses

−

person with glasses

1. a photo of a hat0.054
2. a photo of glasses-0.012
3. a photo of a mountain-0.012
4. a photo of a person-0.014
5. a photo of a bicycle-0.014
6. a photo of a banana-0.015
7. a photo of a dog-0.021

I hope this demonstrates just how revolutionary CLIP was, and how it helped pave the way for future image models. Of course, CLIP itself doesn’t generate images. What it gave researchers was something equally important: a way to represent the relationship between language and visual concepts, as you can see in the example above.

This idea became especially powerful when combined with generative models. A generative model could use a text representation to guide the generation process toward the visual concepts described by the prompt. This connection between language representations and visual representations became a fundamental ingredient in the systems that eventually produced the text-to-image models we use today.

But there are many more nuances to how these text-to-image models actually work and that’s a story for another time.

Follow me on if you want to stay updated and learn how these image models work. But this is what I meant when I said:

Before AI could draw, it had to learn how to see