Reverse engineering GPT-4o image gen via Network tab – here’s what I found

I am very intrigued about this new model; I have been working in the image generation space a lot, and I want to understand what's going on

I found interesting details when opening the network tab to see what the BE was sending – here's what I found. I tried with few different prompts, let's take this as a starter:

"An image of happy dog running on the street, studio ghibli style"

Here I got four intermediate images, as follows:

https://preview.redd.it/ew32o34z4ere1.png?width=2048&format=png&auto=webp&s=0ea0551b7cb262d4b167911201011e43fa9d5fe6

We can see:

• The BE is actually returning the image as we see it in the UI
• It's not really clear wether the generation is autoregressive or not – we see some details and a faint global structure of the image, this could mean two things:
  • Like usual diffusion processes, we first generate the global structure and then add details
  • OR – The image is actually generated autoregressively

If we analyze the 100% zoom of the first and last frame, we can see details are being added to high frequency textures like the trees

https://preview.redd.it/a2z8iozz4ere1.png?width=2608&format=png&auto=webp&s=7b64dffff754007d811642d4611fddd98efaf294

This is what we would typically expect from a diffusion model. This is further accentuated in this other example, where I prompted specifically for a high frequency detail texture ("create the image of a grainy texture, abstract shape, very extremely highly detailed")

https://preview.redd.it/gz6alyp05ere1.png?width=2048&format=png&auto=webp&s=fcb0be6ec1b38a0f7986b1a3ed11ea07073bccae

Interestingly, I got only three images here from the BE; and the details being added is obvious:

https://preview.redd.it/l3l0i4e15ere1.png?width=2058&format=png&auto=webp&s=1bfb7da6db385e7914711c8241c43dff3c83b857

This could be done of course as a separate post processing step too, for example like SDXL introduced the refiner model back in the days that was specifically trained to add details to the VAE latent representation before decoding it to pixel space.

It's also unclear if I got less images with this prompt due to availability (i.e. the BE could give me more flops), or to some kind of specific optimization (eg: latent caching).

So where I am at now:

• It's probably a multi step process pipeline
• OpenAI in the model card is stating that "Unlike DALL·E, which operates as a diffusion model, 4o image generation is an autoregressive model natively embedded within ChatGPT"
• This makes me think of this recent paper: OmniGen

There they directly connect the VAE of a Latent Diffusion architecture to an LLM and learn to model jointly both text and images; they observe few shot capabilities and emerging properties too which would explain the vast capabilities of GPT4-o, and it makes even more sense if we consider the usual OAI formula:

• More / higher quality data
• More flops

The architecture proposed in OmniGen has great potential to scale given that is purely transformer based – and if we know one thing is surely that transformers scale well, and that OAI is especially good at that

What do you think? would love to take this as a space to investigate together! Thanks for reading and let's get to the bottom of this!