Hardik's Notes · AI, simply
How Google steers an AI image maker without rebuilding it
Ask for a lizard in sunglasses and you may get a lizard with none. Google Research built a small add-on that steers the model.
What goes wrong?
Ask an image tool for “a lizard wearing sunglasses”. You may get a fine lizard with no sunglasses. Push the tool harder and the lizard’s face may come out warped.
The tool is trying to do two things at once: follow your words, and keep the picture looking good.
What did Google Research build?
On 29 September 2026, Google Research wrote about a method called Diffusion Controller. The paper first went online on 7 March 2026.
An image maker like this starts from random static (noise) and cleans it up step by step until a picture appears. Diffusion Controller adds a small second network (a small set of learned numbers) beside the big model. At each cleaning step it nudges the picture towards what you asked for. The big model is not touched.
The usual way
- Change the big model itself
- Needs full access to its insides
- Risk: it forgets how to make good pictures
Diffusion Controller
- Leave the big model frozen
- Add a small network that nudges each step
- A penalty stops the nudges from wrecking the picture
The blog compares it to a steering damper on a motorcycle. The motorcycle is not rebuilt. A small part is added to keep it on course.
Did it work?
About 7 in 10
The add-on beat the unchanged model in picture-preference scores, with the big model frozen.
A “win” here means a program that predicts human taste (HPS-v2) liked the new picture more than the old one. Humans also compared pictures in one of the setups.
In two of three training methods, the add-on also beat LoRA. LoRA is a popular way to tune a model by changing a small part inside it. It needs access to the model’s insides. The add-on does not.
What’s the catch?
- The test used one older open model, Stable Diffusion v1.4. Results on today’s big tools are unknown.
- Google’s blog says it can work with closed models. The paper did not test one.
- The very high scores (90 in 100) come from versions that also change the model.
- No code was linked on 8 October 2026, so you cannot try it yet.
Try it
Until tools like this are in your hands, you can steer them yourself:
3 habits when an image tool ignores your prompt
- Ask for one hard thing at a time, then add the next.
- Say the missing part first: 'sunglasses on a lizard'.
- Keep what worked. Change one word per try and note what changed.
Sources
- Diffusion Controller: Framework, Algorithms and Parameterization, arXiv, 7 March 2026
- How Diffusion Controller unifies and simplifies AI image generation, Google Research, 29 September 2026
Keep reading
Three more from the notebook.
5 October 2026
Open now: 5 contests for students (October 2026)
Five contests students in India can enter now, from a one-weekend NASA hackathon to a 10,000 dollar environment prize. Deadlines from 29 October.Read it22 September 2026
How to find the one thing that is broken
Your circuit will not light. Your code will not run. Here is the method engineers use, and it works on homework too.Read it
22 September 2026The Arduino UNO Q, explained in five levels
The board you already know has a Linux computer in it now. Explained five times: to a child, a teen, a college student, a grad student, an expert.Read it
Beyond the notebook
This is what we do in schools.
A year-long programme for classes 6 to 12. Creativity, taught as a skill, inside the classroom you already have, led by your teachers.