Generative AI models have an enemy: humans
LLMs learned from us. The challenge now is for them to learn beyond us: RLHF, RLVR, and world models, explained without jargon.
Generative AI models are very good at working with, and generating, text, video, or audio based on information we humans already know and hand to them.
They can summarize, write, program, translate, reason over documents, and combine ideas better than almost any tool we had before. But:
What happens when we want AI to discover things humans do not know yet?
- LLMs and multimodal models are extremely powerful. Most of us already use these tools every day, and some of us have developed something close to a dependency on ChatGPT, Claude, or Gemini.
- But these models have a serious problem. The main one, at least so far, is that they depend on us, humans, to keep improving.
That is a ceiling. The problem is that humans are not a feature in this story. We are a bug.
Fei-Fei Li, one of the most recognized researchers in the field, has pointed at the problem with the obsession with LLMs:
📁 Fei-Fei Li, former Google Chief Scientist, says the industry is dangerously fixated on language models.
Most of the real economy is physical, perceptual and spatial.
Once AI fully understands the visual world, it stops being a chatbot and starts becoming infrastructure.
When human knowledge is the ceiling
- Imagine for a moment that ChatGPT had existed thousands of years ago.
With the knowledge we had at the time, if you had asked whether the Earth was flat, it would have said yes. Why? Because that was the established truth in the books and conversations of the era.
- Another example. If you had asked ChatGPT before 1954 whether a human could run a mile in under four minutes, it would have said no.
Because no human had done it, and the widespread belief was that it was physically impossible. When Roger Bannister did it, it was not because human physiology changed. It was because someone dared to try.
What do these two examples have in common? That the "truth" a model trained on human data can give you is always limited by what humans know at that moment.
And this is exactly what happens with LLMs today. They are as good as the data we gave them. But also as limited.
This has a technical name: RLHF, Reinforcement Learning from Human Feedback.
It is the process by which models are adjusted based on human feedback.
We are the teachers, the curators, the judges. That is the strength and, at the same time, a flaw in the system.
What if models could learn on their own?
This is exactly the question the Google DeepMind team asked when it built AlphaGo, a model that ended up beating the world Go champion.
The truly impressive part of that match was not just that it won. It was that the model made a move (the now famous "move 37") that no human would have considered.
During the game, the commentators thought it was a mistake. Later they realized it was a move the model had discovered on its own, after millions of iterations playing against itself. A move outside humanity's repertoire.
Today, we share a breakthrough on the planar unit distance problem, a famous open question first posed by Paul Erdős in 1946.
For nearly 80 years, mathematicians believed the best possible solutions looked roughly like square grids.
An OpenAI model has now disproved that
A few weeks ago something similar happened in another domain. OpenAI reported that one of its models solved an open math problem. Again: an answer that was not in the human training data. It emerged from the model learning beyond us.
This has a new name:
- Reinforcement Learning from Verifiable Rewards (RLVR)
The premise is to imitate the way humans learn: by reward.
Every action we take has a consequence. A positive or negative signal, depending on the context.
- If you put your hand in the fire, your nervous system sends a negative signal. You learn not to do it again.
- If you get a perfect score in class and your parents congratulate you, that is a positive signal. You learn that studying is worth it.
How RLVR works (simplified)
- You build an environment. Today this can mean many things, but it is basically a space where a task has rules, state, and clearly verifiable rewards (positive or negative).
- You let the model explore. As it interacts with the environment, the model learns. For example, how to solve an addition or a multiplication. If it makes many wrong decisions, it gets penalized. If it makes the right ones, it gets rewarded. And it moves toward wherever there is more positive signal.
The important detail: for this to work, rewards have to be verifiable.
For example: 2 + 2 = 4 is easy to verify.
- Solving a math problem is too.
- Writing code that passes a test suite is too.
- But "write an elegant email" or "give relationship advice" is not verifiable in the same way.
That is why RLVR works better today in math, code, and games than in, say, creative writing.
But careful: even here we are still inside the original problem. Humans design the environments and the rewards. And that brings in our biases and our limits.
If the environment is badly designed, the model learns strange things. There is an effect called "reward hacking," where models find absurd ways to maximize the reward that the human designer did not anticipate.
The truly bitter-lesson-pilled version would be a model that learns on its own, in the real world, without us telling it what is right and what is wrong. That is where the next concept comes in.
World models: learning in a simulation of the world
Ideally, models could interact with a real environment, just like we do, and discover for themselves what is a positive signal and what is a negative one.
This is particularly relevant for physical-world problems. For example: autonomous driving.
Companies like Waymo, Tesla, and Cruise use world models today. They are models that learned to represent the world (physics, roads, pedestrians, weather) with enough fidelity that an AI system can train inside them and learn without crashing real cars.
Even more: these models can predict the second-order consequences of an action. For example:
If a Waymo is thinking about turning left, it first predicts what will happen next: is someone crossing the street? Will the oncoming car brake? Is there an obstruction further down the road? The model simulates several possible futures, evaluates which one has the best outcome, and only then acts.
The same principle applies to robots of any kind. If you want a robot to learn to fold clothes, you do not want it to fail a million times with real clothes. You want it to fail a million times in a simulation, and then come down to the physical world with the behavior already polished.
The future of AI models
The next five to ten years in AI will not look like the last three. LLMs gave us a new interface for using human knowledge. But that is not necessarily the same as creating new knowledge.
Entire classes of problems will only be solved when models can learn from the world on their own: physical control, robotics, complex sequential decisions. In short: results that go beyond the patterns humans already know.