AI training is currently a process of making the AI remember the dataset. It doesn't involve the AI thinking about the dataset and drawing (and remembering) conclusions.

It can probably remember more facts about a topic than a PhD in that topic, but the PhD will be better at thinking about that topic.

What about actively obtained data - models seeking data, rather than being fed. Human babies put things in their mouths, they try to stand and fall over. They “do stuff” to learn what works. Right now we’re just telling models what works.

What about simulation: models can make 3D objects so why not give them a physics simulator? We have amazing high fidelity (and low cost!) game engines that would be a great building block.

What about rumination: behind every Cursor rule for example, is a whole story of why a user added it. Why not take the rule, ask a reasoning model to hypothesize about why that rule was created, and add that rumination (along with the rule) to the training data. Providing opportunities to reflect on the choices made by their users might deepen any insights, squeezing more juice out of the data.

The only real important thing in AI is data, not infrastructure, not fancy methods.

I don't know if it matters. Even if the best we can do is get really good at interpolating between solutions to cognitive tasks on the data manifold, the only economically useful human labor left asymptotes toward frontier work; work that only a single-digit percentage of people can actually perform.

There are new ideas, people are finding new ways to build vision models, which then are applied to language models and vice versa (like diffusion).

The original idea of connectionism is that neural networks can represent any function, which is the fundamental mathematical fact. So we should be optimistic, neural nets will be able to do anything. Which neural nets? So far people have stumbled on a few productive architectures, but it appears to be more alchemy than science. There is no reason why we should think there won't be both new ideas and new data. Biology did it, humans will do it too.

> we’re engaged in a decentralized globalized exercise of Science, where findings are shared openly

Maybe the findings are shared, if they make the Company look good. But the methods are not anymore

> It’s not crazy to argue that all the underlying mechanisms of these breakthroughs existed in the 1990s, if not before.

That's not super relevant in my mind. It's because they're showing fruit now that will allow research to move forward. And the success, as we know, draws a lot of eyeballs, dollars, resources.

If this path was going to hit a wall, we will hit it more quickly now. If another way is required to move forward, we are more likely to find it now.

There are new things being tested and yielding results monthly in modelling. We've deviated quite a bit from the original multi head attention.

I think this is reflective of current state, but does not mean this will be the future. I think there is a lot of innovation to come on revisiting some of the 1990s principles of back propagation and optimization. Imagine if you could train current models to optimal weights in 1 day or 1 hour instead of weeks/months?

Just a hypothesis of mine

At Axon, we see the same pattern: data quality and diversity make a bigger difference than architecture tweaks. Whether it's AI for logistics or enterprise automation, real progress comes when we unlock new, structured datasets, not when we chase “smarter” models on stale inputs.