this is maybe one of the most bonkers machine learning research i've seen all year.

Whaaaaat!? This is crazy right?

I'm just blown away. Editing weights? Writing weights from scratch!?

So how would this be in the training? Does OpenAI train models on their own epochs?

How long until OpenAI will disable this feature, though?

(of course for users, not themselves)

I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.

...I think this may be what it takes for me to finally admit that LLMs must have their own, alien form of consciousness. I am struggling to properly articulate my exact thoughts on this, so I may have to come back later with some barely comprehensible ramblings about what-it's-like-ness and how an LLM being able to read weights _surely_ means it's conscious to anyone who isn't a solipsist.

... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.

This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.

This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.

[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...

This could set us on a path to weights becoming thew new compilation target.

Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

This is straight black magic. llama 70b, let’s see what we can push you to