Posted by milkshakes 4 hours ago
The most interesting question to me is what will be consumed by the exponential like math seems to be undergoing, and what won’t. Writing has been quite stubborn, but I’ve noticed Fable to be quite a big step up there. How about politics? Will we develop new ways to let people express their own values in democracies, or will we just get much better at manipulation? How about experiment driven domains like biology?
What’s new about LLMs is that you can scalably manipulate people individually. It used to be that you could either have scale (speeches, tweets, interviews, website, etc.) or individual engagement (replying to mail/tweets/town hall questions.)
Now you can pull the history and preferences of an individual, then shape a message—in real time—to them, specifically. You can have conversations on social media with a single person and shape your message specifically to them.
Part of this can be good (you talk about what they care about, where 90% of broadcast messaging might not apply) and part of it can be bad (manipulation.)
My guess is that, in the US, the right will cynically adopt manipulation to great effect and the left will take a moral stand against shady practices and lose elections.
I think that statement may itself highlight how prevalent manipulation is.
I fully anticipate all groups to continue maximal manipulation they can. One thing with LLMs is that it'll be a far less unified view, so a "divide and conquer" strategy is what I anticipate.
Yes.
In general, you can think of the process as generating massive rollouts in generation N, and then compiling in the verifier/human feedback("gradient") signal into generation N+1. The time taken to make the rollout in generation N, and separately the time taken to get the same rollout in generation N+1, each grows constant in some tasks, linear in more, and exponential in some.
In the end, this becomes bottlenecked by time. Today, we can make statements like "I generated all these successful trajectories with 2 weeks of compute, in the next model it will be able to do it in 7 hours of compute", but very soon you'll find yourself making statements like "I generated.... with 8 months of compute, in the next model it can do it in 6 months", which isn't really enticing the same way you can _technically_ brute force passwords but it just needs prohibitive amounts of time and money. That is the "plateau". Note that, this point is quite far away. For example, at any point if we agree it plateaus, today's known hardware techniques such as fixed function accelerators give you a 10x timeline reduction immediately allowing for a few more cycles of improvement. This is not to mention future innovations, but of course none of that is helping with the benchmarks where the time needed is growing superlinearly.
In many math and coding benchmarks, we are still in the constant phase. These are the massive improvements we see every few months. I'm not making any prediction of what will plateau and what will not as it's not possible to make an informed prediction about these things IMO. But the observed fact is that some have already plateaud as in, they don't improve with reasonable inference time (likely superlinear growth).
> will we need mathematicians to translate
Let's take a sudoku analogy. The model is initially just doing the random value algorithm, but lets say you the human are watching it. You make one of the usual reductions and interject "hey you can stop trying 8 here because of ....". Over enough examples, you get to a point where the model is _forced_ to learn the logical pattern. Next generation, it will skip that number. After this, you can peak the distribution using simple 1/0 RL. Doing _pure_ 1/0 RL works decent, but its not frontier as its a very sparse signal.
For that lift, human (or even a better LLM, but if you're trying to improve a frontier LLM, there is by definition no better LLM) feedback becomes necessary. This is _why_ it is crucial that these models interface in natural language and is also why the labs are hiring AI tutors by the hundreds.
> But the long term is completely bewildering if you believe any of these trends can continue at a similar pace for the next few years.
For math and coding, for now we are in the phase where the times are just ... constant, so there's little reason to think it will stop soon. We still need humans to expand the frontier. It just becomes a matter of if its worth the cost of compute for running this generalized The Algorithm or not.
Given how well chess players internalized _many_ (not all) of alphazero's emergent chess knowledge, I am confident we wont have too much trouble figuring out any new math LLMs come up with, which will let us keep expanding the frontier by giving the LLM the next "lift". Only when we reach the stage where the time growth become exponential will this stop, IMO.
Is there even the tiniest reason to suspect that the people steering this progress will use it for the democratic good of all?
isn't it clearly split between verifiable not verifiable ? what is interesting about that question.
what did you notice ?
Whilst current models can't 'intuit' and come up with conjectures, they can certainly disprove some of them very quickly through the kind of grind that humans can't do. I suppose there really are some mathematicians out there today, whose last few years of study, have just been up-ended by this.
--
"Yes we are," insisted Majikthise. "We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!"
"What's the problem?" said Lunkwill.
"I'll tell you what the problem is mate," said Majikthise, "demarcation, that's the problem!"
"We demand," yelled Vroomfondel, "that demarcation may or may not be the problem!"
"You just let the machines get on with the adding up," warned Majikthise, "and we'll take care of the eternal verities thank you very much. You want to check your legal position you do mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we're straight out of a job aren't we? I mean what's the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?"
That's how they are finding these solutions though, unless we are just going to label intuition as something only humans can do. Like a submarine being unable to swim or whatever that example is.
The two places were seeing lots of movement are:
* Updates to lower/upper bounds. In many cases, these kinds of problems are the deep-math equivalent of calculating more digits of pi. Yes, if you throw time at it you'll break the record, but it may not be terribly worthwhile.
* Finding counter examples which disprove conjectures. This is really useful, and helps offset some positivity bias on the human side, often bringing together known tools from distant silos.
If you read the list of ten results, almost all fall into one of these buckets.
I disagree. I routinely let LLMs speculate or generate hypotheses along the way of helping with technical research. Sometimes they can prove the correctness of a concrete math idea but other times even an unproven conjecture helps with the numerical algorithm implementation and the result is then simply supported by additional data. I guess that any autoresearch-adjacent application has LLMs intuiting and coming up with hypotheses/conjectures—as do the steps/lemmas along a complex proof. In my opinion the modern LLMs are powerful intuitive thinkers that generate lots of conjectures of varying quality or importance.
Of course computers can grind in a way that humans can't. But now we have systems that convert the human-comprehensible ideas into a computer's plan of attack, in a way that greatly expands the frontier of ideas thus treatable.
People keep saying this. Why?
Surely the AI can complete the prompt “Generate new research questions based on these observations”?
When I read the reasoning traces of coding models they are constantly asking themselves questions and attempting to answer them.
They can't exit the hull until the "intuition" starts spawning points outside the convex hull.
Humans can “intuit” based on a much larger, if not unlimited, context. Also I just want to say that human cognition is something so insanely complex and deep that we will not understand it at all in my lifetime. To attribute all, or really any, aspects of human cognition to a machine at this point is silly to me.
Most humans are dumber than a box of rocks. Here in Seattle we had one of many light rail-related fuckups where they had to replace part of the line with buses. People piled into the front of one when it was full. When people got out they never moved back. As the driver struggled to close the door and people struggled to get in the wad of people never moved back to fill the ample space.
Chatgpt was smarter than the average person a while ago
This does not demonstrate a lack of intelligence. It demonstrates laziness and a lack of interest in spreading apart. Or just lack of consideration (or even malice) on the part of those at the back of the wad.
> Chatgpt was smarter than the average person a while ago
This is an absurd claim that fundamentally misunderstands what it means to be "smart". Reasoning that would get you to this conclusion would equally well apply to Google's search engine over a decade ago.
There are processes at work there that we don’t even have the language to describe.
(86 billion is the number ChatGPT, ironically enough, has given me a couple of times. I remember hearing for a long time that it was estimated to be somewhere in the ballpark of 100 billion. This is not my field of study.)
For the same reason that you can't draw a 15 of Diamonds from a regular card deck.
check here : 1. high dimensional sphere packing https://muchmirul.github.io/conjectures/sphere-packing/
2. multicolor ramsey number https://muchmirul.github.io/conjectures/multicolor-ramsey
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.
Maybe you’re the one who needs breaking out of your cached beliefs.
That’s not a credible position, but there isn’t anything that I or anyone else can say to someone who simply doesn’t want to believe something.
In my experience modern models are better at all tasks than models from two years ago, especially complex multi-step tasks.
I suspect GPT 5.6 would be even better at it, if given the same sycophantic system prompt and lack of guardrails.
It wasn't "better" it was better at kissing your ass which matches what a lot of people want in a partner.
There is irony here
I agree with the parent that we need to acknowledge that we're at a turning point in history. I lived through some of them (internet, ubiquitous personal computing). But it's somewhat difficult to comprehend the impact of this one for many people.
I do biomedical research at one of the top European research institutions. We're very well-funded, but I can clearly see the gap between us (say, top-100) and top-10. I also realize this gap is going to get so much wider unless we invest heavily in AI access (and I'm not so sure I can sell anything more expensive than $20 Claude subscription to the leadership).
I think people having 6-7 figure SOTA AI budgets will move exponentially faster than those who don't. That makes me worried.
So, for me, it's not a question of recalibrating expectations. We're way past that.
That's not what people mean when they say "moving the goalposts". It means that people are adamant that something wasn't important/hard/impressive once the "AI" solves it. And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
It's also very much not a new phenomenon. It's been happening since the 1980s. As you can see from this quote from GEB by Hofstadter:
> There is a related "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem: "AI is whatever hasn't been done yet."
-Alan Turing (allegedly)
I don't see how that's any better.
In recent years, I have commonly seen the phrase "you're moving the goalposts" deployed by the "it might be sentient" crowd to shoot down the "it's a stochastic parrot" crowd when the latter respond to a new development with "OK but...". In a well-understood field of inquiry, that would be a clear case of goalpost-moving, in the commonly-understood meaning of the phrase where requirements are retroactively changed in response to them having been met. Thank you OP. 'Artificial Intelligence', and indeed intelligence in general, is very much not a well-understood field of inquiry - in fact we don't even have a common agreement about what 'intelligence' is. We are therefore learning as we go (even after all this time!) but making rapid progress in recent years. When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change? This is arguably one of the most pathological development projects ever - what are the requirements? 'It thinks like a human'? What does that mean? And the answer is we don't know what that means, and we're working it out as we go - moving the goalposts. If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
Side note that, in case it's not obvious, none of this detracts from how impressive LLMs are. They're a marvel of the modern age, all the problems notwithstanding. However I reserve the right to stay sceptical about their capabilities.
It's in how they change, not the fact that they change. The skeptics seem to have secret definitions for intelligence, sentience, consciousness, creativity, etc. that amounts to "a thing only humans have". Often that thing is equivalent to a soul. When yesterday's challenge (LLMs don't have X because they can't do Y!) is met, Y changes but X stays the same. This is not the process by which a field matures, it is a rhetorical technique used by skeptics to avoid honestly stating or confronting their internal definitions. That can be revealed by asking the skeptic the following:
"Forget LLMs. What if we made a completely physically accurate simulation of a human being?"
Many say no, that simulated human being still couldn't have (intelligence, consciousness, sentience, creativity, ...). This reveals that there is a necessary metaphysical component to those attributes, at which point any scientific-minded person will leave the debate.
> If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
The criticisms are directed toward people who did clearly act like they knew, not the ones who were honest that they did not know.
To me, the goalposts were already defined by the person you were responding to. "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
Note that this does not depend on things like AI sentience or defining "intelligence" more rigorously, it just depends on AI impact.
The fundamental argument that I've personally made since the early days of this is that LLMs are not reasoning, in the way that word is commonly understood.
There are lots of reasons why that argument needs to evolve that could certainly appear to be "moving the goalposts", but let's take an example.
A lot of AIs were tripped up by the question "Should I walk or drive 50m to the carwash?" Several folks liked to use that as an example that illustrates that LLMs aren't reasoning, but as the models have been trained on that specific example, it's of course less useful. An AI can mostly nail it now.
So a different example is needed. A new demonstration of how these things fail at basic reasoning a child can do.
Did I move the goalposts? I don't think so. The fundamental argument stays the same. It's not hard to find lots of examples that trip up LLMs, because they are what they are: statistical inference machines. Nothing more and nothing less.
Useful, sure. But also commonly misapplied to areas for which they are inappropriate solutions.
The motte is "AI useful". The bailey is "Singularity is nigh".
(I'm personally still skeptical about this, but I'm being pulled towards accepting it).
"AI is useful" is too low of a bar, and "singularity is nigh" is too high. "AI is on its way to upending society" is about in the middle, and still vastly contentious among laypeople.
But there are people like Ed Zitron, frequently posted and cited here, who disagree even with the former.
Personally I prefer to follow explorers rather than swamp-sitters.
"it isn't clear whether generative AI actually provides much business value at all"
"cannot seem to find a product that people will pay for, in part because the results are so mediocre"
"Last week, we got our first real, definitive glimpse of what’s around that corner that future. And boy, was it underwhelming."
"OpenAI claims that o1 “performs similarly to PhD students on challenging benchmark tasks in physics, chemistry, and biology.” Just not in geography, it seems. Or basic elementary-level English language tests. Or math. Or programming. "
"Worse still, it's kind of hard to explain why anybody should give a shit about o1."
"o1 shows that OpenAI is both desperate and out of ideas."
"the software is not becoming more useful"
Honestly, every other line is quotable in this context.
But it seems we have somehow optimized away shame. It wasn’t good for profits, I guess.
Not long ago many folks were saying AI was the same as the crypto bubble. No real useful technology and only hype.
I think AI is clearly both revolutionary and useful. Revolutionary insofar as the job I do has changed almost completely in a year or so span.
We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
How does intellectual property work in an AI generated future? What about healthcare advances, who gets to own those?
What about meaning, what about purpose? How do we replace the work ethic that tells us we are our jobs and idleness is immoral? How do you replace “What do you do?” As one of the first questions you ask a new person?
That sort of thing.
If you are correct, I expect corporations to reap massive profits while most Americans try to find a way to survive in a world where they are obsolete.
And it's sad, really, because I think these two groups would make a great pairing if they could stop arguing against one another for a moment. They'll both be impacted about as much and probably have the same ultimate goals (to lead dignified lives).
But it seems these days everyone is more interested in Kayfabe and feeling like they're in the right than working together, so maybe I should just keep quiet rather than attract the ire of both groups...
I don't know if it is fair to say they're in denial. For my part, I don't expect life to get much better for regular people (especially short term), but that doesn't mean we shouldn't work to try to make it happen.
What a lot of people want to do, and I’m not saying that you’re one of them, is to assume that a positive outcome is impossible and either do nothing or loudly yell that the world is ending. Neither is particularly useful.
Or, as I said above, others just deny that there’s anything to see here and try to get people to move along.
Shane Legg (DeepMind co-founder), one of the more intelligent and thoughtful people you'll find in the industry, could only offer "it's a tough problem - we need to think about it" when recently interviewed by Hannah Fry.
On the surface the most likely outcome for AI allowed to replace jobs is extraordinarily negative, especially since it is a general capability technology, not a specific one where displaced workers can just move to another field. Once AI becomes more capable it will be able to do the vast majority of white collar jobs, including any new ones that may appear as a result of AI. As Shane Legg put it, "if your job can be done remotely, sitting in front of a computer, then it can probably be replaced by AI".
Not only does AI threaten to replace ALL the white collar jobs, but it is rapidly going after blue collar (factory jobs, driving jobs) and pink collar ones (Japanese robotics for elder-care) as well.
If a positive outcome (which doesn't include putting displaced workers on welfare - UBI) is possible, then it sure would be nice to hear it, and the silence from the AI companies, and government for that matter, is deafening.
Eventually UBI will be the norm, and if the living standards of a person on UBI is as good as yours or mine today, that will be an enormous win for everyone. It’s like pensions, once these were only for the elderly poor, now they’re a right for everyone in most developed countries.
It’s also interesting that for most of human history leisure time was the point of life, and only in recent modernity has work come to be the meaning of someone’s existence.
UBI has to be commensurate with production being automated. That’s a big logistical problem, if you think building datacenters is a challenge try bringing about radical abundance, but even so it’s not insurmountable. It just needs to be taken on as project and not seen as an impossibility.
So much of this is not about what is possible so much as what people believe is possible. We can do anything if we try.
See "Machines of Loving Grace" by Dario Amodei: https://darioamodei.com/essay/machines-of-loving-grace.
"Massive Economic Abundance: Because AI will exponentially grow the total economic pie, overall resource scarcity will diminish. The fundamental challenge shifts from producing wealth to distributing wealth."
So how do we go from everyone out of work, no income to spend on food, or the goods and services that the AI is producing, to "massive economic abundance"?!
It's like the meme:
Step 1: Create AI
Step 2: AI takes all the jobs
Step 3: ???
Step 4: Profit! (massive economic abundance)
What is step 3?
> In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
> We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
UBI in the United States is never going to happen in time. If it happens at all. We don’t even get universal healthcare. I think people who think AI will be a net positive for humanity are also in some sort of denial.
In a different US political climate I would entertain it. If these frontier labs weren’t so clearly going after the money, I would entertain it.
LLMs are clearly a step up for capitalists so I just can’t see any inclusion of LLMs move towards more progressive ideologies.
My own $0.02 on the economics piece - every country should have a sovereign wealth fund. Governments should block market access from automated[0] companies until those companies provide equity contributions to the wealth fund for that country. This aligns regulator and corporate interests. Dividends flow into the sovereign wealth funds and then can be allocated locally from there - UBI, job programs, etc. Let different jurisdictions explore different ways to structure a post-labor society.
On the broader social front - I think a lot of lack of meaning discussion boils down to the overemphasis we have on your job as your self-worth. We need to realign our societal expectations - and people need to spend more time with their families.
[0] for this to work, I think we would need well accepted metrics for 'how automated' a company is - and that probably needs a 3rd party auditing industry.
The only way to win is to wield the AI.
Humanity survives (but we reading this probably don't), the AI treats the living humans like the Emperor's favorite pets (probably a pretty good life), and then the AI does whatever else it deems important.
Bad news for you -- there's a 100% chance we all die. Sorry to be the one to tell you.
And you accuse the 'other side' of 'suicidal apathy'??
You should put down the AI and do some self-reflection on how you came to hold these views.
Does it bother you that the people who are publicly cocksure that P(doom) is moments away are the same people that have profited most handsomely from that pronouncement?
That the 'humanists' that want to do 'altruism' for 'potential future humans' and are the same people that commit fraud and theft at a civilizational scale, then sell this 'intelligence' to any child-incinerating militaries with spare cash?
It's not wrong to want to do good, but if a system that is branded 'do (the most) good' commits great evils, you are morally and intellectually obligated to step back and reconsider how you are spending your time.
Also, I asked a chicken and a feral rock dove what it's like to be not be 'apex' and they burbled at me and kept eating millet and sunflower seeds.
Would you like me to follow up with them? I'm not sure what point you expected them to make.
You hallucinated the "preemptive nuclear war". He didn't say anything about nukes. That's your own invention.
> Does it bother you that the people who are publicly cocksure that P(doom) is moments away
20% is not "cocksure". The "moments" is again an exaggeration.
Every problem is a search problem. Nuke the data centers.
P(doom) = 98.9% (Aug-2026) P(doom) = 98.2% (July-2026) P(doom) = 98.2% (Jun-2026) P(doom) = 98.5% (May-2026) P(doom) = 98.8% (mid-April-2026) P(doom) = 98.7% (April-2026) P(doom) = 98.7% (March-2026) P(doom) = 98.5% (mid-Feb-2026) P(doom) = 97% (Feb-2026) P(doom) = 94% (Jan-2026) P(doom) = 93% (Dec-2025) P(doom) = 95% (July-2025)
Powerful word, `if`. "You're not only wrong you're a fulminating psychopath" is a perfectly valid response to getting it wrong like a fulminating psychopath.
What can be asserted without evidence can also be dismissed without evidence.
Now you are again postulating that there would be no more extreme progress in the near future. That's actually more "insane".
not sure how many will get this reference but "AI" for science and math is like super-shoes for runners
at first we are blown away by the impossible improvements including sub-2-hour realworld marathon and every other PR/CR/WR is dialed down
but then the improvements slow and reach a stall point because of the limit of technology and the source of the achievement
ie. sub-2-hour marathon yes, sub-1-hour never happening (rollerblade inline-skate record is 1-hour marathon)
The fact we see a lift is not the same as evidence that the lift is unbounded.
The lift being finite is supported by the fact improvements have come at the edges: improvements from human feedback, improvements in harnesses, improvements on model compatibility with harnesses, improvements in inference efficiency with new architectures, etc. If we were just training better models from scratch that would be one thing, but we are just making better use of a tool we've developed.
As a programmer, I am mostly interested in whether my role is sustainable long-term and whether the models will get better. I don't feel in jeopardy yet, but two more years like this and the calculus of hiring software engineers could shift even further. QAs are already overwhelmed with work
I’m not a mathematician so I have zero clue what “ New upper bounds on sphere-packing density down to the Cohn–Elkies thresholds” means.
https://garymarcus.substack.com/p/two-critical-updates-re-as...
As always, PR hype. Goalposts have not moved.
Guys, please use critical thinking. The haters don't hate by default, we hate because we're gaslit about this stuff every day and it's annoying. Extraordinary claims require proof, and they're not giving us information that would be essential to knowing if this is actually significant or not.
Remember October 2024 Pelicans [1] ? It's been only less than 2 years.
We don't know what will come in the next 2 years. But the progress doesn't seem to stop for now.
[1] https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
People are skeptical of the announcement because the room include several PHDs in math and physics. The prompts are not published so we can see how generic the starting prompt is.
He literally says it's an impressive feat in the second article.
The only way that is PR hype is if you're invoking the insane conspiracy that frontier AI labs are just buying off results that would otherwise be career defining for a mathematician, just for marketing.
The posts you linked are urging caution regarding the exaggerated e/acc-esque lies peddled by people like Musk, not that the models haven't proven themselves as having genuine ability to contribute to research in some areas.
They still do things that I find incredibly annoying and “dumb”. And I still have to clean up messes they make quite often.
But on the whole they are clearly smarter than before. No extraordinary claims needed. I just try to learn how the tool works and how to use it effectively.
I remember the time when he insisted that diffusion-based image generators trained on Internet scale data will never be able to make an image of a horse riding an astronaut. Today you can generate 4K video of that.
I heard that Gary Kasparov was impacted by AI chess, but at least he still seems to have a job, so don't give up.
It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem?
Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.
But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
For example, despite our best efforts, the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n). In contrast, our best algorithms for the task run in time roughly O(2^n). That’s an exponential gap. This is despite decades of trying to find lower bounds.
The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more.
The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.
a colleague was telling me that the base idea for proving that something is not sofic already appeared in the literature around 2019 or so (this is the "expanders graphs" that are mentioned in OpenAI s paper. no one had managed to find a concrete example though. this doesn't make the result less impressive in any case.
The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.
I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
Mundane incremental research is cobbled from existing citations that already appear nearby in the record.
Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record.
So we can know pretty conclusively.
Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on)
It also links to a paper written by an LLM where the model "reconstructs how the proof came together" based on the unpublished reasoning traces: https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf
I wish they'd publish the prompts though!
As long as you are not missing important information, how you word the prompt does not have any effect.
but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance. in some sense this fits our intuitions. when top tech companies use math PhD type employees, they have them stop doing pure math research and instead focus on software engineering. these people are often very good at software engineering but not due to recent discoveries in academic mathematics, it's due to their general intelligence. to me, this is evidence that the models are getting better but does not make me think we are on the cusp of a foom style fast takeoff enabled by revolutions in frontier math (i also posted this on twitter @mlipman13)
Like, these would be best-paper awards at many top CS conferences.
Incredible?
> open ai announced like 15% improvement by fixing gpu kernel issue
That is... ordinary software optimization.
Edit: also here’s a opencl 30% compute perf increase documented here : https://m.hexus.net/tech/news/graphics/74425-haswell-systems... that i just googled for
> we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B)
i think you have misunderstanding of what mathematicians do
They get to make cool 3D plot visualizations of functions so obscure to me that they’re named after someone who is still alive - and/or get to work on cryptography for the NSA - I think?
> You are an expert in the field of mathematics, with decades of experience. You are a reviewer of proofs, etc etc.etc.
It is indeed true that all models are, at their core, predictors of what occurs next in a sequence. But I think it's worth exploring the implication of what that means. Because when fed tiny pieces of information for a few tasks at a small scale, this results in something that sorta, kinda works. Or, works surprisingly well.
But when scaled... When the amount of information starts approaching the sum of all human knowledge, the tasks start approaching all useful applications of that human knowledge, and the fidelity of the predictor approaches incomprehensible sizes, the starts encodes / becomes (I'd argue it becomes) something that can model all human knowledge.
It feels wrong to say that, but let me explain, what is the best way to predict the behavior of a ball constrained in two directions that bounces with initial vertical velocity v(y) (y is up / down axis) and horizontal velocity v(x) (x is side by side in 1d) ?
If we purely look at it via a graph, it's by modelling the function of acceleration under earth's gravity.
If only a few points are given to you for this and you can't make something really sophisticated, then you'll make something that's rough that kinda sorta works and then call it a day.
But... if the number of points keeps increasing in number, precision and accuracy as well as the number of examples (assumed that data about air pressure, velocity and all other factors is included alongside these points), the fidelity with which you can replay / tweak the function keeps improving, and the number of times you can iterate keeps increasing, you'll eventually create a function that models that process so well that it intrinsically contains a good enough model of the deformation of the ball (provided the dataset contains information about elasticity of the ball's material, its dimensions and mass etc..), the nearly negligible (under normal conditions) effects of the ambient environment (provided there's diversity in the number of environments supplied), the oblateness of the Earth and minute changes in the gravitational field (the length of a seconds pendulum varies depending on where the experiment happens. It's presumed that all of the prior set of experiments were repeated across the Earth and the subtle, but real deviations were faithfully recorded)... and so much more.
A machine trained on the above with a large number of parameters, measures to prevent "laziness" and enough reps for high fidelity across a large enough dataset would start to approach a simulation of the ball falling. Because to predict what happens next in the sequence, you must model what's occurring in the sequence.
Now imagine doing that for other tangible and intangible things in this world. For all of human knowledge across all fields of endeavor. All experiences. No matter how noble, ignoble, notable or ignorable. But putting all of it into the soup that's this machine. Then at larger and larger scales, you eventually start encountering "good enough" models (in modelling the falling ball sense) for even the most hard to quantify / qualify things like grief and joy. At some point, by simply trying to predict what it has been taught ought to be the next part of the sequence in say... human interaction, it starts to make a model of something that hews ever closer to a full fidelity theory of mind.
Is there evidence for this? Kind of, yes. There are early indications that as machines are trained for an ever larger number of tasks at larger and larger scales, their internal representations converge. It's called the Platonic Representation Hypothesis. Overview and paper here, https://phillipi.github.io/prh/
It is my opinion that these machines are displaying a new form of intelligence that human beings haven't quite encountered before. They are the sum of all human knowledge made manifest and given voice by processes that nudge (bit-by-bit) what kind of step it ought to predict for the next part of whatever sequence it displays.
In my mind this means that, of course, these models can create new knowledge. This strains the analogy, but with the sum of all human mathematics within them, they can "reason" via the act of predicting what ought to come next.
Of course, these machines are "surprisingly" good at a lot of things the larger they get, because what the labs have created here is a rough version of humanity's collective knowledge given form and the ability to say hello.
I suspect that the current generation isn't close to the "true frontier" of what these machines could be. They are nowhere close to the sum of all human knowledge and endeavor. They are quite a way there, but they haven't yet achieved true completeness for domains where the data isn't so public.
I think it's the most exciting scientific and technological breakthrough of my lifetime. And I can't wait for us to get close to the true frontier of all domains.
