And what software developers should do?

Basically people are vibe coding their personal apps and anything that's expensive is being vibe coded open in the public. I don't see many software companies staying profitable for long.

DHH is the biggest proponent of AI and let me know which of the 37 signals products can't be vibe coded in a month at a $200 plan that are suitable for that organization alone that just has to be accessible internally only? Hence scale and security aren't such an issue.

In that climate - for how long software companies would stay profitable and when not, who'll be employing developers?

PS: Don't underestimate vibe coded apps. Take a look at PDFCraft, VectorCraft, WordCraft. And imagine the feature parity in a year.

Tao is continuously updating the cancer slide with the help of Claude in the past few hours.

https://github.com/teorth/tao-web/commits/main/

It reduces the badness somewhat.

"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial.

Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""

Yes, absolutely an AI solution could exploit a weakness in the trial process or its math models. Would it remain uncaught? Unclear.

Folks really want to decide if this is pro or anti and respond to that. It's neither.

In a world with only LLMs doing the day-to-day of proving, you still want more than just proofs dumped out as soon as they pass adversarial review or formalization. The first proof code that passes checks may be a mess that's hard to build on; the path to a result, including failures, is full of interesting stuff; naming, generalizing, linking, and organizing things are tasks that move us from today's raw material to tomorrow's future directions.

Folks get this about code: "it works" isn't the same as "it's good." We knew it before LLMs, out of practical necessity as projects became hard to build on.

An odd thing to me is it would not be that big a concession for the labs to openly talk about the difference between the math artifacts they're releasing and what mathematicians do. It's analogous to the Claude's C Compiler post noting it wouldn't replace a prod C compiler. They could talk about paths from a rough initial product to 'production' math, say they hope future models will continue to improve as math collaborators and expositors, and so on.

For the compiler, the gaps would be trivially obvious to their customers, so it was an obvious call to acknowledg them. With their math output, the difference between a proof dump and the kind of well-written, contextualized mathematical program the community pursues is apparently subtle and deep enough the companies can't get themselves to acknowledge there is any difference. So the companies self congratulate and then are yelled at on the Internet instead.

(Ant seems to have wised up a tiny bit about this compared to OAI, and e.g. Ant handed a recent result to some experts in complexity theory to understand and refine and publish rather than just dumping a Claude-written PDF.)

And yes the drug discovery piece was weird. There are good relevant points: it'd be good to understand pharmaceuticals better (helps us find others, understand side effects, etc.); when we get a mystery result, it's worth effort to try to work our way to proper understanding; anything that could help us understand (reasoning etc.) shouldn't be hidden. But as presented it's both not a lot like math and accidentally a lot like those "would you choose..." questions that always get so much engagement on social media.

One thing that AI can give me: time. Would I like to see all the rest of the Millennium Problems solved during my lifetime? Absolutely.

Also how many math and scientific discoveries do I want to see during my lifetime? "Just a little bit more."

It’s funny to see people in the STEM field focus more on the human aspect of creation. It used to be that the result mattered more than your feelings. Now we are moving the goal posts about how things should be done.

I wonder if we will begin to actual value human creation more at the end of all of this

It's quite noticeable how when Terence tao was sounding pro-AI the sentiment was much more positive and he was being held up as an authority to listen to. Then he puts out some limitations of AI in a presentation about how to work with it as an expert and suddenly on HN he is just some out of touch killjoy trying to hold back progress etc.

I think people are missing the point that terrence is trying to make, especially on the cancer drug.

For nuance lovers - here are some basics for how you get your drugs: there is an established chain of trust from the first basic science paper to the phase 3 trial and the subsequent availability of the drug to general public

- someone publishes the first paper (basic science) explaining some biological phenomenon, which leads to 10s or 100s of other papers with some tweaks in conditions,

- after the above papers the pathway of the phenomenon is understood by researchers, they try therapies at cell level to see if they can control some behavior, 10s or more papers get published,

- then someone tries this in mice and other models, 10s and more papers get published.

- then researchers at pharma companies + hospitals create this therapy for human trials - phase 1, 2, 3 etc - data collections, then FDA - then approval.

Now, the people who worked on the phase 3 trial might not know the people who wrote the first seminal paper and they often don't exist in the same decade - but it absolutely does not mean that we (humans) don't know how these drugs work - if you take 1-2 researchers from each phase and put them in a room and ask them how that particular drug works - they will quickly be able to build a consensus. that is what the chain of trust means here. now of course there can be fraud in scientific research, but that happens in every human endeavor and is a separate topic.

back to terrence - he is saying that if there is suddenly a drug that nobody knows the origin of; passed phase 3 but it's unclear who conducted the phase 3 or if the phase 3 even happened or if it's fabricated - you would not want to take the drug. usually when doctors recommend these kinds of drugs - there is already a lot of information available about where the drug came from, if there are any case studies, which doctor tried it first, which country- they often even call those other doctors and find out who was behind the first trials going back as far as the university professors.

Your MD doctor might not know the chemistry and physics behind the drug you are taking but there is deifnitly a group of people, when put together, can tell how that drug is working. My wife is a fundamental researcher - understanding physics at DNA level and my brother is a MD doctor; our conversations are super fun.

Side effects are a completely different thing - they involve the above cycle on repeat.

It is nice to see some sanity back in the conversation!

I'm still not sure about math-2.0 (humans+AI will make fundamentally more progress):

- AI and computer usage take a mental toll on humans and humans will overlook radical improvements.

- AI may be good at finding useless things like "P==NP, but the complexity is O(n**4242424242424242)". In other words, useless.

- Humans become formalists and lose traditional sources of inspiration. Maybe interacting with Lean should be left to specialists, but not to creative blackboard mathematicians.

- AI exposure will further intellectual conformity, more than the Internet did.

As to the last point, a lot of progress (real, not measured in publications) seems to have been made when communication was slower and there were several different schools and approaches.

The workflow/understanding graphs really helped me understand his viewpoint on the field.

Can someone explain why there wouldn't be arrows from all three types of solutions back to human understanding?

> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands– even partially the mechanism behind this cure

I mean... yes, most people will find it reassuring, but history has proven that's not necessary. People have been using medicines of which the mechanism wasn't understood for a very long time and greatly enjoyed their benefits. Even widely used one (e.g. Paracetamol).