Top
Best
New

Posted by johnbender 10 hours ago

LLMs are making formal software verification feasible at a much larger scale(www.anthropic.com)
335 points | 189 commentspage 3
mjorgers 9 hours ago|
It might be that I’m unfamiliar with the matter, but as I understand it, in order to prove a statement, the statement must correctly model the reality of the code. In that case, how can you be truly sure that, when formally verifying a program, you have correctly modelled the software you’re trying to verify?

Couldn’t it very well be that you proved something that wasn't actually your program?

gizmo686 8 hours ago||
If you have a model of the programming language, then you can make proofs about the source code itself. At this point, your assumption is that you are accuratly modelling the compiler.

You can go a step farther and model the instruction set and architecture, and which point you can make proofs about the final binary directly (thereby sidestepping the problem of compiler bugs). At this point, your assumption is just that your physical hardware actually matches your model.

geraneum 8 hours ago|||
> Couldn’t it very well be that you proved something that wasn't actually your program?

Correct. It can happen, where the actual code, could diverge from a lean representation of it, in a glaring or subtle way.

anon291 1 hour ago|||
I mean ... The thing about formal verification is that you can write the extremely slow but correct version of your code and then have an optimizer write a much more efficient version with a proof of correctness.

For things like programming languages we often have tedious, but slow, semantics that can be directly implemented in a suitable language. Then really all that a compiler or interpreter is is a faster version of those semantics.

If you are able to show correspondence between the fast and the the correct version, then you can toss the correct and slow version without even ever running it and just defer to the provably correct fast version.

Most software is ultimately like this

david-gpu 8 hours ago||
> how can you be truly sure that, when formally verifying a program, you have correctly modelled the software you’re trying to verify?

You can do it backwards. You declare the properties you want the resulting program/function to have, using a proof assistant like Lean, then prove that a program exists that meets those requirements, and the proof assistant spits two things: a confirmation that the proof is valid, and a piece of code that implements the proof, in the form of computer code.

At least that's how the proof assistant Coq would often be used. You still have the challenge of ensuring that you have defined the properties you really want in the first place, but at least it solves the part where you want the code to meet the specification.

cadamsdotcom 7 hours ago||
One wonders how deep is Anthropic's queue of marketorials.
adyavanapalli 7 hours ago||
> Claude can write a proof showing that a compiled program’s behavior satisfies the requirements of a formal spec (which is verified by an external proof checker). However, a proof checker can’t tell you whether your spec matches reality.

Imagine having a formal spec for reality!

thinkindie 1 hour ago||
is it just me, or is Anthropic posting more articles like this (and similar other topics) in the past few days than before? And more of them are ending up in HN front page?

If I recall correctly the only news they used to post were new models updates, while now they have expanded their topics, or so I feel.

braiamp 8 hours ago||
TLA mentioned, previous related discussions about the topic:

"What TLA+ can and can't check" https://news.ycombinator.com/item?id=49909056

"The internet discovers TLA+. Now what?" https://news.ycombinator.com/item?id=49863600

gnabgib 8 hours ago|
What a tiny subset of all TLA+ articles (a very popular topic on HN)...

2017: Leslie Lamport: Video course on TLA+ (328 points, 74 comments) https://news.ycombinator.com/item?id=13918648 (219 points, 2018, 17 comments) https://news.ycombinator.com/item?id=16956778

2017: Writing correct lock-free and distributed stateful systems in Rust, with TLA+ (374 points, 94 comments) https://news.ycombinator.com/item?id=14373359

2019: TLA+ model checking made symbolic (197 points, 51 comments) https://news.ycombinator.com/item?id=21662484

2019: Introduction to TLA+ Model Checking in the Command Line (176 points, 44 comments) https://news.ycombinator.com/item?id=18937214

2019: Learn TLA+ (387 points, 92 comments) https://news.ycombinator.com/item?id=19661329 (310 points, 2020, 58 comments) https://news.ycombinator.com/item?id=22393653 (254 points, 2022, 64 comments) https://news.ycombinator.com/item?id=33010645 (350 points, 2022, 75 comments) https://news.ycombinator.com/item?id=31952643

2020: Using TLA+ in the Real World to Understand a Glibc Bug (166 points, 76 comments) https://news.ycombinator.com/item?id=24958504

2026: Hunting a 16-year-old SQLite WAL bug with TLA+ (239 points, 29 comments) https://news.ycombinator.com/item?id=48730953

...

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

braiamp 7 hours ago||
Yeah... usages of TLA+ within the realm of LLM is kinda newish, wouldn't you agree?
noiv 2 hours ago||
> Specifically, Claude proved that these programs will always exit normally without triggering memory-safety errors or failing assertions, for every input less than 64 MB and ...

Did they solve Halting Problem?

bmacho 2 hours ago||
I don't get what you mean? Granted, I didn't even try reading your message, since the language that you're using, English, is not sound, complete, decidable (or even defined).
vanschelven 2 hours ago||
_these_ programs not _all_ programs
noiv 58 minutes ago||
Right, 123 libs have been found to behave...
j16sdiz 8 hours ago||
Did they remember what OpenAI did with Lean and Math Proofs?

OpenAI wrote some lean that pass the validate, write a math paper that feel plausible, but the paper don't match what's in lean.

Anthropic is now translating programs into Rocq proof. The same can happen.

Don't get me wrong -- this process can help catch some bug. but this is far from a proof

cmrx64 2 hours ago||
About time, too. I studied formal software and hardware verification “before AI” and automated reasoning and cognitive architectures were just in need of scale. People are struggling with the inherent meaningmaking of taking formalism seriously in the world. It’s easy to roll your eyes (and I do a lot of that myself) but I have a lot of sympathy. It took many years of dedicated focus for me to convince myself the validity of automated reasoning / externalized cognition.
ChicagoDave 5 hours ago||
I designed Sharpee and generated the code with Claude Code. About six months in I had Claude generate a manual for the language. Then I had this crazy idea of testing the manual with a completely dummy instance of Claude in a VM.

Nothing solidified Sharpee more than “compiling” this manual chapter by chapter.

But even that left gaps. I then designed a true DSL called Chord that compiles to Sharpee and this really ramped up the solidity of Sharpee.

dominotw 5 hours ago|
what is sharpee. chord ?

no idea what you are even talking about

ChicagoDave 2 hours ago||
You could infer I created two programming languages or you could infer I tested AI generated constructs in a way only AI could.
PandaRider 8 hours ago|
I'm curious if increased adoption of formal verifications would increase migration of legacy COBOL to modern systems.
y1n0 5 hours ago||
It seems like it’s a good fit. Proving equivalence is much easier than writing a spec without a golden reference.
js8 3 hours ago||
Except COBOL systems are often easier to formally verify than modern systems.

They're often simpler, and whole classes of security vulnerabilities are mitigated by design. My favorite example is SQL injection mitigated by having proper access controls in the DB. Why prove about every SQL statement that it cannot be nefarious, when we can just prove it about the connecting user? The user access carries the intent of what the application program wants to do with the database, in a separate channel, independently of the complexity of the application.

More comments...