Top
Best
New

Posted by bearsyankees 12/3/2025

Reverse engineering a $1B Legal AI tool exposed 100k+ confidential files(alexschapiro.com)
821 points | 288 commentspage 5
fallinditch 12/3/2025|
> ... after looking through minified code, which SUCKS to do ...

AI tends to be good at un-minifying code.

a_victorp 12/3/2025||
Legit question: when working on finding security issues, are there any guidelines on what you can send to LLMs/AI?
CER10TY 12/3/2025|||
Personally, I'd just use common sense and good judgment. At the end of the day, would you want someone to hand your address, and other private data to OpenAI just like that? Probably not. So don't paste customer data into it if you can avoid it.

On the other hand, minified code is literally published by the company. Everyone can see it and do with it as they please. So handing that over to an AI to un-minify is not really your problem, since you're not the developer working on the tool internally.

fallinditch 12/3/2025|||
I got downvoted, so maybe that means someone thinks un-minifying code is not advised for dealing with security issues? But on reflection surely you can just use the 'format code' command in the ide? I am no expert but surely it's ok to use AI to help track down and identify security issues with the usual caveats of 'don't believe it blindly, do your double checking and risk assessing.'
nodesocket 12/4/2025||
Doesn't Chrome Developer tools automatically un-minify?
2ndatblackrock 12/3/2025||
now that's just great hacking
quapster 12/3/2025||
[flagged]
j45 12/3/2025|
It's a little hilarious.

First, as an organization, do all this cybersecurity theatre, and then create an MCP/LLM wormhole that bypasses it all.

All because non-technical folks wave their hands about AI and not understanding the most fundamental reality about LLM software being fundamentally so different than all the software before it that it becomes an unavoidable black hole.

I'm also a little pleased I used two space analogies, something I can't expect LLMs to do because they have to go large with their language or go home.

jimbokun 12/3/2025|||
My first reaction to the announcement of MCP was that I must be missing something. Surely giving an LLM unlimited access to protected data is going to introduce security holes?
j45 12/4/2025||
Agree.

It’s assuming and estimating it will behave like other software before it when it’s nothing like the software that came before it.

LLMs today won’t behave like the software we’re used to where 1+1 will equal 2 every time.

jimbokun 12/4/2025||
It’s like handing access to your sensitive data to a complete stranger and hoping for the best.
dogman144 12/3/2025||||
Assuming a 101 security program past the quality bar, there are a number of reason why this can still happen at companies.

Summarized as - security is about risk acceptance, not removal. There’s massive business pressure to risk accept AI. Risk acceptance usually means some sort of supplemental control that’s not the ideal but manages. There are very little of these with AI tools however - small vendors, they’re not really service accounts but IMO best way to monitor them probably is that, integrations are easy, eng companies hate devs losing admin of some kind but if you have that random AI on endpoints becomes very likely.

I’m ignoring a lot of nuance but solid sec program blown open by LLM vendors is going to be common, let alone bad sec programs. Many sec teams I think are just waiting for the other shoe to drop for some evidentiary support while managing heavy pressure to go full bore AI integration until then.

j45 12/4/2025||
You missed risk creation vs reward creation.

And then folks can gasp and faint like goats and pretend they didn’t know.

It reminds me of the time I met an IT manager who dint have an IT background. Outsourced hilarity ensued through sales people who were also non-technical.

dogman144 12/4/2025||
What am I missing? Risk acceptance is what you’re referring to - risk creation and reward creation.

Sec lead might have a pretty darn clear idea of an out of whack creation of risk v reward. CEO disagrees. Risk accept and move on.

When you’re technical and eventually realize there’s a business to survive behind the tech skills, this is the stuff you learn how to do.

People “will know” as you say because it’s all documented and professionally escalated.

Aurornis 12/6/2025||||
> I'm also a little pleased I used two space analogies, something I can't expect LLMs to do because they have to go large with their language or go home.

Speaking of LLMs, did you notice the comment you were responding to was written by an account posting repetitive LLM-generated comments? :)

stronglikedan 12/3/2025||||
Nitpick, but wormholes and black holes aren't limited to space! (unless you go with the Rick & Morty definition where "there's literally everything in space")
j45 12/3/2025||
Not a nit pick at all friend, it is even more rabbit holes to explore.
RansomStark 12/3/2025|||
Maybe this is the key takeaway of GenAI: that some access to data, even partially hallucinated data, is better than the hoops that the security theatre puts in place that prevents average Joe doing their job.

This might just be a golden age for getting access to the data you need for getting the job done.

Next security will catch up and there'll be a good balance between access and control.

Then, as always security goes to far and nobody can get anything done.

It's a tale as old as computer security.

j45 12/5/2025||
This is not at all what I am saying.

"GenAI" is nothing new. "AI" is just software. It's not intelligent, or alive, or sentient, or aware. People can scifi sentimentalize it if they want.

It might simulate parts of things, hopefully more reliably.

It's however a different category of software which requires management that doesn't exist yet how it should.

Cybersecurity security theatre for me is using a web browser to secure and administer what was previously already done and creating new security holes from a web interface.

Then, bypassing it to allow unmanaged MCP access to internal data moats creating it's own universe of security vulnerabilities, full stop. In a secured and contained environment, using an MCP to access data to unlock insight is one thing.

It doesn't mean dont' use MCPs. It means the AI won't figure out what the user doesn't know about security around securing MCPs which is a far more massive vulnerability because users of AI have delegated their thinking to a statistics formula ("GenAI"), because it is so impressive on the surface, but no one is checking the work to make sure it stays that way. Managing quality however, is improving.

My comment is calling out effectively letting external paths have unadulterated access to your private and corporate data.

Data is the new moat. Not UI/UX/Software.

A wormhole that exposes your data makes it available for someone to put it into their data moat far too commonly, and also for it to be mis-interpretted.

larrysanchez77 12/11/2025||
[dead]
kitschman 12/4/2025||
[dead]
electric_muse 12/3/2025||
[flagged]
tomhow 12/4/2025||
Please don't do this here. If a comment seems unfit for HN, please flag it and email us at hn@ycombinator.com so we can have a look.

We detached this subthread from https://news.ycombinator.com/item?id=46137863 and marked it off topic.

simonw 12/3/2025|||
That comment didn't read like AI generated content to me. It made useful points and explained them well. I would not expect even the best of the current batch of LLMs to produce an argument that coherent.

This sentence in particular seems outside of what an LLM that was fed the linked article might produce:

> What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals.

Aurornis 12/3/2025|||
The users' comment history does read like generic LLM output. Look at the first lines of different comments:

> Interesting point about Cranelift! I've been following its development for a while, and it seems like there's always something new popping up.

> Interesting point about the color analysis! It kinda reminds me of how album art used to be such a significant part of music culture.

> Interesting point about the ESP32 and music playback! I've been tinkering with similar projects, and it’s wild how much potential these little devices have.

> We used to own tools that made us productive. Now we rent tools that make someone else profitable. Subscriptions are not about recurring value but recurring billing

> Meshtastic is interesting because it's basically "LoRa-first networking" instead of "internet with some radios attached." Most consumer radios are still stuck in the mental model of walkie-talkies, while Meshtastic treats RF as an IP-like transport layer you can script, automate, and extend. That flips the stack:

> This is the collision between two cultures that were never meant to share the same data: "move fast and duct-tape APIs together" startup engineering, and "if this leaks we ruin people's lives" legal/medical confidentiality.

The repeated prefixes (Interesting point about!) and the classic it's-this-not-that LLM pattern are definitely triggering my LLM suspicions.

I suspect most of these cases aren't bots, they're users who put their thoughts, possibly in another language, into an LLM and ask it to form the comment for them. They like the text they see so they copy and paste it into HN.

balamatom 12/3/2025|||
Or maybe these are people who learned from a LLM that English is supposed to sound like this if you want to be permitted to communicate a.k.a. "to be taken into consideration"! Which is wrong and also kinda sucks, but also it sucks and is wrong for a kinda non-obvious reason.

Or, bear with me there, maybe things aren't so far downhill yet, these users just learned how English is supposed to sound, from the same place where the LLMs learned how English is supposed to sound! Which is just the Internet.

AI hype is already ridiculous; the whole "are you using an AI to write your posts for you" paranoia is even more absurd. So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere. Just like most non-AI-generated thoughts, except perhaps the one which leads to the fridge.

Aurornis 12/3/2025||
Or maybe the 2 month old account posting repetitive comments and using the exact patterns common to AI generated comment is, actually, posting LLM generated content.

> So what if they are? Then they'd just be stupid, futile thoughts leading exactly nowhere.

FYI, spammers love LLM generated posting because it allows them to "season" accounts on sites like Hacker News and Reddit without much effort. Post enough plausible-sounding comments without getting caught and you have another account to use for your upvote army, which is a service you can now sell to desperate marketing people who promised their boss they'd get on the front page of HN. This was already a problem with manual accounts but it took a lot of work to generate the comments and content.

That's the "so what"

balamatom 12/4/2025||
Wasn't there some sort of escape hatch for situations like that - for when it becomes impossible to trust the agora?

It would be massively funny if that escape hatch just sort of disappeared while we were looking at something else.

Your point stands, though.

>exact patterns common to AI generated comment

How can there be exact patterns to it?

LoganDark 12/3/2025|||
> I suspect most of these cases aren't bots, they're users who put their thoughts, possibly in another language, into an LLM and ask it to form the comment for them. They like the text they see so they copy and paste it into HN.

Yes, if this is LLM then it definitely wouldn't be zero-shot. I'm still on the fence myself as I've seen similar writing patterns with Asperger's (specifically what used to be called Asperger's; not general autism spectrum) but those comments don't appear to show any of the other tells to me, so I'm not particularly confident one way or the other.

balamatom 12/3/2025||
That's ye olde memetic "immune system" of the "onlygroup" (encapsulated ingroup kept unaware it's just an ingroup). "It don't sound like how we're taught, so we have no idea what it mean or why it there! Go back to Uncanny Valley!"

It's always enlightening to remember where Hans Asperger worked, and under what sociocultural circumstances that absolutely proverbial syndrome was first conceived.

GP evidently has some very subtle sort of expectations as to what authentic human expression must look like, which however seem to extend only as far as things like word choice and word order. (If that's all you ever notice about words, congrats, you're either a replicant or have a bad case of "learned literacy in USA" syndrome.)

This makes me want to point out that neither the means nor the purpose of the kind of communication which GP seems to implicitly expect (from random strangers) are even considered to be a real thing in many places and by many people.

I do happen to find that sort of thing way more coughinterestingcough than the whole "howdy stranger, are you AI or just a pseud" routine that HN posters seem to get such a huge kick out of.

Sure looks like one of the most basic moves of ideological manipulation: how about we solved the Turing Test "the wrong way around" by reducing the tester's ability to tell apart human from machine output, instead of building a more convincing language machine? Yay, expectations subverted! (While, in reality, both happen simultaneously.)

Disclaimer: this post was written by a certified paperclip optimizer.

samdoesnothing 12/3/2025|||
It's probably a list of bullet points or disjointed sentences fed to the LLM to clean up. Might be a non-English speaker using it to become fluent. I won't criticize it, but it's clearly LLM generated content.
snapdeficit 12/3/2025|||
“This comment is AI” is the new “First Post” from /. days. Please stop unless you have evidence or a good explanation.
rootusrootus 12/3/2025||
That was literally the same thought that crossed my mind. I agree wholeheartedly, accusing everything and everyone of being AI is getting old fast. Part of me is happy that the skepticism takes hold quickly, but I don't think it's necessary for everyone to demonstrate that they are a good skeptic.

(and I suspect that plenty of people will remain credulous anyway, AI slop is going to be rough to deal with for the foreseeable future).

lordnacho 12/3/2025||
Also, an AI comment might have a worthwhile point to be addressed. Pointing out something was written in a new way is not addressing the point.
Aurornis 12/3/2025||
Spammers use AI comments to build reputation on a fleet of accounts for upvoting purposes.

That may or may not be what's happening with this account, but it's worth flagging accounts that generate a lot of questionable comments. If you look at that account's post history there's a lot of familiar LLM patterns and repeated post fragments.

Conasg 12/3/2025|||
Yeah, you have a point... the comment - and their other comments, on average - seem to fit quite a specific pattern. It's hard to really draw a line between policing style and actually recognising AI-written content, though.
snapcaster 12/3/2025|||
What makes you think that? it would need some prompt engineering if so since ChatGPT won't write like that (bad capitalization, lazy quoting) unless you ask it to
lazide 12/3/2025||
“Chat, write me a blog article that seems like a lazy human who failed English wrote it”?
legostormtroopr 12/3/2025|||
What’s worse being accused of an AI post or being defended because your post is so bad that AI wouldn’t have written it?
FrustratedMonky 12/3/2025|||
Well then that's everything.
samdoesnothing 12/3/2025|||
Ya ur right, it's either LLM generated, LLM enhanced, or the author has been reading so much LLM output that its writing style has rubbed off.
syndacks 12/3/2025|||
or, they wrote it and asked an LLM to improve the flow
koumou92 12/3/2025|||
You are right, it's 100% AI written
vkou 12/3/2025|||
What? It doesn't read that way to me. It reads like any other comment from the past ~15 years.

The point you raised is both a distraction... And does not engage with the ones it did.

jfindper 12/3/2025||
We finally have a blog that no one (yet) has accused of being ai generated, so obviously we just have to start accusing comments of being ai. Can't read for more than 2 seconds on this site without someone yelling "ai!".

For what it's worth, even if the parent comment was directly submitted by chatgpt themselves, your comment brought significantly less value to the conversation.

probably_wrong 12/3/2025||
It's the natural response. AI fans are routinely injecting themselves into every conversation here to somehow talk about AI ("I bet an AI tool would have found the issue faster") and AI is forcing itself onto every product. Comments dissing anything that sounds even remotely like AI is the logical response of someone who is fed up.
jfindper 12/3/2025||
Every other headline and conversation having ai is super annoying.

But also, its super annoying to sift through people saying "the word critical was used, this is obviously ai!". not to mention it really fucking sucks when you're the person who wrote something and people start chanting "ai slop! ai slop!". like, how am i going to prove is not AI?

I can't wait until ai gets good enough that no one can tell the difference (or ai completely busts and disappears, although that's unlikely), and we can go back to just commenting about whether something was interesting or educational or whatever instead of analyzing how many em-dashes someone used pre-2020 and extrapolating whether their latest post has 1 more em-dashes then their average post so that we can get our pitchforks out and chase them away.

anonymous908213 12/3/2025|||
LLMs will never get good enough that no one can tell the difference, because the technology is fundamentally incapable of it, nor will it ever completely disappear, because the technology has real use cases that can be run at a massive profit.

Since LLMs are here to stay, what we actually need is for humans to get better at recognising LLM slop, and stop allowing our communication spaces to be rotted by slop articles and slop comments. It's weird that people find this concept objectional. It was historically a given that if a spambot posted a copy-pasted message, the comment would be flagged and removed. Now the spambot comments are randomly generated, and we're okay with it because it appears vaguely-but-not-actually-human-like. That conversations are devolving into this is actually the failure of HN moderation for allowing spambots to proliferate unscathed, rather than the users calling out the most blatantly obvious cases.

jfindper 12/3/2025||
Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot?

The only spam I see in this chain is the flagged post by electric_muse.

It's actually kind of ironic you bring up copy-paste spam bots. Because people fucking love to copy-paste "ai slop" on every comment and article that uses any punctuation rarer than a period.

anonymous908213 12/3/2025|||
> Do you think the original comment posted by quapster was "slop" equivalent to a copy-paste spam bot?

Yes: the original comment is unequivocally slop that genuinely gives me a headache to read.

It's not just "using any punctuation rarer than a period": it's the overuse and misuse of punctuation that serves as a tell.

Humans don't needlessly use a colon in every single sentence they write: abusing punctuation like this is actually really fucking irritating.

Of course, it goes beyond the punctuation: there is zero substance to the actual output, either.

> What's wild is that nothing here is exotic: subdomain enumeration, unauthenticated API, over-privileged token, minified JS leaking internals.

> Least privilege, token scoping, and proper isolation are friction in the sales process, so they get bolted on later, if at all.

This stupid pattern of LLMs listing off jargon like they're buzzwords does not add to the conversation. Perhaps the usage of jargon lulls people into a false sense of believing that what is being said is deeply meaningful and intelligent. It is not. It is rot for your brain.

jfindper 12/3/2025||
"it's not just x, it's y" is an ai pattern and you just said:

>"It's not just "using any punctuation rarer than a period": it's the overuse and misuse of punctuation that serves as a tell."

So, I'm actually pretty sure you're just copy-pasting my comments into chatgpt to generate troll-slop replies, and I'd rather not converse with obvious ai slop.

anonymous908213 12/3/2025||
Congratulations, you successfully picked up on a pattern when I was intentionally mimicking the tone of the original spambot content to point out how annoying it was. Why are you incapable of doing this with the original spambot comment?
jfindper 12/3/2025||
I'm not replying to your slop (well, you know, after this one).

Anyways, if you think something is ai, just flag it instead so I don't need to read the word "slop" for the 114th fucking time today.

Thankfully, this time, it was flagged. But I got sucked in to this absolutely meaningless argument because I lack self control.

anonymous908213 12/3/2025||
Ironically, you were the first person in this thread to use the word "slop". You have become what you hate.
jfindper 12/3/2025||
jokes on you, I already hate me, that’s why I spend so much time on HN arguing about nothing

oh shit I’m supposed to be done replying

slop-cop 12/4/2025|||
[dead]
Despyte 12/3/2025|||
Cultural acceptance of conversation with AI should've come because of actual AI that are indistinguishable from humans, being forced to swallow recognizable if not blatant LLM slop and turn a blind eye feels unfair
jfindper 12/3/2025||
the original comment in this chain is not blatant llm slop.
chunk1000 12/3/2025||
Thank you bearsyankees for keeping us informed.
observationist 12/3/2025||
I think this class of problems can be protected against.

It's become clear that the first and most important and most valuable agent, or team of agents, to build is the one that responsibly and diligently lays out the opsec framework for whatever other system you're trying to automate.

A meta-security AI framework, cursor for opsec, would be the best, most valuable general purpose AI tool any company could build, imo. Everything from journalism to law to coding would immediately benefit, and it'd provide invaluable data for post training, reducing the overall problematic behaviors in the underlying models.

Move fast and break things is a lot more valuable if you have a red team mechanism that scales with the product. Who knows how many facepalm level failures like this are out there?

croes 12/3/2025||
> I think this class of problems can be protected against.

Of course, it’s called proper software development

jeffbee 12/3/2025|||
The techniques for non-disclosure of confidential materials processed by multi-tenant services are obvious, well-known, and practiced by very few.
venturecruelty 12/3/2025|||
And jail time for executives who are responsible for data leaks.
marginalx 12/3/2025||
Are you saying executives cannot make mistakes ever (ask because you didn't qualify your statement)?
venturecruelty 12/3/2025|||
I'm saying that if executives get praise and bonuses for when good things happen, they should also have negative consequences when bad things happen. Litigate that further how you wish.
dghlsakjg 12/4/2025||||
The key word in that is "responsible".

The legal world has plenty of ways for determining if you are legally responsible for the outcome of an event. Right now the standard is civil punishments for provable negligence.

It sounds like GP is proposing a framework where we tighten up the definition of negligence, and add criminal penalties in addition to civil ones.

pbhjpbhj 12/4/2025|||
Are you saying the OP was just a single error, effectively an executives typo.
dghlsakjg 12/4/2025||
In this case, AI was a red herring.

This was just plain terrible web security.

imvetri 12/3/2025|
Legal attacks engineering - font type license fee on japan consumers. Engineering attacks legal - AI info dump in above post.

How does above sound like and what kind of professional write like that?