Top
Best
New

Posted by jakelsaunders94 6 days ago

I got hacked: My Hetzner server started mining Monero(blog.jakesaunders.dev)
604 points | 409 commentspage 3
egberts1 6 days ago|
This Monero mining also happened with one of my VPS over at interserv.net, when I forgot to log out of the root console in web-based terminal console to one of my VPS and closed its browser tab instead.

It has since been fixed: Lesson learned.

tgsovlerkhgsel 6 days ago|
> when I forgot to log out of the root console in web-based terminal console to one of my VPS and closed its browser tab instead.

This should not enable the compromise of your session, i.e. either there was some additional vulnerability (and not logging out widened the window during which it could be exploited), or the attack was totally unrelated. Either way, you should find out the real cause.

egberts1 4 days ago||
And they did.
CGamesPlay 6 days ago||
I took issue with this paragraph of the article, on account of several pieces of misinformation, presumably courtesy of Claude hallucinations:

> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

1. Files of running programs can be deleted while the program is running. If the program were trying to hide itself, it would have deleted /tmp/.XIN-unix/javae after it started. The nonexistence of the file is not a reliable source of information for confirming that the container was not escaped.

2. ps shows program-controlled command lines. Any program can change what gets displayed here, including the program name and arguments. If the program were trying to hide itself, it would change this to display `login -fp ubuntu` instead. This is not a reliable source of information for diagnosing problems.

It is good to verify the systemd units and crontab, and since this malware is so obvious, it probably isn't doing these two hiding methods, but information-stealing malware might not be detected by these methods alone.

Later, the article says "Write your own Dockerfiles" and gives one piece of useless advice (using USER root does not affect your container's security posture) and two pieces of good advice that don't have anything to do with writing your own Dockerfiles. "Write your own Dockerfiles" is not useful security advice.

3np 6 days ago|
> "Write your own Dockerfiles" is not useful security advice.

I actually think it is. It makes you more intimate with the application and how it runs, and can mitigate one particular supply-chain security vector.

Agreeing that the reasoning is confused but that particular advice is still good I think.

minitech 6 days ago||
> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

  /tmp/.XIN-unix/javae &
  rm /tmp/.XIN-unix/javae
This article’s LLM writing style is painful, and it’s full of misinformation (is Puppeteer even involved in the vulnerability?).
jakelsaunders94 6 days ago||
Yeah fair, I asked claude to help because honestly this was a little beyond my writing skills. I'm real though. Sorry. Will change
minitech 6 days ago|||
Seconding what others have said about preferring to read bad human writing. And I don’t want to pick on you – this is a broadly applicable message prompted by a drop in the bucket – but please don’t publish articles beyond your ability to fact check. Just write what you actually know, and when you’re making a guess or you still have open questions at the end of your investigation, be honest about that. (People make mistakes all the time anyway, but we’re in an age where confident and detailed mistakes have become especially accessible.)
sincerely 6 days ago||||
Just a data point - I would rather read bad human writing than LLM output
croemer 6 days ago||||
It still says Puppeteer in multiple places.
seafoamteal 6 days ago|||
Hi Jake! Cool article, and it's something I'll keep in mind when I start giving my self-hosted setup a remodel soon. That said, I have to agree with the parent comment and say that the LLM writing style dulled what would otherwise have been a lovely sysadmin detective work article and didn't make me want to explore your site further.

I'm glad you're up to writing more of your own posts, though! I'm right there with you that writing is difficult, and I've definitely got some posts on similar topics up on my site that are overly long and meandering and not quite good, but that's fine because eventually once I write enough they'll hopefully get better.

Here's hoping I'll read more from you soon!

jakelsaunders94 6 days ago||
Thanks for the encouragement! I find it difficult to write articles beyond simply stating a series of facts.

I tried handwriting https://blog.jakesaunders.dev/schemaless-search-in-postgres/ bit I thought it came off as rambling.

Maybe I'll have a go at redrafting this tomorrow in non LLM-ese.

lewiscollard 6 days ago|||
> I tried handwriting https://blog.jakesaunders.dev/schemaless-search-in-postgres/ bit I thought it came off as rambling.

There is nothing wrong with this article. Please continue to write as you; it's what people came for.

LLMs have their place. I find it useful to prompt an LLM to fix typos and outright errors and also prompt them to NOT alter the character or tone of the text; they are extraordinarily good at that.

lmf4lol 6 days ago|||
This is much more pleasent to read and it gives a great insight into your actual thought process. Thanks for sharing and great writeup.
jakelsaunders94 6 days ago||
I fixed it, apologies for the misinformation.
3np 6 days ago|||
It still says:

> IT NEVER ESCAPED.

You haven't confirmed this (at least from the contents of the article). You did some reasonable spot checks and confirmed/corrected your understanding of the setup. I'd agree that it looks likely that it did not escape or gain persistence on your host but in no way have you actually verified this. If it were me I'd still wipe the host and set up everything from scratch again[0].

Also your part about the container user not being root is still misinformed and/or misleading. The user inside the container, the container runtime user, and whether container is privileged are three different things that are being talked about as one.

Also, see my comment on firewall: https://news.ycombinator.com/item?id=46306974

[0]: Not necessarily drop-everything-you-do urgently but next time you get some downtime to do it calmly. Recovering like this is a good excercise anyway to make sure you can if you get a more critical situation in the future where you really need to. It will also be less time and work vs actually confirming that the host is uncontaminated.

jakelsaunders94 6 days ago||
I did see your comment on Firewall, and you're right about the escape. It seems safe enough for now. Between the hacking and accidentally hitting the front page of HN it's been a long day.

I'm going to sit down and rewrite the article and take a further look at the container tomorrow.

3np 6 days ago|||
Hey, thanks for taking the time to share your learnings and engage. I'm sure there are HN readers out there who will be better off for it alongside you!

(And good to hear you're leaving the LLMs out of the writing next time <3)

microtonal 6 days ago|||
Before rewriting the article, roll out a new server. Seriously. It seems you do not have the skills yet to do a proper audit. It’s better to roll out a pristine server. If that is a lot of work, it is a good moment to learn about declarative system configuration.

At any rate, this happening to you sucks! Hugs from a fellow HN user, I know that things like this can suck up a lot of time and energy. It’s courageous to write about such an incident incident, I think it’s useful to a lot of other people too, kudos!

Eduard 6 days ago|||
I still see Puppeteer mentioned several times in your post and don't understand what that has to do with Umami, nextjs, and/or CVE-2025-66478.
rishikeshs 3 days ago||
What a coincidence. I also received a similar email and upon investigating, found the same Monero mining on my server. Tried rebuilding the image, but same thing was poping up. Then I did a npm update and that fixed everything!
andix 6 days ago||
I was surprised by the same thing, a tool I run that uses Next.js without me knowing before.

I noticed this, because Hetzner forwarded me an email from the German government agency for IT security (BSI) that must have scanned all German based IP addresses for this Next.js vulnerability. It was a table of IP addresses and associated CVEs.

Great service from their side, and a lot of thanks to the German tax payers wo fund my free vulnerability scans ;)

broken_broken_ 6 days ago||
I am not an expert in incident reaction, but I thought the safe way was to image the affected machine, turn it off, take a clean machine, boot a clean OS image with the affected image mounted read only in a VM, and do the investigation like that ?

Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?

dewey 6 days ago|
Maybe if your company infrastructure is affected but not the server you use to host your side projects on with “coolify” unless IT security is your hobby.
hughw 6 days ago||
You can run Docker Scout on one repo for free, and that would alert you that something was using Next.js and had that CVE. AWS ECR has pretty affordable scanning too: 9 cents/image and 1 cent/rescan. Continuous scanning even for these home projects might be worth it.

[*] https://aws.amazon.com/inspector/pricing/

jlengrand 6 days ago||
TY for the article. You made me look into my server, and also found some strange activity on my docker containers. Good call!
nodesocket 6 days ago||
I also run Umami, but patched once the CVE patch was released. Also, I only expose the tracking js endpoint and /api/send via Caddy publically (though, /api/send might be enough to exploit the vul). To actually interact with Umami UI I use Twingate (similar to Tailscale) to tunnel into the VPC locally.
hoppp 6 days ago|
This nextjs vulnerability is gonna be exploited everywhere because its so easy. This is just the start
christophilus 6 days ago|
I didn’t think it was possible for me to dislike nextjs any more, but here we are. It’s the Sharepoint of the JS ecosystem.
More comments...