If you can, do not expose ports like this 8080:8080, but do this "192.168.0.1:8080:8080" so its bound to a private IP. Then use any old method expose only what you want to the world.

In my own use I have 10.0.10.11 on the vm that I host docker stuff. It doesn't even have its own public IP meaning I could actually expose to 0.0.0.0 if I wanted to but things might change in the future so it's a precaution. That IP is only accessible via wireguard and by the other machines that share the same subnet so reverse proxying with caddy on a public IP is super easy.

  {
    "log-driver": "json-file",
    "log-opts": {
      "labels": "production_status",
      "tag": "{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}",
      "env": "os,customer",
      "max-size": "10m"
    },
    "bip": "172.17.1.1/24",
    "default-address-pools": [
      {"base": "172.17.0.0/16", "size": 24}
    ]
  }

  > Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. 

Like I said in another comment, drop Docker, install podman.

  loginctl enable-linger

    systemctl --user enable podman.socket loginctl enable-linger <USER>

?

it doesn't matter what netfilter frontend you use if you allow outbound connections from any binary.

In order to stop these attacks, restrict outbound connections from unknown / not allowed binaries.

This kind of malware in particular requires outbound connections to the mining pools. Others downloads scripts or binaries from remote servers, or try to communicate with their c2c servers.

On the other hand, removing exec permissions to /tmp, /var/tmp and /dev/shm is also useful.

I strongly disagree, firewall-cmd is way too complicated. I mean it's probably fine if your main job is being a firewall administrator, but for us who just need to punch a hole in a firewall as a tiny necessary prerequisite for what we actually want to do, it's just too much.

On ufw systems, I know what to do: if a port I need open is blocked, I run 'ufw allow'. It's dead simple, even I can remember it. And if I don't, I run 'ufw --help' and it tells me to run 'ufw allow'.

Firewall-cmd though? Any time I need to punch a hole in the firewall on a Fedora system, I spend way too long reading the extremely over-complicated output of 'firewall-cmd --help' and the monstrous man page, before I eventually give up and run 'sudo systemctl disable --now firewalld'. This has happened multiple times.

If firewalld/firewall-cmd works for you, great. But I think it's an absolutely terrible solution for anyone whose needs are, "default deny all incoming traffic, open these specific ports, on this computer". And it's wild that it's the default firewall on Fedora Workstation.

Personally I find just using nftables.conf straightforward enough that I don't really understand the need for anything additional. With iptables, it was painful, but iptables has been deprecated for a while now.

> Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Depending on your configuration, those firewall rules in OP may not actually do anything to prevent docker from opening incoming ports.

This sounds like great news. I followed some of the open issues about this on GitHub and it never really got a satisfactory fix. I found some previous threads on this "StrictForwardPorts": https://news.ycombinator.com/item?id=42603136.

Hetzner has a free firewall service outside of your machine. You can use that as the first line of defence.

The problem with firewalld is that it has the worst UX of any program I know. Completely unintuitive options, the program itself doesn’t provide any useful help or hint if you get anything wrong and the documentation is so awful you have to consult the Red Hat manuals that have thankfully been written for those companies that pay thousands per month in support.

It’s not like iptables was any better, but it was more intuitive as it spoke about IPs and ports, not high-level arbitrary constructs such as zones and services defined in some XML file. And since firewalld uses iptables/nftables underneath, I wonder why do I need a worse leaky abstraction on top of what I already know.

I truly hate firewalld.

Why people use these abstractions over nftables? Use nftables, it's easy to learn, effective and if you know it you know everything about firewalls.

Do either of them work for container-to-container traffic?

Imagine container A which exposes tightly-coupled services X and Y. Container B should be able to access only X, container C should be able to accesd only Y.

For some reason there just isn't a convenient way to do this with Docker or Podman. Last time I looked into it, it required having to manually juggle the IP addressed assigned to the container and having the service explicitly bind to it - which is just needlessly complicated. Can firewalls solve this?

UFW and Firewall-CMD both just use iptables in that context though. The real upgrade is in switching to nftables. I know I'm going to need to learn eBpf as the next step too, but for now nftables is readable and easy to grok especially after you rip out the iptables stuff, but technically nftables is still using netfilter.

And ufw supports nftables btw. I think the real lesson is write your own firewalls and make them non-permissive - then just template that shit with CaC.

also docker bypasses ufw

You might be interested in ufw-docker: https://github.com/chaifeng/ufw-docker

I've not used firewalld but I have used ufw on my desktops and servers going back to 2006 and can guarantee I have no plans to change from it.

One of those rare HN comments that's just pure gold.

I’ll just mention Foomuuri here. Its bit of a spiritual successor to shorewall and has firewalld emulation to work with tools compatible with firewalld

Does it have a gui?

..backed by xml?

Frontend churn has chilled out so much over the last few years. The default webapp stack today has been the same for 5 years now, next.js (9yo) react (12yo) tailwind (8yo) postgres (36yo). I'm not endorsing this stack, it just seems to be the norm now.

Compare that to what we had in the late 00's and early 10's we went through prototype -> mootools -> jquery -> backbone -> angularjs -> ember -> react, all in about 6 years. Thats a new recommended framework every year. If you want to complain about fads and churn, hop on over to AI development, they have plenty.

You can write web apps without touching the hottest JS framework of the week. I've never touched these frameworks that try to blur the line between frontend and backend.

Pick a solid technology (.NET, Java, Go, etc...) for the backend and use whatever you want for your frontend. Voila, less CVEs and less churn!

I had a Pangolin instance compromised by this: https://github.com/orgs/fosrl/discussions/2014

I'm hearing about it like crazy because I deployed around 100 Next frontends in that time period. I didn't use server components though so I'm not affected.

For my Java based sites, I use HTML/CSS/JS (vanilla js), no frameworks.

Use Svelte? (:

Another is running containers in read-only mode, assuming they support this configuration... will minimize a lot of potential attack surface.

This is true, but it's also easy to set at one point and then later introduce a bursty endpoint that ends up throttled unnecessarily. Always a good idea to be familiar with your app's performance profile but it can be easy to let that get away from you.

While this is a good idea I wonder if doing this could allow the intrusion to go undetected for longer - how many people/monitoring systems would notice a small increase in CPU usage compared to all CPUs being maxed out.

Soft and hard memory limits are worth considering too, regardless of container method.

This is a great shout actually. Thanks for pointing it out!

The other thing to note is that docker is for the most part, stateless. So if you're running something that has to deal with questionable user input (images and video or more importantly PDFs), is to stick it on its own VM and then cycle the docker container every hour and the VM every 12, and then still be worried about it getting hacked and leaking secrets.

Firewalls in the majority of cases don't get you much. Yes it's a last line of defense if you do something really stupid and don't even know where or what you configure your services to listen on, but if you don't the difference between running firewalls and not is minuscule.

There are way more important things like actually knowing that you are running software with widely known RCE that don't even use established mechanisms to sandbox themselves it seems.

The way the author describes docker being the savior appears to be sheer luck.

But the firewall wouldn't have saved them if they're running a public web service or need to interact with external services.

I guess you can have the appserver fully firewalled and have another bastion host acting as an HTTP proxy, both for inbound as well as outbound connections. But it's not trivial to set up especially for the outbound scenario.

The only time I have ever had a machine compromised in 30 years of running Linux is when I ran something exposed to the internet on a well known port.

I know port scanners are a thing but the act of using non-default ports seems unreasonably effective at preventing most security problems.

Password auth being enabled is also very brave. I don’t think fail2ban is necessary personally, but it’s popular enough that it always come up.

I don't whitelist IPs for ssh anymore, but I always run sshd on randomly selected port, in order to not get noticed by port scanners.

I do it for a really long time already, and until now I am not sure if it has any benefit or it's just umbrella in a sideways storm.

I have SSH blocked altogether and use wireguard to access the server. If something goes wrong I can always go to the dashboard and reenable SSH for my IP. But ultimately your setup is just as secure. Perhaps a tiny bit less convenient.

For the record this is only available for their VPS offering and not dedis. If you rent a dedi through their server auction you still need to configure your own firewall.

Yup. All my servers are behind Tailscale. The only thing I expose is a load balancer that routes tcp (email) and http. That balancer is running docker, fully firewalled (incl docker bypasses). Every server is behind herzner’s firewall in addition to the internal firewall.

App servers run docker, with images that run a single executable (no os, no shell), strict cpu and memory limits. Most of my apps only require very limited temporary storage so usually no need to mount anything. So good luck executing anything in there.

I used, way back in the day, to run Wordpress sites. Would get hacked monthly every possible way. Learned so much, including the fact that often your app is your threat. With Wordpress, every plugin is a vector. Also the ability to easily hop into an instance and rewrite running code (looking at you scripting languages incl JS) is terrible. This motivated my move to Go. The code I compiled is what will run. Period.

Honestly fail2ban is amazing. I might doa write up on the countless of attempts on my servers.

While this is true, the general security stance on this is: Docker is not a security boundary. You should not treat it like one. It will only give you _process level_ isolation. If you want something with better security guarantees, you can use a full VM (KVM/QEMU), something like gVisor[1] to limit the attack surface of a containerized process, or something like Firecracker[2] which is designed for multi-tenancy.

The core of the problem here is that process isolation doesn't save you from whole classes of attack vectors or misconfigurations that open you up to nasty surprises. Docker is great, just don't think of it as a sandbox to run untrusted code.

1. https://gvisor.dev/

2. https://firecracker-microvm.github.io/

Firstly, the attacker just wants to mine Monero with CPU, they can do that inside the container.

Second, even if your Docker container is configured properly, the attacker gets to call themselves root and talk to the kernel. It's a security boundary, sure, but it's not as battle-tested as the isolation of not being root, or the isolation between VMs.

Thirdly, in the stock configuration processes inside a docker container can use loads of RAM (causing random things to get swapped to disk or OOM killed), can consume lots of CPU, and can fill your disk up. If you consider denial-of-service an attack, there you are.

Fourthly, there are a bunch of settings that disable the security boundary, and a lot of guides online will tell you to use them. Doing something in Docker that needs to access hot-plugged webcams? Hmm, it's not working unless I set --privileged - oops, there goes the security boundary. Trying to attach a debugger while developing and you set CAP_SYS_PTRACE? Bypasses the security boundary. Things like that.

You really need to use user namespaces to get this kind of security protection -- running as root inside a container without user namespaces is not secure. Yes, breakouts often require some other bug or misconfiguration but the margin for error is non-existent (for instance, if you add CAP_SYS_PTRACE to your containers it is trivial to break out of them and container runtimes have no way of protecting against that). Almost all container breakouts in the past decade were blocked by user namespaces.

Unfortunately, user namespaces are still not the default configuration with Docker (even though the core issues that made using them painful have long since been resolved).

>there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something?

non necessary vulnerability per. se. Bridged adapter for example lets you do a lot - few years ago there were a story of something like how a guy got a root in container and because the container used bridged adapter he was able to intercept traffic of an account info updates on GCP

If the container is running in privileged mode you can just talk to the docker socket to the daemon on the host, spawn a new container with direct access to the root filesystem, and then change anything you want as root.

Docker containers with root have rootish rights on the host machine too because the userid will just be 0 for both. So if you have, say, a bind mount that you play fast and loose with, the docker user can create 0777 files outside the docker container, and now we're almost done. Even worse if "just to make it work" someone runs the container with --privileged and then makes the terminal mistake of exposing that container to the internet.

Container escapes exist. Now the question is whether the attacker has exploited it or not, and what the risk is.

Are you holding millions of dollars in crypto/sensitive data? Better assume the machine and data is compromised and plan accordingly.

Is this your toy server for some low-value things where nothing bad can happen besides a bit of embarrassment even if you do get hit by a container escape zero-day? You're probably fine.

This attack is just a large-scale automated attack designed to mine cryptocurrency; it's unlikely any human ever actually logged into your server. So cleaning up the container is most likely fine.

I think a root container can talk to docker daemon and launch additional containers...with volume mounts of additional parts of file system etc. Not particularly confident about that one though

    podman run -u$(id -u) --userns=keep-id

There would be, but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape.

Also, if you've been compromised, you may have a rootkit that hides itself from the filesystem, so you can't be sure of a file's existence through a simple `ls` or `stat`.

Either docker or a kernel level exploit. With non-VM containers, you are sharing a kernel.

TFA mentions it’s mostly a transcript of a Claude session literally in the first paragraph.

It's almost like... the problem wasn't the tool at all :P

Yeah I did consider just killing it, I'm going to keep an eye on it for a few days with a gun to it just in case.

I was lucky in that my DB backups were working so all my persistence wax backed up to S3. I think I could stand up another one in an hour.

Unfortunately I didn't keep an image no. I almost didn't have the foresight to investigate before yeeting the whole box into the sun!

Luckily umami in docker is pretty compartimentalized. All data is in the and the DB runs in another container. The biggest thing is the DB credentials. The default config requires no volume mounts so no worries there. It runs unprivileged with no extra capabilities. IIRC don't think the container even has bash, a few of the exploits that tried to run weren't able to due to lack of bash in the scripts they ran.

Deleting and remaking the container will blow away all state associated with it. So there isn't a whole lot to worry about after you do that.

Nothing in that container luckily, just what Umami needed to run, so no creds at all. Thanks for the info though!

I love mining malware - it's reasonably visible and causes almost no damage. Essentially, it's like a bug bounty program that you don't have to manage, doesn't generate costly bullshit reports, and only costs you a few bucks of electricity when a vulnerability is found.

If you have decent network or process level monitoring, you're likely to find it, while you might not realize the vulnerable software itself or some stealthier, more dangerous malware that might exploit it.

I had to nuke my Oracle Cloud box that runs my Umami server. It got hit. Was a good excuse to upgrade version and upgrade all my backup systems etc. Lost a few hours of data while it was returning 500 errors.