The relative proximity of the words "done right" and "split-horizon DNS" makes my insides hurt a little bit.

Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term.

Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.

Split horizon DNS is not something I'm willing to do. I'll just rock out with .internal or .home.arpa, have step-ca and bind communicate to each other, either in step-issuer in kube or maybe even rfc2136 if i feel like a bit of the dns-01 strategy is in order. I slap the internal ca root certs everywhere, and keep my home infra out of the crt.sh logs.

I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..

Hmm. I don't really care enough about leaking home network host names because they are all super generic names like 'router', 'laptop', 'tv', 'nas'. So I use my public zone on cloudflare. I just use internal ip addresses (eg: nas.example.com = 10.1.2.3) on the public zone and DNS01 challenge for let's encrypt. Anyone can resolve the ip for any of my hosts, but obviously you'd need to be on the wireguard vpn to hit them.

This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc

This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.

My preferred procedure is to use DNS-01 validation and have no publicly accessible "A" or "AAAA" record for internal services.

Or even a more extreme example: https://crt.sh/?id=27555237869 (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.

The real answer here is that configuring HTTPS clients to trust a self-signed cert (or signed by an internal CA) shouldn't be as difficult as it is. I find it extremely annoying that every programming language has it's own idea of where certificates should live instead of just checking the os trust store.

I use the acme dns-1 challenge on my public domain. That gives you certificates you can use as you see fit, without needing to expose anything else to the public internet.

I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.

I was under impression that I understand networking and DNS resolution. It was really hard to follow, the OP did worked hard, just not sure what exact problem was being solved with the proposed solution that isnt already been solved.

This is crazy. If you have a home network with a few internal services, or some sort of network where you don't control the endpoints, just use DNS validation. That's why it exists.

But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.

Personally, I hate split horizon DNS. I prefer the "BeyondCorp" model. I MUCH prefer putting an mTLS cert in my trusted devices over relying on VPNs in same devices. I've yet to see a "clever" DNS setup not cause annoyances.

Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones.

I bet you can do it with HA-Proxy, but I use https://github.com/ThomasHabets/sni-router

I don’t know much in this space, but I find myself wishing there was a dead simple self hosted CA solution and also that trust on first use (à la ssh) was A Thing for self-managed root certs in client implementations. TOFU is such an elegant, good-enough solution for these use cases. Fixed deployment is always still an option, but in this day and age it feels so much like we are unnecessarily still dealing with solved problems