Top
Best
New

Posted by hyperknot 10/26/2024

Understanding Round Robin DNS(blog.hyperknot.com)
394 points | 123 commentspage 2
solatic 10/27/2024|
Multiple A records is not for load balancing, a key component of which is full control over registering new targets and deregistering old targets in order to shift traffic. Because DNS responses are cached, you can't reliably use DNS to quickly shift traffic to new IP addresses, or use DNS to remove traffic from old IP addresses.

As OP clearly shows, it's also not useful for geographically routing traffic to the nearest endpoint. Clients are dumb and may do things against their interest, the user will suffer for it, and you will get the complaints. Use a DNS provider with proper georouting if this is important to you.

The only genuinely valid reason for multiple A addresses is redundancy. If you have a physical NIC, guess what, those fail sometimes. If you get a virtual IP address from a cloud provider, guess what, those abstractions leak sometimes. Setting up multiple servers with multiple NICs per server and multiple A records pointing to those NICs is one of those things you do when your usecase requires some stratospherically high reliability SLA and you systematically start to work through every last single point of failure in your hot path.

neuroelectron 10/26/2024||
We used to do this at Amazon in the 00's for onsite hosts. At the time round robin DNS was the fastest way to load balance as even with dedicated load balancers of the time, the latency was a few milliseconds slower. A lot of the decisions didn't make sense to me and seemed to be grandfathered in from the 90's.

We had a dedicated DNS host and various other dedicated hosts for various services related to order fulfillment. A batch job would be downloaded in the morning to the order server (app) and split up amongst the symbol scanners which ran basic terminals. To keep latency as low as possible the scanners would dns round robin. I'm not sure how much that helped because the wifi was by far the biggest bottleneck simply for the fact of interference, reflection and so on.

With this setup an outage would have no effect the throughput of the warehouse since the batch job was all handled locally. As we moved toward same day shipping of course this was no longer a good solution and we moved to redundant, dedicated fiber and cellular data backup then almost completely remote servers for everything but app servers. So what we were left with was million dollars hvac to cool a quarter rack of hardware and a bunch of redundant onsite tech workers.

hypeatei 10/26/2024||
The browser behavior is really nice, good to know that it falls back quickly and smoothly. Round robin DNS has always been referred to as a "poor mans load balancer" which it seems to be living up to.

> Curl also works correctly. First time it might not, but if you run the command twice, it always corrects to the nearest server.

This took two tries for me, which begs the question how curl is keeping track of RTT (round trip times), interesting.

freitasm 10/26/2024||
Interesting. The author starts by discussing DNS round robin but then briefly touches on Cloudflare Load Balancing.

I use this feature, and there are options to control Affinity, Geolocation and others. I don't see this discussed in the article, so I'm not sure why Cloudflare load balancing is mentioned if the author does not test the whole thing.

Their Cloudflare wishlist includes "Offline servers should be detected."

This is also interesting because when creating a Cloudflare load balancing configuration, you create monitors, and if one is down, Cloudflare will automatically switch to other origin servers.

These screenshots show what I see on my Load Balancing configuration options:

https://cdn.geekzone.co.nz/imagessubs/62250c035c074a1ee6e986...

https://cdn.geekzone.co.nz/imagessubs/04654d4cdda2d6d1976f86...

hyperknot 10/26/2024|
I briefly mention that I don't go into L7 Load Balancing because it'd be cost prohibitive for my use case (millions of requests).

Also, the article is about DNS-RR, not the L7 solution.

nielsole 10/26/2024||
> Curl also works correctly. First time it might not, but if you run the command twice, it always corrects to the nearest server.

I always assumed curl was stateless between invocations. What's going on here?

barrkel 10/26/2024|
My hypothesis: he's running on macOS and he's seeing the same behavior from Safari as from curl because they're both using OS-provided name resolution which is doing the lowest-latency selection.

Firefox and Chrome use DNS over HTTPS by default I believe, which may mean they use a different name resolution path.

The above is entirely conjection on my part, but the guess is heavily informed by the surprise of curl's behavior.

plagiat0r 10/27/2024|||
But this does not make sense. How Mac operating system resolver are supposed to test the latency of (A)ddress records? Browser use this network address to actually make a tcp connection on 443 and measure latency here. Or udp/443 when using http3/quic.

But operating system resolver only speak with DNS servers. It does not make https connections to calculate latency which would pick "the closest server". Also dns had no way to tell what port you will be using, maybe service is on 8443 or something.

For geo DNS I've built a custom backed for powerdns with geo DNS capabilities and healthckecks to quickly remove a broken vps from the DNS responses.

barrkel 10/27/2024||
If I had to hypothesize further, I'd say that macOS may let its DNS resolver cache interact with its TCP stack. It's not inconceivable that the TCP handshake is used to make a rough estimate of network latency.
plagiat0r 10/27/2024||
A bold hypothesis. The problem is, nowhere in the tcp handshake you will find the string, a.k.a fqdn. And one IP can host hundreds of fqdns.

No way MacOS parse tls clienthello looking for SNI.

Also I doubt a DNS resolver runs in the Mac kernel, ring 0 to pull this off.

The thing with DNS is that it works on layer 3. Hold on, what? Yes, layer 3 because you obtain network address for layer 3 (ip4, ipv6) but latency can be measured only in layer4 (tcp, quic). Of course I know that common wisdom says DNS is a layer 7 but from functional perspective, you are yet to establish your destination network address, therefore functionally it's like layer 3 to me. Or even lower, because without destination, you can't even start creating a packet and inspecting your routing table entries figuring out if you can even reach it ;)

There is zero chance Mac resolver libraries can connect you to the fastest responding server - unless there is no Berkeley sockets but something that allows you to do a connect(char * fqdn) and system library return you two pipes, one for write, other for read, and that you can close them independently. I doubt it there is such a thing, but don't know Mac os API.

hyperknot 10/26/2024|||
Correct. I'm on macOS and I tried turning off DoH in Firefox and then it worked like Safari.
mlhpdx 10/26/2024||
Interesting topic for me, and I’ve been looking at anycast IP services and latency based DNS resolvers as well. I even made a repo[1] for anyone interested in a quick start for setting up AWS global accelerator.

[1] https://github.com/mlhpdx/cloudformation-examples/tree/maste...

why-el 10/26/2024||
Hm, I thought Happy Eyeballs (HE) was mainly concerned with IPv6 issues and falling back to IPV4. I didn't think it was this RFC in which finally some words were said about round-robin specifically, but it looks like it was (from this article).

Is it true then that before HE, most round-robin implementations simply cycled and no one considered latency? That's a very surprising finding.

LikeBeans 10/27/2024||
Another way to solve for clients that stick with an IP after resolving is to use a combination of DNS RR and Anycast (if you have control over the physical infra). That means you resolve with RR to an IP in the regional data center and then use Anycast for local delivery. That way if the server goes down these clients can continue to operate.
zamalek 10/26/2024||
Take a look at SRV records instead - they are very intentionally designed for this, and behave vaguely similarly to MX. Creating a DNS server (or a CoreDNS/whatever module) that dynamically updates weights based on backend metrics has been a pending pet project of mine for some time now.
jeroenhd 10/26/2024|
Until the HTTP spec gets updated to include SRV records, using SRV records for HTTP(S) is technically spec-incompliant and practically useless.

However, as is common with web tech, the old SRV record has been reinvented as the SVCB record with a smidge of DANE for good measure.

chasil 10/27/2024|
I actually use round robin into a set of ssh servers.

There is never a delay if one of them is down.

I am using a closed-source client (Bluezone Rocket), but I'm assuming that it pulled a lot of code from PuTTY as it uses the PPK format.

More comments...