Top
Best
New

Posted by TangerineDream 12 hours ago

Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache(blog.cloudflare.com)
625 points | 189 commentspage 2
bhouston 11 hours ago|
I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.
MayeulC 10 hours ago||
AFAIK (at least it worked like that some 10 years ago) the captive portal just intercepts the HTTP page load and inserts its own content (most often a 302). So it just has to be a http web page. Firefox uses http://detectportal.firefox.com/canonical.html

Relevant support page, though light in details: https://support.mozilla.org/en-US/kb/captive-portal

Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.

Edit 2: Wikipedia says both methods are used: https://en.wikipedia.org/wiki/Captive_portal and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.

My point was: that domain is not treated any differently from other domains.

comprev 8 hours ago|||
I've had reliable success by using http://neverssl.com to force a basic HTTP connection for kickstarting a public WiFi portal login, although I have to disable NextDNS (iOS) too.
Sohcahtoa82 6 hours ago|||
The ironic part of neverssl.com is that it does indeed now support SSL.
bpicolo 4 hours ago|||
I use good ol example.org
fc417fc802 9 hours ago|||
Can we take a minute to appreciate how utterly broken this state of affairs is? The dogged over centralization of DNS is an endless source of problems.
deathanatos 2 hours ago|||
Dumb captive portals, which do still exist in some places, usually do MitM attacks on the connection, so you need some http(no-s) site that you can abuse as "yeah, this can get attacked by the WiFi" to then answer the portal.

The right way is that there's DHCP option for the network to signal "I have a captive portal", that's been standardized for over a decade.

… or … IDK … just stop shoving ads down people's throats just because they want WiFi.

brians 11 hours ago||
That’s a Mac bug if so—it should be always using dumb udp/53 for captive detection, not some fancy DoH thing.
Dylan16807 3 hours ago||
So they optimized from Vec to Box, but they're still using Box all over and spending 16 bytes on it? The things they're boxing need 2 bytes for length, and their memory use is low enough that they could cram the pointers into 4 bytes. Trying to pack that into 6 bytes is probably too much fuss for the benefit, but I see no reason to use more than 8 bytes.
0xAstro 10 hours ago||
It's weird that it took so long for these trivial optimizations but it might just be that they were working on optimizing other stuff.
sergq 10 hours ago|
this applies to more than DNS caches. In 1998 I mailed Microsoft a proposal to replace search engine crawlers with a push-based filesystem monitor (detect change → extract → compress → push to index). Got a 5-line rejection letter. They built the same thing 20 years later as IndexNow. Full story with the original letter: https://dev.to/andrew_vl/in-1998-i-proposed-push-based-searc...
fulafel 2 hours ago||
Where are their users coming from? Besides the few manually putting 1.1.1.1 in their settings.
didgetmaster 3 hours ago||
Why do people seem to think that optimization is something you only have to deal with once the software scales so much that 100s of TB of memory or disk space (or thousands of hours of processing time) are being wasted.

It is almost like nobody even thought during the design phase about what might happen down the road.

This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).

jacquesm 3 hours ago|
It can be quite hard to predict where particular usage patterns will take a piece of software under extreme load, especially with things that have lots of internal state. Obviously when you get to spend 100 T or more the pay off of an optimization is much larger than what it is in the case of 1T or less, and your typical developer is not going to have that kind of memory even in aggregate to play with. I tend to be forgiving when it comes to watching software bloat that I did not cause myself (and yet, I'm frustrated that Ubuntu's start-up greeting message takes a whopping 500 M).

In the case of internet infrastructure I don't think there was anybody even up to the year 2000 who had any idea of how bit this was going to be. And even now we have IPV4 and lots of legacy to deal with. Cloudflare is not my favorite company, let's put it like that, but in this case they show how the sausage is made and I think that should be applauded. Much better than 'why were down again for X hours'.

edflsafoiewq 10 hours ago||
General theme: A programming language's native in-memory object format is typically optimized for random access, uniformity, and mutability (fields at fixed offsets, etc). Serialization formats for network or disk tend to be designed explicitly to be more compact. But you can design your own in-memory representation too, with the properties you need.
kccqzy 8 hours ago|
That’s the old school of thought. These days, designers of newer serialization formats realize that designing a more compact format doesn’t really buy much on modern CPUs and modern networks. See for example Cap’n Proto (whose inventor, kentonv, also works at Cloudflare) and flatbuffers.
inigyou 3 hours ago||
That's also the ancient school of thought, before compaction was viable and before portability was needed.
9bot 11 hours ago||
The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....
ManBeardPc 6 hours ago||
The Record struct contains rtype and data where RecordData is a tagged union. Aren’t those two always in sync? Not a DNS expert, just wondering if this is redundant or there is a reason both are there. Doesn’t matter anymore if they store it already serialized but I would be interested why it was this way.
rfgplk 10 hours ago||
Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd

> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.

jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?

jandrese 3 hours ago||
An approach like that would be at constant war with the borrow checker in Rust. Apparently it is possible but there is enough friction that these guys went a different route.
senderista 10 hours ago|||
I would also have instinctively reached for a large VM reservation to exploit demand paging. I have used that pattern a lot in C++ but not in Rust, so I don't know how difficult it would be to implement there.
cobalt 6 hours ago||
Rust supports punning via pointer casting, but you'll want to use #[repr(C)] on any data types used
pocksuppet 8 hours ago|
> 56% A records, 25% AAAA, and 19% TXT

And they say nobody uses IPV6.

shric 7 hours ago|
It’s finally gaining some traction…

https://www.google.com/intl/en/ipv6/statistics.html

hnav 6 hours ago||
no doubt because of scraping and the cost of IPv4
More comments...