Top
Best
New

Posted by soheilpro 10 hours ago

My monitor was cutting my download speed by 75%(blog.tymscar.com)
379 points | 194 commentspage 2
zodwick 5 hours ago|
>It’s run by this one guy

This was super surprising, have always thought ISPs are a super hard businesses. But good for you mate, good ISPs are lifesavers.

I have had the opposite experience with my ISP guys. One of the VMs I got as part of work only had IPv6 and to my shock my internet didn't have one. When I contacted my ISP they legit told "You can either have IPv4 or IPv6, but not both". Unfortunately I was on a 12 month plan and couldn't easily cancel so ended up working around using tailscale.

eska 4 hours ago|
There are communities of small towns and villages running their own ISP and providing help materials for others who want to try.
zodwick 3 hours ago||
Do you have any pointers ? Idt this is a thing where I live tbh
array4277 45 minutes ago||
Having the correct driver is crucial for any 2.5Gb+ adapter. I know because I've run into this problem with 2.5 and 5Gb dongles too.
srcreigh 8 hours ago||
Nice post. Unsatisfying conclusion - did you ever figure out why the original adapter was throttled when plugged into the monitor, and not when plugged into the computer?
cibyr 8 hours ago||
The additional latency from going through the monitor made the bandwidth-delay product exceed the amount of data the adapter was able to send before receiving an ack. I thought that was fairly clear from the explanation, but now I notice that he didn't mention the latency without the monitor.
nh2 5 hours ago||
That doesn't seem to be described by the article at all.

The latency added in the monitor is described as around 8 us, completely negligible compared to any Internet connection's latency.

So the latency without the monitor should be completely irrelevant.

toast0 2 hours ago|||
> The reason is that if you add up the average time it takes to get the go-ahead through my monitor’s hub, which is around 7.6 µs, to the average time it takes to send 4 KB, around 8.2 µs, you can find what the maximum speed is.

This part is saying the NIC with this monitor can transfer 4 KB to the host through the USB hub every 15.8 us. You can do the math and it's about 2gbps.

Remove the hub, and reduce the delay solves the problem.

Increase the burst size would work: 8 KB per 24.0 us would be sufficient for 2.5 gbps.

Increase the burst size and the transfer rate as done with the 5 gbps adapter, and that should work out, although that one is probably going to be insufficient for full rate 5 gbps as well.

nh2 1 hour ago||
Yes, that is correct. I had misunderstood which connection cibyr was referring to, see sibling post.
db48x 3 hours ago|||
Negligible compared to the RTT, sure. But they’re talking about the latency between when the network adaptor signals that a packet has arrived and when the CPU actually services the interrupt to tell the card to send the packet(s) to main memory. The NIC’s buffer could only hold 4 packets, so by the time the NIC got a DMA buffer to copy the packets into a fifth or sixth packet had arrived and been dropped.
nh2 1 hour ago||
I see now, indeed. What cibyr posted is entirely correct.

I thought their "latency without the monitor" had referred to the overall end-to-end connection to the other IP endpoint, but they were talking about `NIC <-USB-> laptop`.

nh2 5 hours ago|||
The article answers this

"if the speed of the drain from my adapter to my Mac is half a gig slower than the speed of the input, then you can imagine the buffer getting filled more and more until it overflows. Those are dropped packets, which result in a hard backoff when the router resends them, so you drop loads in speed. And that’s the reason for all of this."

With diagram https://blog.tymscar.com/studio-display-ethernet/bottleneck....

The monitor decides to send the adapter's packets on at only 2 Gbit/s, which leads to buffer overflow, thus packet loss, thus e.g. TCP congestion-control backoff.

The only thing we don't know is _why_ the monitor chooses to limit the bandwidth to those 2.07 Gbit/s (computed as the 41 KiB blocks per (7.6+8.2) µs). If it sent e.g. 81 KiB blocks, doubling the bandwidth, the problem wouldn't happen.

That'd be between Apple and the monitor.

db48x 2 hours ago|||
No, you’ve misunderstood. Any time the NIC signals that one or more packets have arrived the CPU must reply back to the NIC to give it a DMA buffer to copy the packet(s) into. By the time that reply arrived at the NIC it was already too late. The NIC could only buffer four packets and the fifth or even sixth packet had already arrived and been dropped. The NIC then sent the four packets it had to the DMA buffer and started over.

The monitor was not really at fault. All USB hubs must read USB packets on all of their ports and copy them over to the upstream port. Those packets must all go into a buffer first so they the hub can send them individually even when they arrive simultaneously. That introduced just enough delay to allow the NIC’s internal buffer to fill up. Any hub would have caused the same problem. The solution was to buy a NIC designed for a higher speed because it has a bigger internal buffer. That new NIC will probably have the same problem with USB hubs, but at 5Gbps instead of 2.5Gbps.

nh2 1 hour ago|||
> The NIC could only buffer four packets

Where does the "4" come from? The article's diagram suggests that the NIC's buffer is "~35 KB", though I don't know where that number comes from either (I supposed measuring how long it takes until frames are dropped). "4" seems to be the number of 1 KB packets (the 3+1 from "packet size 1024 bMaxBurst 3"), so a transmission size (protocol-level "window size"), not a buffer size. I guess if you are referring to that as a different "buffer", that makes sense, but it doesn't seem to be the buffer the article is talking about.

But you are right with your main point regarding latency. My statement

> The monitor decides to send the adapter's packets on at only 2 Gbit/s

isn't really right: It's not that the monitor "decided" to only send 41 KiB instead of larger sizes; the 41 KiB is the maximum that the NIC can send because its `bMaxBurst = 3`, and the latency added by the monitor being a USB hub then limits the bandwidth to the mentioned ~2 Gbit/s.

I guess the key point not said anywhere is that `bMaxBurst = 3` and `wMaxPacketSize = 1024 B` are fixed protocol-level parameters that hubs like the monitor have no control over. So the 4 KiB windows are fixed from the NIC through the monitor to the Mac; the monitor cannot e.g. decide to buffer some of those to compensate for the latency it adds.

brigade 2 hours ago|||
Not all hubs and computers are equal here. Having experienced this with RTL8153, the combination of the Studio Display and ARM MacBooks is especially bad. Using an Intel MacBook with the display, or a different USB hub instead of the display, gets the full NIC bandwidth.
db48x 1 hour ago||
Fair enough, but I still wouldn’t say that he monitor is at fault. It’s probably not operating outside the USB specification in any way, even if it takes longer to forward messages than other hubs. It may even be multiple hubs, internally.

It’s really the NIC’s fault because its buffer is too small to handle a little extra latency.

nh2 1 hour ago|||
Hm, HN garbled my syntax involving asterisks. I meant to write

    computed as the 4*1 KiB blocks per (7.6+8.2) µs). If it sent e.g. 8*1 KiB blocks
not

    computed as the 41 KiB blocks per (7.6+8.2) µs). If it sent e.g. 81 KiB blocks
archerx 8 hours ago||
I’m guessing it’s a USB bandwidth issue. I learned about these kind of limitations when I tried to have 3 kinects running at the same time on the same USB bus.
swiftcoder 7 hours ago|||
> I’m guessing it’s a USB bandwidth issue

It's explicitly not a bandwidth issue (that would also affect the SSD). It's a latency issue - the chip doesn't have a big enough buffer to keep sufficient packets in flight at once.

nh2 5 hours ago||
> It's explicitly not a bandwidth issue

That's explicitly the opposite of what the author says.

"the speed of packets coming into the adapter is 2.5 Gbps. That’s what the router pushes in. The speed going out is around 2 Gbps."

Of course latency is always on the denominator side in the "bandwidth = data / time" equation, but it seems wrong to claim it's latency when it could also be improved by fixing the numerator; it's the entire fraction that's insufficient, thus bandwidth. As written:

"The reason is that if you add up the average time it takes to get the go-ahead through my monitor’s hub, which is around 7.6 µs, to the average time it takes to send 4 KB, around 8.2 µs, you can find what the maximum speed is"

And later, increasing the numerator is exactly what fixes it with the other hardware: "I can see that instead of sending 4 1 KB packets per burst, it would send 8!"

> the chip doesn't have a big enough buffer to keep sufficient packets in flight at once

How does this make sense? Even if his buffer was 100x larger, if the fill bandwidth exceeds the drain bandwidth, it'll eventually fill up. The article says that explicitly:

"Well, if the speed of the drain from my adapter to my Mac is half a gig slower than the speed of the input, then you can imagine the buffer getting filled more and more until it overflows."

swiftcoder 5 hours ago|||
> Even if his buffer was 100x larger

"buffer" was perhaps a poor choice of words on my part, because "buffer" is ambiguous. In this case I am referring to the notional "transmit buffer", which is 4 in the non-working case and raised to 8 in the working adapter.

> "I can see that instead of sending 4 1 KB packets per burst, it would send 8!"

Increasing the number of packets in flight does not change the bandwidth, it only improves latency sensitivity (i.e. you can fire more data before requiring an ACK).

> it seems wrong to claim it's latency when it could also be improved by fixing the numerator.

Increasing the bandwidth without increasing the transmit buffer would not fix this issue. The adapter he ends up buying does both (allow 2x the packets in flight, and 2x the bandwidth).

The bottom line here is that the original adapter handles full duplex 2.5g when plugged into the laptop directly, and the SSD gets full duplex bandwidth in excess of 2.5g through the same intermediate link (i.e. the display), so both the adapter and the display have sufficient bandwidth. The added hop of latency however makes the short transmit buffer insufficient to cover the latency.

nh2 1 hour ago||
> In this case I am referring to the notional "transmit buffer", which is 4 in the non-working case

Then I agree with what you were saying. Those 4 * 1 KiB are a protocol-level size (similar to the TCP window size), not an actual "buffer".

> Increasing the number of packets in flight does not change the bandwidth, it only improves latency sensitivity (i.e. you can fire more data before requiring an ACK).

That I don't fully follow. To me, the measured bandwidth includes all the effects of ACKs etc. If the the protocol demands that you can have at max 4 * 1 KiB unACKed in flight so you're waiting around for ACKs and not snding data, then increasing the number of packets in flight to 8 would 2x the bandwidth.

> The added hop of latency however makes [it] insufficient to cover the latency.

I agree with that, see also my newer reply https://news.ycombinator.com/item?id=50592931

I this context, I also agree with your original statement; I misunderstood what you referring to with "buffer" (meaning the window, not actual storage), and "bandwidth" (where I think you're referring to the 5 Gbit/s nominal bandwidth of the bus, irrespective of the fact that the effective bandwidth of data sent is much lower due to requiring ACKs for small data).

chrisweekly 5 hours ago|||
This primer on latency and bandwidth is 13 years old but still worth sharing (for the sake of all readers)

https://hpbn.co/primer-on-latency-and-bandwidth/

KeplerBoy 7 hours ago||||
He states that high speed SSDs through the monitor are fine though.
prepend 7 hours ago|||
Uploads were fine.
js2 2 hours ago||
Usually people discover the bandwidth delay product on long fat pipes. Neat to read someone discover it on a very fast very short link.

https://en.wikipedia.org/wiki/Bandwidth-delay_product

throw0101d 6 hours ago||
What's the difference in BOM cost for Realtek RTL8153 (1GigE) versus RTL8156 (2.5G) versus RTL8157 (5G) versus RTL8159 (10G)?
Palomides 5 hours ago|
chips are probably like $2 to $50 across that range, hard to tell since realtek sucks
ryankrage77 1 hour ago||
Maybe I need to look into frames and dropped packets. My LAN can't quite reach 1Gbps, it's more like 7-900Mbps. Last time I investigated it, I came to the conclusion it's sub-par ethernet cables, but perhaps there's more to it.
brigade 2 hours ago||
> because the issues don’t start at 1 gig

Actually, the speed of gigabit Ethernet adapters are also known to cut 60% behind a studio display. TBH I assumed it was because USB ECM sucks and NCM (which RTL8156 implements) was supposed to fix it.

glitchc 3 hours ago||
I suspect the problem here is the UGreen ethernet adapter. I have soured on UGreen after a couple of their recent hardware bits proved flaky and one damaged two laptop motherboards (it was a usb-c hub).
linker3000 4 hours ago||
Used to get this with dial-up modems when they were either placed on top of the CRT (just to get very toasty?), or the incoming phone line ran too close to some CRTs.

This would reduce the established connection speed due to electrical noise from the monitor.

cozzyd 6 hours ago|
Would bbr congestion control help here to.get speeds to 2 Gbps?
nh2 5 hours ago|
Seems likely. BBR (v1) does not do the hard 30% "multiplicative decrease" on packet loss (caused by the frame drop in the article's diagram), so I'd expect it to be able to correctly estimate the bandwidth to the ~2 Gbit/s bottlenecked by the monitor-Mac connection.
cozzyd 4 hours ago||
Yeah, I'd expect it too, but doing some research it seems Apple still doesn't support BBR. But I wonder if a speedtest with quic would show better results for the author.
toast0 2 hours ago||
It doesn't really matter what the client OS supports because the problem is inbound. You'd want fast.com to support BBR (although... I think they do, fast.com is Netflix and afaik they are a significant BBR user)
More comments...