Posted by soheilpro 10 hours ago
At the start of the game after an update, BG3 compiles the (Vulkan) shaders. To avoid hammering the computer, it checks the load, including I/O pressure, and then throttles itself. It turns out that Ghostty uses io_uring, and even if it's not doing anything this somehow counts as pressure. This was made worse by the fact that I usually have a dozen or so Ghostty windows.
End result, BG3 compiles shaders using 1 core, throttled, instead of all available 20 cores. Instead of a minute or so, it'd take several hours.
(For closure: I configured ghostty to use epoll instead. Probably not optimal but works around the problem.)
If I am reading the thread correctly, the io_uring maintainer's position is that applications trying to use the metric that "poll via io_uring" handles differently from "poll via epoll or select" are fundamentally misguided and measuring something that doesn't make sense, so they don't want to slow down io_uring to placate those apps.
> iowait is fundamentally incompatible with sane accounting in the first place, that is the problem. It really tells you nothing, and was deprecated since the days of uniprocessor systems. The problem here is not how iowait is done in the kernel, it's that people are using this metric in the first place. Which is why you just want to NOT do that, and do sane accounting on your disk drives, if you care about that kind of thing.
> Which is why I refuse to cater to any kind of iowait accounting metrics, it's just adding overhead to support a metric that is largely nonsensical in the first place. It doesn't answer any questions that an application/tier owner would want to know. Those questions might be things like "am I I/O bound?" or "is the device I'm using being slow?". I wish people would realize that, rather than continue to place value in iowait as a metric for ANYTHING, and stop trying to work around it.
It sounds more like their position is "iowait is a useless metric" and that polling with io_uring completely breaks iowait metrics but fixing it would add a minor slowdown so they refuse to do so because they don't think it's a valuable metric to begin with.
It looked at fossilize code (open source component used in the process), found a couple of threads online that suggested io_uring/iowait and told me first to try closing the ghostty windows and then to try to switch to epoll. This did the trick.
I have a very vague understanding of what's going on (actually better now that I've look at the links/resources from the replies in this thread!), and would probably have been able to figure it by googling in my own. But, my first hunch was that it's a problem with steam, proton, or the game, so I'd probably waste a lot of time in that direction first.
Especially for something like compiling shaders. It literally never happens unless the user is about to go play a game, which is going to consume most resources anyway. To the user, there's nothing more important running right now.
You in particular should probably disable a couple cores until your hardware stops being defective. Have you looked into updates and voltage droop compensation settings?
Users can be unreasonable, but in this case I agree - if I reported an issue to a vendor and their response was "return your motherboard" I wouldn't work with that vendor again.
> I wouldn't work with that vendor again.
Possibly rightly so, if it's bad motherboard hardware. Could also be a bad power supply, bad ram, bad CPU, user misconfiguration, etc. It's a complex system, where everything has to NOT be bad for it to be deterministic.
There's a necessary level of competence required with self built systems, or any other task in human life that you're not paying someone else to do for you. That's fine. Go buy a MacBook or a pre built system if that threshold of competence can't be met. That's why they exist, for people that can't or don't want to bother.
You're examples are flawed. Those are to safely halt operation on failure. What you're suggesting with the software is to limp along pretending everything is ok.
The proper test, more in line with your others, would be to catch failure and halt operation. Since, in your case c the human isn't competent enough for that, it would maybe be a self test initiated by the operating system, then a refusal to continue when the hardware problem is identified, emulating all the periodic inspections that are part of all the proper operation of all the physical systems your mentioned.
I think the elevator analogy to this bug is that the elevator goes too fast and triggers the brakes if you input target floors more than 8 apart. You don't fix that by having the user or user software be more careful about button placement, you fix the hardware/firmware.
If they want to cater to customers who foolishly reject reality and take every claim from manufacturer’s at face value… then foolishly misaimed blaming seems expected?
But if they just roll over and swallow the foolish blaming without even challenging customers to double check, then they are also facilitating more nonsense.
Why not just re-nice the process or whatever equivalent for the threads that do the compiling? Tell the kernel what’s going on, it will handle it for you.
Don’t hack your own solution.
Well, to a certain degree. I have a relatively uncommon GPU, and run games on Steam (and Linux), so literally every game does shader compilation for me. Even with a beefy CPU and beefy GPU, if the shader compilation uses more than 16 threads, my computer basically freezes until its done with it, as it's so performance intensive, kind of have to throttle things if I want to be able to use the desktop for the 10 minutes the shader compilation takes sometimes.
Edit: "Do you use Linux? That only happened to me when I was using a dram-less ssd for swap and doing IO heavy operations on the NFS mounted drive"
I do! I do have NFS drives, but not for Steam games or anything else relevant. I'm also not using swap at all. RAM seems fine during this, it's just pegging all cores to 100% usage immediately and stays there, making everything else suffer.
Strange. This never happens to me on Windows. I only notice that the fans spin up a little because all cores are pegged.
Sounds like a scheduling problem that should be adressed at the OS level.
1. Have you tried enabling full preemption as well? I run Debian, and before I did that I had lots of stutters from any kind of background load thanks to dynamic preemption.
2. Even then, ZFS was causing stutters until I lowered it's priority as well (spl_taskq_thread_priority=0)
3. On non-ZFS drives, try setting your I/O scheduler to Kyber for NVME or BFQ for everything else. "mq-deadline" isn't great for interactive use on a SATA SSD, even for non-boot drives, and even NVME sometimes stutters under "none" for some people.
> I'm also not using swap at all.
4. zswap (combined with zstd compression, "min_ttl_ms=1000", and "vm.page-cluster=0") and "defer+madvise" memory compaction can help. (Something about the kernel really liking having swap on-hand.)
Steam runs shader compilations at batch policy and nice 19, so without RAM or I/O bottlnecks even gaming shouldn't be too bad (though from personal experience it still induces some stuttering). It's just most distros are tuned horribly for this. (Not to sound flippant, I understand the frustration of "just try this" and it doesn't work, but it's everything I can think of RN.)
But it's also possible that the problem is in the kernel and the shaders are actually doing some system call that's locking up the kernel. Like maybe there's a bug in your cuda or nvidia driver that's causing the kernel to lock.
Also if you're running untrusted code that attacks your not-very-common terminal emulator, you're probably in more danger from the "untrusted code" part. Threat model and all that stuff.
The fact that it's a monitor is a little bit irrelevant.
The fix is to cut the overhead by sending more than 4 frames per interrupt. Unfortunately that required buying a new adapter with a different chip.
So it certainly seems possible to have an additional device in the signal path.
I think I'd have accepted having two cables from the get-go. And if I observed this problem, the first diagnostic I would do is to plug the Ethernet directly into the laptop.
Fewer technical pieces in a puzzle sometimes trumps ease of use.
I understand that not everybody likes this, because it can feel like a trick, or patronizing, or lying. But in my experience, it works very well for people.
Now I feel as though my time was wasted by a dishonest author trying to get me to believe that their story was interesting when it was not.
In the grand scheme of things, it’s not a very big deal, but I feel quite annoyed since I was baited into expecting a fun read.
To pick a not-RAID example, I've got an old (2012-vintage) 180GB SATA SSD, from Intel, on my desk. It gets used as scratch space for big-ish file operations, where it gets hammered hard periodically, usually in bursts. It's plugged into a cheap SATA <> USB 3 adapter, and is otherwise-unadorned. It is bus-powered.
It's usually connected to the back of the PC, but it's been plugged into different USB ports (with different controller types) and also external hubs.
I don't recall what its exact status was when I starting doing that with it after pulling it out of a gently-used Thinkpad T530 about 5 years ago, but I do recall thinking that it had lots of life left.
Here's where it is today:
- 10.6 years powered on
- 81TB written (of 36.5TB promised); ~222% over spec
- Zero reallocated sectors
- One singular hardware-reported error (corrected)
- Zero sector reallocations
And the pièce de résistance: - Zero reported errors according to ZFS checksums
That last one is important because in this application, ZFS automatically and continuously provides end-to-end verification, and it's been doing...perfectly fine.The other USB storage devices here that are used 24/7 are similar in this way. It's not hellish at all. It's quite lovely.
I'm inclined to further extrapolate that it would not have better data integrity when connected directly to a SATA port on the motherboard, or in a disk shelf attached via iSCSI or SAS or some other more-stodgy thing. It's hard get results that exceed beyond perfection.
(But in this example, I'm not chasing down minute bus timing issues with multi-gig networks. That's a very different kind of discussion.)
Your assertion doesn't seem to align with the article itself.
Need more proof?
A different adapter which required less frequent acknowledgement fixed the problem for him.
It is quite a nice post which gets to the bottom of incredibly opaque behaviour of USB hubs.
> Somehow I am one of those few very lucky people to have the opportunity of owning a 2.5 gig symmetrical line for my internet.
I'm not going to complain against more bandwidth (as a GenXer I stared my online journey at (IIRC) 9600bps (maybe lower): much celebration with V.32/V.90), but I'm curious to know how many residential use-cases can take advantage of that much bandwidth. Linux ISOs?
How many services allow that much speed on a single stream / (TCP) session? Maybe you could have five at 500 Mbps?
Certainly downloading security updates or games may be quicker, but I would think that above 500Mbps (and certainly 1 Gbps) there would be major diminishing returns. Is anyone regularly getting close to that utilization in their daily work / hobbies (and not just a momentary spike on patch Tuesday)?
I think going from something like 100 Mb download to 1 Gb is slightly overrated. You hit heavy diminishing returns. Symmetric Upload, for me, is what really changes your quality of life a lot. Uploading a youtube video suddenly isn't an event, it takes a few seconds. Want to upload an image in any chat? Done before you consciously see it uploading. Heavy commits are instant, even with a bunch of resources. It changed my behavior. I started uploading more stuff to remote servers because the cost in time got so much lower.
It’s great when experimenting with LLMs because I can quickly try new models and download from the internet almost as fast as if coming from my NAS.
Game updates are also a great use case. I don’t game often so when my friends want to play I often have to update the game. Some patches are 60GB or more. When they come down at full line speed that’s valuable time I’m getting back to spend with my friends in game.
But I run servers at home and regularly download enormous video files so I can think of plenty of ways to max out a 2.5 Gbps connection (alas my own is nowhere near as fast).
The benefits are two-fold:
1. Background tasks don't impact your own issue (as you say).
2. Many CDNs can saturate 4 Gbps, so if you download something, the waiting time is much shorter than say 100Mbps.
Whether it's worth it all depends on the price difference.
4Gbit is kind of the sweet spot currently. 2.5Gbit equipment became quite cheap (e.g. Ubiquiti Flex Mini 2.5Gbit managed switches are often around 50 Euro), existing Cat 5e cabling still works reliably, and if you have a router with 10Gbit upstream and multiple 2.5Gbit downstream ports, none of the subnets can completely saturate the upstream bandwidth.
Also, these bandwidths rock for doing an initial backup, downloading a full backup, pulling many Docker images, large and frequent NixOS rebuilds (unstable-small, yay). Downloading GTA 6 (not me though, not a gamer).
This also locks me out of hosting my own “cloud” storage at home where it belongs - I don’t want to be out and waiting forever to get a file, so I stick to traditional cloud services for now. Slower than local storage on the LAN of course though.
A fiber isp is digging trenches soon. Cannot wait!
Even better is those can saturate as much bandwidth as possible while the rest of the family are uninterrupted with Netflix or video calls.
An interesting side effect is that bufferbloat becomes less of a problem - you’re more likely able to pull 1gbps without spiking your ping.
But to be honest unless you have a household with multiple people, it’s something you mainly notice occasionally with updates, backups, large assets, etc.
Yeah definitely Linux ISOs. It's nice to settledown in an evening, pick a new Linux ISO I haven't used yet, and veg on the sofa watching it install.
Of course unlike the netinstall images all the normies use, you have to download it up front before you start wa, uhm, installing it. But I prefer it that way so I can watch the install without being interrupted... Then again I guess at 2.5Gbit the only thing that's going to interrupt you is a crappy integrated network interface, aka Apple monitor. I don't have enough money to have that problem though.
Fetching from Google or anything on S3 can definitely fill your pipe
Let's say you're on a Meet call with a dozen of people, one of them presenting their screen to explain an ongoing incident. You're following along while checking your dashboards and code on a remote server. You're of course keeping Slack sessions in the background.
I don't think you'd be saturating a 2.5G bandwidth, but perhaps about half of it ?
If your family is also using it in parallel that could get close to full use.
Where I work, group call participants send about ~1Mbps of video, and maybe more for screen share, could be ~2Mbps.
So you're looking at 15 Mbps inbound and 1 Mbps outbound for the call. Plus some overhead for congestion control. That's not even 1% of your 2.5gbps connection. IIRC Meet uses higher limits, so maybe double it to 30 Mbps.
You don't need gigabit for video conferencing, unless you're running the conference out of your house. Or you have 20 people all on conferences. You may want some QoS if your usage is near your bandwidth limits. Conferencing is sensitive to jitter and loss.
At my current ISP, speed is either 100M or 1G, so if you had 3+ people who conference simultaneously, 100M might well be limiting, so you would probably want 1G ... otoh, conferencing is usually bandwidth adaptive too, so it would still work, as long as everything plays nice with congestion.
This is closer to 1-10Mbps than 2.5Gbps. Slack is negligible bandwidth. Video conferencing is high compression, low bandwidth video.
The 1G and 2.5G lines benefit bulk file downloads.
I have been on Zoom and Google Meets calls with that many people on my laptop which has a Wifi connection of ≤400 Mbps (sometimes <200Mbps). You don't need >500 Mbps for online meetings.
Are you in meetings where everyone is sending 4Kp60?
lowres, please.
only 4K if the feed is from a full room with +10 people in it.
I only upgraded because my isp charged $90 for 1gig and offered 5 gig at $65 for two years so for the price of a pci nic I thought why not. It’s also nice that every device in the house can max out their capability simultaneously but that’s a rare event.
Run it in background, don’t update so often perhaps; I guess I just update as tic.
I don't know what you have in mind regarding what qualifies as residential usecases. I can tell you that I use my residence for work, and as part of my work I have to hold video calls and download all kinds of packages from the internet, including pulling container images, including media asset bundles that take several GB.
This often takes place when other members of my household are doing their work or watching streaming services.
If you have a 2.5Gbps connection, you often are not looking into sustainable transfer rates but bursts. A difference between 500Mbps and 2.5Gbps is whether it takes you half a minute to pull a package or 5-10 minutes.
These tend to not support sending pause frames to its SFP+ side either, so the best option I found was to set bandwidth limit on the NIC…
I experienced this problem with a macbook and a thunderbolt dock. I had tried multiple different brands of docks and ethernet adapters, and various macbooks. One interesting thing I discovered is if I plugged a non-thunderbolt cable between the dock and the laptop, I would achieve full speed with no packet loss, so interestingly the issue doesn't occur over pure USB but only over thunderbolt.
This has got to be a bug on Apple's side right? Would be nice if they fixed it.
All the more irritating is that all this dock and dongle bullshit was popularlized by them.
Anyway, the exact same situation arises, where some network condition or physical configuration (there’s a lot of hardware in between these two devices, including MoCA adapters) meant that the network adapter buffer fills up and starts dropping packets. The buffer size is configurable through some terminal command and maxing that out makes the issue disappear immediately.
[1] https://www.speedrun.com/Pragma_Twice
[2] https://developer.mozilla.org/en-US/docs/Web/API/Window/requ...
Immediately 10X'd my bandwidth, turned out I was using a cheap cable that wasn't shielded like it was supposed to be, and it was running right past the WiFi antenna and causing loads of interference.
One thing that can happen with pause frames (and why I hate them) is pause propagation. If you have a switch between your server and client and the client sends the switch a pause frame, the switch may send a pause frame to the server. If it does (and the server accepts pause frames), the server will pause sending to all destinations.
So your one slow client can disrupt all the other flows from the server. Which is totally unhinged.