Posted by zdw 4 days ago
This whole thing is basically snake oil. The best thing backend can do instead is have dedicated thread pool where each real thread has its own queue of limited size. Each element in queue would contain input and output state of request and code to deal with those. Once queue grows over certain size the backend should simply immediately return error code (too busy). Much more sound strategy in my opinion.
There are more complex cases of course (like computationally expensive requests with no io that take long time). Handling those would require some extra logic. Async stuff however will not help here either
Sure, it comes with its own issues like large stacks (10k copy of nearly the same stack?) and I predict a memory coloring in the future (stack variables or even whole frames that can opt out from being copied across virtual threads).
My trite slop bashing was days ago: