Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
Wait what? I never heard of that. I call that a botnet
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
Without something like cookies (which are almost certainly tossed after the IP is rotated) or some other persistent identifier, you are stuck have to apply mitigations that scale with the load you're encountering, which means longer challenges for everyone or degraded functionality, like removing some of the fancier cgit features.
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.