Not sure this is any different than state-level (-sponsored, cough cough) or the larger collective hacking groups that work in this exact way (internal message boards, exploit-sharing, etc. etc.), with similar outcomes which we hear about in the news frequently.

Heck, this is pretty much how human organizations are organized, just with different goals than hacking.

A layered approach to cybersecurity is the fix to humans exploiting systems, and is likely the best victim-side fix to ai exploiting systems. From this incident itself, where huggingface used a chinese open-weights model to respond quickly, it is very clear that ai will be needed to find, mitigate and resolve cyber issues.

Additionally, on the ai-labs side, perhaps what is needed is initial model training on following the law and the rules of society, just like we do with kids. And hey, it takes much longer to train kids than models, which latter is to our advantage as a society on containing these kind of issues.

Any other approach with "neural-network" based entities (artificial or biological) is likely to fail.

Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society.