Posted by paimapi 12 hours ago
They aren't necessarily a sign of bias in the underlying model however. Many samples would be required for that.
there is a well studied scenario where humans are asked to hire people from four groups. These groups will be judged in their performance on a job and the humans rated on their hiring abilities. Unbeknownst to the human participants, all applicants are drawn from a single skill distribution, with groups assigned essentially randomly. Stastically, all groups have identical performance. Despite this, humans generalize over their early experiences, and develop biases towards specific groups.
While not identical, I relate this to the experience I have playing Fire emblem with random growths. A unit can get lucky and favored early despite being overall mediocre (hello Diamant from my first run through engage).
The researchers recreated this experiment with LLMs, and showed that the LLMs reproduce the human behavior of overgeneralizing early and failing to, as the paper says, sufficiently explore the space[0].
[1]: They instead exploit in the technical sense (https://en.wikipedia.org/wiki/Multi-armed_bandit), but exploit based on incomplete information.
Perhaps because there is only a real drawback to doing so if avoidance of bias is explicitly rewarded for some external reason? Like, by definition, if the groups are equal to each other, there's no loss from such exploitation (a larger candidate pool only helps if you have a working screening process, and a same-sized sample across the groups doesn't actually even confer the benefits of a larger candidate pool under the assumptions). Whereas if the observed clustering on a small sample isn't illusory, then ignoring it (or even actively going against it) would be clearly suboptimal. The probability of being actively misled by the clustering is necessarily less than the probability of being led correctly.
Going back to the example, of course bad FE units are less likely to overperform than good ones; that's what's bad about them. (But units can also be situationally good or bad for many reasons beyond their base stats and growth rates. And in FE we can typically directly observe that data and don't have to rely on anecdotes.) So the overperformance you saw was legitimate Bayesian evidence.
Please just go read the abstract; your first reactions to the headline may not be relevant.
I used to despise this kind of thing but it sheds light on the enormous generalization problems that aren't even close to being solved.
LLMs do not make decisions, or hold beliefs. Can we please stop anthropomorphizing the token generator?