Posted by Brajeshwar 1 day ago
For reference, the median hourly wage is $27/hour.
Many experts are holding out, and I don't blame them. Why would you want to train AI to replace your job?
It has never been a successful strategy to try and fight new technology. Never.
How do I join?
Honest question: of course, everybody would prefer to work with "lovely" stuff, but I really have difficulties getting what people find so much difficult/hard about jobs where you encounter such content on a screen (the same holds for moderation jobs).
I would claim that I have seen the internet, and I guess many people of my generation have, too (just to be insanely clear: of course not the kind stuff that is hardcore criminal in basically all jurisdictions worldwide - I don't want to get more explicit here).
I wouldn't say I am blunted, but I do think I could handle this stuff without any serious problems as part of my job. I'd thus rather compare it in terms of emotional comfort with a toilet cleaner who sometimes also has to clean very filthy toilets - which is just an ordinary job that some people in society have to do.
Also im seeing the same trend as you at my company, roles replaced overseas while people only focus on AI taking the jobs i think this is the more sinister thing happening quietly (by that i mean not getting much news coverage)
Personally I would love to live in a more rural place, but until I am self sufficient enough, this is not an opportunity I am willing to take.
If you don't want people to ask, don't mention it.
> “At first they told [me]: ‘Don’t worry about time – it’s quality versus quantity,’” she said.
> But before long, she was pulled up for taking too much time to complete her tasks. “I was trying to get things right and really understand and learn it, [but] was getting hounded by leaders [asking], ‘Why aren’t you getting this done? You’ve been working on this for an hour.’”
And:
> Dinika said he’s seen this pattern time and again where safety is only prioritized until it slows the race for market dominance. Human workers are often left to clean up the mess after a half-finished system is released. “Speed eclipses ethics,” he said. “The AI safety promise collapses the moment safety threatens profit.”
Finally:
> One work day, her task was to enter details on chemotherapy options for bladder cancer, which haunted her because she wasn’t an expert on the subject.
This reminds me of the early voice-to-text start ups in the 00's that had these miraculous demos that required people in call centers to type it all up and pretend it was machine.
Great to see that they have not learned from this experience, and are repeating the mistake with Gemini.
RLHF providers:
1. Surge. $1b+ revenue bootstrapped. DataAnnotation is the worker-side (you might've seen their ads), also TaskUp and Gethybrid.
2. Scale. The most well known. Remotasks and Outlier are the worker-side
3. Invisible. Started as a kind of managed VA service.
4. Mercor. Started mostly as a way to hire remote devs I think.
5. Handshake AI. Handshake is a college hiring network. This is a spinout
6. Pareto
7. Prolific
8. Toloka
9. Turing
10. Sepal AI. The team is ex-Turing
11. Datacurve. Coding data.
12. Snorkel. Started as a software platform for data labeling. Offers some data as a service now.
13. Micro1. Also started as a way to hire remote contractor devs
Are there companies that focus on labeling of inputs rather than RLHF of outputs?
There are a whole lot of organizations training competent LLMs these days in addition to the big three (OpenAI, Google, Anthropic).
What about Mistral and Moonshot and Qwen and DeepSeek and Meta and Microsoft (Phi) and Hugging Face and Ai2 and MBZUAI? Do they all have their own (potentially outsourced) teams of human labelers?
I always look out for notes about this in model cards and papers but it's pretty rare to see any transparency about how this is done.
The business process outsourcing companies labelling things for AI training are often the same outsourcing companies providing moderation services to facebook and other social media companies.
I need 100k images labelled by the type of flower shown, for my flower-identifying AI, so I contract a business that does that sort of thing.
Facebook need 100k flagged images labelled by is-it-an-isis-beheading-video to keep on top of human reviews for their moderation queues. They contract with the same business.
The outsourcing company rotates workers between tasks, so nobody has to be on isis beheading videos for a whole shift.
Is that an assumption on your side, a claim made by the business, a documented process or something entirely different?
There are some open-weights NSFW detectors [1] but even if your detector is 99.9% accurate, you still need an appeals/review mechanism. And someone's got to look at the appeals.
A lot of these suppliers provide on-demand workers - if you need 40 man-hours of work on a one-off task, they can put 8 people on it and get you results within 5 hours.
On the other hand, if you want the same workers every time, it can be arranged. If you want a fixed number of workers on an agreed-upon shift pattern, they can do that too.
Even when there is a rotation, the most undesirable tasks often pay a few bucks extra per hour, so I wouldn't be surprised if there were some people who opted to stay on the worst jobs for a full shift.
Even if you can afford only a couple of people a month and it takes 5x as long, do it. It's much eaiser to deal with high quality data than to firefight large quantities of slop. Your annotators will get faster and more accurate over time. And don't underestimate the time it takes to review thousands of labels. Even if you get results l in 5 hours, someone has to check if it's any good. You might find that your bottleneck is the review process. Most shops can implement a QA layer for you, but not requesting it upfront is a trap for young players.
Given the number of labs that are competing these days on "open weights" and "transparency" I'd be very interested to read details of how some of them are handling the human side of their model training.
I'm puzzled at how little information I've been able to find.
Time Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic
https://time.com/6247678/openai-chatgpt-kenya-workers/
Beyond that, I think the reason you haven't heard more about it is that it happens in developing countries, so western media doesn't care much, and also because big AI companies work hard to distance themselves from it. They'll never be the ones directly employing these AI sweatshop works, it's all contracted out.
https://nymag.com/intelligencer/article/ai-artificial-intell...
unwalled: https://archive.ph/Z6t35
Generally seems similar today just on a bigger Scale. And much more focus on coding
Here in the US DataAnnotation seems to be the most marketed company offering these jobs
https://www.theverge.com/features/23764584/ai-artificial-int...
it explores the world of outsourced labeling work. Unfortunately hard numbers on the number of people involved are hard to come by because as the article notes:
"This tangled supply chain is deliberately hard to map. According to people in the industry, the companies buying the data demand strict confidentiality. (This is the reason Scale cited to explain why Remotasks has a different name.) Annotation reveals too much about the systems being developed, and the huge number of workers required makes leaks difficult to prevent. Annotators are warned repeatedly not to tell anyone about their jobs, not even their friends and co-workers, but corporate aliases, project code names, and, crucially, the extreme division of labor ensure they don’t have enough information about them to talk even if they wanted to. (Most workers requested pseudonyms for fear of being booted from the platforms.) Consequently, there are no granular estimates of the number of people who work in annotation, but it is a lot, and it is growing. A recent Google Research paper gave an order-of-magnitude figure of “millions” with the potential to become “billions.” "
I too would love to know more about how much human effort is going into labeling and feedback for each of these models, it would be interesting to know.
Is it possible in 2025 to train a useful LLM without hiring thousands of labelers? Maybe through application of open datasets (themselves based on human labor) that did not exist two years ago?
https://finance.yahoo.com/news/surge-ai-quietly-hit-1b-15005...
Their continued revenue growth is at least one datapoint to suggest that the number of people working in this field (or at least the amount of money spent on this field) is not decreasing.
Also see the really helpful comment above from cjbarber, there's quite a lot of companies providing these services to foundation model companies. Another datapoint to suggest the number of people working providing labeling / feedback is definitely not decreasing and is more likely increasing. Hard numbers / increased transparency would be nice but I suspect will be hard to find.
Is it just to dodge labor laws?
Even theoretically.
To counter your question, what makes you think that's not the case? Do you think Mistral/Moonshot/Qwen/etc. are all employing their own data labelers? Why would you expect this kind of transparency from for-profit bodies that are evaluated in the billions?
"what makes you think that's not the case?"
I genuinely do not have enough information to form an opinion one way or the other.
Sure, but the way you're formulating the question is already casting an opinion. Besides, no one could even attempt to answer your questions without falling into the trap of true diligence... one question just asks how all (with emphasis!) LLMs are trained:
> Are all of the LLMs trained using human labor that is sometimes exposed to extreme content?
Who in the world would even be in such a position?
Depends how you look at it. I think a brand like Google should vet a mere one level down the supply chain.
to ensure the AI models are more aligned with Google's values and preferences.
FTFY
It does not have to have anything ro do with cyberpunk. Corporations are not people, but if they were people, they would be powerful sociopaths. Their interests and anybody elses interests are not the same.
> "Massive privacy invasion: The core of modern adtech runs on tracking your behavior across different websites and apps. It collects vast amounts of personal data to build a detailed profile about your interests, habits, location, and more, often without your full understanding or consent."
And which are these universal human values and preferences ? Or are we talking about silicon valley's executives values ?
"AI raters at GlobalLogic are paid more than their data-labeling counterparts in Africa and South America, with wages starting at $16 an hour for generalist raters and $21 an hour for super raters, according to workers. Some are simply thankful to have a gig as the US job market sours, but others say that trying to make Google’s AI products better has come at a personal cost."
There are lots of jobs out there that suck and people do them anyway. Because the freedom that they supposedly have is not as free as you imagine.
They're making more money than minimum wage. They're free to leave. It's not violating any safety regulations. There aren't any complaints of harassment.
So what precisely is the complaint here around worker's rights?
You'll be hard-pressed to find any 'documentation' of this other than journalists trying to raise hysteria around AI. It's just ragebait. Content moderation and data sorting jobs of this kind are as old as the internet itself. If you don't like it, find another job.
How is this not a straight up lie? For this to be true they would have to throw away labeled training data.
That's how validation works.
It does so indirectly, so it's a true albeit misleading statement.
This doesn't sound as bad to me as the Facebook moderator job or even a call center job, but it does sound pretty tedious.
Lots of people would do anything to get such work.
Congratulations, you just described most jobs. And many backbreaking laborers make about the same or less, even in the U.S., not to mention the rest of the world.
These types of articles always have an elitist view of the workers hired. That's a big source of the right (in the US) despising the left. The left don't say it directly, but when they talk about how shitty their town is and how the job they have is exploitative, there's an implicit judgment on the persons who live/work there.