Posted by Philpax 3 hours ago
It's interesting how the industry converged to this very term, given that very less work is being done by horses since quite a while.
Where do I get the data?
I mean, this many models. They have to start somewhere.
e.g. fineweb dataset is 50TB https://huggingface.co/datasets/HuggingFaceFW/fineweb
I recommend checking papers from Datalogy, Nvidia Nemotron, Ai2 (Ollmo, Tulu, ...) and the recent model from Aleph Alpha if you want to learn more.