here’s a shovel

They're not claiming controlling robot arms with language models is a sensible or efficient thing to do, they're trying it to see what happens.

Just like how you needed to train machine learning models for specific tasks?

Actually, you don't really need a dedicated model as long as you have the proper adapter like the projector model for vision inputs you just need another for robotic outputs, after that it is just a matter of having the training data.