Posted by mplappert 7 days ago
A few decisions I'm least settled on, and would love some pushback/feedback on:
- single arm vs. bimanual (I went single for cost/space, knowing it rules out things like folding cloth)
- not calibrating camera extrinsics/intrinsics for now
- RGB vs. RGB-D for from-scratch policies (ACT / Diffusion Policy)
And one I'm more confident about but expect disagreement on: not building on ROS 2 / LeRobot, and writing my own stack instead. Happy to get into the reasoning.
I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)
- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?
- Any issues you've been running into with this setup?
- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?
Re your questions:
- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.
- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.
- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that
That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.
Time is one of the hard problems in robots, because they are inevitably but non-obviously distributed systems.
Robots are annoyingly, wonderfully difficult.
It would be interesting to explore how RL can be applied on top of my (flawed) human demos to optimize beyond what I’m able to do.
Tell me more! I am slightly biased in that direction. But can’t fully justify it at this point.
I am not an official supporter of the library but am asking out of curiosity.
On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.
At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.
The app did a decent job at surfacing problematic comments that a mod can do something about.
It was cool to optimize llama-cpp arguments for throughput. During the slightly off-peak hours the post processing was pretty much real time. I suspect a second 3090 would’ve be enough for peak posting hours too.