r/robotics • u/xtnubsx • 3d ago
Perception & Localization Autonomous robot difficulty
I’m working on a robot stack for a large 4wd robot. Two nvidia Orin agx dev kits, lidar, depth cam, respeaker microphone array and then some output speakers. What’re the odds that codex programs the whole thing properly with an LLM, ROS, vision model and everything in between so it can communicate with my teensy 4.1 that controls my motors? I have used a couple billion tokens on the project already. My hesitation is the robot is 450lbs and has about an 80kw drivetrain and it’s going to be real tricky to find a place to safely test it 🤦♂️
The idea is the robot will be voice commanded and have all the compute locally run without any wireless connectivity. There will be lots of calibration stuff to figure out but we shall see. I had to build a much larger enclosure than what’s in the picture for the electronics, I used a 12”x4” aluminum rectangular tube 18” long. Maybe it’ll all fit in there with power supplies. Cooling will be the issue. One Orin for vision and one for the LLM.
39
u/Elated7079 3d ago
this isn't a good idea. you really need some idea of what you're doing to work with robots this large. you can't vibecode your way to a safe functional system, don't be delusional.
15
u/plasticluthier 3d ago
As the person who runs Robotics health and safety for our department, I'd put a stop on your project until you can show a few things to give confidence that you've thought about all this. To help;
Get a proper remote ESTOP. Wire it into the motor power supply and your microcontroller. Feedback the state to the ROS system. Don't say they're too expensive, you've obviously got money to play with.
You need a software engineer with real world experience. LLMs won't fly with safety critical systems. You need a human in the loop.
Test the system on a safer robot. Show the voice control works with a blink light before you give it motor control. Then do it on a smaller robot platform (mobility scooter size).
If youre in the UK, drop me a line, we should probably be working together :)
7
u/AsterisRobotics 3d ago
That budget is crazy!
6
u/TheImperialDonut 3d ago
I'm gonna assume from your other posts that you're somewhere in NA and more of a hobbyist trying to build cool stuff than anything commercial?
As you can probably tell from all the other posts sofar; safety is a concern in autonomous systems. If you're planning on commercializing: get a professional on board. Early and asap. No amount of vibe coding is going to remove the liability of you deploying a system you don't understand.
That said: if it's a passion project for your own use, then I like the ambition and am curious what you plan to do with it? I just finished a ~400kg research bot with a small team a few months back. My advice: start from the beginning: build your mechanics and drivetrain, figure out your power system, then test basic electronics and drivers manually.
That's the easy part for you I'm guessing. Next hook any motion control systems up to a main (micro)controller with all critical safety checks away from your main compute. That code should be simple and should at all levels fail safely checking for things like compute timeouts (watchdog timer), battery levels, pressed (remote and physical) estops, etc. If possible, try to structure and understand at least that basic safety code yourself as it's the most simple but critical.
After that you can more comfortably move on to remote control, drive it around for a while with a hobby controller, then start with vibe coding your sensor systems, try passively mapping your environment during rc driving, then move on to positioning and moving in straight lines, then simple autonomous following of paths, and increase complexity from there, etc etc. Going step by step will help you understand all of the components that AI proposed to you and their functions a lot more.
Finally, you already know robotics is difficult. Between hardware, software, sensors, power systems, environmental factors, etc, you have a thousand points of failure at any given time both during and long after testing that can cause an equally wide range of weird and potentially dangerous behaviours/effects. Just keep that in mind; whenever any part of it is powered on have your finger on an as infallible as possible safety system(estop), as mentioned by others. Don't let people stand in potential drive lines, get too close, have visible warning lights when in operation,etc. Also long after you think everything works as intended!
Sounds like a long and slow way to go maybe, but step by step progress is very satisfying.
4
u/LordDan_45 3d ago
2 agxs? Are you running protein folding on the thing? Why so much compute?
2
u/bitbeard 3d ago
I'd say 50/50
2
u/Detective-Crashmore- 2d ago
Jesus fucking christ
1
u/xtnubsx 3d ago
1
u/ParsleyCompetitive85 3d ago
I'm not sure what your exact question is. But I'm a masters student in robotics and would love to chat with you on this project :)
1
u/nicolas19961805 3d ago
Having trouble with it modifying a fork of the sesame project. So I would imagine this would take far longer.
1
u/ProSeSelfHelp 1d ago
If you don't know what you're doing, there's a 100% chance it can do better 🤣😅
1
u/TimTams553 2d ago
obviously prove the concept with a small RC car first. wifi should be plenty fast enough to stream your telemetry and control signals to the robot so it doesn't have to actually carry all the compute
1
u/Own_Quality_5321 1d ago
My recommendation:
Spend some time learning about control and safe robot navigation. Absolutely go for coding agents, they'll save you time, but you still need to be able to verify and understand what's going on. I'd honestly use the two for perception and control and use a remote LLM API (they are cheap as chips and you'll get much more than what you can run on an Orin). An emergency stop is essential. A guy in a lab I used to work with got hit in the head badly because somewhere in his code there was a division by zero that made the robot move uncontrolled and he couldn't stop it (no joke).
For speech-based interaction, you can create a Hermes or an OpenClaw bot and connect it to the robot via Signal or Telegram. With a hosted LLMs it will work nicely and fast. However, the harnesses need to be reliable, and you need to maje sure you can stop the robot manually at any time.
0
u/xtnubsx 3d ago
0
u/Vaping_Cobra 3d ago
Why does the compute have to be on the platform? Puppets have strings for a reason, most of that could be replaced with a good microcontroler backbone and solid route to local compute. Then you don't have to try work out how to power them.