Researchers working on large artificial intelligence models like ChatGPT have vast swaths of internet text, photos and videos to train systems. But roboticists training physical machines face barriers: Robot data is expensive, and because there aren鈥檛 fleets of robots roaming the world at large, there simply isn鈥檛 enough data easily available to make them perform well in dynamic environments, .
Some researchers have turned to simulations to train robots. Yet even that process, which often involves a graphic designer or engineer, is laborious and costly.
Two new studies from 91探花 researchers introduce AI systems that use either video or photos to create simulations that can train robots to function in real settings. This could significantly lower the costs of training robots to function in complex settings.
In the first study, a user quickly scans a space with a smartphone to record its geometry. The system, called RialTo, can then create a 鈥渄igital twin鈥� simulation of the space, where the user can enter how different things function (opening a drawer, for instance). A robot can then virtually repeat motions in the simulation with slight variations to learn to do them effectively. In the second study, the team built a system called URDFormer, which takes images of real environments from the internet and quickly creates physically realistic simulation environments where robots can train.
The teams presented their studies 鈥� the and the 鈥斅燼t the Robotics Science and Systems conference in Delft, Netherlands.
鈥淲e鈥檙e trying to enable systems that cheaply go from the real world to simulation,鈥� said , a 91探花assistant professor in the Paul G. Allen School of Computer Science & Engineering and co-senior author on both papers. 鈥淭he systems can then train robots in those simulation scenes, so the robot can function more effectively in a physical space. That鈥檚 useful for safety 鈥� you can鈥檛 have poorly trained robots breaking things and hurting people 鈥� and it potentially widens access. If you can get a robot to work in your house just by scanning it with your phone, that democratizes the technology.鈥�
While many robots are currently well suited to working in environments like assembly lines, teaching them to interact with people and in less structured environments remains a challenge.
鈥淚n a factory, for example, there’s a ton of repetition,鈥� said lead author of the URDFormer study , a 91探花doctoral student in the Allen School. 鈥淭he tasks might be hard to do, but once you program a robot, it can keep doing the task over and over and over. Whereas homes are unique and constantly changing. There鈥檚 a diversity of objects, of tasks, of floorplans and of people moving through them. This is where AI becomes really useful to roboticists.鈥�
The two systems approach these challenges in different ways.
RialTo 鈥� which Gupta created with a team at the Massachusetts Institute of Technology 鈥� has someone pass through an environment and take video of its geometry and moving parts. For instance, in a kitchen, they鈥檒l open cabinets and the toaster and the fridge. The system then uses existing AI models 鈥� and a human does some quick work through a graphic user interface to show how things move 鈥� to create a simulated version of the kitchen shown in the video. A virtual robot trains itself through trial and error in the simulated environment by repeatedly attempting tasks such as opening that toaster oven 鈥� a method called .
By going through this process in the simulation, the robot improves at that task and works around disturbances or changes in the environment, such as a mug placed beside the toaster. The robot can then transfer that learning to the physical environment, where it鈥檚 nearly as accurate as a robot trained in the real kitchen.
The other system, URDFormer, is focused less on relatively high accuracy in a single kitchen; instead, it quickly and cheaply conjures hundreds of generic kitchen simulations. URDFormer scans images from the internet and pairs them with existing models of how, for instance, those kitchen drawers and cabinets will likely move. It then predicts a simulation from the initial real-world image, allowing researchers to quickly and inexpensively train robots in a huge range of environments. The trade-off is that these simulations are significantly less accurate than those that RialTo generates.
For more information, read or聽visit the teams鈥� websites:
鈥淭he two approaches can complement each other,鈥� Gupta said. 鈥淯RDFormer is really useful for pre-training on hundreds of scenarios. RialTo is particularly useful if you’ve already pre-trained a robot, and now you want to deploy it in someone鈥檚 home and have it be maybe 95% successful.鈥�
Moving forward, the RialTo team wants to deploy its system in peoples鈥� homes (it鈥檚 largely been tested in a lab), and Gupta said he wants to incorporate small amounts of real-world training data with the systems to improve their success rates.
鈥淗opefully, just a tiny amount of real-world data can fix the failures,鈥� Gupta said. 鈥淏ut we still have to figure out how best to combine data collected directly in the real world, which is expensive, with data collected in simulations, which is cheap, but slightly wrong.鈥�
On the URDFormer paper additional co-authors include the UW鈥檚 , , Alex Fang 鈥� all聽doctoral students in the Allen School; , an undergraduate in the Allen School; , a masters student in the Allen School; and , a research scientist at NVIDIA. , a professor in the Allen School, was a co-senior author. On the URDFormer paper additional co-authors include MIT鈥檚 , , 鈥� all doctoral students; , a research assistant; and , an undergraduate. , an assistant professor at MIT, was a co-senior author. The URDFormer research was partially funded by Amazon Science Hub. The RialTo research was partially funded by the Sony Research Award, the U.S. Government and Hyundai Motor Company.
For more information, contact Gupta at abhgupta@cs.washington.edu and Chen at qiuyuc@uw.edu.