Robot Showcase Project

In our lab, we have a robot demonstration team who implements SOTA robot learning technologies on our robots. Here are some projects from the team.

Learning Parkour with Simulators (2024)

Demo Developers: Hikariu Wada, Momo Hanawa, Koki Fukuda, Yoshihiro Noumi

In the high-performance simulation environment “Isaac Gym,” reinforcement learning was applied to a quadruped robot, enabling it to acquire complex parkour-like movements. The advancements in simulation technology have made it possible to learn intricate behaviors in a short time and develop models that can adapt to real-world environments. This approach allows for the rapid collection of large amounts of training data, facilitating the creation of models capable of performing effectively in real-world scenarios.

REFERENCES: Cheng, Xuxin, et al. “Extreme parkour with legged robots.” 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

Robot Foundation Model Demo (2024)

Demo Developers: Yuya Ikeda, Aoi Horo, Hiroki Ishii, Yoshihiro Noumi

Using the robot foundation model “Octo,” we demonstrated multiple robots performing diverse tasks with a single model. Data combining camera images, joint angles, and instruction texts was collected and used for additional training of Octo. It was confirmed that the model could generalize and execute multiple tasks on xArm and Sawyer robots.

REFERENCES: Octo Model Team, Ghosh, D., et al. “Octo: An open-source generalist robot policy.” Proceedings of Robotics: Science and Systems, 2024.

Approach to GPSR Tasks and Their Solutions (2024)

Demo Developers: Aoi Horo, Hikaru Wada, Koki Fukuda, Yoshihiro Noumi

We used several foundation model technologies such as a large language model (GPT-4), a speech recognition model (Whisper), an object detection model (Detic), and a multimodal foundation model (CLIP). By integrating various foundation models and implementing them in a robot, it can comprehensively recognize the real world and generate appropriate actions based on its abilities in response to commands.

REFERENCES: Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery

Flexible Object Manipulation Demonstration Using a World Model (2022)

Demo Developers: Yuya Ikeda, Takuya Okubo, Koki Fukuda, Yoshihiro Noumi

The state estimation and prediction capabilities of “world models,” which acquire the structure of the external environment based on observational data, are expected to have promising applications in robotics. By reproducing the implementation of two research papers related to world models, we validated the potential of applying world models to robotics through a real-world robot arm task of spreading a cloth.

REFERENCES:

• Lin, Xingyu, Yufei Wang, Zixuan Huang, and David Held. “Learning Visible Connectivity Dynamics for Cloth Smoothing.” Conference on Robot Learning (CoRL), 2021.
• Yan, Wilson, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. “Learning Predictive Representations for Deformable Objects Using Contrastive Estimation.” Proceedings of Machine Learning Research (PMLR), 2021.