SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems

ICRA 2024

Joonhyung Lee¹, Sangbeom Park¹, Jeongeun Park¹, Kyungjae Lee², Sungjoon Choi¹

¹Korea University, ²Chungang University

[Project Page] [Paper] [Code] [Video]

📃 Abstract

Pick-and-place is one of the fundamental tasks in robotics research. However, the attention has been mostly focused on the “pick” task, leaving the “place” task relatively unexplored. In this paper, we address the problem of placing objects in the context of a teleoperation framework. Particularly, we focus on two aspects of the place task: stability robustness and contextual reasonableness of object placements. Our proposed method combines simulation-driven physical stability verification via real-to-sim and the semantic reasoning capability of large language models. In other words, given place context information (e.g., user preferences, object to place, and current scene information), our proposed method outputs a probability distribution over the possible placement candidates, considering the robustness and reasonableness of the place task. Our proposed method is extensively evaluated in two simulation and one real world environments and we show that our method can greatly increase the physical plausibility of the placement as well as contextual soundness while considering user preferences.

News

We are happy to announce that SPOTS has been accepted to ICRA2024! 😆🎉🎉 Codes will be uploaded soon.

Overview

Official Implementation of the paper SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems

Simulation

Dish Rack Scene

Bookshelf Scene

Category Scene

Code Explanation

⚠️ Note: Before starting, you need to add your OpenAI API key in /key/my_key.txt. This is required for the receptacle reasoning process.

We have 3 Scenarios in this repository. Each scene consists of MJCF (MuJoCo XML) files for environment configuration and Jupyter notebooks for execution.

A) Dish Rack Scene

The task is To place the dish into a dish-rack. There are three types of dish-racks available.

Environment Files (MJCF):

• scene_kitchen_dish_rack_black.xml
• scene_kitchen_dish_rack_white.xml
• scene_kitchen_dish_rack_small.xml

Execution Files (Jupyter Notebooks): You can find the execution code here:

• Black-dish-rack.ipynb
• White-dish-rack.ipynb
• Small-dish-rack.ipynb

B) Bookshelf Scene

The task is To place the book into a bookshelf.

Environment File (MJCF):

• scene_office_bookshelf.xml
• scene_office_bookshelf_genre.xml

Execution File (Jupyter Notebook): You can find the execution code here:

• office_bookshelf.ipynb
• office_bookshelf_genre.ipynb

C) Category Scene

The task is To place the object into a shelf.

Environment File (MJCF):

• scene_category.xml

Execution File (Jupyter Notebook): You can find the execution code here:

• cateogry.ipynb

Utility Code

• mujoco_parser.py: Contains all functions related to the MuJoCo engine
• util.py: Contains helper functions and utilities