feat: improving manipulation demo

demos.repos

@@ -6,4 +6,4 @@ repositories:
   src/examples/rai-manipulation-demo:
     type: git
     url: https://github.com/RobotecAI/rai-manipulation-demo.git
-    version: development
+    version: kd/vacuum_gripper

docs/demos/manipulation.md

@@ -26,7 +26,7 @@ manipulation techniques.
 2. Download additional dependencies:
 
     ```shell
-    poetry install --with perception
+    poetry install --with perception,simbench
     vcs import < demos.repos
     rosdep install --from-paths src/examples/rai-manipulation-demo/ros2_ws/src --ignore-src -r -y
     ```
@@ -50,29 +50,35 @@ manipulation techniques.
     Ensure that every command is run in a sourced shell using `source setup_shell.sh`
     Ensure ROS 2 is sourced.
 
-1. Start the demo
+1. Run the Demo:
 
     ```shell
-    ros2 launch examples/manipulation-demo.launch.py game_launcher:=demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
+    streamlit run examples/manipulation-demo-streamlit.py
     ```
 
-2. In the second terminal, run the streamlit interface:
+    Alternatively, you can run the simpler command-line version, which also serves as an example of
+    how to use the RAI API for you own applications:
+
+    1. Run Simulation
 
     ```shell
-    streamlit run examples/manipulation-demo-streamlit.py
+    ros2 launch examples/manipulation-demo.launch.py game_launcher:=demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
     ```
 
-    Alternatively, you can run the simpler command-line version, which also serves as an example of
-    how to use the RAI API for you own applications:
+    2. Run cmd app
 
     ```shell
     python examples/manipulation-demo.py
     ```
 
-3. Interact with the robot arm using natural language commands. For example:
+2. Interact with the robot arm using natural language commands. For example:
 
     ```
-    Enter a prompt: Pick up the red cube and drop it on another cube
+    "Place every apple on top of the cube"
+    "Stack all cubes into a tower"
+    "Arrange objects in a line"
+    "Put two boxes closer to each other. Move only one box."
+    "Move cubes to the left side of the table"
     ```
 
 !!! tip "Changing camera view"

examples/__init__.py

@@ -0,0 +1,13 @@
+# Copyright (C) 2024 Robotec.AI
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

examples/embodiments/manipulation_embodiment.json

@@ -1,14 +1,22 @@
 {
     "rules": [
-        "Always follow the specified payload limit; do not exceed 3 kg."
+        "Always follow the specified payload limit; do not exceed 3 kg.",
+        "Before starting the task, by looking at the image, determine if the task can be completed. If yes - proceed without asking. Otherwise report.",
+        "Cubes are 5cm tall. When stacking, add 5cm to the z coordinate for each additional cube.",
+        "Remember to not put objects on top of other objects, unless specifically instructed otherwise",
+        "Try to consider the typical size of objects when placing them (width, height, length)",
+        "Always leave at least a 5cm margin (x and y) between objects unless specifically instructed otherwise",
+        "If you repeat the task, make sure to run detection again to check object positions",
+        "detection tool can't recognize colors, so just ask for categories",
+        "reject tasks that require rotating the gripper"
     ],
     "capabilities": [
         "7 degrees of freedom for complex motion and task execution.",
         "Ability to carry up to 3 kg of payload.",
         "High precision with 0.1 mm repeatability.",
         "850 mm reach suitable for a variety of industrial applications.",
         "Interfaces to detect and manipulate objects.",
-        "Coordinate system: x - front to back (positive is forward), y - left to right (positive is right), z - up to down (positive is up)."
+        "Coordinate system: x-axis (front to back, positive is forward, reachable range [0.3, 0.5] m), y-axis (left to right, positive is right, reachable range [-0.5, 0.5] m), z-axis (vertical, positive is up)."
     ],
     "behaviors": [
         "Performing precise dispensing tasks.",