Pick April Tag Labeled Object

warning

We are currently migrating this example to our new simulator. This tutorial's Gazebo robot configuration and example objectives may not be fully functional, but still provide a useful starting place.

AprilTags are a type of fiducial marker widely used in applications of robotics.

They enable the use of a single RGB camera to detect 3D poses by using high-contrast, easy-to-detect, markers of known size and unique identifiers.

In this guide, you will learn how to use MoveIt Pro to detect AprilTags and use the resulting poses for a simple pick and place example.

Setup

For this example, we will use a Universal Robots UR5e simulated in Gazebo. To launch this configuration, run:

moveit_pro run -c picknik_ur_gazebo_config

From the default starting position of the robot arm, you should see the following AprilTags from the wrist-mounted camera feed.

Teaching Grasp Offsets to AprilTags

MoveIt Pro includes Behaviors and Objectives to find and calculate the pose of AprilTags, and also teach grasp poses relative to them. Before teaching a grasp offset it is recommended you run Take Snapshot so that it is easier to see where the grasp pose is located in the scene. To teach a grasp offset locate the Teach Grasp Offset From Apriltag Objective from the Objectives tab by expanding the All Objectives button in the top right of the UI. Running the Objective will prompt you to click on the object you would like to teach. An interactive marker of the robot's end effector will appear in the scene, which you can position as desired. By default the Teach Grasp Offset From Apriltag Objective is set to teach AprilTag ID 1 which is the left tag just below the red cube.

To explore and edit how this Objective works click on the Objective Builder tab and locate the Teach Grasp Offset From Apriltag Objective.

To estimate the AprilTag pose this Objective uses the Sample April Tag Subtree which takes in the tag ID we are searching for and several other parameters to tune the estimation and averaging algorithms. We sample the pose of the AprilTag multiple times because the pose estimation is not perfect and has noise that we would like to average out, making the pose estimation more consistent. If you are experiencing inconsistencies in the pose of the tags or taught grasps please see the Tips and tricks to using AprilTags section below. After the pose of the AprilTag has been averaged it is then transformed to the world frame and the offset between the interactive marker and the tag is calculated. The last step uses the Write Pose To YAML Behavior where we can specify the name of the yaml file we would like to save so that the user can teach different offsets for each tag or even apply it to a range of tags.

Picking With AprilTag Offsets Example

From here, you can run the Pick April Tag Labeled Object Objective that uses the pose the previous Objective wrote to a yaml file.

note

If you know the offset that you would like to apply to an AprilTag you can always write your own yaml file containing the offset and use it with the TransformPoseFromYaml Behavior.

Tips and tricks to using AprilTags

To experiment with the AprilTag detector behavior and visually inspect the accuracy, you can run the Loop Detect April Tag Objective. Then, open a separate terminal to launch RViz:

moveit_pro rviz

1. While the Objective is running you can edit the apriltag_detection_config.yaml file in the ${HOME}/.config/moveit_pro/picknik_ur_gazebo_config/objectives folder on your system and see the effect live.
2. This configuration sets the AprilTags to be from the 36h11 family and have a size 0.08 (8 cm which is the size of the tags in the Gazebo simulation).
3. You can create your own YAML parameter file by specifying a different file name and adding it to the ${HOME}/.config/moveit_pro/picknik_ur_gazebo_config/objectives folder on your system.

As shown below, the pose estimation returned from the AprilTag detector can have significant error or noise when observed at larger distances from the camera. For example here is a 5 cm tag being rotated at a distance of approximately 0.3 m from the RGB camera.

If you are experiencing noisy measurements, MoveIt Pro includes a Subtree (Sample April Tag) that can sample the pose a specified number of times and return the average. Averaging multiple estimates can help increase the repeatability of the observed pose. If you are experiencing large offsets between the pose and depth cloud produced by the 3D camera you should ensure you have the correct settings in your AprilTag configuration yaml (see this guide guide for tuning the detector parameters). You can also try to increase the camera resolution of the RGB image or increase the AprilTag size to reduce the error. For example you can see here a more accurate pose estimation when the tag is 10 cm square and at the same distance of around 0.3 m.

If you need to create or test tags of different sizes/encodings, refer to this repository for a set of pre-generated tags.