Skip to main content
Version: 8

Visual Servoing

Many applications require aligning the arm end effector with respect to some (possibly moving) visual reference. Visual Servoing is a technique commonly used in these cases.

This how-to guide demonstrates an advanced Visual Servoing example that uses the PoseJog Behavior to track and intercept a moving satellite in space.

Setup​

For this example, we use the space_satellite_sim configuration which demonstrates visual servoing with a moving target. To launch this configuration, run:

moveit_pro run -c space_satellite_sim

Example: Grapple Moving Satellite with State Estimation​

The Grapple Moving Satellite with Fuse Objective demonstrates visual servoing by tracking and intercepting a moving satellite using state estimation from AprilTag detection.

This Objective shows how to:

  1. Track a moving target using odometry information
  2. Compute velocities to align with the target in real-time
  3. Use PoseJog for safe, smooth Cartesian motion control
  4. Maintain continuous visual servoing until alignment is achieved

Key Behaviors​

The main Behaviors involved in this visual servoing approach are:

  • ComputeVelocityToAlignWithTarget: Computes the Cartesian velocities needed to align the end effector with a target pose. It takes into account the target's motion state and applies proportional gains to generate smooth control commands.

  • PublishVelocityForceCommand: Publishes velocity and force commands to the robot's velocity-force controller. This Behavior bridges the computed velocities to the robot's control interface.

  • PoseJog: Provides safe Cartesian teleoperation with advanced features including collision avoidance, singularity handling, and workspace boundary management. Unlike the deprecated ServoTowardsPose, PoseJog offers improved safety and control fidelity.

Running the Example​

  1. Edit the space_satellite_sim config.yaml mujoco_viewer param to launch Pro with the MuJoCo viewer.
  2. Launch the space_satellite_sim configuration as shown in the Setup section
  3. In the MuJoCo viewer, load Key 1 (by clicking on Key 0 and dragging to the right) to initiate satellite motion or reset its position
  4. In the MoveIt Pro web interface, run the "Grapple Moving Satellite with Fuse" Objective
  5. Observe how the robot tracks and intercepts the moving satellite using visual servoing

The Objective will:

  1. Open the gripper
  2. Switch to velocity-force control
  3. Continuously track the satellite's position using odometry
  4. Compute alignment velocities and execute them through PoseJog
  5. Close the gripper once alignment is achieved

Space satellite grappling example

For more details on configuring PoseJog for your own applications, see the Configure Pose Jog guide.