Segment an image using SAM3 with multimodal prompts. More...

#include <get_masks_2d_from_exemplar.hpp>

Inheritance diagram for moveit_pro::behaviors::GetMasks2DFromExemplar:

[legend]

Collaboration diagram for moveit_pro::behaviors::GetMasks2DFromExemplar:

[legend]

Classes
struct	InferenceHandle
	Interface for SAM3 segmentation inference. More...

struct	SAM3DetectHandle
	Default implementation using SAM3Detect. More...

Public Member Functions
	GetMasks2DFromExemplar (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< moveit_pro::behaviors::BehaviorContext > &shared_resources, std::unique_ptr< InferenceHandle > inference_handle=std::make_unique< SAM3DetectHandle >())
	Constructor for SAM3 multimodal segmentation behavior.

	~GetMasks2DFromExemplar () override

Public Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase
	AsyncBehaviorBase (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources)

	~AsyncBehaviorBase () override=default

BT::NodeStatus	onStart () override
	Required implementation of BT::StatefulActionNode::onStart().

BT::NodeStatus	onRunning () override
	Required implementation of BT::StatefulActionNode::onRunning().

void	onHalted () override
	Required implementation of BT::StatefulActionNode::onHalted().

void	resetStatus ()
	Resets the internal status of this node.

Public Member Functions inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode >
	SharedResourcesNode (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources)
	Constructor for SharedResourcesNode. Called by BT::BehaviorTreeFactory when creating a new behavior tree containing this node.

Static Public Member Functions
static BT::PortsList	providedPorts ()
	Provides port definitions for BehaviorTree.

static BT::KeyValueVector	metadata ()
	Provides behavior metadata for UI.

Protected Member Functions
tl::expected< bool, std::string >	doWork () override
	Execute multimodal segmentation.

tl::expected< void, std::string >	doHalt () override
	Cancel an in-flight SAM3 inference via the inference handle.

Protected Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase
void	notifyCanHalt ()
	Called when runAsync() finishes to notify onHalted() that the async process has finished.

Additional Inherited Members
Protected Attributes inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode >
std::shared_ptr< BehaviorContext >	shared_resources_

Detailed Description

Segment an image using SAM3 with multimodal prompts.

This behavior segments an image using text, an optional exemplar image, and/or bounding box prompts. Supports any combination of prompt types for flexible segmentation.

Data Port Name	Port Type	Object Type
target_image	input	sensor_msgs::msg::Image
text_prompt	input	std::string
exemplar_image	input	sensor_msgs::msg::Image (optional)
target_bboxes	input	std::vector<vision_msgs::msg::BoundingBox2D>
exemplar_bboxes	input	std::vector<vision_msgs::msg::BoundingBox2D>
encoder_model_path	input	std::string
text_encoder_model_path	input	std::string
geometry_encoder_model_path	input	std::string
decoder_model_path	input	std::string
model_package	input	std::string
confidence_threshold	input	double (optional, default 0.0)
masks2d	output	std::vector<moveit_studio_vision_msgs::msg::Mask2D>
mask_count	output	int
confidence_scores	output	std::vector<double>
confidence_scores_str	output	std::vector<std::string>

Constructor & Destructor Documentation

◆ GetMasks2DFromExemplar()

moveit_pro::behaviors::GetMasks2DFromExemplar::GetMasks2DFromExemplar	(	const std::string &	name,
		const BT::NodeConfiguration &	config,
		const std::shared_ptr< moveit_pro::behaviors::BehaviorContext > &	shared_resources,
		std::unique_ptr< InferenceHandle >	inference_handle = `std::make_unique<SAM3DetectHandle>()`
	)

Constructor for SAM3 multimodal segmentation behavior.

Parameters

name	The name of a particular instance of this Behavior. This will be set by the behavior tree factory when this Behavior is created within a new behavior tree.
config	This contains runtime configuration info for this Behavior, such as the mapping between the Behavior's data ports on the behavior tree's blackboard and the Behavior's input/output ports.
shared_resources	Provides access to common resources such as the node handle, failure logger, and TF buffer that are shared between all behaviors inheriting from moveit_pro::behaviors::SharedResourcesNode.
inference_handle	Custom inference implementation (default uses SAM3DetectHandle)

◆ ~GetMasks2DFromExemplar()

moveit_pro::behaviors::GetMasks2DFromExemplar::~GetMasks2DFromExemplar ( )

overridedefault

Member Function Documentation

◆ doHalt()

tl::expected< void, std::string > moveit_pro::behaviors::GetMasks2DFromExemplar::doHalt ( )

overrideprotectedvirtual

Cancel an in-flight SAM3 inference via the inference handle.

Forwards to InferenceHandle::requestTerminate(), which aborts the current ONNX Session::Run() at the next per-operator boundary. doWork() is responsible for calling clearTerminate() and notifyCanHalt() before invoking predict() so this halt is observable.

Reimplemented from moveit_pro::behaviors::AsyncBehaviorBase.

◆ doWork()

tl::expected< bool, std::string > moveit_pro::behaviors::GetMasks2DFromExemplar::doWork ( )

overrideprotectedvirtual

Execute multimodal segmentation.

Returns: Success or error message

Implements moveit_pro::behaviors::AsyncBehaviorBase.

◆ metadata()

BT::KeyValueVector moveit_pro::behaviors::GetMasks2DFromExemplar::metadata ( )

static

Provides behavior metadata for UI.

Returns: Metadata key-value pairs

◆ providedPorts()

BT::PortsList moveit_pro::behaviors::GetMasks2DFromExemplar::providedPorts ( )

static

Provides port definitions for BehaviorTree.

Returns: List of input and output ports

The documentation for this struct was generated from the following files:

Classes

Public Member Functions

Static Public Member Functions

Protected Member Functions

Additional Inherited Members