Segment an image using SAM3 with multimodal prompts. More...

#include <get_masks_2d_from_exemplar.hpp>

Inheritance diagram for moveit_pro::behaviors::GetMasks2DFromExemplar:

[legend]

Collaboration diagram for moveit_pro::behaviors::GetMasks2DFromExemplar:

[legend]

Classes
struct	InferenceHandle
	Interface for SAM3 segmentation inference. More...

struct	SAM3DetectHandle
	Default implementation using SAM3Detect. More...

Public Member Functions
	GetMasks2DFromExemplar (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< moveit_pro::behaviors::BehaviorContext > &shared_resources, std::unique_ptr< InferenceHandle > inference_handle=std::make_unique< SAM3DetectHandle >())
	Constructor for SAM3 multimodal segmentation behavior.

	~GetMasks2DFromExemplar () override

Public Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase
	AsyncBehaviorBase (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources)

	~AsyncBehaviorBase () override=default

BT::NodeStatus	onStart () override
	Required implementation of BT::StatefulActionNode::onStart().

BT::NodeStatus	onRunning () override
	Required implementation of BT::StatefulActionNode::onRunning().

void	onHalted () override
	Required implementation of BT::StatefulActionNode::onHalted().

void	resetStatus ()
	Resets the internal status of this node.

Public Member Functions inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode >
	SharedResourcesNode (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources)
	Constructor for SharedResourcesNode. Called by BT::BehaviorTreeFactory when creating a new behavior tree containing this node.

Static Public Member Functions
static BT::PortsList	providedPorts ()
	Provides port definitions for BehaviorTree.

static BT::KeyValueVector	metadata ()
	Provides behavior metadata for UI.

Protected Member Functions
tl::expected< bool, std::string >	doWork () override
	Execute multimodal segmentation.

Protected Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase
virtual tl::expected< void, std::string >	doHalt ()
	Optionally implement additional work needed to cleanly interrupt the async process.

void	notifyCanHalt ()
	Called when runAsync() finishes to notify onHalted() that the async process has finished.

Additional Inherited Members
Protected Attributes inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode >
std::shared_ptr< BehaviorContext >	shared_resources_

Detailed Description

Segment an image using SAM3 with multimodal prompts.

This behavior segments an image using text, an optional exemplar image, and/or bounding box prompts. Supports any combination of prompt types for flexible segmentation.

Data Port Name	Port Type	Object Type
target_image	input	sensor_msgs::msg::Image
text_prompt	input	std::string
exemplar_image	input	sensor_msgs::msg::Image (optional)
target_bboxes	input	std::vector<vision_msgs::msg::BoundingBox2D>
exemplar_bboxes	input	std::vector<vision_msgs::msg::BoundingBox2D>
encoder_model_path	input	std::string
text_encoder_model_path	input	std::string
geometry_encoder_model_path	input	std::string
decoder_model_path	input	std::string
model_package	input	std::string
confidence_threshold	input	double (optional, default 0.0)
masks2d	output	std::vector<moveit_studio_vision_msgs::msg::Mask2D>
mask_count	output	int
confidence_scores	output	std::vector<double>
confidence_scores_str	output	std::vector<std::string>

Constructor & Destructor Documentation

◆ GetMasks2DFromExemplar()

moveit_pro::behaviors::GetMasks2DFromExemplar::GetMasks2DFromExemplar	(	const std::string &	name,
		const BT::NodeConfiguration &	config,
		const std::shared_ptr< moveit_pro::behaviors::BehaviorContext > &	shared_resources,
		std::unique_ptr< InferenceHandle >	inference_handle = `std::make_unique<SAM3DetectHandle>()`
	)

Constructor for SAM3 multimodal segmentation behavior.

Parameters

name	The name of a particular instance of this Behavior. This will be set by the behavior tree factory when this Behavior is created within a new behavior tree.
config	This contains runtime configuration info for this Behavior, such as the mapping between the Behavior's data ports on the behavior tree's blackboard and the Behavior's input/output ports.
shared_resources	Provides access to common resources such as the node handle, failure logger, and TF buffer that are shared between all behaviors inheriting from moveit_pro::behaviors::SharedResourcesNode.
inference_handle	Custom inference implementation (default uses SAM3DetectHandle)

◆ ~GetMasks2DFromExemplar()

moveit_pro::behaviors::GetMasks2DFromExemplar::~GetMasks2DFromExemplar ( )

overridedefault

Member Function Documentation

◆ doWork()

tl::expected< bool, std::string > moveit_pro::behaviors::GetMasks2DFromExemplar::doWork ( )

overrideprotectedvirtual

Execute multimodal segmentation.

Returns: Success or error message

Implements moveit_pro::behaviors::AsyncBehaviorBase.

◆ metadata()

BT::KeyValueVector moveit_pro::behaviors::GetMasks2DFromExemplar::metadata ( )

static

Provides behavior metadata for UI.

Returns: Metadata key-value pairs

◆ providedPorts()

BT::PortsList moveit_pro::behaviors::GetMasks2DFromExemplar::providedPorts ( )

static

Provides port definitions for BehaviorTree.

Returns: List of input and output ports

The documentation for this struct was generated from the following files:

Classes

Public Member Functions

Static Public Member Functions

Protected Member Functions

Additional Inherited Members