Queries Google Gemini with a text prompt and a ROS image to locate 2D points in the image. More...

#include <get_points2d_from_gemini_query.hpp>

Inheritance diagram for moveit_pro::behaviors::GetPoints2DFromGeminiQuery:

[legend]

Collaboration diagram for moveit_pro::behaviors::GetPoints2DFromGeminiQuery:

[legend]

Public Member Functions
	GetPoints2DFromGeminiQuery (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources, gemini::GeminiPostFn http_post_fn=nullptr)

Public Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase
	AsyncBehaviorBase (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources)

	~AsyncBehaviorBase () override=default

BT::NodeStatus	onStart () override
	Required implementation of BT::StatefulActionNode::onStart().

BT::NodeStatus	onRunning () override
	Required implementation of BT::StatefulActionNode::onRunning().

void	onHalted () override
	Required implementation of BT::StatefulActionNode::onHalted().

void	resetStatus ()
	Resets the internal status of this node.

void	notifyCanHalt ()
	Called when runAsync() finishes to notify onHalted() that the async process has finished.

Public Member Functions inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode >
	SharedResourcesNode (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &behavior_context)
	Constructor for SharedResourcesNode. Called by BT::BehaviorTreeFactory when creating a new behavior tree containing this node.

const std::shared_ptr< BehaviorContext > &	getBehaviorContext () const

Static Public Member Functions
static BT::PortsList	providedPorts ()

static BT::KeyValueVector	metadata ()

Detailed Description

Queries Google Gemini with a text prompt and a ROS image to locate 2D points in the image.

Uses libcurl to POST a multimodal request containing both the text prompt and a JPEG-encoded image to the Gemini generative language API. The API key is read from the GOOGLE_GEMINI_API_KEY environment variable. Because the HTTP call can take several seconds, this behavior is asynchronous and will return RUNNING until the request completes.

The request constrains Gemini via generationConfig.responseMimeType + responseSchema to emit a JSON object with two fields: narrative (a human-readable answer) and points (a list of normalized 2D image coordinates with short labels). The narrative is placed on the response output port, the parsed points on the detected_points output port, and the labels associated with each point on the detected_labels output port (one label per point, same order). Labels are useful when prompts ask Gemini to tag each location with metadata such as an object ID, color, or class name.

Data Port Name	Port Type	Object Type
prompt	input	std::string
image	input	sensor_msgs::msg::Image
model_name	input	std::string (optional)
save_debug_image	input	std::string (optional)
response	output	std::string
detected_points	output	std::vector<geometry_msgs::msg::PointStamped>
detected_labels	output	std::vector<std::string>

Parameters

name	The name of the behavior node in the behavior tree.
config	The BehaviorTree.CPP node configuration.
shared_resources	Shared resources provided to the behavior at runtime.
http_post_fn	Optional HTTP-POST function used to talk to Gemini. Defaults to a libcurl implementation when null. Tests inject a fake to avoid real network traffic.

Constructor & Destructor Documentation

◆ GetPoints2DFromGeminiQuery()

moveit_pro::behaviors::GetPoints2DFromGeminiQuery::GetPoints2DFromGeminiQuery	(	const std::string &	name,
		const BT::NodeConfiguration &	config,
		const std::shared_ptr< BehaviorContext > &	shared_resources,
		gemini::GeminiPostFn	http_post_fn = `nullptr`
	)

Member Function Documentation

◆ metadata()

BT::KeyValueVector moveit_pro::behaviors::GetPoints2DFromGeminiQuery::metadata ( )

static

◆ providedPorts()

BT::PortsList moveit_pro::behaviors::GetPoints2DFromGeminiQuery::providedPorts ( )