 |
MoveIt Pro API
Core Behaviors for MoveIt Pro
|
|
MoveIt Pro API
Core Behaviors for MoveIt Pro
|
Queries Google Gemini with a text prompt and a ROS image to locate 2D points in the image. More...
#include <get_points2d_from_gemini_query.hpp>
Public Member Functions | |
| GetPoints2DFromGeminiQuery (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources, gemini::GeminiPostFn http_post_fn=nullptr) | |
 Public Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase | |
| AsyncBehaviorBase (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources) | |
| ~AsyncBehaviorBase () override=default | |
| BT::NodeStatus | onStart () override |
| Required implementation of BT::StatefulActionNode::onStart(). | |
| BT::NodeStatus | onRunning () override |
| Required implementation of BT::StatefulActionNode::onRunning(). | |
| void | onHalted () override |
| Required implementation of BT::StatefulActionNode::onHalted(). | |
| void | resetStatus () |
| Resets the internal status of this node. | |
| Public Member Functions inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode > | |
| SharedResourcesNode (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources) | |
| Constructor for SharedResourcesNode. Called by BT::BehaviorTreeFactory when creating a new behavior tree containing this node. | |
Static Public Member Functions | |
| static BT::PortsList | providedPorts () |
| static BT::KeyValueVector | metadata () |
Protected Member Functions | |
| tl::expected< bool, std::string > | doWork () override |
| User-implemented function which handles executing the potentially-long-running process. | |
| std::shared_future< tl::expected< bool, std::string > > & | getFuture () override |
| Gets the shared future which is used to monitor the progress of the async process. | |
| Protected Member Functions inherited from moveit_pro::behaviors::AsyncBehaviorBase | |
| virtual tl::expected< void, std::string > | doHalt () |
| Optionally implement additional work needed to cleanly interrupt the async process. | |
| void | notifyCanHalt () |
| Called when runAsync() finishes to notify onHalted() that the async process has finished. | |
Additional Inherited Members | |
| Protected Attributes inherited from moveit_pro::behaviors::SharedResourcesNode< BT::StatefulActionNode > | |
| std::shared_ptr< BehaviorContext > | shared_resources_ |
Queries Google Gemini with a text prompt and a ROS image to locate 2D points in the image.
Uses libcurl to POST a multimodal request containing both the text prompt and a JPEG-encoded image to the Gemini generative language API. The API key is read from the GOOGLE_GEMINI_API_KEY environment variable. Because the HTTP call can take several seconds, this behavior is asynchronous and will return RUNNING until the request completes.
The request constrains Gemini via generationConfig.responseMimeType + responseSchema to emit a JSON object with two fields: narrative (a human-readable answer) and points (a list of normalized 2D image coordinates). The narrative is placed on the response output port and the parsed points on the detected_points output port.
| Data Port Name | Port Type | Object Type |
|---|---|---|
| prompt | input | std::string |
| image | input | sensor_msgs::msg::Image |
| model_name | input | std::string (optional) |
| save_debug_image | input | std::string (optional) |
| response | output | std::string |
| detected_points | output | std::vector<geometry_msgs::msg::PointStamped> |
| name | The name of the behavior node in the behavior tree. |
| config | The BehaviorTree.CPP node configuration. |
| shared_resources | Shared resources provided to the behavior at runtime. |
| http_post_fn | Optional HTTP-POST function used to talk to Gemini. Defaults to a libcurl implementation when null. Tests inject a fake to avoid real network traffic. |
| moveit_pro::behaviors::GetPoints2DFromGeminiQuery::GetPoints2DFromGeminiQuery | ( | const std::string & | name, |
| const BT::NodeConfiguration & | config, | ||
| const std::shared_ptr< BehaviorContext > & | shared_resources, | ||
| gemini::GeminiPostFn | http_post_fn = nullptr |
||
| ) |
|
overrideprotectedvirtual |
User-implemented function which handles executing the potentially-long-running process.
This function is called within an async process in a separate thread.
Implements moveit_pro::behaviors::AsyncBehaviorBase.
|
inlineoverrideprotectedvirtual |
Gets the shared future which is used to monitor the progress of the async process.
Classes derived from AsyncBehaviorBase must implement getFuture() so that it returns a shared_future class member.
This exists to prevent destruction of the derived class while the async process is still in-progress. If the derived class is destroyed, the definitions of the functions used within doWork() will be destroyed too, which will result in the virtual functions in the base class being called instead and cause a fault.
This function will force derived classes to add an instance of this type and return a reference to it. The base class can then use this virtual function to access the shared future in functions like onStart.
By adding this virtual function we're properly demonstrating how this future depends on things from the derived class and the natural flow of object lifetimes will do the hard work for us. The std::shared_future destructor will get the value of the future before the derived class is destructed assuming it's the last reference to the shared state. Doing it this way means neither the base nor derived class should need to implement a destructor which is a nice property to have.
Implements moveit_pro::behaviors::AsyncBehaviorBase.
|
static |
|
static |
1.9.8