GetPoints2DFromGeminiQuery Class
Queries Google Gemini with a text prompt and a ROS image to locate 2D points in the image. More...
Declaration
Included Headers
Base class
| class | AsyncBehaviorBase |
|
A base class for behaviors which need to asynchronously run a function that might take a long time to complete. More... | |
Public Constructors Index
| GetPoints2DFromGeminiQuery (const std::string &name, const BT::NodeConfiguration &config, const std::shared_ptr< BehaviorContext > &shared_resources, gemini::GeminiPostFn http_post_fn=nullptr) | |
Private Member Functions Index
| tl::expected< bool, std::string > | doWork () override |
|
User-implemented function which handles executing the potentially-long-running process. More... | |
| tl::expected< void, std::string > | doHalt () override |
|
Optionally implement additional work needed to cleanly interrupt the async process. More... | |
| std::shared_future< tl::expected< bool, std::string > > & | getFuture () override |
|
Gets the shared future which is used to monitor the progress of the async process. More... | |
Private Member Attributes Index
| gemini::GeminiPostFn | http_post_fn_ |
| std::shared_future< tl::expected< bool, std::string > > | future_ |
| std::stop_source | stop_source_ |
Public Static Functions Index
| static BT::PortsList | providedPorts () |
| static BT::KeyValueVector | metadata () |
Description
Queries Google Gemini with a text prompt and a ROS image to locate 2D points in the image.
Uses libcurl to POST a multimodal request containing both the text prompt and a JPEG-encoded image to the Gemini generative language API. The API key is read from the GOOGLE_GEMINI_API_KEY environment variable. Because the HTTP call can take several seconds, this behavior is asynchronous and will return RUNNING until the request completes.
The request constrains Gemini via generationConfig.responseMimeType + responseSchema to emit a JSON object with two fields: narrative (a human-readable answer) and points (a list of normalized 2D image coordinates with short labels). The narrative is placed on the response output port, the parsed points on the detected_points output port, and the labels associated with each point on the detected_labels output port (one label per point, same order). Labels are useful when prompts ask Gemini to tag each location with metadata such as an object ID, color, or class name.
| Data Port Name | Port Type | Object Type |
|---|---|---|
| prompt | input | std::string |
| image | input | sensor_msgs::msg::Image |
| model_name | input | std::string (optional) |
| save_debug_image | input | std::string (optional) |
| response | output | std::string |
| detected_points | output | std::vector<geometry_msgs::msg::PointStamped> |
| detected_labels | output | std::vector<std::string> |
- Parameters
-
name The name of the behavior node in the behavior tree.
config The BehaviorTree.CPP node configuration.
shared_resources Shared resources provided to the behavior at runtime.
http_post_fn Optional HTTP-POST function used to talk to Gemini. Defaults to a libcurl implementation when null. Tests inject a fake to avoid real network traffic.
Definition at line 56 of file get_points2d_from_gemini_query.hpp.
Public Constructors
GetPoints2DFromGeminiQuery()
|
Declaration at line 59 of file get_points2d_from_gemini_query.hpp, definition at line 185 of file get_points2d_from_gemini_query.cpp.
Private Member Functions
doHalt()
| virtual |
Optionally implement additional work needed to cleanly interrupt the async process.
The default implementation of this function is a no-op which will not interrupt the async process. This will mean that onHalted will wait until the process finishes before returning.
- Returns
Return an empty tl::expected<void, std::string> if the additional work to halt was successful. Return an error message if something failed while doing additional work to halt.
Declaration at line 70 of file get_points2d_from_gemini_query.hpp, definition at line 281 of file get_points2d_from_gemini_query.cpp.
doWork()
| virtual |
User-implemented function which handles executing the potentially-long-running process.
This function is called within an async process in a separate thread.
- Returns
A tl::expected which contains a bool indicating task success if the process completed successfully or was canceled, or an error message if the process failed unexpectedly.
Declaration at line 68 of file get_points2d_from_gemini_query.hpp, definition at line 223 of file get_points2d_from_gemini_query.cpp.
getFuture()
| inline virtual |
Gets the shared future which is used to monitor the progress of the async process.
Classes derived from AsyncBehaviorBase must implement getFuture() so that it returns a shared_future class member.
This exists to prevent destruction of the derived class while the async process is still in-progress. If the derived class is destroyed, the definitions of the functions used within doWork() will be destroyed too, which will result in the virtual functions in the base class being called instead and cause a fault.
This function will force derived classes to add an instance of this type and return a reference to it. The base class can then use this virtual function to access the shared future in functions like onStart.
By adding this virtual function we're properly demonstrating how this future depends on things from the derived class and the natural flow of object lifetimes will do the hard work for us. The std::shared_future destructor will get the value of the future before the derived class is destructed assuming it's the last reference to the shared state. Doing it this way means neither the base nor derived class should need to implement a destructor which is a nice property to have.
- Returns
Returns the shared_future, which should be owned by the child class.
Definition at line 72 of file get_points2d_from_gemini_query.hpp.
Private Member Attributes
future_
|
Definition at line 79 of file get_points2d_from_gemini_query.hpp.
http_post_fn_
|
Definition at line 78 of file get_points2d_from_gemini_query.hpp.
stop_source_
|
Definition at line 80 of file get_points2d_from_gemini_query.hpp.
Public Static Functions
metadata()
| static |
Declaration at line 65 of file get_points2d_from_gemini_query.hpp, definition at line 218 of file get_points2d_from_gemini_query.cpp.
providedPorts()
| static |
Declaration at line 63 of file get_points2d_from_gemini_query.hpp, definition at line 193 of file get_points2d_from_gemini_query.cpp.
The documentation for this class was generated from the following files:
Generated via doxygen2docusaurus 2.2.2 by Doxygen 1.9.8.