Detect corn stalks in an environment and calculate the best grasping point for a robot arm to insert a needle sensor.
This pipeline is meant as a plug-and-play system, meaning it can be applied to varying similar problems with little interference. Each part of the pipeline can be swapped out with different alternatives, which may work better under certain environment conditions, like camera field of view, ground plane visibility, ground plane smoothness, point cloud density and accuracy, etc.
You can select the desired options for each pipeline step in src/config.py.
The options are described in more detail below.
This ROS node registers a ROS service, which allows a request-reply communication between two nodes. In this case, this node publishes the service get_stalk, which receives a number of frames and a timeout, and returns the stalk positions and other information. This service is defined here.
A node can wait for this service to be available and then "call" it and await a response. Alternatively, a node can make a persistent connection to a service, which maintains the communication between nodes and does not re-search for the node at each subsequent service request (this can increase performance when the service is being called multiple times from the same node).
For example, a node could call this service once using the following code:
rospy.wait_for_service('get_stalk')
get_stalk = rospy.ServiceProxy('get_stalk', GetStalk)
try:
resp1 = get_stalk(num_frames=5, timeout=10.0)
except rospy.ServiceException as exc:
print("Service did not process request: " + str(exc))
Alternatively, you can call the service from the terminal:
rosservice call /get_stalk "num_frames: 5
timeout: 10.0"
The code is separated into 3 steps, as shown below.
- In "Stalk Detection", the images are converted to image masks representing the stalk locations in 2D. Currently, the only option available is Mask-R-CNN, which performs well on this task. The model is run in
model.pyand called by the ROS service here. See Dataset, Training, and Labeling for tools on how to get your model trained easily.
To get this node up and running, you'll need to prepare the following:
- Install Detectron2, which requires CUDA and torch. Detectron2 also works with torch installed for CPU, if you do not have a GPU. We recommend building Detectron2 from source.
- Install all package dependencies, which are in requirements.txt.
- If desired, replace the first line of
main.pywith your Python interpreter path - Fill in the configuration file with your desired parameters: specifically, make sure to edit the
MODEL_PATH, the TF frames, the image and depth image topics, and the constraints on stalk distance. - Ensure the
main.pyscript is executable, runcatkin_make, start up yourroscore, and run:
rosrun stalk_detect main.py
We recommend training on a large number of images for robustness in deployment. With the labeling tool provided below, labeling is very quick, and 500 images should take no more than a few hours.
For labeling, we use the Segment Anything model to predict masks around objects given a position contained within the mask. This tool is forked from the commonly-used SALT package (Segment Anything Labeling Tool), and fixes some small bugs, adds progress bars, and adds a setup file for automatic configuration.
Follow instructions described in the tool's README.
To train and test your Mask-R-CNN model, you may use this tool, which describes the needed data format and other configuration. Training is very quick with a decent GPU, since we use the pre-trained checkpoint to start.
- Mark Lee for assistance in testing, data collection, drivers, and overall assistance
- Kantor Lab for general assistance.