-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backbone change #18
Comments
Hi @vietpho, Sorry for the delay, I wanted to make sure I address all of your points.
We make heavy use of Hydra and its auto-instantiating feature to run our controlled experiments. This means all of our backbones are instantiated from configuration files. For example, if you want to instantiate the SILK_BACKBONE = ParametricUNet(
n_channels=1,
n_classes=128,
input_feature_channels=input_feature_channels,
bilinear=False,
use_max_pooling=True,
n_scales=4,
length=1,
down_channels=(32, 64, 64, 64),
up_channels=(64, 64, 64, 128),
kernel=5,
padding=0,
) We understand this is not ideal for now, but I'm working on a tool that will make the backbone selection easier. I'll release that soon.
The
We tend to use
In case both are enabled, the keypoint selection will work as follow. First use threshold to get your keypoint set. If that set is smaller than the required
This is simply a scaling factor of the descriptors ( Moreover, changing that value won't change the MNN matching, nor the ratio-test. BUT, it will affect the double-softmax matching, as changing
You can change
Raw descriptors coming from the descriptor head are L2-normalized. We simply divide them by their norm
We used the default matcher recommended for each model. From memory, we used MNN for SiLK, R2D2, DISK, SIFT and SuperPoint. We use the learned matcher for LoFTR.
MNN (Mutual Nearest Neighbors) discards keypoints that are not mutually neighbors, which reduces the original set of keypoints coming from the model.
Tuning for :
The threshold operates on distances (not similarities), that's why the logic is reversed. Here you can see how we convert matching probabilities / similarities into a distance. This is required since our MNN implementation operates on distances, and the threshold is the maximum distance allowed for matching, anything higher will be discarded. |
Thank you for your wonderful and heartfelt response! |
Hello and thank you for your outstanding work!
As a newcomer to deep learning, I am interested in experimenting with other checkpoints beyond pvgg-4.ckpt. Specifically, I would like to use punet-disk.ckpt, but I am unsure if I need to change the backbone. Would you be able to provide an example of which backbone model would be appropriate? I have seen options such as Unet and parametric Unet, but I am uncertain which one to use. Additionally, for the coco-rgb-aug.ckpt checkpoint, I was wondering which backbone was used. Do I need to change the backbone each time I use different checkpoints?
On a related note, I have been exploring the impact of different values for the SILK_THRESHOLD parameter in your deep learning model. Specifically, I attempted to change the threshold to 0.5 and 0.8 while keeping the default inference setting (top-k = 10000), but I did not observe any significant difference compared to the default value of 1.0.
Based on my understanding, the SILK_THRESHOLD parameter represents a confidence level, and I expected to see more keypoints appearing with a lower threshold value. I suppose that the images in my dataset already have more than 10000 keypoints, and I am wondering if this might be why the threshold did not seem to have a significant impact.
In addition, I was hoping you could provide some additional information about the SILK_SCALE_FACTOR parameter. Could you explain what this parameter does and how it might affect the performance of the model?
And I don't know how to obtain the sparse positions and descriptors and dense positions and descriptors. I have come across the 'get_dense_positions' function, but I can't find a similar function for the sparse positions and descriptors. Additionally, I have noticed an argument labeled 'normalized_descriptor,' and I am curious as to its meaning. Is there a function available to normalize the original descriptor?
I also had a couple of questions related to your paper.
Firstly, I am curious to know if you used the same matcher for SILK as you did for other models such as SIFT. Could you provide some information on the matching technique used in your experiments?
Secondly, in your paper, you referred to pre-match and post-match keypoints. While I understand that the pre-match keypoints are the keypoints extracted from an image, I am unclear on how you obtained the post-match keypoints. Specifically, since the number of keypoints is already limited to the 'top-k' value during the matching process, I am not sure how you obtained post-match keypoints. Could you please elaborate on this process?
Lastly, could you suggest which parameters I should tune if I want to use the results for 3D reconstruction?
I've tried changing the matcher to 'double softmax' with the same value of temperature (0.1) but different threshold values. However, I don't understand why I'm getting less matches with a low threshold value. In my understanding, since the threshold value has been lowered, there should be more matches that are above the threshold value. Can you explain why it's like this?
Thank you for your time and assistance.
The text was updated successfully, but these errors were encountered: