sam2 real time inference?

[Karmik25]

can someone explain to me how to use sam2 for real-time inference, such that the model can predict objects while my webcam is open.

[heyoeyo]

The existing code doesn't support live/streaming video. Instead you'd have to first record a video on webcam and then process that video using the existing video predictor and/or web demo. It is possible to handle streaming video by modifying the code. It's mostly a matter of re-implementing the existing frame loop outside of the sam2_video_predictor.py script, and handling the frame reading (e.g. from a webcam) in that new loop.

If you don't want to implement this manually, there is an existing repo that has done this:
https://github.com/Gy920/segment-anything-2-real-time
I also have a script that works with webcams (though it's worth noting the repo above has code much more similar to the original, in case that's important):
https://github.com/heyoeyo/muggled_sam?tab=readme-ov-file#run-video-or-webcam

[277845265]

https://www.bilibili.com/video/BV1TNkEYrEnb/?spm_id_from=333.33.top_right_bar_window_dynamic.content.click&vd_source=1aa85c9fa1298c2b3a033e3bd0932f3a

[asakhare-zdata]

We also adapted the code in video_predictor.py for real-time tracking. This implementation can easily be paired with YOLO for real-time tracking of detected objects. However, the number of objects to track must be specified during initialization.

The code can be found here:
https://github.com/zdata-inc/sam2_realtime

The core tracking logic is implemented in object_tracker.py

Check out the following notebook for a usage example:
https://github.com/zdata-inc/sam2_realtime/blob/main/notebooks/realtime_detect_and_track.ipynb