Adding features track functionality to the project #7

kalwalt · 2023-03-25T20:39:46Z

Tracking feature

The code can only compute keypoints and descriptors for the trackable. I will implement also the tracking functionality.
This codes will be derived most from webarkit-testing and wasm-ar, even if they are coded with C++, i think it can be easily translated to js.
I will add inside WebARKitCV a track method, with a trackables argument and a Promise structure.
So you will be able to pass matrix data to trackers; it will be independent from any 3d is rendering library.

List of features:

track method.
Tracker class and ITracker interface.
~~simple VideoCapture method for video stream.~~ cameraViewRenderer class for video stream to be exported
tracking code implemented inside the Worker, event listeners to pass data from/to.

For the future...

Things to improve in the near future.

We need to display correctly the found pattern, a sort of draw debug function to dispaly a rect around it.
improve the detection and tracking phase in the worker.
docker image to build the whole project.

- post vWidth and vHeight to the worker - missed ConfigData

- increased number of ORB features - disabled blurring for now

kalwalt · 2023-03-26T18:43:42Z

src/Workers/Worker.ts

@@ -43,7 +43,12 @@ const loadTrackables = async (msg: any) => {
    let src = msg.data;
    let refRows = msg.trackableHeight;
    let refCols = msg.trackableWidth;
-    let mat = new cv.matFromArray(refRows, refCols, cv.CV_8UC4, src);


This wasn't correct, infact logging the mat it gives me an Unbinding error, though i have no errors in the console.

kalwalt · 2023-03-26T18:45:45Z

src/Workers/Worker.ts

+    //var mat = new cv.Mat(imageData.height, imageData.width, cv.CV_8UC4);
+    let mat = new cv.Mat(refRows, refCols, cv.CV_8UC4);
+    //mat.data.set(imageData.data);
+    mat.data.set(src.data);


i found this snippet from helpers.js opencv code, see matFromImageData:

Module['matFromImageData'] = function(imageData) { var mat = new cv.Mat(imageData.height, imageData.width, cv.CV_8UC4); mat.data.set(imageData.data); return mat; };

kalwalt · 2023-03-26T18:49:10Z

src/Workers/Worker.ts


    let ksize = new cv.Size(BlurSize, BlurSize);
    let anchor = new cv.Point(-1, -1);
-    cv.blur(mat, mat, ksize, anchor, cv.BORDER_DEFAULT);
+    //cv.blur(mat, mat, ksize, anchor, cv.BORDER_DEFAULT);


Adding blur make more difficult to detect, i will leave out this, maybe i need to adjust the settings.

kalwalt · 2023-03-26T18:49:39Z

src/Workers/Worker.ts

    template_keypoints_vector = new cv.KeyPointVector();

    template_descriptors = new cv.Mat();

    let noArray = new cv.Mat();

-    let orb = new cv.ORB(3000);
+    let orb = new cv.ORB(10000);


Better to increase the number of features, it gives more matches.

kalwalt · 2023-03-26T18:50:05Z

src/Workers/Worker.ts


    let ksize = new cv.Size(BlurSize, BlurSize);
    let anchor = new cv.Point(-1, -1);
-    cv.blur(src, src, ksize, anchor, cv.BORDER_DEFAULT);
+    //cv.blur(src, src, ksize, anchor, cv.BORDER_DEFAULT);


same as above.

kalwalt · 2023-03-26T18:50:15Z

src/Workers/Worker.ts


    var frame_keypoints_vector = new cv.KeyPointVector();

    var frame_descriptors = new cv.Mat();

-    var orb = new cv.ORB();
+    var orb = new cv.ORB(10000);


same as above

kalwalt · 2023-04-05T18:07:39Z

cv.perspectiveTransform is wrapped in a different way from C++ it require cv.Mat as args, struglled a bit with this.
I implemented homographyValid method (not sure if it is correct) but fails with my image test, looks as something is not correct in the code, maybe the same code that can be used in C++ it's needed to adapt. Investigating on this.

kalwalt · 2024-05-15T11:33:21Z

regarding the last comment, when testing the example.html we get this error:

Worker.ts?./node_mod…-loader/index.js:68 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'doubleAt')
    at homographyValid (Worker.ts?./node_mod…ader/index.js:68:19)
    at track (Worker.ts?./node_mod…ader/index.js:157:9)
    at async process (Worker.ts?./node_mod…ader/index.js:99:20)

this error happens only with htat example because the homography is not valid (because at the start it's not recognizing any pattern), it's empty and that's why that message. This routine:

if (homographyValid(homography) == true) {
    var out = fill_output(cv, homography, valid);
  }

should be placed after the findHomography function. i will fix this.

- rebuilding libs and improved github actions scripts

kalwalt · 2024-05-15T12:29:35Z

In check if homograpy is valid commit testing the example_image the result is false. i think that something is not correct maybe this method should be different in javascript? I transformed this snippet from C++ code and maybe should be written in another way?
I wil merge this PR as i would mantain a small Pr's pattern, instead of a BIG one.
Updating the start of this PR thread with a to do list for the future.

kalwalt added 2 commits March 25, 2023 19:04

new Tracker class and ITracker interface

ab1b544

Merge branch 'dev' into features-track-functionality

860d3ae

kalwalt added the 🚀 enhancement New feature or request label Mar 25, 2023

kalwalt self-assigned this Mar 25, 2023

kalwalt added 6 commits March 25, 2023 21:45

Tracker test

d88a2dd

raw process function

6104616

CameraViewRenderer as separate class

e8a18d9

matFromArray was not correct

525a20d

- post vWidth and vHeight to the worker - missed ConfigData

running prettier

df0e4d1

now it can detect and track a simple image

f62fd19

- increased number of ORB features - disabled blurring for now

kalwalt commented Mar 26, 2023

View reviewed changes

kalwalt added 3 commits March 26, 2023 22:06

matrix data is sent accross getMatrix event now

056852f

testing homographyValid and perspectiveTransform

23d70a0

better code for fill_output

b69e273

kalwalt added 3 commits April 5, 2023 20:07

formatting code

f3bf395

samll fixes

91ffcc5

upgrading opencv and emsdk to 3.1.26

904d90e

kalwalt added 5 commits May 15, 2024 13:45

move homgraphyValid after findHomography

de1bfc8

- rebuilding libs and improved github actions scripts

formatting Worker file and upgrading actions scripts

f864d03

check if homography is valid

7821a1b

formatting code

8b1f218

better README file

79280af

kalwalt merged commit 446a655 into dev May 15, 2024
3 checks passed

kalwalt deleted the features-track-functionality branch May 15, 2024 13:10

kalwalt mentioned this pull request May 17, 2024

First steps for improving webarkit-cv #8

Open

4 tasks

kalwalt added ⌨️ Typescript About Typescript issues and related features 🔬 OpenCV Improvements to OpenCV code and building the source code labels May 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding features track functionality to the project #7

Adding features track functionality to the project #7

kalwalt commented Mar 25, 2023 •

edited

Loading

kalwalt Mar 26, 2023

kalwalt Mar 26, 2023

kalwalt Mar 26, 2023 •

edited

Loading

kalwalt Mar 26, 2023

kalwalt Mar 26, 2023

kalwalt Mar 26, 2023

kalwalt commented Apr 5, 2023

kalwalt commented May 15, 2024

kalwalt commented May 15, 2024 •

edited

Loading

Adding features track functionality to the project #7

Adding features track functionality to the project #7

Conversation

kalwalt commented Mar 25, 2023 • edited Loading

Tracking feature

List of features:

For the future...

kalwalt Mar 26, 2023

Choose a reason for hiding this comment

kalwalt Mar 26, 2023

Choose a reason for hiding this comment

kalwalt Mar 26, 2023 • edited Loading

Choose a reason for hiding this comment

kalwalt Mar 26, 2023

Choose a reason for hiding this comment

kalwalt Mar 26, 2023

Choose a reason for hiding this comment

kalwalt Mar 26, 2023

Choose a reason for hiding this comment

kalwalt commented Apr 5, 2023

kalwalt commented May 15, 2024

kalwalt commented May 15, 2024 • edited Loading

kalwalt commented Mar 25, 2023 •

edited

Loading

kalwalt Mar 26, 2023 •

edited

Loading

kalwalt commented May 15, 2024 •

edited

Loading