Markerless location-based AR (more realistic placement of objects)

[nickw1]

Do you want to request a feature or report a bug?

Feature.

What is the current behavior?

Location-based AR works, however what would be nice is more realistic placement of the augmented content, so that it appears to be on the ground rather than floating slightly in the air.

Details

This is an issue to discuss a possible improvement to AR.js as described above. Currently, location-based AR generally works on most devices (subject to a few issues such as #278 and the limitations of the device hardware such as the GPS or sensors). However, while the augmented content appears in approximately the correct place, it would be nice to improve the realism so that it appears to be nicely 'on the ground', as is seen in various proprietary AR libraries.

I have been in email conversation with @ThorstenBux who outlined a general approach, which I will share here. I would like this issue to be a place for general discussion of this proposed improvement, and a place to contribute your own ideas and suggestions.

Anyhow the general approach seems to be as follows:

• Detect feature points. OpenCV can be used but a simpler and more lightweight alternative is tracking.js. In initial tests, this appears to be effective for feature detection.

• Once we have feature points, the intention is to place the augmented content on the feature point. This is where, at present, I am less certain of how to implement it; further research is needed. One approach is to convert the 2D coordinates of feature points into 3D world coordinates, and we can then place the augmented content at the closest feature point where the y (vertical) coordinate is close to zero. This needs refining to deal with such things as slopes and hilly terrain, but it will do for now. Algorithms such as solvePnP can be used for this. However solvePnP appears to need a set of known points (i.e features for which both the 2D screen coordinates and 3D world coordinates are already known) to calculate the current camera pose. Not sure how we can fix this other than by pre-calibration.

That's my thoughts so far, anyhow - would be great to get input on the second point in particular. Would be really nice to add this to AR.js, the world really needs a fully open source solution to markerless AR. At the moment, sadly, most of the solutions are proprietary and often paid-for.

[hiukim]

Hi @nickw1

I would be interested to work together in this topic. I’ve worked on artoolkit before, and I have also written another AR library for image tracking as well - https://github.com/hiukim/mind-ar-js

Just a few initial comments:

I’ve also been playing around different available libraries like tracking.js. I could be wrong, but In my experiences, most of the libraries are generally too slow to be practical because they are running in CPU. This is also one of the bottleneck of ar.js. I think part of the algorithms will need to be delegate to GPU in order to make it run efficiently.

I also think solvePnp is to map 3D coordinates to known 2D coordinates, and not sure how exactly to apply that here. We also need to continuous track the feature points as well. Is it like some kind of SLAM? Another approach I can think of is to do ground detection and tracking, and then place the virtual object on top of the ground. If we can do ground detection and tracking it will open up a lot of potential. But I also don’t know what’s the correct approach.

Thank you for trying to work on this! I’m here if you need someone to bounce off ideas.

[nickw1]

Hi @hiukim thanks for your message. I saw your MindAR project some time ago... looks interesting!

Since my original message I have been looking into this in a bit more detail and have experimented with tracking.js. Yes, we need to track the feature points continuously but you can do this with tracking.js fairly easily. Performance for this part seems reasonable, I have created an (extremely simple) initial experiment at https://hikar.org/tjs (please use portrait, landscape will not work at the moment). Red boxes are detected corners while blue are actual tracked features.

You may well be right about doing some of it on the GPU, I would however like to get the algorithm 'working', albeit slowly, first, and then move onto the GPU later once we've got a proof of concept.

solvePnP, from my understanding, can be used to transform 2D coordinates (the tracked feature points) into 3D coordinates (which can then be used for AR feature placement). However, it needs an initial calibration step first in which it is supplied with a known set of points for which both 2D and 3D coordinates are known, for example a chessboard at a certain distance from the camera.

Talking to Thorsten Bux in more detail about this, one solution he came up with was PTAM which also needs calibration but can perhaps be done one time for all devices.

Ground detection is the other thing that occurred to me, yes.

The big problem I see is how do we convert 2D points (feature points, or ground points, detected with standard computer vision algorithms available in tracking.js or OpenCV) to 3D points. Once we have that, the rest can, I think, be figured out. From what I am reading, a calibration step is needed for this, but I may be wrong.

So if you have any suggestions about converting 2D points to 3D world coordinates easily, that would be great!

[ConcealGeek]

good news?

[nickw1]

Working on this at the moment but after talking to a few people, I am now investigating PTAM. See https://github.com/nickw1/ptam_expts

[dparker2]

Hey, has there been any progress on this or is there any way to maybe help at all?

[nickw1]

@dparker2 still working on this when I have the time. I found that PTAM worked reasonably well indoors but had difficulty detecting surfaces outdoors, and this was a day with good lighting (August, sunny day, northern hemisphere).

Since then I have started experimenting with ORB-SLAM which is probably the state of the art in open source SLAM algorithms. Specifically I'm working on creating a web interface with emscripten. See http://github.com/nickw1/ORB_SLAM3 (fork of the original University of Zaragoza project) and https://github.com/nickw1/orb-slam-expts.

Biggest problem I have is trying to debug an emscripten application (currently getting some issues with unaligned memory access, but finding it hard to track down where exactly this is happening!)

So if you have emscripten experience, or maybe have worked with SLAM algorithms, your input would be most welcome! :-)

[mamoone]

any news ? :/

[kalwalt]

@mamoone and all here you can try this project https://github.com/alanross/AlvaAR

[mamoone]

Thanks,
i Tried the project, it's pretty cool,
Issues:
Unstable performance.
Complexity in integration.
Heavy load.
im looking for somthnig lightweight & more stable

[RichardPickman]

It's been a long time since issue was open. News? There must be something!

[nickw1]

@RichardPickman AlvaAR seems to be the way to go though it hasn't seen any development in a while.