Godot badly projects 3D camera ray if stretch mode is 2d+"expand" (or "keep")

[Zylann]

Godot 3.1.1
Windows 10 64 bits

I have a 3D game in which I use the following code to get a ray from the captured mouse (itself patched to half the size of viewport because of #29559):

	var mouse_pos = get_viewport().size / 2.0
	var ray_origin = _camera.project_ray_origin(mouse_pos)
	var ray_direction = _camera.project_ray_normal(mouse_pos)

Which works fine with the default stretch mode, even when I resize the window.

However, I have a HUD I want to scale nicely, but if I set the stretch mode to 2d and expand, the 3D ray becomes wrong, offset somehow. I believe camera project functions are broken.
Meanwhile, I can't make my UI to stretch properly because of that.

(sorry no time for a repro project yet, I'm in the middle of a game jam)

[KoBeWi]

Can anyone still reproduce this bug in Godot 3.2.3 rc3 or any later release?

If yes, please ensure that an up-to-date Minimal Reproduction Project (MRP) is included in this report (a MRP is a zipped Godot project with the minimal elements necessary to reliably trigger the bug). You can upload ZIP files in an issue comment with a drag and drop.

[Zylann]

I just made a project to reproduce it:
ProjectCameraRay.zip

And I realize Viewport.get_mouse_position() is returning values that follow the 2D stretching behavior, which I didn't know before. When the doc says "relative to viewport", I thought it meant "relative to viewport's output pixels", without including one of the many canvas transformations happening under the hood.

To workaround #29559, and knowing my mouse is locked, I had assumed I could then get the mouse's position from the viewport's center, in final viewport pixels, using this:

	var mouse_pos = get_viewport().size / 2.0
	var ray_origin = _camera.project_ray_origin(mouse_pos)
	var ray_direction = _camera.project_ray_normal(mouse_pos)

That worked without stretching, but 2D expand mode made it wrong because project_ray_normal expects the output of get_mouse_position(), which again, is using a different coordinate system. There is no indication of that in the doc, maybe it should be added?
(a side consequence is that I'm not sure what I should do to convert get_viewport().size / 2.0, to the same coordinate system as get_mouse_position(), or if it's possible to get mouse position in final pixel coordinates).

Otherwise it seems that this issue is fixed.

Note: there might be a remaining problem similar to #29559 though. If you start the test project, you'll notice that mouse position will be reported at (0, 0), or sometimes elsewhere, even though it's supposed to be locked at (400, 300), until you start moving the mouse.

[Cryszon]

I spent a lot of time trying to figure this out and finally found a solution that worked for me. I still don't completely understand it, but I'll just leave the (C#) code here in case anyone stumbles upon this and wants to try it themselves.

Vector2 screenPosition = this.GetViewport().Size / 2;
Vector2 viewportRelativePosition = this.GetViewport().GetFinalTransform().AffineInverse().Xform(screenPosition);
this.Camera.ProjectRayNormal(viewportRelativePosition);

If anyone wants to explain and add this to Godot documentation, that would be appreciated.