Improve Project Manager loading speed by loading rarely edited projects in a thread

[SchimmelSpreu83]

Describe the project you are working on

Any game/project

Describe the problem or limitation you are having in your project

I have so many projects and downloaded projects in the Project Manager.
When I want to open Godot for the first time in a day, it takes long time to look for all the projects and thus remains longer in the boot screen.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I would suggest making 5 previously used projects load instantly (how it currently works) and all other projects would load (pop) in afterwards (with an indicator that Godot is still looking for projects of course).

5 is just an arbitrary number, I suggest it could be set in the Editor Settings.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

When opening Godot, it opens up fast with the first 5 previously used projects in the list and an indicator that it's still trying to find other projects.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, because it's the Project Manager.

Is there a reason why this should be core and not an add-on in the asset library?

It's the Project Manager.

[Ivorforce]

I'm not sure what makes project loading slow. If it's just a path, name and some tags in a cache file, it should load almost instantly in all cases, even if you have tens of thousands of projects.
It would be appropriate to profile launch with many projects to see if it can't be improved more effectively without threading.

[Ivorforce]

Alright, I created a list of 10k projects with the following script:

python -c "import pathlib; pathlib.Path('projects.cfg').write_text('\n'.join(f'[/Users/lukas/dev/godot/project{n}]\n\nfavorite=false\n' for n in range(10000)))"

Then, I profiled the project selector launch using Apple Instruments (CPU). It took about 10 seconds. Scrolling is also miserably slow.

Here's the call tree:

As I suspected, project load does not appear in the profile result. Instead, most of the time spent is spent on UI, specifically sizing and TextServerAdvanced calculations.
For example, 1.67 Gc  6,8 % u_strFromUTF8WithSub_76_godot is the largest entry in the inverse call tree with a self-time of 6.8%. I have an open PR to mitigate this problem (godotengine/godot#102129), but it is unlikely it will lead to noticeable speed ups of load.

All in all, I don't see a simple single point of attack with just performance optimizations for this issue, unfortunately.
Instead, I it may be best to introduce a fixed-item-height scrolling table view for the project launcher. I've used this concept extensively back with the Cocoa framework (NSTableView). It is an incredibly effective way to render fast loading and scrolling table views with millions of entries.
It would also be possible to contribute structural changes to UI and / or TextServerAdvanced. However, this may be difficult to achieve, and requires deep insight into these parts of Godot, and will be less effective.

[SchimmelSpreu83]

I think it must be something to do with the Godot .exe and all my projects being on an HDD drive. My guess would be that when I open the Project Manager, it looks through all project paths to see if they are valid (where the bottleneck probably lies), which is then stored in a temporary cache so godot opens up very fast when I open it again quickly afterwards.

I'm not sure however since I don't actually know the logic behind that; it's just a wild guess.

[Ivorforce]

Oh yeah, I suppose it is possible that your end has a different bottleneck than mine... Would be good to profile for someone that does have problems right now.
@SchimmelSpreu83 Mind sharing how many projects are listed in your project viewer? Just so I have an idea what kinds of numbers we're working with.

[SchimmelSpreu83]

Somewhere around 112. Could probably be more. I haven't counted them manually. It's based upon a quick and dirty look at the number of folders I have in my Projects- and Downloaded Projects folder.

EDIT: I've counted them now. I have 126 projects (9 of them being missing)

[Calinou]

Instead, I it may be best to introduce a fixed-item-height scrolling table view for the project launcher. I've used this concept extensively back with the Cocoa framework (NSTableView). It is an incredibly effective way to render fast loading and scrolling table views with millions of entries.

See also #9678.