Godot 4.4 Beta 1 Takes Long Time To Open/Load Massive Scripts as opposed to Godot 3.6

[stryker313]

Tested versions

Reproducible in v4.4.beta1.official,
Not reproducible in v3.6 (notably, #74733 seems relevant)

System information

Godot v4.4.beta1 - Windows 10 - OpenGL 3 (Compatibility)

Issue description

I am currently working with massive scripts (100k+ lines).
For massive scripts, typing and saving works fine in 3.6 and 4.x. However...

In Godot 3, I can load the script in 1-3 seconds.
In Godot 4.4 beta 1, it takes 4x the time to load the scripts (anywhere from 8-15 seconds)
In other Godot 4 builds (4.0, 4.1, 4.2, and 4.3), this problem is worse (it can be 10 - 30 seconds).

G3scriptbug.mp4

G4Scriptbug.mp4

Normally, you shouldn't be using scripts this long. However, Godot 3.6 is able to load massive scripts very fast, whereas Godot 4.4 beta 1 is not able to, so this is where my concern is.

#74733 seems particularly relevant; was there ever a 4.x counterpart? (it seems to deal more with actual typing than loading, but maybe relevant?)

Steps to reproduce

1. Create a script with 100k lines. Fill it with "########################################################################", or any other text...
2. Load the script in Godot 3.6 (double click the script, open in the in game text editor)
3. Load the script in Godot 4.4 beta 1
4. Compare the loading times...
5. For an even worse example, use any Godot 4 build PRIOR to 4.4 (4.0, 4.1, 4.2, 4.2), and you will see that the script loading is MUCH MUCH slower.

Minimal reproduction project (MRP)

Godot36ScriptLoad.zip
godot43beta1scriptload.zip

[stryker313]

For reference, here is a video of loading this script in 4.3

godot43bug.mp4

[YYF233333]

Profile result:

So this is yet another editor performance issue bottlenecked by TextServerAdvanced. I feel like there are enough similar issues to create a tracker for them.

Some issues that come to my mind immediately:

• In TextServerAdvanced::_shape_run, there is a Vector<Glyph>::push_back that appears to be a bottleneck and spends a significant amount of time on memory allocation because it does not use reserve or resize before the loop. The same applies to TextServerAdvanced::_shape_substr.
• The CodeEdit::_update_delimiter_cache function accounts for about 10% of the execution time. The performance bottleneck lies in the String::length call, which leads to memory access (the length of the String is stored in CowData, located at the beginning of the data structure).

Haven't tested but I guess switch to text_server_fb can address this issue.

[redsett]

As a person who does most of my gdscript dev in the editor, these slow downs has been fairly painful in big projects. To the point where I started using an external editor.

[Ivorforce]

I've also profiled the problem (20s lag spike on load on my end), to see if I can glean anything different.

Let's start with the call stack:

Here we can see that each set_text and shape_lines take up ~40% of performance, while GDScriptCache takes ~10%. It is possible that unnecessary repeated calculations are performed here, but I cannot say for sure.
All in all it's not super unexpected for this to be slow, but there it is possible that top-down improvements could be made to avoid repeated calculations. However, this may be a difficult change requiring deep insight on the advanced text server.

Let's look at the inverted call stack too:

This one is even more fragmented, but there are practical suggestions made by it:

• 3.83 Gc  8,9 % u_strFromUTF8WithSub_76_godot
  • I have recently improved this (unmerged as of yet): Optimize text rendering by caching UBreakIterator instances. #102129
    • Notably, my PR comes with a risk of increased RAM use.
  • There may be more improvement to be had by using pools for break iterators.
• 3.22 Gc  7,5 % std::__1::recursive_mutex::unlock() / 2.58 Gc  6,0 % std::__1::recursive_mutex::lock()
  • It should be possible to restructure TextServerAdvanced to use non-recursive mutexes to improve performance here.
    • Edit: I tested just exchanging it, unfortunately it does seem like the current code does require on recursion. Would require a deeper dive to get rid of it.
• 2.76 Gc  6,4 % CodeEdit::_update_delimiter_cache(int, int)
  • It should be possible to benchmark this function to squeeze better performance from it.

[stryker313]

@Ivorforce

All in all it's not super unexpected for this to be slow

I wouldn't have made an issue, as I understand that opening a 100K file is not ideal in any way, but the thing is that Godot 3.6 can load the script almost instantaneously (see above, using 3.6....)

Furthermore, there is a drastic speed difference between 4.4 beta 1 and other 4.0 builds. (Try something like 4.0,4.1 and you will see that the loading speed is FAR FAR worse; I assume that the speedup in 4.4 is due to the new editor performance upgrades in 4.4).

The script load speed difference/downgrade between Godot 3 and Godot 4 is very interesting; is it truly just a "TextServerAdvanced" thing?

[YYF233333]

The script load speed difference/downgrade between Godot 3 and Godot 4 is very interesting; is it truly just a "TextServerAdvanced" thing?

Correct me if I'm wrong, but I think TextServerFallBack is exactly the text server we use in Godot 3, it lacks the support for some advanced functionalities like BiDi so it is not the default choice but it is really fast :)

[Ivorforce]

I don't mean to say it's supposed to be slow :)
By "not unexpected" i just mean there's nothing obvious that's going wrong, so a fix is unfortunately harder to conjure than if there was.

[Calinou]

Note that TextServerFallback is not compiled in by default, so you need to compile the editor with module_text_server_fb_enabled=yes. Make sure to use other optimizations when compiling to match official builds (production=yes).

After compiling it, change the text driver in the project settings to Fallback. This will affect the editor on the current project after restarting.

You can also check performance without any text rendering by using the Dummy text driver (--text-driver Dummy on the command line). No text will appear when using this text driver.

[akien-mga]

CC @bruvzg

As pointed out, Godot 4 has more advanced text features than Godot 3, and these features come at a cost. So it's not unexpected that Godot 4 has worth performance on huge texts than Godot 3.

That being said, if profiling can identify some low hanging fruits to improve performance of TextServerAdvanced, that would be great.

For extremely long files, we could also consider disabling support for some features when detecting such files to avoid too high a performance cost. And show a popup informing the user about it, with the possibility to re-enable them (Kate does this e.g. when loading files with line lengths that exceed a reasonable number).

[bruvzg]

For extremely long files we probably should implement threaded loading, there might be parts of TextServer that can be optimized (a lot of optimizations already done for 4.4, and it should significantly faster than 4.3), but it will always be slower than zero processing in 3.x.