AGS 4: properly support managed pointers in managed structs

[ivan-mogilko]

WARNING: this requires #1922 to work.
(pr's branch is based on top of that pr for easier testing)

This resolves a long existing problem of AGS script not supporting managed pointers inside user's managed structs.
Such syntax feature was explicitly disabled, because the runtime interpreter did not have any information on the struct's contents, only about it's size. When the user's managed object is disposed, engine cannot unreference the pointers contained in such struct, which leads to nested objects never disposed, occupying program memory (sort of "memory leak").

With addition of RTTI it is now possible to gather information about pointers contained within a type, and correctly unref all them on type's instance's disposal.

This PR introduces new script op code: SCMD_NEWUSEROBJECT2, which has 2 args: typeid and size.
The typeid is a local type id from the script's RTTI table.
Runtime interpreter converts local type id to a global one when creating a user object.

After loading scripts, engine generates a "flat" table of managed pointer offsets per struct. This speeds up user object disposal, as we don't have to analyze all the nested types all the time. Instead we do that only once the script is loaded, and then use a plain list of offsets.

Managed pointers in managed structs are enabled in both compilers, old and new one.

Both compilers declare extension "NESTEDPOINTERS", and a script author can use SCRIPT_EXT_NESTEDPOINTERS macro to test whether this feature is supported.

TESTING

The idea is that a structure of any complexity should be correctly disposed and removed from the memory when all references to it are gone. For example, you may have a managed struct that contains a particularly large dynamic array, and continuously create instances of that struct, observing program's memory usage.

#ifdef SCRIPT_EXT_NESTEDPOINTERS
managed struct ManagedStruct
{
    String Name;
    int BigArray[];
};

managed struct NestedPointers
{
    ManagedStruct* sp;
    ManagedStruct* sparr[];
};

function on_key_press(eKeyCode k)
{
    if (k == eKeySpace) {
        NestedPointers *t = new NestedPointers;
        t.sp = new ManagedStruct;
        t.sp.BigArray = new int[1000000];
        t.sp.Name = "some name";
        t.sparr = new ManagedStruct[10];
        for (int i = 0; i < 10; i++)
        {
            t.sparr[i] = new ManagedStruct;
            t.sparr[i].BigArray = new int[1000000];
            t.sparr[i].Name = String.Format("name %d", i);
        }
        // NestedPointers should be fully disposed here
    }
}
#endif // SCRIPT_EXT_NESTEDPOINTERS

ISSUES / TODO:

• Also might need a new op code for creating a dynamic array with typeid, for knowing which type does they store. This may potentially allow dynamic arrays of regular structs, as they may contain nested managed pointers with more pointers inside, but neither compiler nor engine won't know how to properly unref these.
• Must update the savegame format of a "user object", right now these will loose the type id when restoring a save.
• Must update new compiler's tests.
• Might add explicit compiler option for generating & saving rtti / allowing pointers in man structs.
• Save game problem (see explanation in comments below).

[ivan-mogilko]

So... in overall this is ready for testing. The only mandatory thing missing is compiler tests update, where the bytecode tests are failing because of the new opcodes (and different number of args). I will look into this soon.

But while writing this, I suddenly realized there's a serious loophole with game saves. I will explain this later, in the second part of the comment.

For now, the manual tests might include following:

• A script that includes managed user structs and dynamic arrays.
• The structs now can be composed of any combination of nested regular and managed structs.
• Dynamic arrays cannot contain regular structs at the moment (see AGS 4 / new compiler: support dynamic arrays of normal structs #1818), so there's no way to test these, but the engine has (supposedly) a necessary code to dispose these correctly (will have to test that separately when AGS 4 / new compiler: support dynamic arrays of normal structs #1818 is resolved).
• The managed structs containing nested pointers should be correctly disposed as soon as the last reference to them is removed, leaving no memory leaks.
• Serialization: storing and restoring game saves should work correctly with managed structs in the script mem. (But see a note below.)
• Structs should still dispose correctly after restoring a save, this would mean that they keep type ids.

NOTE
Old saves won't work, because the UserObject and DynamicArray serialization has changed. But it's possible to add compatibility fix and allow to restore these too. This may be done because I assigned new slightly different type names to these script objects, so it's possible to distinguish from the previous ones (unfortunately there's no other reliable way now).

---

The game save issue

While I've been writing this, it suddenly came to me that the (un)serialized typeids may get broken if RTTI changes. And more...
There are following major cases here that I may think of:

1. New types added or their order changes, in which case the numeric IDs may shift. To counter this we may write a simple typename-typeid (string to number) map into the save. That should not take too much extra space. When restoring a save, we first use typenames to find matches in the current game's RTTI, and then use this map of matches to remap loaded user objects and dynamic arrays to the new RTTI. This is similar to how individual script's RTTI is remapped to a global table on script load.

2. Existing type names changed in script. That will break the object matching completely, because there will be no way to find out if it's the same type or not. I don't know if anything may be done with this, or should be... Given problem 1 is resolved, I might throw an error if no match was found, or size have changed, etc. But this is somewhat annoying... because this is a new restriction that did not exist before.

3. The struct's contents change... Now, this could be an issue before as well, where a restored object data could've been less or larger than the type's, or contain fields in an old order. After this addition, it will also may cause memory leaks, if pointers are added, removed, or reordered, as the RTTI will contain the new information. Hypothetically, this may be resolved by storing RTTI, or necessary bits (like managed field offsets per type) in game's save. But I don't know if all this is necessary at this point. EDIT: although, this may come necessary for Version-independent save games #1371, where a save file would have to have a table of types and field names/offsets probably.

[ivan-mogilko]

Eh, thinking more about game saves problem; there's another major issue: the types declared in room scripts.

Because no room scripts are loaded at the time when the save is restored (and only 1 particular room will be loaded later), we won't be able to "match" the objects created from room types.

So, the only solution I see at the moment is to have either full RTTI written in saves, or at least a limited pieces of data necessary for supporting the features (like nested pointers in our case). Then, upon restoring a save, create "temporary types", and match & remap these as more scripts are loaded.

I'll have to think more about this.

[ivan-mogilko]

Hmm, well, with the latest changes the "saves problem" seems to be formally resolved in a minimal necessary way; but the code is quite ugly, so I'll take couple more days to tidy it, and then write a complete explanation of what it does.

Note: by "minimal necessary" solution I mean handling following cases:

• type's numeric id change due to reordering or adding new types;
• objects of types declared inside rooms. For example: there's an object of type declared in room A, the save is made, then quit, restart game and restore a save in another room B before room A was ever loaded.

[ivan-mogilko]

Alright, it took me unexpected extra time to figure out something about restoring saves (because I did not realize this problem early enough), but I suppose that current implementation has a "minimal necessary" solution for it. By that I mean that it will formally work, except for a few niche cases. It will definitely work for a unchanged (finalized) game script. There are options for enhancing this in the future, both resolving some of the niche cases, and improving performance.

But, general words aside, following is a quick recap of problems with saves, and how the engine will deal with that now.

1. As the script compiler passes typeid as a number in the new opcode, this number is local (relative) to the current script. That's why it's called a local typeid. Local typeids are resolved into global ones at runtime, because this is when we have a reliable joint RTTI table (this is similar to how global imports are resolved). This is done using their actual type names.
2. All the managed structs, and arrays, are created by storing these global numeric ids. They are also saved and restored with these global ids.
3. The problem arises if the game script was changed between save and restoring that save. Both local and global numeric ids are tied up to the order of registration (declaration at compile time). This means that if you move the types around, or add more types somewhere, the order of previously existing types registration would change.
4. The second problem comes from types declared in room scripts. As room scripts are loaded only when player is in the room, there may be objects in memory which have such "room type" loaded before the room was loaded, so we basically do not know this type yet. This problem is specifically important if the room's type is a child of a parent global type. In such case the object of room's type may be assigned to a pointer of parent's type in a global variable, addressed by the global script, and even deleted in the global script. And we must know some info about the actual type for deletion (so that we dispose all the nested managed pointers properly).

This is how I tried to solve this.

1. Write a "stripped" global RTTI into the save. In theory we could dump whole RTTI there if necessary, but I went with a partial for now, to save disk space. For instance, it does not write full struct fields info, but only the precalculated list of managed pointer offsets per types which have these. The RTTI is written in a separate save component, so it will be easier to change or completely removed, for example, if another solution will be wanted in the future.
2. When restoring the save, this RTTI will be restored, and joined with the current global RTTI. This does two things: if a type with the same name already exists (normally that would be every type declared in the regular script modules), then it only adds the restored -> real typeid conversion to the map. This map is then used to remap typeids in all the loaded managed objects (that need it), so they will now contain global typeids corresponding to the current game.
3. If a type is not known yet (will usually happen for types declared inside a room script), then it adds a placeholder type (I called them "generated") marked by a special flag. If this placeholder type contained any managed pointers, it will also generated pseudo-fields for this type, just enough for the engine to subref these on object's disposal.
4. As the game progresses, and rooms are loaded, the room's own RTTI will be further joined, and once we have a real type from the room script, they will replace the "placeholders", while keeping the global typeids (so no more typeid remapping is needed).

This way

• the managed objects always have typeids corresponding to the current global RTTI table
• the objects of types declared in unloaded (room) scripts may still be safely disposed.

[ivan-mogilko]

Well, I think this is working, overall.

I made certain that the new opcodes and RTTI generation itself could be disabled completely from compilation, in which case compilers will write old opcodes and also error on nested managed pointers (see compiler options SCOPT_RTTI and SCOPT_RTTIOPS. So the whole feature may be turned off if something bad happens.

It is also possible to write a fallback in script, using SCRIPT_EXT_NESTEDPOINTERS macro. (Although that's rather a formality, as it may be difficult to write two full variants of your script with and without this feature.)

The object disposal is fine, at least according to my tests. My only doubts at the moment are related to saves problem, as mentioned above, but this is maybe because the whole situation about AGS saves relying on exact game data state is ugly. The implemented solutions results in save files being several tens of Kilobytes larger, and having extra processing after the load. It might be possible to optimize probably, and also possible to replace with some other solution at the cost of changing the save format in particular component.

I think the biggest TODO after this will be to implement some kind of a managed pool statistics, which are gathered and optionally printed somewhere. Testing proper disposal manually may be a tedious task, and only suitable if you have few big objects in memory - then you can clearly notice the memory change using trivial system tools. But this kind of test may not be suitable for a regular game run.

In regards to this PR, the only thing remaining is fixing compiler tests.

[ivan-mogilko]

After the #1990 the script performance got improved significantly, and it's time to compare once again:

Testing the 3D space sim prototype
Initial 3.6.0 release: ~65 fps
Latest master branch: 88-90 fps
Latest ags4 branch, old compiler: 87-88 fps
This PR, old compiler: 85-86 fps
Latest ags4 branch, new compiler (optimized bytecode): 110-111 fps (@fernewelten :) )
This PR, new compiler: 107-109 fps

So, it looks like with this feature there's a stable drop by 2-3 fps, but compared to the new performance level it's mere %. As i mentioned above, there are couple of places which may be optimized probably in the future.

---

Also tested racing game, it's a little difficult because the fps jump all the time, but the "stable" highest values are:
Initial 3.6.0 release: ~150 fps
Latest master branch: 210-215 fps
Latest ags4 branch, old compiler: 200-210 fps
This PR, old compiler: 200-205 fps
Latest ags4 branch, new compiler: 225-235 fps
This PR, new compiler: 223-230 fps

[fernewelten]

From originally 65 fps up to 110 fps: that's great improvement!

So the combination seems to do it best – a more efficient bytecode interpreter and more efficient bytecode.

[fernewelten]

I'm not quite up-to-date yet: Do we do both reference counting and trace-and-sweep garbage collection to reclaim unused memory? If so, can we perhaps do away with the reference counting and only rely on the collector? Because I think I read somewhere some time ago that reference counting can potentially be slower than garbage collecting, amortized over bytecode instructions executed. I don't know whether that could be relevant in our system.

[ivan-mogilko]

I'm not quite up-to-date yet: Do we do both reference counting and trace-and-sweep garbage collection to reclaim unused memory? If so, can we perhaps do away with the reference counting and only rely on the collector?

The GC I wrote uses reference counter to find out which objects are still attached to pointers in script (or from internal storage in the engine). It uses the idea described here: https://devguide.python.org/internals/garbage-collector/

In order to do this without ref counter, we'd have to keep records of all the pointer variables in global script memory at least, and then trace managed objects starting from those variables that have a valid handle. From what I understood this uses concept of a "gcroot" object. For that to work we'd need to generate a list of pointers in script data, similarly to how the RTTI is generated now. This was my first thought, but I decided to use gc refcount method described in the linked article, as it did not require any additional script data.

I guess later someone could experiment with the "gcroot" option and see if it works faster in AGS games.

One thing to consider, because the nested or cross-referenced managed structs were not a thing before, right now there are ZERO released games where detached objects can be a thing, so there's practically no real test case ready.

[ivan-mogilko]

@fernewelten oh, by the way, there are not explicit bytecode instructions that do ref count change. Ref count is changed on pointer assignment, that would be MEMWRITEPTR, MEMINITPTR and MEMZEROPTR. So, there's no bytecode to save, only internal calls to the managed pool.

[ivan-mogilko]

Well, this is generally working, the only issue I see here is testing. Manual testing will only show that the pointers themselves work.

What could be a convenient way to add some automated aid here? For example, I could add a compilation flag for logging managed pool stats, and then print number of allocated objects, once per N object allocations.

[ericoporto]

It all depends on the problems you want to detect, so what kind of problems the GC can have?

• an object is never collected, but should
• time taken to run GC methods (maybe they could eat a significant time at some cliff of objects/relationships?)
• something memory related (I think this would be hard to measure without special allocators so we can skip this)

In general my head can't think many possible things that could go wrong there - beyond a case where the behavior itself is wrong, but the n I don't know how logging something could help.

There could also be something similar to Ctrl+G to print some stats in a Display (maybe add to the Debug(a,b) things).

[ivan-mogilko]

It all depends on the problems you want to detect, so what kind of problems the GC can have?

Primarily the correctness of GC and memory leaks, to know that this feature works formally at least.

[ericoporto]

In theory, if we could run it on a script and get some picture of it's state, say before and after some script point, we could write a test to verify it's not incorrect.

If there was some way to dump the GC state to an String at runtime these tests could be written in AGS Script itself with regular Editor and Engine.

Problem is, I think the different states even on an empty game would have a lot of elements you don't care I think, so one may need to figure something to parse out the parts of the state that don't matter. Alternatively these could only spit these states to a text file.

Of course this testing, for now, is something we would make and run at one time, and not something to continuously verify games in general, I don't have many ideas on that.

[ivan-mogilko]

Added some trivial collected stats to the managed pool, printed every now and then after N GC runs, on room change and game exit.