Editor: change game project's Rooms to open format

[ghost]

Current situation is this:

• in the game project rooms are stored as compiled binary blobs, which include room description, background images, area masks and compiled room scripts.
• some of the operations (need to reinvestigate to see which ones) in the Editor rely on C++ code in AGS.Native dll, where rooms are accessed in their runtime format.

This causes number of issues, including inability for multiple users to work on same room in parallel (because changes are impossible to merge).

The suggestion

1. All the room management should be reimplemented in .NET.

2. Have rooms in the game project serialized as a group of files:

• XML for the room description (we have a ready generic serializer for that);
• image files for backgrounds and masks;
• room script is already available in text source, so probably nothing should be changed here (perhaps, that may have sense to cache compiled room scripts, but that's another problem).

Possible steps

1. We already have a Room class in .NET application. We need to research how Editor works with rooms, and make sure that only that and accompanying classes are used in all room editing operations. In the end, Editor should not rely on AGS.Native.dll when working with rooms (except for compilation, for now).
   Editor should be fully able to load up a Room class from the runtime format.

2. Editor has a class-to-XML serialization mechanism based on reflection, which is currently used to save main game settings. Same could be used to store rooms. Implement Room class serialization using that serializer.
   NOTE: if it would be desired to use other format than XML, then we'd only need to modify serializer later. Then all game data will save in a new format at once. Of course in such case we'd additionally have to implement conversion from the old game projects.

3. Since room source will probably consist of several files, I'd assume we need to move them into their own folder (e.g. "Rooms") and maybe even have subfolders per each room there. So, the next task is to invent a good file/folder structure and complete room serialization in the new way.

4. Necessary changes to game compilation: currently it relies on updating binary room files. This again need to be researched, what would be better - to keep cached binary rooms (or parts of these) and update them as user saves one in the editor, or fully rebuild a room every time when game binaries are generated.

5) Bonus step: reimplement room compilation in .NET app.
UPDATE: it appears more desirable to have stand-alone compilation tools, so the room files -> CRM compiler could as well be a stand-alone tool run by the editor. In this case it may as well be written in C++ for max platform compatibility.

[ghost]

Editor already uses game sources in XML. We have a generic serializer class, so there won't be any new code related to XML, we may use same one.
Similarily, if we are going to change format to something else, only serializer has to be modified to write into different format.

PS. But change of format will only complicate this task, because then we'd have to support multiple ones in one project, or change the rest of data, which is beyond the scope of this particular issue.

[ghost]

Okay, I renamed the ticket to just "open format", not to accentuate on XML in particular.
Also elaborated task steps.

[morganwillcock]

I'm also not too keen on XML (too many diagonal lines for my eyes), but that is easiest in the short term. Ideally if the data files were not full of padded empty values, using something like YAML would probably let you write the room data by hand (at that point just a cross-platform game compiler is enough to make a game on another platform).

Rooms being folders on disk is good, as masks can just be files within there (for direct editing, rather than import / export).

[CisBetter]

Ideally if the data files were not full of padded empty values, using something like YAML would probably let you write the room data by hand

Indeed, YAML is very easy for humans to write, relatively easy for computers to analyse, and contains more features (like comments) than JSON. The only real problem I always had when writing configuration files in YAML was most parsers' inability to understand tabs and a relatively loose specification that makes some parsers parse scripts in ways you wouldn't expect (I don't think this is going to be a huge problem though).

(at that point just a cross-platform game compiler is enough to make a game on another platform)

Moreover, having a cross-platform GUI is by far more convenient than just telling developers (and artists!) to write their games in YAML. Many choose AGS because of its simple and intuitive UI. Even though creating a cross-platform editor might be easier with YAML room files, forcing people to create games in "Notepad++" (or Gedit etc.) would make AGS as useful as ScummC (which I like btw.; still people don't use it because of its limited possibilities).

[persn]

Experimental work. For now it just takes all the room data and dumps it to the project folder in a sub-folder "Rooms" when the project is loaded into AGS studio. https://github.com/persn/ags/tree/feature/open-rooms

[ericoporto]

@persn can you throw some generated room examples here? Just .zip and upload in this issue comments itself :)

[persn]

rellax_room_data.zip

[ericoporto]

This is magical, great work! Is there anything missing I am not seeing?

It appears this could be input to generate the room binary that the engine consumes - except I think for the headers that are on it? I think it uses to link them in the room Script in some way.

[persn]

I think I got it all. If there's something missing I need to know about it so I can add it. What are these headers you speak of?

We'll see what blows up when I get so far I actually try using this data, but I'm not quite there yet, soon I think, but not yet. Today I started experimenting with the "Export room background" button. It's low hanging fruit, but it allows me to strip away some of the old AGS.Native code which will now become obsolete, and that in turn means there is less code for me to analyze when I try to figure out what does what.

[ivan-mogilko]

Naturally, room script requires all the other script headers for compilation, including built-in API header and any plugin script headers, and so on, if this is what you are refering to.
This is what #1007 is for.

[ericoporto]

0deb8ae...persn:feature/open-rooms

The headers for the Rooms could be written at the Rooms/ directory (for the issue mentioned).

Then a possible next step would be use DeserializeFromXML to load the room from the xml room files if they exist.

And saving the room itself would not use the native proxy: Editor/AGS.Editor/Components/RoomsComponent.cs#L603

With this, if the crm files need to be built it could happen just in the process that happens when the user hits compile.

[ivan-mogilko]

Sorry, but I still cannot understand which headers are you referring to?

I am completely puzzled; if script headers then why writing them into Rooms directory if these are global and not only for rooms? If this is something else then please elaborate.

[ericoporto]

No worries. I noticed that when loading a .crm in an hex editor to inspect it's contents there's a section where some headers appear. That's why I thought these were needed. But you are right, they are global and not only for rooms.

[ivan-mogilko]

I guess this is part of compiled room script, bundled into crm.

You can find what room consists of in room loading code here:
https://github.com/adventuregamestudio/ags/blob/master/Common/game/room_file.h
https://github.com/adventuregamestudio/ags/blob/master/Common/game/room_file.cpp

notice enum RoomFileBlock, this is supported content blocks.

[ericoporto]

From reading the source it appears the headers in the crm file are not used - at least in the engine... I did a test removing them from the .ags file generated (using the hex editor) in the rellax project and nothing broke. :/

[ivan-mogilko]

These are header names which are inserted into script bytecode either for reference or debug purposes perhaps. Headers themselves become a part of the compiled bytecode during compilation, just like in C or C++ for example.

[ericoporto]

The editor would operate text data and the text->binary could be done by a separate tool.

To clarify, for this to work #1138 needs to be merged since it's part of this path to operate the text files on the Editor.

I just don't think that a future text (xml descriptors + images) -> crm build would necessarily be in the Editor side of the code. I am not opposed to this tool being written in C# if it is done as a separate command line tool or library (that one can write a command line front end later), because then it would not need winforms.

I have an implementation sketch idea here that is just using cpp because it was what I was more comfortable with at the time. For now though the only things it does is bundle script binaries and I used it to test running and compiling ags script in other OSes.

https://github.com/ericoporto/ags/tree/min-ags-compiler/AgsTools/RoomTool

[rofl0r]

I have an implementation sketch idea here that is just using cpp because it was what I was more comfortable with at the time.

i too think that anything that involves command line tools for building a project should be implemented in C++ instead of C# because getting mono to work on non-windows platforms is non-trivial.

[ericoporto]

C# now doesn't need mono since newer dotnet works seamless on Linux and it can be convenient to operate on text files because of reflection. What I meant is just having data->binary rules not mixed with Editor code would be useful.

[persn]

Yeah the plan was that the editor is responsible for room compiling until someone replaces it with CLI tools, and I will refactor away as much as possible of the other room handling code which should hopefully make it easier. I think I remember telling you as much on discord months ago. And C or C++ is probably the right tool to implement that, because the struct data from the crm files already exists in header files.

Also cross platform C# these days is pretty trivial, unless you mean upgrading the editor code base, but C++ is probably still the right choice for the task being discussed.

[ivan-mogilko]

Resolved by #1278 in ags4 branch.