Tool: Generate script files from Game.agf

[ivan-mogilko]

Summary

A stand-alone tool which may be run from command-line (no GUI), which parses Game.agf xml and generates script files necessary for game compilation.
Presumably C++ is better choice here, as it's more portable than C#.

NOTE: this task is exclusively in writing this tool, adjusting Editor is a separate task, so no need to concern yourself with that if you are doing this. In fact it's best to assume that Editor will not be present to ensure tool's work result has no reliance on it.

Input:

• an xml file in AGS 3.* project format (path to file);
• path to output directory;

Output:

• _AutoGenerated.ash - contains declarations (imports) of global game objects, arrays, macros and so on;
• _GlobalVariables.ash - contains imports of global variables found in "GlobalVariables" xml section;
• _GlobalVariables.asc - contains definitions and export statements for "GlobalVariables";
  Optionally?
• dialog scripts, each in a separate file (to be converted and merged by a different tool as a later step) - verbatim content of <Dialogs> - <Dialog> - <Script> xml sections, excluding <![CDATA[ tag; need to figure out how to call these temporary files to avoid conflicts.

---

Details

_AutoGenerated.ash

Process could be found in the Editor code:
https://github.com/adventuregamestudio/ags/blob/master/Editor/AGS.Editor/Tasks.cs#L526
(RegenerateScriptHeader function)

Note that the header takes global declarations from game data itself (read from Game.agf), but also room declarations from the given room. This is because room scripts need both.
My proposal is to ignore room declarations for the purpose of this tool, and only make it work with global ones. The generated _AutoGenerated.ash may then be used everywhere, and for room scripts there will be a separate tool (or separate task) which reads room files (CRM) and creates additional headers per room.

_GlobalVariables.ash/asc

Process could be found in the Editor code:
ash: https://github.com/adventuregamestudio/ags/blob/master/Editor/AGS.Editor/Components/GlobalVariablesComponent.cs#L89
asc: https://github.com/adventuregamestudio/ags/blob/master/Editor/AGS.Editor/Components/GlobalVariablesComponent.cs#L57
(UpdateScriptHeader and GenerateGlobalVariablesScriptModule functions)

To be amended.

---

See Also

#1007 - a previous ticket on subject, I guess it may be treated as a separate task for replacing Editor's code that does all the above with calls to standalone tool(s) when it's ready.
General overview of the game building process: https://github.com/adventuregamestudio/ags/wiki/AGS-Game-Build-process-(3.5.*)

[ericoporto]

I can write this. The best way in my view is to remove Winforms bits from AGS Types, since AGS Types has the rules for Game.agf, and write the thing in C# already.

If maintaining the rules for Game.agf in two places is not a problem, to quickly write the code I do have a .xsd schema for Game.agf, and boost has tools to generate the plain cpp classes for each .xml object/tag and it can then have the serialization/desserialization be switched to use Cereal to avoid boost - Cereal is tiny.

Then it's about navigating the class objects at runtime and writing the needed header files and script.

For this, I would separate the core functionality from the command line front end and allowing to return the needed files as string in the core and as files in the front end.

agsgen.cpp - cmd line front end
agsgen_core.cpp|hpp - library front end
agsgen_core/
           core_objs/
                     character.cpp|hpp
                     inventory_item.cpp|hpp
                             ...
                     ags_core_objs.hpp
            core_generators/
                           global_variables_gen.cpp|hpp
                           autogenerated.cpp|hpp
                           ...

This allows for validation when desserializing the Game.agf data.

If I didn't make sense in a sentence please ask about. I can also roll a little example of the above in the weekend if it would make clear to explain.

But I still think the best approach is fixing #1092 and using C# since most of the code is already written and would be shared with the current Editor, easing future maintenance.

[ivan-mogilko]

The best way in my view is to remove Winforms bits from AGS Types, since AGS Types has the rules for Game.agf, and write the thing in C# already.

This tool does not need to read whole objects. It practically needs one field from xml per object: the script name. Second piece of data is total number of objects. That's enough to generate the script headers.

For "global variables" it does need full data though (they consist of 3 fields).

I missed the previous discussion, what was the everyone's opinion on the language? Maybe I got wrong impression that C++ was chosen for this in the end. Is the primary reason for suggesting C# is existing code?

[rofl0r]

ivan, thanks for opening this ticket.

boost has tools

please, could we try not to pull in yet more huge dependencies into the project ?
if it's just a single header then there would be nothing speaking against it, but isn't xml already processed currently by C++ code? if so existing code/libraries should be reused.
other than that, the XML parsing libraries that are already installed on almost every system are expat and libxml2. though i think for this purpose even simple regex matchers should be sufficient (like the ones that should already be part of libc (regcomp/regexec) or whatever std:: has to offer). can someone attach a demo game.agf file for evaluation/testing purposes ?

[rofl0r]

I got wrong impression that C++ was chosen for this in the end

indeed, this seemed to be the consensus

[ericoporto]

Rofl0r, it's a single step once in a lifetime code generation to avoid writing code by hand.

[rofl0r]

it's a single step once in a lifetime code generation

you mean you'd use boost only for generation of some temporary data in a throw-away "script" ? and later that data is used instead of boost ?

[ericoporto]

It writes plain classes with validation of the data if they are present in the xsd schema. The resulting code has no boost in the classes but it has a little boost helper for serialization and desserialization of such classes, which can be cut out easily. I would then use a library to provide the serialization and desserialization instead (striping out any boost dependency) - there are a bunch with pros and cons, some sell themselves as reflection libraries instead of serialization/desserialization libraries. So the rest of the code just navigates plain cpp classes to obtain the data and doesn't have to know about the original data - let's say .xml is switched to yaml, or single big file is switched for multiple smaller files, it's easy to adapt.

The big con of this approach is maintain these classes.

In Java we use POJO (plain java objects) for describing these type of classes that are only meant to store data, but I don't know how to name these in cpp. :/

Anyway, maybe the plain text processing approach is preferable but I just thought to clarify what I wrote - not sure I did hahaha.

[ivan-mogilko]

Ah, no, I forgot ofcourse.

This particular tool only needs couple of fields from the xml, but this is only for script compilation. If we aim to do full game compilation with command line tools, then there will be next tool which must parse whole xml and write "main game data" from it. That tool will definitely need to have full knowledge of Game.agf.

PS. Please ignore close/reopen issue, I misclicked...

[ivan-mogilko]

but isn't xml already processed currently by C++ code

No, only C# code does this in the Editor.

can someone attach a demo game.agf file for evaluation/testing purposes ?

I have these on github:
https://github.com/ivan-mogilko/ags-camdemo/blob/master/Game.agf
https://github.com/ivan-mogilko/ags-lastfurious/blob/master/Last'n'Furious/Game.agf
but they are missing Dialogs and GlobalVariables, so maybe need to create a demo project with little bit of everything.

[ericoporto]

In the past I used oxygen when I was studying the Game.agf format in the past and made this visualization:
https://ericoporto.github.io/ags_schema/
The xsd generated from Game.agf that was used for that visualization is this: AgsSchema.xsd

It has some missing fields and I never hand corrected the .xsd. I ended up buying the previously mentioned tool, so correcting the xsd is possible if needed - it's really boring, but not hard because of the interface. Anyway, the visualization helped me read the format - a vertical monitor is also useful because some elements are huge.

[morganwillcock]

If this isn't C# then presumably it is OK if an additional input to the tool is a symbol representing a set of engine limits for a given engine version? All the constants used to define the array sizes are in AGS.Types.

[ivan-mogilko]

Sorry, looks like I did not get notification of the last couple commits for some reason...

If this isn't C# then presumably it is OK if an additional input to the tool is a symbol representing a set of engine limits for a given engine version? All the constants used to define the array sizes are in AGS.Types.

I did not think about that... Perhaps it could be a list of constants to parse instead? even if also in form of xml (since it parses xml anyway), or other simplier text format.
BTW, since engine has "--tell" feature now, could a way to generate these values be querying engine to retrive this data?

[rofl0r]

If this isn't C# then presumably it is OK if an additional input to the tool is a symbol representing a set of engine limits for a given engine version? All the constants used to define the array sizes are in AGS.Types.

this is actually something i looked into recently with regard to recompiling old games.

agsedit version 2.72 created a header for internal use that had roughly these contents:

        #define AGS_MAX_CHARACTERS 300
        #define AGS_MAX_INV_ITEMS 301
        #define AGS_MAX_GUIS 50
       enum CursorMode {
          eModeWalkto = 0,
   ...
};
 import Character character[1];
        #define EGO 0
        import Character cEgo;
        import Hotspot hotspot[30];

etc. full list in commit message here: rofl0r/agsutils@10e16bf

ags 2.62 otoh exported only a much smaller list of macros:

        #define EGO 0
        #define STATUSLINE FindGUIID("STATUSLINE")
        #define ICONBAR FindGUIID("ICONBAR")
        #define INVENTORY FindGUIID("INVENTORY")
        #define VIEW1 1
        #define VIEW2 2

this header was created in memory and then sent to the compiler.
i figured this out by compiling the demo game and then dumping the memory contents of the agsedit process.

does anything speak against creating the same kind of header to include by the build tools?

[ivan-mogilko]

does anything speak against creating the same kind of header to include by the build tools?

I am unsure if I understand the question; are you referring to including overall header when compiling game script, or suggest to make a precreated header with fixed set of macros for these tools?

[rofl0r]

suggest to make a precreated header with fixed set of macros for these tools?

yeah, this

[ivan-mogilko]

There seem to be two general alternatives here:

• simply provide this in a precreated header along with the editor release; perhaps even merge this data into API header.
• request engine to report this data (when it's related to its limits and capabilities), and generate such header using engine's reply.

First is ofcourse trivial and may as well be used as a starting solution.
The second will maybe potentially allow for one editor to work with multiple engines... or maybe not. I don't know if it's worth to follow this path now.

[rofl0r]

simply provide this in a precreated header along with the editor release; perhaps even merge this data into API header.

there are some generic parts in that header, like limits, but most of the generated code is game specific - view names, character names, etc. if the latter is still required then i don't think a generic version of the header can be provided. btw, i've seen references to "_BuiltinScriptHeader.ash" somewhere, and i think that's what this is.

[ivan-mogilko]

there are some generic parts in that header, like limits, but most of the generated code is game specific - view names, character names, etc

I think we are talking about different things then.
There are two parts:

• the symbols that are not game specific but related to engine capabilities.
• the game specific symbols.

This very ticket is about tool that should generate game specific symbols. I explained where to get and how to generate this information in the first post.
But what morganwillcock asked few comments above is engine limits symbols, and this is something I forgot about when writing this ticket.

PS. _BuiltinScriptHeader.ash is the file that contains script API, and it corresponds to this file from editor resources:
https://github.com/adventuregamestudio/ags/blob/master/Editor/AGS.Editor/Resources/agsdefns.sh
(_BuiltinScriptHeader.ash itself exists only in memory at the moment, it's never really written to disk afaik)

So the first variant is basically to put all missing generic declarations there as well, as it already has many of them.