Implement qextensions

[g0mb4]

Quake Extensions

1. About

Quake Extensions (qextensions) are Dynamic Link Library (DLL) files on Windows and Shared Object (SO) files on Linux. They allows external code to be executed within QC programs.

In theory it works on macOS using the Dynamic Librariy (dylib), but I do not have a Mac to test with.

2. Usage

The extensions cvar must be set to a non-zero value to enable qextensions. If it is set through the command line +extensions 1 must come before +map.

In order to use qextensions the new built-in functions must be defined.
This usually happens in defs.qc:

...
float(string path) OpenExtension = #93;
string(float id, string cmd, string arg) CallExtensionString = #94;
float(float id, string cmd, float arg) CallExtensionNumber = #95;
vector(float id, string cmd, vector arg) CallExtensionVector = #96;
float(float id, string cmd, entity arg) CallExtensionEntity = #97;
...

To use a specific extension e.g. hello, the following steps needs to be made:

1, Open/load the extension:

This needs to be done only once. The void() worldspawn is a perfect place to do that:

    ....
    float hello_ext;
    ...
    void() worldspawn = 
    {
        ...
        hello_ext = OpenExtension("hello");
        ...
    }
    ...

Note that the value returned from OpenExtension is the extension descriptor. It needs to be saved globally, since it will be used for calling the extension.

2, Call the extension functions:

    ...
    num = CallExtensionNumber(hello_ext, "hello", num);
    ...

This call is blocking, meaning that the game will wait for the extension to return before continuing. This may cause FPS drop if extension is not optimised. If extension takes too long, consider making asynchronous extension, where the result of the work of the extension is collected in a separate call.

There are 4 function that can be called:

    string(float id, string cmd, string arg) CallExtensionString;
    float(float id, string cmd, float arg) CallExtensionNumber;
    vector(float id, string cmd, vector arg) CallExtensionVector;
    float(float id, string cmd, entity arg) CallExtensionEntity;

The functions can not modify their arguments. More then one qextension can be used simultaneously.

3. Creating

qextensions can be created using any programming language that can output native .DLL/.SO files. This example code is written in C.

The qextension must export exactly 5 functions:

    int qextension_version (void)
    const char* qextension_string (const char *cmd, const char *arg)
    float qextension_number (const char *cmd, float arg)
    float* qextension_vector (const char *cmd, vec3_t arg)
    float qextension_entity (const char *cmd, edict_t *arg)

In Windows the function must be preceded with __declspec (dllexport) if MSVC is used.

The qextension_version must return 1. It is the version of the qextension system, not the specific extension.

Since the qextension is a native file, different platforms require different library files. OpenExtension("hello") will try to find the underlying library file depending on the platform quakespasm is running on:

• hello_x86.dll on 32 bit Windows systems

• hello_x86_64.dll on 64 bit Windows systems

• hello_x86.so on 32 bit Linux systems

• hello_x86_64.so on 64 bit Linux systems

If support of multiple platforms is required, each library file must be installed. This requires cross-compilation or different build systems.

4. Installation

qextensions use the filesystem of Quake. They can be placed into the game directory or embedded into a .PAK file, e.g.:

• id1/hello_x86.dll -> OpenExtension("hello");

• id1/pak0.pak/hello_x86_64.dll -> OpenExtension("hello");

• id1/ext/hello_x86.dll -> OpenExtension("ext/hello");

• mymod/hello_x86_64.so -> OpenExtension("hello"); (only if -game mymod was used)

5. qext_hello

This is an example qextension. It supports 64 bit Linux and Windows systems.

• src/ contains the source code of the qextension
• qsrc/ contains the source code of the mod uses the qextension

5.1. Building

qcc is required to be in the $PATH/%PATH% in order to compile progs.dat.

5.2. Linux

gcc is required to compile the qextension. Simply type:

make

A Win64 version also can be compiled, it requires the gcc-mingw-w64 toolchain.
Remove the following line from Makefile, if it is not present in your system:

cd src; make -f Makefile.w64

5.3. Windows

Open an x64 Native Tools Command Prompt for VS X ant type:

build.bat

5.4. Running

To run the hello qextension copy the generated hello folder to the folder of quakespasm (where id1 lives) then type:

quakespasm --args -game hello +extensions 1 +developer 1 +map start

If you see the outside of the start map the extension is working. Open the console and you should see:

world of Quake
11
'    2.0    3.0    4.0'
1

[sezero]

CC @ericwa, @andrei-drexler, @Shpoike

[alexey-lysiuk]

Taking security considerations into account, it would be nice to have a way to disable this feature.

[g0mb4]

Sure, good idea.

[vsonnier]

Maybe put the present dlopen-like code hidden away in pl_linux/win.c ?

[g0mb4]

Dear @vsonnier! I like the idea. I did not know about the SDL functions, but they allow more portable code as I see. If I drop the mechanism to map a memory region as a file I can merge the platform-specific code. I'll experiment a bit.

[vsonnier]

The problem with SDL_LoadObject is that it won't be able to load from a Quake .pak file, is it a problem ?

Small detail remark : Replace the super-generic extension CVAR and naming around the dynamic QC functionality with something more explicit, idk pr_dynamic_extension

For instance, we have pr_checkextension 0/1 in vkQuake/QSS to enable or disable support to FTE extensions, where pr_ stands for "program" to refer to QC logic.

[g0mb4]

I'd like to keep the ability to load from .pak files, I think it is neat.

Thank you for the suggestions, I tried to follow the style of the code, but yep, the names are a bit generic.

FTE extensions? Am I reimplementig the wheel?

[vsonnier]

FTE extensions? Am I reimplementig the wheel?

Think of it at an additional extended library of more extensions, but still hard-coded in the regular C source :

https://github.com/Novum/vkQuake/blob/ea6d5eb86e5186b622f3ddf909ba57233a79430f/Quake/pr_ext.c#L4999

In mods like Arcane Dimensions, QC checks dynamically for the existence of a particular extension "category", listed here :
https://github.com/Novum/vkQuake/blob/ea6d5eb86e5186b622f3ddf909ba57233a79430f/Quake/pr_ext.c#L5278

Ex, in .qc Aracane Dimensions pesudo-code:

if (checkextension("FTE_PART_SCRIPT") {
  // call qc FTE api to have FTE particules effects and such.
} else {
   // use classic Quake particles...
}

So for ports supporting FTE particles and such, Arcane Dimensions effectivelly show sophisticated particles effects, and classic particles on other ports.

With your improvement, it goes beyond that and new APIs are part of the loaded native library + compagnon .qc code, not the executable so it adds flexibility, unless I'm mistaken ?

I suppose with additionnal effort one mod may implement 2 variants of code like Arcane Dimensions did to be compatible with non-supporting ports, but that seems a lot of work.

[sezero]

I'd like to hear others' opinions, but yes, this does feel like re-inventing the wheel.

Close this? Any ideas to pick from this?

[g0mb4]

Thank you for the detailed explanation!

Yes, the "logic" moves to the dynamic library, where the QC calls only provides input.

Back in the day I implemented some basic websocket support for Arma3 this way, this is where I got the idea.

[vsonnier]

In any case, this solution has the same drawbacks as Quake 2 / Quake 3 mods : we need to ship native platform-specific binaries.

The advantage of .qc bytecode is that it is universal and timeless. Once considered a drawback because "slow" leading Carmack to switch to logic implemented in native code with Q2/Q3, this is now a huge advantage I think.

Nowadays we have Q2 mods that can't be run with Q2 Remaster for this very reason, while all Quake 1 creations can be run on any port... (except if using specific extensions, leading to workarounds as above :)

[vsonnier]

I assume that such native extensions can't be Network-aware, (similar to Client-Side QC drawing HUDs and other things) otherwise we'll need to implement protocol extensions on top of that work, like FTE protocol did.

Of course, since it is native code we can do anything including side-stepping Quake clients communication by creating IP side channels, here protocol means the official Quake communication logic : https://quakewiki.org/wiki/Network_Protocols

Note that Quake protocol is also locally used in a single-player Quake client as a stream of commands that drive the engine itself.

[vsonnier]

Advantages of this solution, IMHO :

• Offloading calculation-heavy operations to native code,
• Coding in a familliar language.
• Easier to debug code using usual native IDE and tools ?
• others ?

[Shpoike]

this is likely to result in any mods using it to become platform specific (eg: no support for macs at all, thanks to how much of a pain they are to (cross)compile for). your 'qlib.h' file even has `//johnfitz -- was int` comments in it, which should be a pretty strong indication that even quakespasm has changed this stuff and will likely do so again. in terms of cross-engine and forwards compat this is about as flawed as abusing qccx hacks were. external extensions like this are just that - external. letting them see internal state will make qs maintenance harder, not easier, or even impossible for authors of other engines that might want to run a mod that supports this. externality also means any attempts to provide basic things like tracebox goes from just passing an extra pair of args into having to reimplement the entire tracing code - and without any access to any of the other entities that it might also impact, and probably leaking memory due to no shutdown/close events, and probably getting the wrong base/game dirs/map because none of that info seems to be provided either. having a cvar to stop people from using this stuff is pointless when a malicious mod can just localcmd that cvar to enable it. hiding this stuff inside pak files will hide the risks, while also giving virus scanners a harder time. your choice of builtin numbers conflicts with the registercvar/min/max/bound/pow builtins. your callextension* builtins requiring arg+return value to match, and only supporting a single arg each is clumsy and awkward. if all you plan on using this for is to work around quakespasm's continued refusal to support proper string manipulation then I can understand a use... but even then it still won't really be able to provide strzone etc easily, it'll end up as just a clumsier way to achieve what other engines already support, with easier native code exploit potential. for slightly more constructive criticism of your 'qlib.h' file's internal stuff, if it were replaced with passing just a pointer to the entvars and depending on eg ED_FindField instead of sharing the struct itself then that would be much more robust and forwards compatible, while also allowing for custom fields, too. Otherwise I wouldn't even be able to support this even in QSS (and _definitely_ not in FTEQW), I believe vkquake would similarly have problems. quite frankly, quakespasm would be better served by just merging in support for some of the qc extensions that already exists in quakespasm's forks, instead of something that would be broken in both upstream and downstream forks. as an alternative I've recently been compiling C code directly with fteqcc, specifically doom and vanilla quake (thanks to FRIK_FILE for reading their data files, and fte's r_uploadimage builtin for updating textures to display the resulting graphics onscreen). compiling those game engines into the .dat itself. It 'just' needs a few extra builtins(for a small libc) and a 'few' extra opcodes (primarily int ones as well as uniform ev_pointer/ev_string casts and the ability to load from pointers without having to hack at entity references first). it might be fun to use qccx-style hacks to emulate some of this stuff... it should probably be noted that quakespasm still has the same flaws that proquake does, allowing for all the same 'qccx' hacks (just with different offsets - yay for forwards compat issues), where a .dat can just exploit its way outside of the vm and stomp on the rest of the engine's memory (hence why qss still does not download csprogs.dat files from the server). that stuff needs to be fixed either way. probably I'm going to eventually end up just stripping out qss's qcvm and throwing fte's qclib in there instead, fixing the sandboxing while also giving it more opcodes to play with. 'eventually' being the key word there, sadly.

…

On Thu, 16 Jan 2025 at 21:52, Vincent Sonnier ***@***.***> wrote: Advantages of this solution, IMHO : - Offloading calculation-heavy operations to native code, - Coding in a familliar language. - Easier to debug code using usual native IDE and tools ? - others ? — Reply to this email directly, view it on GitHub <#119 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG2Z7TKXRQ5W2UME7FYO2Z32LAS3HAVCNFSM6AAAAABVJXV4ICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJWHE3TONRSGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[vsonnier]

Right, I've looked a little more closely and wondered how you could load a function with an arbitrary signature, for universality...

Well there is no miracle and you have to know the signatures biforehand to import the symbols. So if I read it correctly, the only functions your current API implementd are the following you listed:

float(string path) OpenExtension = #93;
string(float id, string cmd, string arg) CallExtensionString = #94;
float(float id, string cmd, float arg) CallExtensionNumber = #95;
vector(float id, string cmd, vector arg) CallExtensionVector = #96;
float(float id, string cmd, entity arg) CallExtensionEntity = #97;

Well in this case this is very limited indeed, and if you later grow the amount of signatures I'm afraid @sezero is right :

This is reinventing the wheel and it should be much simpler to add existing QC extensions like FTE and such. That would be equivalent to merge QSS into QuakeSpasm, so why not use QSS directlty ?

Unless we manage to call arbitrary/variadic signature from QC and back, I don't see how it could work.

[g0mb4]

It seems like the community has decided, thank you for your time, though.

[sezero]

OK, closing. Thank you.