Editor: prepare Linux package with executable flags set

[ghost]

The issue was explained on AGS forums before:
http://www.adventuregamestudio.co.uk/forums/index.php?topic=54571.msg636557425#msg636557425

AGS Editor supports preparing game files for Linux with prebuilt binaries (AFAIK these work on a range of "common" systems like Ubuntu, Debian, and so on).

When Linux package is prepared under Windows, these binaries loose "executable" flag, and that have to be manually fixed by the game developer on Linux machine before creating final distribution archive.

The solution may be to (optionally?) zip the game files on Windows into Linux-friendly archive format and set executable flag to appropriate binaries inside the archive.

[ivan-mogilko]

@morganwillcock do you know if this was done since, and if not, is it possible to do?

[morganwillcock]

It isn't done, but if doing it internally to the editor or with C# build tools then it could be done like this:
#658 (comment)

I think it is better done with separate build tools. Hopefully the separated project build system would allow custom hooks for build steps as well as project metadata access, and with this you invoke the build tools and optionally your own scripts. Given lack of flexible archive tools on Windows, this specific task is probably best to be integrated into build tools directly, but technically it can be done externally and this becomes easier if project metadata is more easily available.

[ericoporto]

I don't think we should do this by default, Steam is one that I know picks the files directly from hard drive and applies the necessary flags.

If the Linux build outputs a tar.gz file, then the person has to extract the files and then upload, and extract tar.gz files are a hassle in Windows. And only after the files are extracted then the person can do the upload.

Also for people running AGS Editor on Wine, since debug builds are currently broken on Wine, then the person won't be able to build and run the game using the Build Executables functionality from the Editor without first unpacking.

I would suggest either making this an optional setting on the game project General Settings or postpone this until we are looking into command line tools builds.

[morganwillcock]

If it is implemented right now everything has to be staged to disk anyway, because some of the files that are part of the game are potentially just files that happen to be have been manually copied into particular folders.

[ericoporto]

I have been thinking about just adding bsdtar.exe (from here) in the AGS Editor directory and just running something like @morganwillcock ags-tar.cmd but like, a c# version in the Editor that would call bsdtar.exe...

By calling the exe it may be a good time to think how calling command line tools could be handled - we could have a simple read-only console window to get the command line utility output printed.

Other alternative is to move to dotnet 7 or 8, which have TarEntry support, so we could create tar files from using the standard C# library. It will be necessary to upgrade the Visual Studio for this in the docker image... (see #1508)

[morganwillcock]

I think there will be edge cases around which files to include in the archive, because in some cases files the game uses will be automatically included, and in other cases the user will have to manually update the contents of the compiled directory before distribution. I.e. unless it is safe to delete the compiled directory and have the build process fully recreate it, it wouldn't be possible to just create an archive of that directory for distribution.

It is really part of a wider problem that the project structure and build processes are inconsistent in how files are managed, and without that being fixed I don't think it will ever be safe to assume the compiled platform directories are in the correct state at the end of the build process.

Or to put it another way, the current project format defines what is required to build, but doesn't allow proper definitions of what needs to happen for distribution.

[ericoporto]

I don't understand, every file in the directory goes into the package file. What I asked was what are the feelings around just distributing bsdtar.exe and calling it vs adding an actual C# (or native) library. The advantage I can see of calling a command line tool is it won't eat in the memory restrictions of the 32-bit Editor.

One good thing or having some functionality to add a second step where it auto zip/tar/... things is I see a ton of Windows only distributions of AGS games where people also zip the Data directory, doubling the size of their games in the package unnecessarily. Since there isn't a proper build menu of sorts, I couldn't figure where to put a warning that one doesn't need to ship the Data directory.

[ivan-mogilko]

I don't understand, every file in the directory goes into the package file.

I think what @morganwillcock means is that user may need to add something by hand into the Linux distribution, after the build has completed, and update something by hand without rebuilding whole game.

The Editor now lacks a distinction between two major steps: compiling common files and making a package per platform. It does so in one internal step, but user may sometimes need to be able to do something in between. There's no way to configure this currently.
Perhaps there are other things that I might be missing.

In other words, if we add automatic archiving, then we need to make sure there's a way to either disable this step, or that it won't prevent users from doing some extra changes to the output files.

[morganwillcock]

The way I imagine it working is that the "project tree" would gain additional nodes for data or files which are required to build all variants of a release. For example, if you want to include a text file or a PDF file you would define that in the project, including where to copy those files from (the project directory is a valid and recommended source, no magical caching needed), whether the file should be copied normally or embedded into the game data, and which file-system permisions should be set on them (where applicable). If the files are platform specific then the build settings for the platform need to allow for the selection of which of these files should be included.

I think this way also allows the removal of the current special actions to copy files into the game data. As far I can tell, any .ogv file in the top level of the project directory gets included into the game data whether it is used or not, which doesn't seem very safe (it happily includes anything with a .ogv filename, even when it isn't a video).

It should also simplify the process of changing the game name. The entire "Compiled" directory could just be deleted since it can now be recreated.

For creating an archive with correct file permissions set, the project itself now has the information and can either expose that to an external tool or implement the archiving itself.

[ivan-mogilko]

The way I imagine it working is that the "project tree" would gain additional nodes for data or files which are required to build all variants of a release.

I think that the same may be achieved by having either subdirs of predefined names within the project, or list of files in text files of predefined name; in which case the node in project tree would be an optional UI frontend for these lists.

There seem to be cases where having an inclusion rule is more convenient rather than adding item per item into the project tree. For example, if you're making "levels" for your game, and add/remove/change files often.
Also see #2063.

[morganwillcock]

I think it depends on what happens if the project state is accidentally modified; trusting directory contents to be correct means that it isn't possible to tell when they are incorrect.

If there is a tool which is meant to build the game and package it for distribution I think it is preferable if this process fails when required files are missing or unexpected files would be distributed. Even in the best case where someone is very careful there would likely need to be explicit exclusions for temporary files and version control files/directories.

For example, if you're making "levels" for your game, and add/remove/change files often.

An explicit include seems much safer, and could be a glob match to include the contents of the directory, at the user's risk.