exorbitant size/bloat of new ags games

[rofl0r]

most new games that come out are 100s of MB big. a big share of it are speech and audio files, which the authors usually ship in studio quality, i.e. 44.1KHz, 16bit stereo.
as an example, the new game metaphobia ships in a .rar archive that's > 400MB.
by simply converting the mp3 and ogg files in it to lower bitrate (while pertaining pretty good quality, using the instructions in agsutils' README) my repack here is "only" 80MB.

but that's still a lot. the game uses a resolution of 320x200, so it should be tiny. it seems all graphics are stored in 32bit color (and maybe even uncompressed), instead of 8bit and using a palette. At least that's what i assume, because i was debugging slow rendering speed and noticed that all allegro bitmaps in use (including the screen surface) used costly 32bit ops.

so my question is:
could this be improved by the editor, for example by counting the number of colors of all actors in a given scene as well as the backgrounds, and then creating a palette and 8bit images automatically, and store indexed images ? or does the user even already have to use custom palettes, and the editor just fails to generate proper 8bit indexed images ?

also, how can we make the game authors aware that they can produce a game that's 1/5th of the size simply by resampling their audio tunes a little ?

and can the engine use 8 bit or 16 bit mode by default for faster blitting speeds ?

[sonneveld]

I don't think converting to 8bit is the answer. I'd much rather we encourage 32bit for everything as it much easier to render under OpenGL and provides alpha channels. 8bit and 16bit requires conversion to 32bit anyway under accelerated graphics drawing.

I've had success with disabling the simple sprite compression and storing all resources in a zip instead of the clib format. Even simple deflate compression cut down Unavowed from 2.5 gig to 1.5 gig. There's newer compression formats such as zstd which might help further.

There's also the issue of some games bundling the Windows and Linux data together. It seems some authors might not realise that the Windows exe is bundled already.

[ghost]

This is most ironic, because community members were suggesting removing 16-bit mode from engine for years (and some even suggested removing 8-bit mode). All the game templates were converted from 16-bit to 32-bit recently because 16-bit setting caused confusion between newcomers who tried to use sprites with alpha channel in their games (or had their sprites loosing colors).

Recently I also decided to made 32-bit screen mode default because 16-bit mode did not work well on some newer systems.

AGS does support palette and 8-bit games, but majority of game developers either do not know how to use palettes or do not want to and seem to prefer to make games in true colour.

I don't know if it would be possible to reliably fit whole scene in a 256-colour palette, considering there may be much more colours there; the scene may consist of room itself, any combination of characters (they switch rooms) and GUI. Additionally, such conversion may increase compilation times. So the possibility is rather hypothetical.

I think the general consensus in the past was to eventually have only hardware accelerated graphics which work with textures rather than blitting raw bitmaps on screen; and have 8-bit palette effects simulated using shaders.

[ghost]

As for the unnecessary large audio, that's a good point of course, but personally I have no idea how to suggest this to everyone who uses AGS. AGS's "knowledge base" is spread out into numerous disconnected tutorials and making everyone knowing this can take a lot of time.

We could post this information on forums though and see what happens.

EDIT: we could also add this hint to new manual, perhaps a topic called "resource management" which would explain how to optimise the game's assets.

[sonneveld]

the editor would have to count the colors used and only use 8 bit if possible (maybe with a small warning like "using X colors, suggested to use only 256" being output somewhere)

This sounds like a lot of work for reducing size when we could use lossless image compression or zip compression. Using 8/16bit might help with the software renderer, but I see us dropping that eventually, leaving just the (usually 32bit) 3d accelerated drivers.

but then, it probably makes more sense to store images in a compressed image format such as gif which is optimized for the purpose.

The nice thing about storing the compressed raw sprite data is that you can stream it straight into a buffer. I imagine png (not gif, it's only 8bit unfortunately) would require some processing first.

so having the main game file itself compressed wouldn't really make a difference in how much you gotta download.

Keeping the game in a compressed format can help with loading times where storage is the bottleneck, putting some of the load on the cpu instead of waiting for storage.

[ghost]

I opened the ticket in manual project (where we prepare a new variant of documentation): adventuregamestudio/ags-manual#72

[ghost]

Only for the sake of theorizing, automatic palette building per scene is probably not possible at all in AGS at this point because there's no simple way to tell which sprites will be used within the room. Any object in game can change sprite anytime, characters may use different animations, and animations in AGS are not attached to particular character or object. Plus, sprites may be generated by script commands at runtime and assigned. In other words, the best one could do is to calculate palette for whole game, which includes all sprites.

[i30817]

This is a problem, and i'd be interested in a cmdline tool to downgrade the wasteful audio files. My not really complete but probably around the same magnitude of the total gb of AGS games currrently has about 45 gb compressed. I'm pretty sure that without the recent games pushing 500mb-3gb and such, it'd cut down to a modest 30gb compressed or so, and i'm pretty sure that the most of that is full voiceover audio, probably uncompressed wavs or something similarly silly.

Maybe also a switch that could remove the executable ags bundling with the data, for the people that know that the open source engine can play the game perfectly and don't want to keep paying the windows executable space tax.

[ghost]

Maybe also a switch that could remove the executable ags bundling with the data, for the people that know that the open source engine can play the game perfectly and don't want to keep paying the windows executable space tax.

There's already such switch, in AGS Editor one can select "build target" between "Data", "Windows" and "Linux", where "Data" produces plain game file without any executable attached.
It could also be worth to additionally add another switch/build type which packages windows game with data and executable as separate files which could simplify certain actions (like repacking game) and maybe help vs antivirus trigger.

Btw, note that windows exe is around 2.5 mb large, which is unnoticeable in case of large games (it may become an issue in case of many small bundled together though).

[rofl0r]

This is a problem, and i'd be interested in a cmdline tool to downgrade the wasteful audio files.

the command line tool to downgrade the wasteful audio files was mentioned in my first post, first paragraph:

converting the mp3 and ogg files in it to lower bitrate (while pertaining pretty good quality, using the instructions in agsutils' README

[i30817]

Ugh, sorry about that, i probably misremembered reading it a while ago and thought it was just a proposal when posting now.

[necros2k7]

Read the issue and my thoughts:

1. It would be great if engine would support 7z lzma2, opus formats, zip at last.
2. It would be awesome if Editor had build-in optimization options like asking to perform convertion of wav, mp3 to opus and png optimization with Fileoptimizer or Papas optimizer or using some of their utilities. For people who want to make quick and dirty game / slow PC should be 2 editor modes - Quick build (no size optimization), Optimized build (should be default). As of question of people calling to use these options - floating tooltip system can be implemented, like when author adds sounds - tooltip bubble should inform him that its highly recommended to optimize your music/sounds losslessly to say opus format.

[rofl0r]

It would be great if engine would support 7z lzma2, opus formats, zip at last.

the engine already offers a great deal of tech to achieve reasonable filesizes. the issue is that the game authors do not use their brains or don't care about filesize. even if you throw yet more tech at them, that doesn't solve the issue.

[i30817]

It depends on how convenient the tech is IMO. If the editor can optimize audio creation by default from a lossless format when importing, or can convert art into palette optimized graphics transparently, things will become significantly better.

The major waste is apparently audio, and a better lossy format like opus would help a bit.

Also the DIGITAL MIDI driver not being 'apparently' broken would probably help with devs wanting to use MIDI.

Video is also a problem in the very big games i expect. Does ags use ffmpeg to decode it?

[ericoporto]

I agree with @rofl0r that maybe this is more about developers not caring that much about filesizes. On the documentation side I will try to have a page written as discussed in the other issue so at least that is in and people can know how ogg files can be very good at compressing audio.

Any introduced compression like zip will make binary patching harder -> bigger download sizes on Steam when updating.

I think midi is a bad audio format to lean on (it doesn't ship instrument samples with the music) and the Engine already support some tracker formats if people would like to use them.

[ghost]

Any introduced compression like zip will make binary patching harder -> bigger download sizes on Steam when updating.

How they are making it harder, do not zip containers support modification/addition of data?

[ghost]

I agree with @rofl0r that maybe this is more about developers not caring that much about filesizes. On the documentation side I will try to have a page written as discussed in the other issue so at least that is in and people can know how ogg files can be very good at compressing audio.

I have a pretty pessimistic view on this, imho not many people would actually read and use these, except maybe long-term commercial developers who are interested in optimizing their games to max.
Therefore I find a sense in idea to have some average losless compression as a default, so long as it does not damage the runtime perfomance of course.

[ghost]

Video is also a problem in the very big games i expect. Does ags use ffmpeg to decode it?

You mean literally video files, or graphics in general? I ask because I do not know a lot of games which would use video more than in few cutscenes (but I could miss something).

To answer the question, AGS currently uses Theora library, which plays OGV. There's also DirectX Media support, but it works only on Windows and may require particular codecs installed, thus not recommended.

[i30817]

Videos. Big games i suspect use it include Nelly Cootalot Fowl Fleed, Detective Gallo, All Emmo, Downfall Redux, Mage Initiation etc.

Most commercial ags games in short. Though maybe it's not as bad as i thought before. I doubt video ever gets more than 100 mb regularly.

edit: checked Spoonbeaks Ahoy HD and the two videos are a mere 30 mb, so i was probably extremely incorrect about it being important to size nowadays.

Which makes games like Adventure - Welcome to the Genre taking 2.3 gb probably caused mostly by audio...

[necros2k7]

Another option is popup window after author click to build project explaining it`s strongly recommended to optimize resources before release offering "Optimize now?"

[rofl0r]

another thing i've noticed in the game nocturne 191 (link here rofl0r/agsutils#15 ) is that the game ships with unused sprites which appear to be from the AGS template. it could be helpful if the build process removed unused sprites (if possible). the second thing is that these sprites start with index 2000, which means for each sprite in between the 22 that are actually used and the "default" ones, acrsprset.spr needs to have an empty slot for the non-existing sprites in between. in case that index 2000 is the default for the ags game template, maybe it should be considered to make it start at 100 instead so less space is wasted. also if a user picks up this idiom of grouping sprites into number groups aligned to 1000, he might easily use an index of 2 million instead of 2000 where the spritecache is effectively almost empty, yet has a huge amount of empty entries.

[AlanDrake]

Unfortunately it's impossible to determine unused sprites because they could be referenced by scripts at runtime.
acrsprset will stop being a problem once we move to a different format for storing the sprites (one day, in the future).

[ivan-mogilko]

the second thing is that these sprites start with index 2000, which means for each sprite in between the 22 that are actually used and the "default" ones, acrsprset.spr needs to have an empty slot for the non-existing sprites in between. in case that index 2000 is the default for the ags game template, maybe it should be considered to make it start at 100 instead so less space is wasted.

Lowering template sprite indexes may be a good idea, this may have something to do with hardcoded sprite placeholders in the old games (they were used for some gui buttons, iirc).

I'd note just in case that empty sprite slot afaik takes 2 bytes (a single short), which is a colour depth (I honestly dont know why it was made 2 bytes though as 1 seem enough). An empty sprite entry thus is defined by colour depth field = 0. That's still something, but it's not a full sprite description fortunately.

@AlanDrake

acrsprset will stop being a problem once we move to a different format for storing the sprites (one day, in the future).

Not sure what particular problem you are refering to, it might be possible to fix few things in acrsprset as well if necessary.

he might easily use an index of 2 million instead of 2000 where the spritecache is effectively almost empty, yet has a huge amount of empty entries.

This was a reason why when asked to raise sprite limit I decide to only increase it from 30k to 90k, as at that moment I was not ready to remake sprite cache and save format to something more suitable for random indexes.
There's a comment I left regarding that: #499

And yes, many users, at least in the past, prefered to be able to set up sprite indexes in groups, perhaps not starting with each 1000, but they said it's useful for organization and writing algorithms in script.

[AlanDrake]

Right, sorry. Even if we change format in the editor, it will still have to be compiled for the game in some way.

[ivan-mogilko]

Well it could be compiled in another way too, my meaning was that if there's a good idea of how to improve this within existing format, then it may also be applied.

[rofl0r]

This was a reason why when asked to raise sprite limit I decide to only increase it from 30k to 90k, as at that moment I was not ready to remake sprite cache and save format to something more suitable for random indexes.

ah, good to know. spriteindex was also something i implied in the above as iirc every empty sprite number also needs a couple bytes in the index, which now with 64bit offset is probably at least 12 byte per sprite.

[ivan-mogilko]

UPDATE: FYI, as of 3.6.0 we support automatic packing of 16/32 bit sprites to 8-bit paletted sprites and unpacking to 16/32-bit in memory when they are loaded in game: #1461

Same could have been done for room backgrounds probably, but was not at the time. So this is still an option to add.

Also 3.6.0 now supports a selection of sprite compression options between RLE (original) and LZW. The latter works much better for games where sprites have many different colors (RLE still works best for sprites with limited number of colors).

UPD: In 3.6.1 we support Deflate compression, saving even more space.

[ivan-mogilko]

Although there has been a significant improvement to the sprite compression since AGS 3.6.0, an audio issue is may be one remaining unsolved, and it's not clear whether we may or should do anything with this in AGS itself.

Strictly speaking, if a user does not care at all about optimization, then there's not much we can do rather than complaining, it seems. In the past I've seen couple of gigantic games which had over few GBs of compressed sprites, even though the games themselves were something like 320x200. So its mostly about people pushing unrestricted amounts of content and making unoptimized game scenes.

Perhaps, if we cannot enforce optimization, we could at least provide a help to those who might care about one.
What may be a step into this direction is a analysis of the compiled game content, presented to the user in the editor (or as an output of a standalone tool too), which would show a table of sizes per type of assets (sprites, regular audio, voice-overs, compiled scripts, etc). There's a ticket on this for some time: #1967.

[ivan-mogilko]

About audio, something that we might consider, is the game project settings that defines optional audio encoding settings, and have these settings applied to the files that are packed into the game. I might write a feature proposal for discussion, although don't know how popular this idea may be. But I think something like that may have other uses, like ensuring the game audio has uniform format.

[ivan-mogilko]

Opened #2319, I shall close this now, because this discussion has been stalling. If necessary, it may be restarted in a new ticket.