Sharing experience on direct asset export

[mortalis13]

Hi. Wanted to share some info on how I could export a single asset using the direct asset file loading. Since I used UnrealPak for extracting the package already, I just needed to get data from the assets themselves. And I didn't have usmap also.

Basically, I used LegacyPackageReader which accepts uasset stream and also had to modify a bit the PackageReader.py so it doesn't give me errors for missing provider for example.

So the base code for loading an asset would be:

from UE4Parse.Assets.PackageReader import LegacyPackageReader
from UE4Parse.BinaryReader import BinaryStream

uasset = BinaryStream('Texture.uasset')
uexp = BinaryStream('Texture.uexp')
reader = LegacyPackageReader(uasset, uexp)

I created the following commit in my fork so it should be easier to follow: mortalis13@0c78cda
And the Readme has a sample for loading and exporting a texture.

What I did was adding version parameter to specify the UE version directly without provider, which I don't pass.
And as the provider is optional, I added branches for using it when it's not None.
Also I don't have mapping files, so uasset will not have the mappings attribute. (Not sure what it is, maybe it's present in the UE5 packages)

And also added full versioned dependencies in requirements.txt so it should install in a venv without problems.

[joric]

@mortalis13 slightly related, did you try exporting uasset that's NOT in JSON format using pyUE4Parse? I'm trying to export raw binary asset (.cfg) but it seems it only supports jsonable uassets and textures (it fails at try_load_package despite the asset name it's in provider.files). I don't want to use other tools.
@MinshuG how to export .cfg files (I just need raw binary data)? Looks like DefaultFileProvider only supports .umap/.uasset ?

Upd. My raw asset export discussion moved to #26

[mortalis13]

not sure what .cfg files are, don't remember seeing them in the folders with data i was exporting,
as i'm aware of, the .umap, .uasset, .uexp are official (so to speak) formats for UE assets

i'll try to remember what game i was working with ) and check what it has,

i suppose you're extracting a .pak file with UnrealPak first? this should give you all the asset files inside
and also as i remember, the .uasset are not json, they contain encoded data,
.cfg on the other hand, does seem like a config file possibly in json or other text format

[joric]

Comment moved to #26

[mortalis13]

ok, responding to the initial question, i didn't work with other types other that the mentioned.
.cfg files are inside the .pak, or inside the .uasset files, or are directly located in a game subfolder?

[joric]

directly located in a game subfolder

Of course not. They are inside the .pak they are similar to .uasset files but text. I just wish pyUE4Parse were able to extract them.

[mortalis13]

What error does it return? Anyway, it's better to move this to a separate issue, for cleanness...

[joric]

@mortalis13 I'd love this option (mostly because local ubulk unpacker is slow) but I'm having troubles with your fix: mortalis13@0c78cda

  File "D:\Projects\github\pyUE4Parse\UE4Parse\Assets\PackageReader.py", line 102, in __init__
    self.PackageFileSummary = FPackageFileSummary(self.reader)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\github\pyUE4Parse\UE4Parse\Assets\Objects\FPackageFileSummary.py", line 30, in __init__
    raise InvalidMagic("Not a UE package")
UE4Parse.Exceptions.Exceptions.InvalidMagic: Not a UE package

Apparently the legacy code reads .uasset here self.PackageFileSummary = FPackageFileSummary(self.reader) and checks it for magic bytes in the beginning, see FPackageFileSummary.py:

PACKAGE_FILE_TAG = 0x9E2A83C1
PACKAGE_FILE_TAG_SWAPPED = 0xC1832A9E
class FPackageFileSummary:
    FileVersionUE4: EUnrealEngineObjectUE4Version
    FileVersionLicenseeUE4: EUnrealEngineObjectLicenseeUE4Version
    bUnversioned: bool = False
    CustomVersionContainer: FCustomVersionContainer

    def __init__(self, reader: BinaryStream) -> None:
        Tag = reader.readUInt32()
        if Tag != PACKAGE_FILE_TAG and Tag != PACKAGE_FILE_TAG_SWAPPED:
            raise InvalidMagic("Not a UE package")

You can trace it back to direct .uasset reading. But neither of my .uasset have those bytes in the beginning (I used multiple unpackers, e.g. FModel and ZenTools). So I'm worried about what's happening here. Do your .uasset have these signatures? My assets are from UE 5, looks like LegacyPackageReader is incompatible. Maybe try loading through IoPackageReader then? It has similar API.

[mortalis13]

In my case, .uasset's I used all have that exact signature (the first bytes are c1 83 2a 9e), and I used files from UE4.X. So cannot tell anything for other versions, sorry.
For UE5 I suppose the signature could be different, don't know where to check exactly though, not familiar with the UE specifications too much (if they mention this info somewhere anyway...)

Interestingly, I searched for the "0xC1832A9E" on the internet and it gave me some pages where it's mentioned, without concrete solutions of course...

You could maybe try asking on this forum for example: https://www.gildor.org/smf/index.php/board,3.0.html
It seems there's a viewer tool, haven't found the clear info about the support for UE5+ though...

It seems to be a completely new format, so it requires investigation I suppose, maybe UE source code studying, trial and error etc.

[MinshuG]

what are we talking about here?

[joric]

what are we talking about here?

Parsing pre-exported assets. I tried to implement virtual textures, but it starts unpacking a 3GB ubulk with oodle (?) and it takes forever for some reason (chunks are too small, perhaps?), so I had to unpack it with zentools. I don't even need ubulk to read the asset header, but it can't be skipped. See https://github.com/joric/maps/wiki/Stalker-2#tileset

[MinshuG]

if uasset is from UE5+ game and from iostore (.ucas/.utoc) it's most likely an IoStore package and it can't be read without it's container file (.ucas/.utoc and global.utoc/ucas)

[joric]

The uasset in question was unpacked with Zentools as .uasset + .uexp + .ubulk and I only needed .uexp (it's like an external part of the header) to read virtual texture properties. I didn't need .ucas/.utoc after export.