add XAR header format

[armijnhemel]

The XAR format is an archive that consists of a header, a zlib compressed XML document and then the payload. The structure of the payload (the files, etc.) is described in the XML file. Because Kaitai Struct does not have a built in XML parser this parser is limited to just the header.

[armijnhemel]

The version from Apple (which is in current use) has the same version number in the header but supports additional checksum algorithms. I am not sure how to wiggle these into this particular specification. Basically, it should ignore one field, but only if that field is not a valid strz. Ideally I would use some sort of "lookahead" operation, or some other conditional.

[KOLANICH]

Because Kaitai Struct does not have a built in XML parser this parser is limited to just the header.

You can try to implement the approach I use in numpy_npy format.

[generalmimon]

@KOLANICH:

Because Kaitai Struct does not have a built in XML parser this parser is limited to just the header.

You can try to implement the approach I use in numpy_npy format.

@armijnhemel:

The structure of the payload (the files, etc.) is described in the XML file.

This is a very similar concept to the glTF binary format (#445), where one needs to parse JSON to be able to parse the binary structure. However, this does not mean it can't be described with the KS language - you can either outsource the XML parsing to opaque types (which I don't really recommend though, because you'll have lose any type checks / derivations that the compiler does and you'll have to resort to unsafe .as<> type casts, which are easy to screw up) or create a parametric type that can be instantiated from the application code. The application code will serve as a "bonding agent": first, it will invoke the initial parsing, access the XML string, parse it using a native or external XML parsing library of choice and then puts the needed data pulled from the XML "back" to the KSY-generated parser via the parametric types.

Which means that although the KSY won't read the whole file down to bottom (so it won't be so enjoyable to use in the Web IDE, for example), it's quite easy to use and you don't lose any type checks and you won't have to use type castings. For the glTF binary format, I have created a proof of concept for this approach, so you can see what I mean: #445 (comment)

[armijnhemel]

I fixed a few things. One thing I am not happy about is that currently the toc is defined as part of the header, but it shouldn't be.

I tried this:

seq:
  - id: magic
    contents: 'xar!'
  - id: len_header
    type: u2
  - id: header
    type:
      switch-on: len_header
      cases:
        28: apple_header
        _: regular_header
  - id: toc
    size: header.len_toc_compressed
    process: zlib
    doc: zlib compressed XML further describing the content of the archive

but that doesn't seem to work:

Call stack: undefined io.kaitai.struct.precompile.ErrorInInput: xar_header: /seq/3/size: don't know how to call method 'len_toc_compressed' of object type 'KaitaiStructType'

I don't know where I am going wrong there.

[KOLANICH]

It would require interfaces proposal to implement it the way you wanted. There may be another way to implement it though

[generalmimon]

@armijnhemel Sample files, both with the Apple header and the regular one?

[armijnhemel]

@armijnhemel Sample files, both with the Apple header and the regular one?

Test files with gzip, bzip2 and no compression:

https://github.com/armijnhemel/binaryanalysis-ng/blob/master/src/test/testdata/unpackers/xar/test-bzip2.xar
https://github.com/armijnhemel/binaryanalysis-ng/blob/master/src/test/testdata/unpackers/xar/test-gzip.xar
https://github.com/armijnhemel/binaryanalysis-ng/blob/master/src/test/testdata/unpackers/xar/test-none.xar

I would need to search for test files for the Apple version.

[generalmimon]

Thanks!