Kaitai-struct-compiler producing different files

[arun-singhal]

I was trying to parse a PE (executable) file via kaitai struct. When I tried to generate the code from .ksy for Javascript, I saw a difference in code between my local generated code and the code generated by webIDE.
Why there is a difference given that I am using the latest version only.
kaitai-struct-compiler version : 0.9

[generalmimon]

If you use the stable Web IDE at https://ide.kaitai.io and the stable version of the local kaitai-struct-compiler (0.9) with the same configuration, the output must (and will) be exactly the same. There are two tabs with the JS generated code in the Web IDE, as you may have noticed - JS code and JS code (debug).

The JS code tab shows the code generated with the default configuration, i.e. with debug mode (#332) disabled. This is equivalent to running just kaitai-struct-compiler -t javascript microsoft_pe.ksy.

The JS code (debug) tab shows the JS code generated in debug mode, i.e. as if you passed --debug option (or the separate --no-auto-read and --read-pos options - see #332) - an equivalent is running kaitai-struct-compiler -t javascript --debug microsoft_pe.ksy locally. This is the actual code run by the Web IDE when parsing (you can even edit it there for debugging purposes and then press Ctrl+Enter to run the edited code). The code generated in the debug mode stores start and end offsets of each field coming from the byte stream and makes them accessible, so the Web IDE knows where the fields are laid out in the hex dump and it can highlight intervals of bytes in the hex dump.

So make sure you use the right compiler/Web IDE version (there are two of them, see https://doc.kaitai.io/user_guide.html#web-ide) and the compiler options are the same.

Also make sure that the .ksy spec you loaded in the Web IDE is the same as the local one. Note that the preloaded kaitai.io/formats libraries in both the stable Web IDE (https://ide.kaitai.io) and even the devel Web IDE (https://ide.kaitai.io/devel/) are not updated frequently and always contain an older version of the format library than at https://github.com/kaitai-io/kaitai_struct_formats or https://formats.kaitai.io - generally, we only pull the latest format library to the Web IDE at the time of releasing a new major version of Kaitai Struct. It's recommended to check out the most recent format library (website, repo) to see if it doesn't contain an updated version of the spec that you're interested in.

[arun-singhal]

@generalmimon .ksy files are same only. Web IDE is able to parse the file but when I am generating the parser with the same .ksy file in local, it is not able to parse.
I can confirm that I am using the same .ksy file and the same compiler in local.
I generated the parser files in java from Web IDE and tried locally to parse the same file as I did with Web IDE but surprisingly it is not able to parse the file correctly in local.
Below are the screenshots for the response in local and in Web IDE


As you can see, in local I am getting pe instance as null but not in Web IDE.
Only 1 diff can be there that in local I am using java parser files generated from Web IDE and in webIDE I think we are using JS parser files

[generalmimon]

@arun-singhal I think the problem here is that instances in Kaitai Struct are lazy, which means that an instance is only parsed when you explicitly request it (i.e., by calling the instance getter method). Instances that are not explicitly asked for their values have not been parsed and are indeed null. The runtime variable inspector of your IDE suggests this - note that mz has a value, but pe is null. This makes sense, because mz is a seq field (which is parsed just when calling _read(), which is automatically called from the constructor unless you disable it by passing --no-auto-read or --debug) but pe is an instance (https://github.com/kaitai-io/kaitai_struct_formats/blob/8a85dc4/executable/microsoft_pe.ksy#L20-L26):

seq:
  - id: mz
    type: mz_placeholder
instances:
  pe:
    pos: mz.ofs_pe
    type: pe_header

So please call the struct.pe() method and then inspect the property values. This time, the pe property will be initialized.

The laziness of instances is intended to save execution time and memory by parsing data that you actually don't need in your application. But yes, it poses a problem when you want to dump all fields because you need to specifically invoke the instances that you want to parse (this can be automated for a generic KS parser module using reflection, though).

[arun-singhal]

Thanks @generalmimon