Add rekordbox file format specs. #116

brunchboy · 2018-11-20T01:15:01Z

These have been proven to work in the context of my Beat Link Trigger
project, enabling it to retrieve the database over NFS, parse it, and
extract all the track metadata it needs even when it is impossible to
connect to the database server running on the players because there
are four of them in use.

These have been proven to work in the context of my Beat Link Trigger project, enabling it to retrieve the database over NFS, parse it, and extract all the track metadata it needs even when it is impossible to connect to the database server running on the players because there are four of them in use.

database/pdb.ksy

@KOLANICH

Another good suggestion from @KOLANICH

brunchboy · 2018-11-21T03:31:46Z

This is as far as I am able to take these files at the moment. You are welcome to add them if you like, and it will not hurt my feelings if you decide they are not useful, because they are incredibly useful to me. 😄 Thanks in any case for your excellent tools!

KOLANICH · 2018-11-21T09:07:49Z

database/pdb.ksy

@@ -0,0 +1,964 @@
+meta:
+  id: rekordbox_pdb


file name should match its id

If this is true, it should be stated in the style guide.

It is, as far as I remember.

I searched the whole style guide before posting my comment, and could not find anything along those lines. I will rename the files when I have a few minutes though.

All right, I have renamed the files and submitted a PR to the style guide to help future me avoid making the same mistake. 😄 Thanks. I still need to fix something in the rekordbox_anlz.ksy file to avoid Java exceptions when the file contains a fourcc that we have not previously encountered, as I am currently discussing in the gitter channel.

Any chance we can do devicesql_pdb here, as we know that it is not exactly specific to rekordbox?

There was another thread on this topic, @GreyCat. The filename started out not being specific to rekordbox, and @KOLANICH suggested the current rename. I was hesitant at first but ended up agreeing because, while there are certainly aspects of the format which are generally applicable to other DeviceSQL implementations, the specific column types which are used by this mapping (e.g. tracks, artists, playlists, etc.) are clearly used only by rekordbox.

Ok, I agree with rekordbox_pdb, but please rename the file as database/rekordbox_pdb.ksy then?

while there are certainly aspects of the format which are generally applicable to other DeviceSQL implementations

Can they be moved into a separate ksy and used via import?

Whoops! @GreyCat I had tried to rename the file to rekordbox.pdb and accidentally renamed dbf.pdb instead. That is fixed now. Phew!

@KOLANICH nobody knows; I don’t know if anyone has seen any other files created by DeviceSQL in recent years.

Trying to use an enum causes unavoidable parse errors in Java and Python when new/unknown FourCC values are encountered. See kaitai-io/kaitai_struct#300

database/pdb.ksy

KOLANICH

👍

KOLANICH · 2018-11-27T06:49:11Z

BTW, you can rewrite history and --amend changes to a single commit.

brunchboy · 2018-11-27T06:50:19Z

Only if the commit hasn’t been pushed anywhere, of course. I do that sometimes. magit makes that sort of thing very easy in Emacs. 😄

@iamtunzor_twitter found media where there were different values for the artist row, which was causing total database parse failure. Now we should be robust as long as there are no actual structural changes.

KOLANICH · 2018-12-06T06:33:24Z

database/rekordbox_pdb.ksy

+      - type: u2
+        doc: |
+          Some kind of magic word? Usually 0x60, 0x00 but have seen
+          0x64, 0x00.


Can it be format version?

Maybe, although the numbers don’t seem plausible to me. It could also be yet another redundant length value of some sort—since I don’t have access to the actual data where this problem occurred, I can’t explore that question. I don’t even know on what continent the nightclub is found. 😄

It turns out that in fact this is some sort of format version. A show producer in Croatia was running into parse problems and was able to get his DJ to share the database file with me. When the row has the 0x64, 0x00 header it means the name is more than 255 bytes long and is stored in yet another weird format! But I have been able to capture this variant in the latest commit.

Thanks to @iamtunzor_twitter in Croatia for getting his DJ to share a copy of the problematic database file with me!

KOLANICH · 2018-12-11T19:33:22Z

database/rekordbox_pdb.ksy

+      ofs_long_name:
+        type: u2
+        if: subtype == 0x64
+        pos: _parent.row_base + 0x0a


How about

ofs_real_name: value: _parent.row_base + (subtype == 0x64 ? ofs_long_name : ofs_name) name: pos: ofs_real_name type: device_sql_string

Also please follow the convention pos [io] type desc if

Around the time you were writing this, I was at my lunchtime swim, and in the pool is often when I review recent code and make improvements and optimizations, and I came up with the same idea, it will make dealing with the value much easier. Thanks! I will fix the order of the fields as well.

Rats, when I try that I am getting a KSC error in the generate phase: mapping values are not allowed here in 'reader', line 410, column 66: ... (subtype == 0x64)? ofs_name_far : ofs_name_near) (column 66 is right after the :. Any ideas?

Ah sorry, that’s a FAQ, need to escape it as a string for the silly YAML format.

I ended up doing it a slightly different way, but it has the same net effect, the struct is easier to understand and much easier to use; the name is always accessed using the same instance regardless of how it is actually stored.
brunchboy@a2898a0

Now has a much clearer structure in the .ksy *and* provides a single, unified API for the struct user to access the name however it was stored.

KOLANICH · 2018-12-13T06:39:15Z

database/rekordbox_pdb.ksy

      name:
+        pos: '_parent.row_base + (subtype == 0x64? ofs_name_far : ofs_name_near)'


ofs_name_far: pos: _parent.row_base + 0x0a

pos: '_parent.row_base + (subtype == 0x64? (_parent.row_base + 0x0a) : ofs_name_near)'

was it intended?

Yes, that is really the weird layout that they have set up.

Note that your substitution is not quite correct. In the case where it is a “far” string the offset from _parent.row_base is the two-byte value found at _parent.row_base + 0x0a. So row-base gets added to the value found there, it is not added twice to the resulting offset.

brunchboy · 2019-03-08T05:23:01Z

I just made some small documentation changes based on things I noticed while finishing the detailed explanation article. These mappings have been used successfully to run shows by a number of people over the past few months, so if you would like to include them in your collection, they are probably ready. If you are not interested in them I can close the pull request as well; just let me know.

KOLANICH · 2019-03-08T07:31:45Z

This value seems to be redundant, because it can be calculated by dividing the offset
of the start of the page by $len_page$, but perhaps it serves as a
sanity check.

Have you tried to replace these values and check the consequences?

_root.len_page * index

I think that there is no reason to link pages into a list if they are organized into an array. So it may be that array is just an impl detail of memory allocation. I mean that you should check if moving the pages and their page_refs don't spoil the format. If it is the case, the list may be just a strange key-value storage with index as a key and looking up a page by its index is crawling the list and checking the index of every page.

brunchboy · 2019-03-08T15:59:57Z

I’m sorry, I am again not sure what you mean. I can’t move pages around, I can only examine them as they are given to me. The code that creates them is in third-party proprietary software designed to run in low-power processors. The pages of a given table are linked together because there might be pages of other tables interspersed between them.

In all files that any of my users have seen the index is equal to the array index of the page. There are many examples of far worse redundancy and inconsistency in the format.

KOLANICH · 2019-03-08T16:30:14Z

Thank you for the clarification, now I understand your spec better.

KOLANICH reviewed Nov 20, 2018

View reviewed changes

database/pdb.ksy Outdated Show resolved Hide resolved

KOLANICH reviewed Nov 20, 2018

View reviewed changes

database/pdb.ksy Outdated Show resolved Hide resolved

KOLANICH reviewed Nov 20, 2018

View reviewed changes

database/pdb.ksy Outdated Show resolved Hide resolved

KOLANICH reviewed Nov 20, 2018

View reviewed changes

database/pdb.ksy Outdated Show resolved Hide resolved

KOLANICH reviewed Nov 20, 2018

View reviewed changes

database/pdb.ksy Outdated Show resolved Hide resolved

James Elliott added 2 commits November 20, 2018 21:11

Rename KSY structures, per @KOLANICH

108c283

No longer fail if the first four bytes are not zero.

84a3455

Another good suggestion from @KOLANICH

KOLANICH reviewed Nov 21, 2018

View reviewed changes

James Elliott added 2 commits November 21, 2018 10:58

Rename files to match top-level ids

7019417

Abandon use of enum for FourCC until issue kaitai-io#300 is resolved

3c7635d

Trying to use an enum causes unavoidable parse errors in Java and Python when new/unknown FourCC values are encountered. See kaitai-io/kaitai_struct#300

GreyCat reviewed Nov 25, 2018

View reviewed changes

database/pdb.ksy Outdated Show resolved Hide resolved

James Elliott added 4 commits November 25, 2018 12:22

Attempt kaitai-io#2 at renaming file, weird!

f4a44d4

Replace file accitentally moved.

00ea701

Simplify storiage of heap position per @GreyCat

7ed34f0

Improved understanding of waveform headers.

c49dea2

KOLANICH reviewed Nov 27, 2018

View reviewed changes

Loosen up constraints on magic row header bytes.

ad378db

@iamtunzor_twitter found media where there were different values for the artist row, which was causing total database parse failure. Now we should be robust as long as there are no actual structural changes.

KOLANICH reviewed Dec 6, 2018

View reviewed changes

Handle longer-string variant of artist rows.

97a5b49

Thanks to @iamtunzor_twitter in Croatia for getting his DJ to share a copy of the problematic database file with me!

KOLANICH reviewed Dec 11, 2018

View reviewed changes

Improve long-offset artist name handling.

a2898a0

Now has a much clearer structure in the .ksy *and* provides a single, unified API for the struct user to access the name however it was stored.

KOLANICH reviewed Dec 13, 2018

View reviewed changes

Incorporate tweaks made while finishing writeup article.

ed80bf6

Add recent discoveries.

52bd8ab

generalmimon mentioned this pull request Jun 9, 2024

Update rekordbox formats #696

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add rekordbox file format specs. #116

Add rekordbox file format specs. #116

brunchboy commented Nov 20, 2018

brunchboy commented Nov 21, 2018

KOLANICH Nov 21, 2018

brunchboy Nov 21, 2018

KOLANICH Nov 21, 2018

brunchboy Nov 21, 2018

brunchboy Nov 21, 2018

GreyCat Nov 22, 2018

brunchboy Nov 22, 2018 •

edited

Loading

GreyCat Nov 25, 2018

KOLANICH Nov 25, 2018 •

edited

Loading

brunchboy Nov 25, 2018

KOLANICH left a comment

KOLANICH commented Nov 27, 2018

brunchboy commented Nov 27, 2018

KOLANICH Dec 6, 2018

brunchboy Dec 6, 2018

brunchboy Dec 11, 2018

KOLANICH Dec 11, 2018 •

edited

Loading

KOLANICH Dec 11, 2018 •

edited

Loading

brunchboy Dec 12, 2018

brunchboy Dec 12, 2018

brunchboy Dec 12, 2018

brunchboy Dec 12, 2018

KOLANICH Dec 13, 2018 •

edited

Loading

brunchboy Dec 14, 2018

brunchboy Dec 14, 2018

brunchboy commented Mar 8, 2019

KOLANICH commented Mar 8, 2019

brunchboy commented Mar 8, 2019

KOLANICH commented Mar 8, 2019

		name:
		pos: '_parent.row_base + (subtype == 0x64? ofs_name_far : ofs_name_near)'

Add rekordbox file format specs. #116

Are you sure you want to change the base?

Add rekordbox file format specs. #116

Conversation

brunchboy commented Nov 20, 2018

brunchboy commented Nov 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunchboy Nov 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KOLANICH Nov 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KOLANICH left a comment

Choose a reason for hiding this comment

KOLANICH commented Nov 27, 2018

brunchboy commented Nov 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KOLANICH Dec 11, 2018 • edited Loading

Choose a reason for hiding this comment

KOLANICH Dec 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KOLANICH Dec 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunchboy commented Mar 8, 2019

KOLANICH commented Mar 8, 2019

brunchboy commented Mar 8, 2019

KOLANICH commented Mar 8, 2019

brunchboy Nov 22, 2018 •

edited

Loading

KOLANICH Nov 25, 2018 •

edited

Loading

KOLANICH Dec 11, 2018 •

edited

Loading

KOLANICH Dec 11, 2018 •

edited

Loading

KOLANICH Dec 13, 2018 •

edited

Loading