Failed to update dictionary data

[rplasson]

Since the update of 10 dec 19, the dictionary fails to update. A warning is set on the icon stating "Failed to update dictionary data". Manual attempt to update the dictionary from the preferences (Kanji directory, "Retry" button close to the error message) gives the following error" "Updated failed. A request was aborted, for example through a call to IDBTransaction.abort". The error can disappear when (randomly) switching back and forth languages and hitting the retry (I installed both english and french dictionaries), but comes back after firefox restart.

Beside the update problem (or the update error problem; as far as I know, the update may have been correctly processed while an erroneous error message is raised), the app works correctly.

[birtles]

@rplasson Thank you so much for this report! I'm really sorry to have introduced this problem. I have been trying to fix problems some have faced recently (see #148) but I suspect today's update made it worse for some.

I have an idea to try to fix this which I will try tomorrow. Please bear with me for another day while I try to work out what is going on. Thank you again for your report and patience.

[birtles]

You can tell if the update has worked by hovering over 弄. If you see an entry for Radical (Nelson) 96 玉 in the components table the update has worked. The version reported in the options window should be 1.1.3.

The error can disappear when (randomly) switching back and forth languages and hitting the retry (I installed both english and french dictionaries), but comes back after firefox restart.

This part has me baffled. The errors I am seeing being logged are for partial updates making me think the underlying the transaction has not completely finished somehow. But if at some point (through switching languages back and forth) you are able to make it succeed, then on a restart it shouldn't need to apply any further updates. So I'm not sure why the error would appear on a restart.

Currently the add-on will automatically retry if there is a network error but it doesn't retry for database errors. It's possible that writing to IndexedDB is just flaky sometimes and it needs to retry for certain database errors too. That would explain why switching languages helps since that will actually cause it to retry.

For now I've released 0.1.12 which fixes a couple of things but mostly adds more logging so I can hopefully understand a little better what is going on.

[birtles]

I got one interesting error report with the following sequence:

• Loading: …35f69f/data/dict.idx
• Successfully updated kanji database
• Successfully updated kanji database (This double message is due to both the sync and async update paths now reporting this)
• Loaded: …635f69f/data/dict.idx
• Loaded: …635f69f/data/dict.dat
• Loading: …5f69f/data/names.dat
• Loading: …5f69f/data/names.idx
• Loaded: …35f69f/data/names.dat
• Loaded: …35f69f/data/names.idx
• Loaded dictionary successfully
• Setting initial language of kanji database to en.
• Successfully set language to en.
• Updating kanji database...
• Download error: Database file https://d1uxefubru78xw.cloudfront.net/kanjidb-rc-en-1.1.1-patch.ljson not accessible (NetworkError when attempting to fetch resource.). Retrying in 4385.597054796728ms.
• Aborting updating kanji database due to DownloadError
• Aborting updating kanji database due to DownloadError

The curious thing here is that even after successfully updating the database it goes and tries to update it again. It's possible we somehow picked up the wrong language from the config, set it back to 'en' and clobbered the database.

More likely, however, the double abort at the end is the key. I suspect we have overlapping calls to maybeDownloadData that are causing issues.

The other observation is that after adding more fine-grained logging, I see that we are hitting a ConstraintError when putting records in the database.

Digging into the Firefox source there are only a handful of places that throw IndexedDB constraint errors. In this particular case, the most likely seems to be that we are getting a NS_ERROR_STORAGE_CONSTRAINT which is a converted SQLITE_CONSTRAINT. If we do have overlapping updates that might trigger it but I don't think that's the case here.

This situation doesn't seem to overlap with the above potentially overlapping updates situation at all. Furthermore, I can see that it happens on different partial updates. I suspect it might just be a "retry until it works" situation. Something to try tomorrow.

[rplasson]

SO:
I only saw your message after the update to 0.1.12. I didn't close firefox, so the icon switched back to the dark on yellow [!] warning with "Failed to update dictionary".

1. I now have another update error: "Update failed: A mutation operation in the transaction failed because a constraint was not satisfied"
2. As I had this update error, checked the 弄 kanji: nothing was seen.
3. Tried to switch back and forth between french and english dictionary, the same error was raised each time, until the point the update process stalled (stopped at "checking for updated dictionary"). Hovering the arrow on rikaichamp icon states that it is still "checking for updated dictionary", but the [!] warning disapear.
4. The 弄 kanji is working. Closing/going back to the module preferences indicates that the kanjidic version is 1.1.3
5. switched back to another language => update error, 弄 kanji no more working

I can reproduce these steps by randomly switching back and forth between languages, when the update stalls, the update is actually working. Trying another modification, the update failed error appears again.

[rplasson]

I also obtained once another "Update failed: Database version (1.1.2) is older than current version (1.1.3) Retrying at..." after random language changes...

[birtles]

Great, thank you again!

1. I now have another update error: "Update failed: A mutation operation in the transaction failed because a constraint was not satisfied"

This much is sort of expected. Version 0.1.12 simply exposing the actual underlying error properly, i.e. the ConstraintError.

The bit I don't understand it why it tried to update at all. Presumably your existing install should have already been up-to-date. And of course, I have no idea why this ConstraintError comes about in the first place.

2. As I had this update error, checked the 弄 kanji: nothing was seen.

Again, presumably your existing install should have already been there so I don't know why it needed to update.

3. Tried to switch back and forth between french and english dictionary, the same error was raised each time, until the point the update process stalled (stopped at "checking for updated dictionary"). Hovering the arrow on rikaichamp icon states that it is still "checking for updated dictionary", but the [!] warning disapear.

The [!] icon should disappear while it's still checking for updates. I'm not sure why it got stuck though.

4. The 弄 kanji is working. Closing/going back to the module preferences indicates that the kanjidic version is 1.1.3

Did it start working by itself? Or did you restart Firefox or something?

I can reproduce these steps by randomly switching back and forth between languages, when the update stalls, the update is actually working.

I tried this over and over again today but unfortunately couldn't reproduce the problem.

Can I ask how you know the update is actually working?

I also obtained once another "Update failed: Database version (1.1.2) is older than current version (1.1.3) Retrying at..." after random language changes...

That's quite odd. It shouldn't happen but it suggests something is racing. I have to dig into the scenarios that could cause that tomorrow.

The other things that puzzles me is this only ever seems to happen when applying the subsequent patch updates. The initial update never seems to fail.

Thank you so much for all the extra information. This is still quite a mystery to me, but I hope I can make more progress tomorrow.

[birtles]

I was diagnosing some other unhandled errors and tracked them down to extension storage failing. Specifically methods like browser.storage.local.set() can randomly fail with "An unexpected error occurred".

After adding logging for these methods it turns out that the users that are hitting them appear to correlate closely with the set of users who are hitting either these ConstraintErrors or the DatabaseUnavailableErrors from #148. In other words, there is probably nothing particularly wrong with how Rikaichamp is using IndexedDB. Rather, IndexedDB, as exposed to the extension itself, is just flaky.

It's not entirely clear if IndexedDB is temporarily unavailable or if something about the users's configuration means it is thoroughly broken but I suspect it's the former because:

• Some users hit both errors with browser.storage.local and DatabaseUnavailableErrors but some users who have problems with browser.storage.local encounter ConstraintErrors, meaning they successfully opened the database but then subsequently hit problems.
• The particular keys / actions that trigger the browser.storage.local errors are varied suggesting the database fails at different points in time. (That is, sometimes we clearly succeed in reading the enabled key on startup but fail to set the lastUpdateKanjiDb key after updating, but on other occasions we'll fail the former.)
• The fact that often these ConstraintErrors are arising when applying a patch suggests the database worked at some point in the past.

So I think the best we can do here is do a few retries for both opening the database and writing to it. My guess is these things happen often at startup because there is contention for some resource (there is also some sort of correlation with users who are encountering errors loading the datafiles into memory) so if we can wait a few seconds between retries we might have more luck.

Somewhat unrelated, as of version 0.1.14 I've made the add-on hide the exclamation work when a patch update fails (since we still have a workable database we can use) so hopefully this is already a little less distracting for the user.

[rplasson]

Well, this is now working for me with version 0.1.14, no more update errors. Note that when I checked the dictionary version, it was actually at 1.1.0 after firefox restart, but I could up hit the "Check for update" to upgrade it flawlessly to 1.1.3 (and the 弄 kanji is indeed seen correctly).
I hope my few information could help you,
thank you.

[birtles]

I'm really glad to hear that it is working properly for you now. Your information has been most valuable. Thank you!

In the process of digging into this, I have come up against some problems with the current setup so I am currently working on a fairly thorough overhaul of the update checking which I hope to release next week. Hopefully I don't make things worse in the process.

[birtles]

I'm closing this for now since I believe this is as fixed as it's going to get. The main outstanding issue is permanent private browsing but that is tracked in #148.