fix import from MoM 1.9.3 on android

[seinsinnes]

Changes to allow Valkyrie to find MoM 1.9.3 data on Android.

The newer versions of the official app moved the locations of the installed data around.

Tested as working on Android 8 - Android 10 (inclusive). It doesn't work on Android 11+ due to scoped storage.
It now has a solution for importing assets on Android 11+ as well.

[seinsinnes]

These changes are on top of the previous fixes for importing from official MoM 1.9.3.
The previous fix was enough for the import to work on mac and windows but on android the data locations also were moved around in the newer versions.

On Android 11+ cross app data access is restricted so the import fails. The only way I can see around this is the "Hosted file import" idea or having the user copy the data manually using a file browser which has full access to the filesystem.

[seinsinnes]

After looking at the Proof of Concept:
https://github.com/folderv/androidDataWithoutRootAPI33
I've been able to get access to import the MoM official app data on Android 11 and Android 13 in Valkyrie.
By asking the user for permission to access specific folders, like so:

This is a build with this permission fix: Valkyrie-android-2.5.9d.apk

[Quantumrunner]

Awesome! It would be fantastic if this would really solve the problem.

Did you also ensure to catch the case that the user revokes the permission to the folder/cancels giving the permission initially?

[seinsinnes]

The permission will be requested each time the user clicks import/reimport. Once the import has completed the permission isn't needed anymore.
Currently if the user jumps out of request window without accepting or changes the "use this folder" to a different folder the import won't work properly so I need to make a way to catch that.

[seinsinnes]

Also I've just realised I only implemented the permission request for MoM and I need to implement it for Road to Legend on Android 11+
That shouldn't be too much extra work.

sigh Android Storage Access Framework and Permissions are kinda painful. Google will probably come up with some new changes that will break things in newer android versions.

I note Road to Legend official app isn't even available on android 13

[Quantumrunner]

I note Road to Legend official app isn't even available on android 13

Oh I did not know that. So it is not even possible to install it on an Android 13 device?

[Quantumrunner]

Some discussion regarding Road to Legend App on android 13 came up here. It is not working on Android 13 phones with 64Bit only but working on older phones with Android 13. Therefore your fix should work but somebody else will need to verify it once a build would be ready (I currently have not Android 13 device to test this).

[seinsinnes]

Right. Agreed. I only have an old device that can run android 11 and device (which is too new) that runs android 13 for testing.
The mechanism for permission and import should work just as well for Road to Legend as for Mansions of Madness but yes it would be good to confirm once I get a chance to complete the fix.

[seinsinnes]

When you go to import. It should look like this:

You need to be able to just click "Use this folder" if you have to navigate around it probably won't work.
If you don't see similar to the images then the file system structure on the tablet must be different than on my phone.

[seinsinnes]

Are you trying to import MoM data?

I wonder where the data is if not Android/data/com.fantasyflightgames.mom

[seinsinnes]

If you see DCIM, Downloads, Videos etc. do you also see the Android folder?

[Quantumrunner]

Yes but the Android folder only shows Media subfolder instead of any other files.

I have installed Total Commander App to see if it has the same problem and it does. In total commander (after giving a lot of extra permissions while already running the app) I can go to /storage/emulated/0/Android/obb but when I then click at com.fantasyflightgames.rtl a dialog opens with the message:

On Android >=13, the external SD card is write protected.

Location: Android/obb/com.fantasyflightgames.rtl

You need to choose the directory from a special Android dialog to give Total Commander write rights

Clicking Yes in the dialog opens the same file picker you implemented and then I'm stuck with the same problem. Instead of choosing a folder automatically it just shows the file picker dialog on basic level and a warning in the top

Can't use this folder

To protect your privacy, choose another folder

Edit: The Can't use this folder might be more related to the fact that it is not possible to select those generic folders like Home and Android because when drilling down to Android/Media the message disappears and I can click Use this folder button.

[Quantumrunner]

Furthermore clicking then back in the file chooser goes back to Valkyrie which is then stuck with a black screen.

We should definitely investigate if clicking the back button in the folder picker dialog can be catched (e.g. show Import failed: Access to folder was denied. and go back to Import screen) so that the user is not stuck with a black screen in Valkyrie afterwards.

With your new method it would still be possible to copy all necessary files to another folder (e.g. downloads) somehow and then select this available folder in the file picker.

[seinsinnes]

Yeah, I'll think about a way to catch the back button (/fail gracefully)
The communication between the calling c# code and the kotlin/java code which deals with the permission request and SAF is a bit of a pain.

If you'd be willing I'd be curious to know if you get the same behaviour with MoM.

I'm not able to test RtL on my Android 13 device (only on Android 11) but MoM import works on my Android 13 device.
It would be useful to know if the error is just RtL on Android 13 or a difference between our two devices.

Perhaps the access rules for Android/obb (where RtL stores its data) and Android/data (where MoM stores its data) are different.

[Quantumrunner]

If you'd be willing I'd be curious to know if you get the same behaviour with MoM.

I'm not able to test RtL on my Android 13 device (only on Android 11) but MoM import works on my Android 13 device. It would be useful to know if the error is just RtL on Android 13 or a difference between our two devices.

Perhaps the access rules for Android/obb (where RtL stores its data) and Android/data (where MoM stores its data) are different.

Yes same behaviour when I try to import MoM. So not a problem depending on the the file location but general access problem. When I try any app in data or obb folder using total commander it is the same behaviour everywhere.

When using the official Files app on the tablet I can go to data and obb but both folders show as empty:

Does it look different on your device?

[seinsinnes]

Ok, something different between the two devices.

But in terms of the file browsers yes this looks exactly the same on my device. The Android/data and Android/obb folders being hidden and/or restricted is what I'm hoping to work around.

[Quantumrunner]

Ok after some more tests I think this is really a device bug because Lenovo might not have correctly implemented this API interface. Seems that no app (I tried with another file explorer app) is able to reach the correct folder but instead always lands in the root folder of the device.

[seinsinnes]

I've updated the apk so you can choose a different folder in the "Use this folder" dialog when importing.
If any file manager will allow you to copy Android/obb/com.fantasyflightgames.rtl to the custom folder and then choose the custom folder for import it should work.

It isn't great since I was trying to remove that kind of manual handling but maybe better than nothing.
Dealing with Android's SAF has been one of the most frustrating development experience I've ever had.

Valkyrie-android-2.5.9d.apk

[seinsinnes]

Unfortunate if it is just down to vendors implementing the SAF differently.

[seinsinnes]

The import works without access to all files on my phone but could be worth trying to give Valkyrie permission:

I don't know. It is hard to debug by proxy.

[Quantumrunner]

It isn't great since I was trying to remove that kind of manual handling but maybe better than nothing. Dealing with Android's SAF has been one of the most frustrating development experience I've ever had.

I would not remove the option where it auto selects the correct folder because I hope for most users this would still be the option that works. This would then heavily reduce amount of support questions because people don't need to understand where to find the files and copy them to other folder manually.

Instead we could split it into two options (e.g. Import from App, Import from other folder)? This could be done with two buttons or instead using a followup page where users can select an option.

The import works without access to all files on my phone but could be worth trying to give Valkyrie permission:

This did not change anything unfortunately. Still no folders showing inside Android folder. I still think it is my device being buggy. :(

[seinsinnes]

I'm sorry to hear. Very disappointing.

No folders inside the Android folder is how it appears to me in the default file browser, but file managers like Total Commander or Cx File Explorer could see inside Android/data.
I set about reproducing what those file managers were doing in Valkyrie.

[seinsinnes]

It is hard to know what the best workflow is here. If users are going to copy data around then they could also just copy in a pre-converted set of assets into Valkyrie's import folders, if they can get their hands on them from somewhere.

[Quantumrunner]

I could solve the import issue now. So at least under Android 13 there is a solution (For android 14 this workaround will not work):

1. Install Total commander.
2. In Total Commander, go to the home folder and from there to "Installed Apps".
3. Search for the app "Files" and click on it.
4. In the properties of the app, click on the 3 dots in the upper right corner and select "Uninstall updates".
5. Repeat the two previous steps if you also have an app called Files by Google.
6. Afterwards try import in Valkyrie again.

Edit: Import for Mansions of Madness was finished in ~20 seconds.
Edit 2: For MoM it seems that some images are not imported correctly but I'm not sure if this was already the case before:
New game > Pick any scenario > Investigator images are missing:

Select content > Expansion images are missing:

[Quantumrunner]

New post for Descent import issues:
Current problem: As you mentioned before:

I note the import takes a pretty long time on the average android device also there maybe a memory leak in the import code. It works but maybe not optimally.

The import for Road to Legend is running know for more than 10 minutes and does not seem to finish. Does it write any log files anywhere that I could check? From what I see in Total commander the .obb file has been copied to Valkyrie folder

but afterwards it seems to get stuck during import (even though some content has been copied to folder /import:

[seinsinnes]

MoM import taking 20 seconds is far too fast it means it didn't grab all the assets.
Also the missing images confirms this.

For Rtl if the .obb file has been copied then the import should work. Like I said the import is extremely slow partly due to number of files contained in the obb which need to be checked for assets.

Valkyrie on Android logs to the standard Android logging system. Viewable by anything that can do a logcat.

[seinsinnes]

Also note that a large number of assets are downloaded by the official MoM as it starts up so if you have installed MoM but never opened it Valkyrie won't get a complete set of assets.

I've discovered that these are pulled from ffg's s3 bucket:
eg.
https://s3.amazonaws.com/ffg-asset-bundles/mansions-of-madness/AssetBundles/1.9.3/Android/scenarios

So theoretically the extra assets could be downloaded from ffg's bucket for MoM bypassing the whole android security issue but it won't work for RtL because all the extra assets for RtL are stuffed in the obb.

[seinsinnes]

I've done some optimisations.
I was taking 5 hours to import Descent on an old phone! It now takes 12 minutes on the same device.

Valkyrie-android-2.5.9e.apk

valkyrie-linux-2.5.9e.tar.gz

valkyrie-macos-2.5.9e.tar.gz

valkyrie-windows-2.5.9e.7z

valkyrie-windows-2.5.9e.exe

valkyrie-windows-2.5.9e.zip

[Quantumrunner]

Since integration of FSAF has been moved to external project this PR is approved now.