Accommodate AppData Redirection for Windows Store Python

[rmartin16]

Describe the bug
Windows will redirect interaction with %LOCALAPPDATA% when Python is installed from the Windows Store. This means that the briefcase data directory will be created at somewhere like %LOCALAPPDATA%\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\Local\BeeWare instead of %LOCALAPPDATA%\BeeWare.

To Reproduce
Steps to reproduce the behavior:

1. Ensure %LOCALAPPDATA%\BeeWare directory does not exist.
2. Run briefcase build android using Python from the Windows Store
3. See error on trying to verify licenses

Expected behavior
Briefcase should prevent this redirection or incorporate it in to sourcing the briefcase data directory.

Of note, if the %LOCALAPPDATA%\BeeWare exists, this redirection will not happen.

Environment:

• Operating System: Windows 10 1903 and later
• Python version: 3.10
• Software versions:
  • Briefcase: 0.3.11

Additional context
python/cpython#84557
https://docs.microsoft.com/en-us/windows/msix/desktop/desktop-to-uwp-behind-the-scenes#file-system

Workarounds

• Use the public Python installer
• Move the %LOCALAPPDATA%\Packages\<Python>\LocalCache\Local\BeeWare directory to %LOCALAPPDATA%\BeeWare

[rmartin16]

Found this documentation today.

Given this documentation doesn't provide any real workaround to this, it seems this behavior is unavoidable. However, this documentation does mention that os.path.realpath() can reveal when this redirection is happening.

Therefore, in lieu of any apparent way to detect and work around this in real time, it may be best to detect this situation, drop an error to the user recommending the workaround above, and prematurely exit Briefcase. I'm sympathetic to users encountering this because the manifesting errors are dumbfounding and seem likely to deter people from using BeeWare altogether since it just looks like BeeWare is broken on their machine.

[freakboy3742]

Completely agree that the manifestations of this are something that we need to work around - the actual problems (like "heat.exe" not working) don't seem to bear any connection to the underlying cause.

However, I'm not clear why we can't use os.path.realpath() to detect the problem, or why moving the folder in code isn't viable as a workaround. Is that because any attempt to move the folder to the "good" location will itself get re-written to use the new location? Can we do an end run around this by calling mkdir via os.system (or similar) to create a Briefcase folder in the "good" location?

[rmartin16]

However, I'm not clear why we can't use os.path.realpath() to detect the problem, or why moving the folder in code isn't viable as a workaround. Is that because any attempt to move the folder to the "good" location will itself get re-written to use the new location? Can we do an end run around this by calling mkdir via os.system (or similar) to create a Briefcase folder in the "good" location?

I was considering the same thing. It's clear that filesystem interactions coming directly from the python.exe are redirected....however, it also seems like this sandbox is a bit leaky and could be subverted altogether. I'll try to do some tests to determine the limits of the sandbox.

[rmartin16]

To confirm the suspicion above, the sandbox can be ostensibly subverted by simply creating the directory in a child process.

Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\python.exe'
>>>
>>> app_path = "C:\\Users\\User\\AppData\\Local\\MyApp"
>>>
>>> import os
>>>
>>> os.path.realpath(app_path)
'C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Local\\MyApp'
>>>
>>> os.system("mkdir " + app_path)
0
>>> os.path.realpath(app_path)
'C:\\Users\\user\\AppData\\Local\\MyApp'
>>>
>>> # manually deleted the app_path directory
>>>
>>> os.path.realpath(app_path)
'C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\LocalCache\\Local\\MyApp'
>>>
>>> import subprocess
>>>
>>> subprocess.run(["mkdir", app_path], shell=True)
CompletedProcess(args=['mkdir', 'C:\\Users\\User\\AppData\\Local\\MyApp'], returncode=0)
>>>
>>> os.path.realpath(app_path)
'C:\\Users\\user\\AppData\\Local\\MyApp'
>>>

Of note, though, this behavior from windows store apps was implemented starting in windows 10 1903. Before that, a "copy on write to a per-user, per-app location" policy was used. That said, even 1903 went EOL at the end of 2020....so, it seems reasonable to not worry too much about this.

[freakboy3742]

Yeah - we don't need to worry too much about supporting software that is 2 years past EOL, unless you've got evidence that the EOL version still represents a significant proportion of the user base.

[rmartin16]

for posterity:

source: https://reports.adduplex.com/#/r/2022-06