Crash when trying to create/open a project on Google Drive

[ding0z]

It happens on linux (doens't matter the distro nor the novelWriter version) but it works perfectly fine in MacOs.
When opening the folder manager to browse the Google drive, the following error pops up.
Checking the Gdrive, few dirs have been created but not the files.

Environment:
novelWriter Version: 1.6-beta1
Host OS: linux (5.15.13-200.fc35.x86_64)
Python: 3.10.1 (0x30a01f0)
Qt: 5.15.2, PyQt: 5.15.6
lxml: 4.6.3
enchant: 3.2.2

OSError:
[Errno 5] Input/output error: '/run/user/1000/gvfs/google-drive:host=gmail.com,user=myemail/0APpBNR1zr4FMUk9PVA/12q-qSbLQro63ptfXuTwA-MVwjDow0qiE/test/test/content/1bd52105ae6ef.nwd~' -> '/run/user/1000/gvfs/google-drive:host=gmail.com,user=myemail/0APpBNR1zr4FMUk9PVA/12q-qSbLQro63ptfXuTwA-MVwjDow0qiE/test/test/content/1bd52105ae6ef.nwd'

Traceback:
  File "/home/testuser/Downloads/novelWriter/./novelWriter.py", line 21, in <module>
    novelwriter.main(sys.argv[1:])

  File "/home/testuser/Downloads/novelWriter/novelwriter/__init__.py", line 293, in main
    nwGUI.showProjectLoadDialog()

  File "/home/testuser/Downloads/novelWriter/novelwriter/guimain.py", line 962, in showProjectLoadDialog
    self.newProject()

  File "/home/testuser/Downloads/novelWriter/novelwriter/guimain.py", line 385, in newProject
    if self.theProject.newProject(projData):

  File "/home/testuser/Downloads/novelWriter/novelwriter/core/project.py", line 293, in newProject
    aDoc.writeDocument(titlePage)

  File "/home/testuser/Downloads/novelWriter/novelwriter/core/document.py", line 165, in writeDocument
    os.replace(docTemp, docPath)

[vkbo]

Hmm, interesting. I've never tested novelWriter with Google Drive.

The line where it crashes is new in 1.6 Beta 1, but if you say it crashes in earlier versions as well, than that is in itself not the issue.

What is happening is that it is writing the document to a temp file, then replacing the original file with that temp file if the write was successful. It's a safeguard against data loss. It seems that the writing is fine, but the replacing fails. Curious.

I'll add a crash safeguard around that line, but it doesn't solve the problem.

I will set up a Google Drive folder and give it a try myself and see if I can reproduce this.

[vkbo]

I've spent quite a bit of time trying to reproduce this, but I cannot for the life of me find a straight forward way to even mount Google Drive on Linux. I can get the link to it in Nautilus, but that's about it. It doesn't seem to be accessible anywhere else, and there is certainly nothing in /run/user/1000/gvfs.

I need something more to go on here ...

[vkbo]

I've added exception handling around the call to prevent the crash in #961, but that doesn't solve the underlying io error. I suspect there is some file locking going on by the gvfs/gio library.

A brute force solution may be to implement a custom function that runs the os.replace call in a for loop of say 5 steps, with a time.sleep call in between, and only fail if the for loop exits with no success. But I'd like to be able to test if it works first.

Edit: I've added a loop that tries 5 times with a sleep time in between that goes in steps of 50 ms from 0 to 200 ms. Would you be able to test the solution from the branch in #961?

[vkbo]

I managed to mount it up in the same way you did in a VM with a different distro. My fix does not work because the issue seems to be with Google Drive encoding the file and folder names. Accessing the files directly in /run/user/1000/gvfs simply does not work. Google does not provide an app on Linux, so the solution there is probably a lot less robust, which is why it may work on other OSes.

I have no solution to this. If novelWriter cannot find the files it just made, it obviously cannot work.

[ding0z]

I will have a look.

[ding0z]

I managed to mount it up in the same way you did in a VM with a different distro. My fix does not work because the issue seems to be with Google Drive encoding the file and folder names. Accessing the files directly in /run/user/1000/gvfs simply does not work. Google does not provide an app on Linux, so the solution there is probably a lot less robust, which is why it may work on other OSes.

I have no solution to this. If novelWriter cannot find the files it just made, it obviously cannot work.

I tried to add a Google Account to the OS using Fedora and PopOS so the GDrive appears directly in the file manager.
I don't know if it could make a difference.

[ding0z]

Update: I tried the branch #961 but still the same problem.

[vkbo]

Yes, the branch doesn't solve it for me on Ubuntu 20.04 either. The problem seems to be more fundamental.

I've read up on the topic a bit, and Google maps file paths to a database in a virtual file system. See for instance this blogpost. As long as you can access that file system through a layer that is POSIX-compliant, it probably works fine. I guess that is what's going on with macOS via Google's own app/driver.

On neither Debian 11 nor Ubuntu 20.04 can I browse to the google drive in the file dialog when creating a project from novelWriter, but the drive does show up in Nautilus. On Ubuntu I was able to navigate to the /run/user/ path, and then I got the same error you did. Adding error handler around the spot where it failed on the current main branch, I managed to get further and see that it cannot actually find the files/folders it created.

The error does in any case not occur within novelWriter itself, but within Python's os library. I am not sure if there is anything I can do to make the os library handle these paths. I don't do anything to bypass symlinks and other types of links either. So I'm at a loss what to do.

Google provides a Python package to interact with Google Drive directly. I fear I would have to write a wrapper using this to be able to make it work. If so, I would need to abstract away all I/O access in order to be able to add a Google layer. I'll look for other solutions, but at the moment I'm at a loss.

[vkbo]

I've merged the fix that solves the crash itself, but it obviously doesn't solve the issue with handling of volatile paths. I don't think I have a solution for that. Even standard OS tools cannot handle this directly without going through the virtual file system tools. You cannot ls such a folder from command line either, for instance. You need to use the gio tool instead.

I think any solution here lies outside the scope of novelWriter. The crash was definitely a bug, but I think the requirement that the projects are saved in a non-volatile location is fair. This is really an issue with the lack of proper support for Google Drive on Linux as far as I can tell. I will make a note of it in the docs though, so I'll leave the issue open until then as a reminder.