EncFS build with WinFsp / Cygwin FUSE

[benrubson]

Hi,

EncFS provides an encrypted filesystem in user-space, using FUSE library.
https://github.com/vgough/encfs

Perfect, let's then give it a try using WinFsp and its Cygwin FUSE compatibility layer !

Fresh new install of Cygwin64 on a Windows 2012 R2 Server.
With following required packages :
cmake
make
gcc-g++
gettext-devel
openssl-devel
procps-ng

Install WinFsp FUSE into Cygwin :
cd /cygdrive/c/Program*/WinFsp/opt/cygfuse
sh install.sh

In /usr/include/fuse/winfsp_fuse.h, modify :
void *cygwin_create_path
To :
char *cygwin_create_path

Get last EncFS version :
https://github.com/vgough/encfs/releases
(encfs-1.9.4.tar.gz at time of writing)
Uncompress the archive and enter the project directory.

In CMakeLists.txt, modify :
-DFUSE_USE_VERSION=29
To :
-DFUSE_USE_VERSION=29 -DCYGFUSE

And manual build :
https://github.com/vgough/encfs/blob/master/INSTALL.md
mkdir build
cd build
cmake ..
make

Let's then test :
mkdir /tmp/enc /tmp/dec
./encfs --standard --extpass="echo test" /tmp/enc /tmp/dec

[benrubson]

So unfortunately, I'm facing the following issue :
Cannot create WinFsp-FUSE file system: mount point in use.

I found quite a similar issue with the build of s3fs (#143).

EncFS uses fuse_main() :
https://github.com/vgough/encfs/blob/v1.9.4/encfs/main.cpp#L185

Thus I'm not sure we have something that checks mountpoint we could modify to make WinFsp-FUSE happy...

Any clue ?

Thank you very much !

Ben

[billziss-gh]

WinFsp FUSE requires that the drive/directory does not exist when mounting. Please see FAQ entry 2 under "FUSE":

https://github.com/billziss-gh/winfsp/wiki/Frequently-Asked-Questions#fuse

[benrubson]

Sorry, I'm confused, I should have red the FAQ...
It now correctly mounts, perfect, thank you again @billziss-gh for your support 👍

Some issues though with stat (I think), I'll try to figure-out what's going on :

[billziss-gh]

Try mounting on a drive using the mountpoint x: to mount to drive x.

When mounting on a directory Windows expects the file system to be case-insensitive. If it is not multiple problems ensue.

[benrubson]

@billziss-gh, thank you, it helped 👍
I'm debugging to make EncFS functional, write does not work (well did).
EncFS directly modifies the buffer given to write() operation (to encrypt it).
But each time it does so, it crashes.
I discovered than duplicating the buffer (and modifying a copy) corrects the issue.
Does WinFsp prevents its buffer to be overwritten ?
Thx 👍

[billziss-gh]

(By crash I am assuming that only the file system process crashes and not the whole OS, otherwise this is a very serious bug.)

This is by design.

WinFsp uses zero-copy techniques where it may map the write buffer in the address spaces of both the originating process (the process that issued the WRITE) and the file system process. For this reason it disallows writing into the WRITE buffer by marking the pages READ-ONLY.

BTW, note that the write operation in fuse_operations has the buffer marked const, so EncFS is not playing nice here ;-)

[benrubson]

I think so too @billziss-gh ! Thank you for your explanation, it confirms what I discovered 👍
I will push a modification to EncFS, simply memcpying the buffer is OK and should be the way to go.
Next bug :)

[benrubson]

OK, I now have a bug deleting files.
Before unlinking a file, EncFS checks that the file is not opened :
https://github.com/vgough/encfs/blob/v1.9.4/encfs/DirNode.cpp#L735

Unfortunately, deleting a file constantly fails on the above check.
Looking for hard_remove in WinFsp code, I found this interesting comment :
https://github.com/billziss-gh/winfsp/blob/v1.2POST1/src/dll/fuse/fuse_intf.c#L1054

Does it means that a file being deleted on Windows is inevitably opened ?
If so, I may have to modify EncFS test, and check for example that the file is opened only once.

Sounds like I encounter quite similar behavior renaming, is a file opened before it is being renamed ?

Thank you 👍

[billziss-gh]

Unfortunately this is indeed the case in Windows. While in POSIX there are distinct namespace operations (like unlink, rename, etc.), on Windows all operations go through a file handle internally. This means that a DeleteFile is really the sequence of Open/CanDelete/Cleanup/Close, a MoveFileEx is really the sequence of Open/Rename/Cleanup/Close and even a GetFileAttributes is really the sequence of Open/GetFileInfo/Cleanup/Close.

However I am surprised the EncFS has the limitation you described. It looks like you would have to modify it to remove it or relax it.

[benrubson]

Thank you for your confirming this in details @billziss-gh 👍

EncFS assumes a file should not be opened before being deleted. Otherwise, hard_remove may have been enforced, so it's better not to delete the opened file :

hard_remove
              The default behavior is that if an open  file  is  deleted,  the
              file  is  renamed  to  a hidden file (.fuse_hiddenXXX), and only
              removed when the file is finally released.   This  relieves  the
              filesystem  implementation  of having to deal with this problem.
              This option disables the hiding behavior, and files are  removed
              immediately  in  an  unlink  operation (or in a rename operation
              which overwrites an existing file).

Well workaround is easy then, just verify that the file is opened only once.

it's a little bit more tricky with rename !
EncFS keeps track of opened files, and how many times they are opened.
In a rename operation foo -> bar, Windows opens foo, EncFS sets its open-counter to 1.
Windows renames foo to bar, internally EncFS moves open-counter foo to bar, to 1 time then.
Windows closes foo, EncFS tries to decrease foo counters but fails, and bar counter is not decreased, thus remains to 1.
So things like deleting a file after having renamed is not possible (because its open-counter is too high).
Quite tricky case, I'm trying to relax it.

[billziss-gh]

I am curious. Why does EncFS care if a file is open or not? POSIX file systems rarely do.

[benrubson]

EncFS has for example an auto-unmount-after-inactivity function, which then unmounts only if there is no remaining opened file.
To each file node is also attached some info such as the encrypted filename etc...
To avoid keeping this into memory, the info is released once all handles to this file are closed.

[billziss-gh]

Thanks.

[benrubson]

So, after some patches, EncFS now works pretty good.

Before digging into performance testing, I have this remaining "bug" :
when I choose a drive as the source drive (for example /cygdrive/c), the mount point (for example U:) is generally not reachable from Windows itself.
I can however browse it (/cygdrive/u) from Cygwin.
If I chmod 755 /cygdrive/u, then U: is available from Windows.
Any reason why mount point is set like this ?

And a tiny question, is kill (still) the common method to unmount ?

Thank you !

[billziss-gh]

If I chmod 755 /cygdrive/u, then U: is available from Windows.
Any reason why mount point is set like this ?

That is indeed weird. Try to mount as u: rather than /cygdrive/u and see if that fixes the problem.

You should use u: and not /cygdrive/u as a mount point to be on the safe side. While FUSE for Cygwin will use Cygwin's cygwin_create_path I am uncertain whether that converts /cygdrive/u to u: or u:\ and the two are different for WinFsp.

Another likely reason is that EncFS reports st_mode==0 or st_uid==-1 etc. in getattr.

And a tiny question, is kill (still) the common method to unmount ?

Yes, there are is no unmount operation on Windows. Killing the file system process will correctly also unmount the file system. This happens even if the process dies unexpectedly.

[benrubson]

I use /cygdrive/u on the command line, but then (once patched) EncFS gives u: to Fuse (this is why we correctly see the mounted U: drive on my screenshot).
/cygdrive/u is however kept as the internal mount point by EncFS to avoid breaking its internal logic.
All other files have their stat info correctly shown, but the mount point.

I just discovered that a first chmod is necessary on the mount point for Windows to access it.
Then, it will be OK, subsequent mounts will no longer require it. I think (confirmed below) that this chmod must certainly update C: mode too.
If for example I chmod 000 the mount point, Windows can't access it anymore, and /cygdrive/c mode is also updated, equally, thus Windows can't access C: anymore !

[billziss-gh]

I use /cygdrive/u on the command line, but then (once patched) EncFS gives u: to Fuse (this is why we correctly see the mounted U: drive on my screenshot).

Fair enough :)

I just discovered that a first chmod is necessary on the mount point for Windows to access it.

By default drives created by WinFsp are accessible and they do not need chmod (or any Windows equivalent). This must be an EncFS peculiarity/bug.

If for example I chmod 000 the mount point, Windows can't access it anymore, and /cygdrive/c mode is also updated, equally, thus Windows can't access C: anymore !

I am suspecting that you may have to change how EncFS handles getattr, etc. for the root directory. And certainly chmod 000 should not be changing the C: drive IMO :)

Your screenshot appears to show Cygwin responding to ls -l /cygdrive with st_mode==S_IFDIR|000 and unknown st_uid, st_gid. This certainly looks incorrect.

[benrubson]

Your screenshot appears to show Cygwin responding to ls -l /cygdrive with st_mode==S_IFDIR|000 and unknown st_uid, st_gid. This certainly looks incorrect.

Yes this is how the mount point /cygdrive/u (U:) is created for the first time.
This is why Windows can't browse it (cygwin can however).

[benrubson]

EncFS may return EBADMSG if underlying data has been corrupted :
#define EBADMSG 77 /* Bad message */
Unfortunately WinFsp does not seem to handle this return code, as EPERM is returned instead.
Could it be possible somehow to map it for example to STATUS_DATA_ERROR ?
Goal is that when EncFS returns EBADMSG, calling app gets EBADMSG.

Other finding, even if I set -oattr_timeout=0, I need to add -oFileInfoTimeout=0 to get the expected behavior (when the underlying file is modified, to get the correct stat info on FUSE side immediately).
Strange because this seems to be done here (however set_FileInfoTimeout is never initialized to 0).

Thank you 👍

[billziss-gh]

EncFS may return EBADMSG if underlying data has been corrupted :
#define EBADMSG 77 /* Bad message */
Unfortunately WinFsp does not seem to handle this return code, as EPERM is returned instead.
Could it be possible somehow to map it for example to STATUS_DATA_ERROR ?
Goal is that when EncFS returns EBADMSG, calling app gets EBADMSG.

I would have thought that EBADMSG is not an error code that a file system should return. Do you have any evidence that this is used for file system purposes in Linux, BSD, etc.?

In any case this is something that should be relatively easy to add.

Other finding, even if I set -oattr_timeout=0, I need to add -oFileInfoTimeout=0 to get the expected behavior (when the underlying file is modified, to get the correct stat info on FUSE side immediately).
Strange because this seems to be done here (however set_FileInfoTimeout is never initialized to 0).

This may indeed be a bug.

[benrubson]

I would have thought that EBADMSG is not an error code that a file system should return. Do you have any evidence that this is used for file system purposes in Linux, BSD, etc.?

EncFS returns this specific code when authentication of decoded data fails (when data is corrupted, even perhaps intentionally by a bad person). This allows to quickly identify an issue with data decoding, whereas the default EIO could refer to many other (sometimes low-level) things.
I just had a look again to Linux errno.h, the following one could also work :
EILSEQ : Illegal byte sequence
Perhaps you prefer it ?

This may indeed be a bug.

Should I open a bug to keep track of this ?

[billziss-gh]

I believe EIO would have been the right error here, for a file system at least. But I understand your reasoning.

I any case WinFsp should not really have an opinion on what error codes FUSE file systems return, so I agree that we should add EBADMSG and other potentially missing error codes. (Provided that they are defined for both Microsoft Visual C and Cygwin; I believe EBADMSG is defined in both.)

Should I open a bug to keep track of this ?

You can open bugs for them of course, although I would prefer PR's even more ;-)

[benrubson]

Thinking again about EBADMSG vs EILSEQ, I would say that EILSEQ would be preferable and seems to better match our need (and also seems to better match the Windows status STATUS_DATA_ERROR).

[billziss-gh]

EILSEQ is already mapped to STATUS_INVALID_PARAMETER (link)

[benrubson]

I close this now, the PR to support EncFS on Cygwin is almost ready :
vgough/encfs#499

I will open individual issues for the remaining (non-blocking) findings.

Bill, thank you very much for you really nice support here 👍

Edit : PR merged to EncFS master, EncFS 1.9.5 will be Cygwin ready.

[billziss-gh]

No problem. Glad to be of help.

[mhogomchungu]

@benrubson,

Shouldn't there be a binary release of encfs for windows? Asking windows users to build themselves is simply not practical.

[benrubson]

@mhogomchungu please open an issue directly in Encfs repository ?
We'll then be able to track it and see what is possible to do.
Thx 👍

[benrubson]

@billziss-gh you can add EncFS to the list, v1.9.5 which supports Cygwin has just been released 👍
Thx !

[billziss-gh]

🎉 🎉 🎉

@benrubson congratulations!

Will add it ASAP.

[billziss-gh]

Done: commit 5ae0804 and wiki change.

[benrubson]

Many thanks Bill 👍 🎉

[ScottRFrost]

Almost a year later and I found this. It's not clear how to set this up @benrubson . Is it possible with a native build or is cygwin required? If so, how do yo use it? It looks like WSL doesn't support FUSE yet, so I assume that's out for now.

[benrubson]

@ScottRFrost follow the wiki ?
Cygwin is required yes for the build. Once built, you should only need some needed Cygwin DLLs.