please clarify precise conditions for concurrent access

[frankseide]

Hi, I need to access the same blob container concurrently from multiple running jobs (each mounting it via BlobFUSE) and my local workstation. Could you please very precisely document the constraints of that? The documentation is not clear on this.

In my case, each running job accesses a different subdirectory inside the same container. They never write to the same files. All file writes are either write-once-immutable-after or write-append (log files). All jobs read the same files (input data and binaries). I might read files from my workstation once they have been written, and it is OK if those become visible only after a timeout; and in the case of log files written incrementally, it is OK if the updates become visible at a delay.

In short, the same blob is never written to by multiple mounts, but may be read from by multiple, where minute-long delays of a new file becoming visible are acceptable.

From the documentation, it is not clear whether this is possible/supported, and what edge cases to consider for this scenario. On the other hand, I see no reason why a caching system might get confused by this.

So, can you please clarify the precise conditions under which concurrent access can happen?

BTW the documentation says on one hand "By setting --file-cache-timeout-in-seconds to 0, you may achieve close-to-open cache consistency like in NFS v3. This means once a file is closed, subsequent opens will see the latest changes from the Blob storage service ignoring the local cache." This indicates it is OK that a blob is changed while the container is mounted.

On the other hand, it says "While a container is mounted, the data in the container should not be modified by any process other than blobfuse. This includes other instances of blobfuse, running on this or other machines. Doing so could cause data loss or data corruption." I read this as individual blobs in the container must not be modified concurrently? Otherwise, one would have to create a new container for every running job. That is not practical.

[amnguye]

Hi Frank,

I think you might have answered your own question. We did document that "While a container is mounted, the data in the container should not be modified by any process other than blobfuse. This includes other instances of blobfuse, running on this or other machines. Doing so could cause data loss or data corruption." And your interpretation of that is correct. The reason why we don't go more into about what the edge cases and the constraints is because this is not supported.

To comment on a new file becoming visible at a delay, the files will become visible on the container once you have closed the file. It will get flushed and uploaded to the container.

[frankseide]

But it is supported, in practice: I am actually doing what I described, and it works just fine. I do that, because it is simply not feasible to create a new set of containers for every single job I run. There also does not seem a reason why it should not work. Consider this scenario:

• one container, say data
• job 1, on machine 1, has data mounted via BlobFUSE, and writes its output to a sub-path, say data/job1/output.txt
• job 2, on machine 2, has data also mounted via BlobFUSE, and writes its output to a sub-path, say data/job2/output.txt
• job 2 does not touch anything in data/job1, and vice versa
• on my desktop machine, I also have data mounted via BlobFUSE. When each job is done, I access the respective files via BlobFUSE from my local machine.

As I understand it, once a job has finished writing its file, it will close it, which will then have it uploaded to blob storage. Since both jobs write into different folders, and never access each other's folders, this seems no different than uploading it with azcopy. Then, my local BlobFUSE mount will either not see the file, or after some timeout, will bother to look and see the file. Why would this not be supported? Where is the issue here, specifically what issue do we have here that we don't already have if it was only a single job?

So my real question is: Since this works just fine, could you please discuss the risks that I am taking, to the level that I can make an informed design that solves my need as much as possible while minimizing unnecessary risks. And it would be super-great if you could you make it supported (under clearly specified constraints). Most likely that will cost no code change, just a specification of the edge cases.

[amnguye]

I'm sure it works for now for you. However if this scales and for other types of scenarios it could potentially cause data corruption. Blobfuse was not written or designed for other processes accessing and modifying the container while it is also trying to access data in the container. We've seen blobfuse reupload files that were deleted by a different blobfuse process before, and not update files accordingly.

It may just work fine for you, however this is an extremely unrecommended use case for blobfuse in the off case that it might cause data corruption. Which is why we don't document the edge cases and constraints if you decide to use it in that scenario. That would only encourage users to use it in this way and potentially cause data corruption.

Supporting this case would require major code change.

[NaraVen]

@frankseide , your scenario " writing to different directories only from different instances" will work if you set --use-attr-cache=false and --file-cache-timeout-in-seconds=0. You are not taking any risks. We have verified this in our latest version 1.3.4