Atomic "Move" Operation for Files

[rbair23]

Sometimes, you may want to take action when a File in the Hedera File Service has been uploaded. However, each file is uploaded in 6k "chunks", because Hedera has a limit on the size of transactions. One idea, is to add a new "move" transaction to the file service. You can upload the file to the file service, and once it is uploaded, "move" it over top another file.

For example, the properties file is file 121. Rather than uploading a new properties file into 121, which may take multiple chunks, we could upload to some new file (maybe it is 0.0.12345678 -- whatever entity id is next), and then do a "move" from 0.0.12345678 to 0.0.121. "move" would remove the old entity and perform its work without actually moving bytes around. This is actually a little tricky in the implementation, but should be doable without any excessive overhead.

If you have "move", you might also want "copy". However this is really tricky, because we would not want to actually copy the bytes, but we would want copy-on-write behavior. This is probably far from trivial in the implementation today. So we wouldn't do this right now.

[bugbytesinc]

I really like this idea, but why just not "copy"? -- I realize storage is an issue (which is addressed via rent), it seems from the outside perspective that it might be nice to have more control over when the source file is destroyed.

[rbair23]

My concern with copy is around the TPS more than the storage or cost. To do a copy would require reading the entire file into memory and write it back out again. If instead the copy could be done by having the target hold a reference to the same bytes as the source, and then if the source is updated, the target would keep the pointer to the bytes and the source would get a new pointer to new bytes, it would be way more efficient. But we don't have the plumbing to support that at the moment.

[rbair23]

If the file service were updated so that file bytes are stored in one place with its own ID, and then file entities have a "hidden" reference to this ID, then it would be trivial to implement all these move / copy operations efficiently.

[rbair23]

☝🏻 Not that hard to do, I think...

[bugbytesinc]

This observation goes beyond this hip, it really applies to the whole of Hedera, but Hedera needs to start thinking from the perspective of the network is a service not the network is a piece of engineering beauty.

To be successful and utilized, the network needs to be focused on a client centric design that makes it easy for adoption, and then work out the minutia regarding the implementation after creating a good HAPI design.

In this concrete example, "moving" a file adds cognitive complexity of "what happens to old file name 0.0.123" -- this really not trivial to document and for clients to grapple with. "Copy" is easy, contents from file 0.0.123 are now in 0.0.456. See? Much easier to understand and grep. These designs should be customer centric first, not what is cool to implement under the hood.

[rbair23]

I'm pulling the term "move" from unix & mac OS. Move and Copy in unix have these exact semantics. I agree completely in designing APIs that are user centric. I do agree that we have some APIs that could have been designed in a more friendly way. The challenge is that users are not a homogenous group. Calling it "move" communicates to the user exactly how this operates --- if the user is from the unix world (of which a very large percentage of the developer community is familiar). I remember struggling with unix looking for "rename" when I was first using it, only to find it was "mv". Go figure :-). Nevertheless, I think "move" and "copy" have very clear semantics that most developers would intuitively understand because we would be using the exact same semantics as Unix / Linux / BSD Unix / Mac OSX / etc.