How deep is a deep copy?

[focustense]

I've been trying this out and seeing results that look more like a shallow copy to me. I read the wiki page on copying but it's a tad ambiguous to me, and also had a somewhat hard time following the code.

My interpretation of "deep" is that it also clones dependencies, so e.g. if I DeepCopyIn an NPC including head parts, it will copy the HDPT records as well. But I think perhaps what "deep copy" really means is that it copies the entire binary blob and will therefore preserve things like arrays containing structs containing arrays, if such a thing exists. But it looks like references are still copied as references, and new masters are added as required. This isn't much different from what XEdit's "copy as new" does, I just had different expectations based on the name.

Is this the intended behavior, or am I somehow using it wrong? Ultimately the behavior I want is something like "recursively copy the targets of any form links, unless those targets are already present in the destination plugin's masters, or there's already a copy sitting in the destination plugin". From what I gather, I will have to do this explicitly, and not be able to rely on DeepCopy semantics?

[Noggog]

Definitions as I see them:

Shallow Copy

A shallow copied npc would be a new C# object, but all references within it would point to the same objects as the original. So you would get this behavior:

var npc = new Npc();
npc.Name = "MyName";
var npcShallowCopy = npc.ShallowCopy();
npc.Items.Add(new ContainerEntry());

// This will print 1, because both npcs point to the same list for their Items member
Console.WriteLine($"Shallow npc has {npcShallowCopy.Items.Count} items and is named {npcShallowCopy.Name}.");
// prints Shallow npc has 1 items and is named MyName

This concept does not exist in Mutagen

DeepCopy

A deep copy is exactly the same above, except each subobject also gets cloned to a new object, but with the same overall contents. Equals would return true, and when exported the bytes would be identical. But this would be out of coincidence. They're identical twins, not siamese twins.

var npc = new Npc();
npc.Name = "MyName";
var npcShallowCopy = npc.DeepCopy();
npc.Items.Add(new ContainerEntry());

Console.WriteLine($"Shallow npc has {npcShallowCopy.Items.Count} items and is named {npcShallowCopy.Name}.");
// prints Shallow npc has 0 items and is named MyName

The name was copied from npc, but since the item Add happened after the deep copy, the shallow version is still empty.

FormLinks

FormLinks get their value copied over. So if the original record pointed to something 012345:Skyrim, the new copy will, too. The actual record 012345:Skyrim points to will not be looked up and interacted with whatsoever.

DeepCopyIn

DeepCopyIn is the same, except the object being filled is given to it, rather than created internally. So actually, a DeepCopy could be seen as using DeepCopyIn:

var newObject = new ObjectOfSomeType();
newObject.DeepCopyIn(rhs);
return newObject;

DeepCopyIn would actually be doing the heavy lifting of cloning the content

Duplicate

This is actually what I'd say is the equivalent of xEdit's "copy as new". It does a DeepCopy, but assigns a new FormKey.

DuplicateFromOnlyReferenced

A variant of this is probably what you're after. It's half done, experimental, and horridly named.

This code will inspect a target mod, and duplicate all contained records of a given type (copy + give it a new FormKey), as well as duplicating all records they point to, recursively. It will then also reroute the FormLinks in all the records to each other's dups (via the RemapLinks call).

https://github.com/Mutagen-Modding/Mutagen/blob/release/Mutagen.Bethesda.Core/Extensions/DuplicateFromMixIn.cs
The current code is an attempt for one of Piranha's projects (https://github.com/Piranha91). It's got a few edge cases to iron out before I'd say it's reliable.

Work to be Done

I imagine DuplicateFromOnlyReferenced will be renamed, solidified, tested a lot more in the coming future. Current state is just experimental proof of concept with some lingering edge cases to iron

I might rename Duplicate in the near future, as it's not a clear name as compared to Copy As New. Could be misinterpreted as DeepCopy, as they're somewhat synonymous terms, id say.

[Noggog]

Is it even possible to have a literal shallow copy?

It is, it's just got a lot of edge cases on what the definition of shallow is, or "how shallow". For example, does a shallow copy of an npc mean the Items list is the same instance of a List in both? Or just that each ContainerEntry on each separate list are the same object instances? It's not clear (hard to even write that out ^, lol), and it's hard to control/direct the behavior without some intense public API.

But yeah, you could do shallow-esque things in mutagen pretty easily:

var npc = new Npc();
var npc2 = new Npc();
var entry = new ContainerEntry();
npc.Items.Add(entry);
npc2.Items.Add(entry);
entry.Item.Count = 4;

Both npcs contain the same ContaineEntry instance, so they could be thought of as "shallow". Similar behavior would happen if a shallow copy was done where object instances were shared.

It would be dangerous

Not necessarily. It's a tool in the toolbox. For example, it would be perfectly safe to add the same ContainerEntry instance to 100+ npcs like shown above. The only difference is while the program is running, do you have to modify one ContainerEntry to affect all? or do you have to loop over each and modify? When exporting a shared setup to disk, the same ContainerEntry instance will get written again and again underneath each Npc record, and the end results would be perfectly valid.

Looks like for now I'll have to port the XEdit-based implementation.

does Mutagen have an equivalent of CleanMasters?

One question I have is why would you want a recursive deep copy? Presumably, you want to make modifications to each of those records, as just doing a deep copy and exporting just makes an ITPO. Having a one-call recursive deep copy means you have to then follow up and track down all the records you deep copied so you can modify them, or trim them out.

Consider these two patterns:

1. DeepCopy everything (or lots of things)
   Track down the copies that were made
   Modify the copies
   Run a CleanMasters after. (this would actually be a clean ITPO call, and clean masters would implicitly follow)
2. Loop over the records in readonly form
   Find the ones you want to modify
   Copy that single record to the outgoing patch
   Modify

Most synthesis patchers follow pattern 2, which doesn't make ITPOs, while also not needing clean functions to exist. I imagine Mutagen will eventually get a CleanItpos call, but it will be out of convenience and completeness, rather than necessity. Mutagen wont get a CleanMasters, as it automatically exports a minimal list on export based on records it contains.

Here's an example of pattern 2, just for reference:
https://github.com/Synthesis-Collective/khajiitearsshow/blob/de52d69359080e2a3cc46af7739ea03e833c1009/KhajiitEarsShow/Program.cs#L23

• loops all armors
• skips ones any aren't interesting
• Adds to patch via GetOrAddAsOverride (make deep copy, or get the one made earlier if asked repeatedly)
• modifies the record, and done.

I imagine the need for CleanItpos or CleanMasters is the result of following the existing xedit code patterns 1:1. I think the same goals can be achieved, but just through slightly different mentalities?

[focustense]

Yeah, I guess shared collections aren't "dangerous" in the sense of causing program crashes or anything like that - they're just very likely to produce results that are surprising to the person who wrote the code. I've debugged many an application in many a language where the root cause was exactly that kind of copying; it's especially prevalent in ES6 where it's so darned easy to shallow-copy an object. In my experience, you generally never want to copy a reference to a collection unless the caller specifically asks for it (and understands what they're asking for).

Anyway, Synthesis is a very cool project and there's even a Synthesis patcher that does something kinda sorta along the same lines as what I'm doing, but not exactly. Part of what this app does is create an uber-merge, so you can install 50 NPC mods and boil them all down to 1. And pretty much every NPC overhaul worth installing defines its own head parts. And wouldn't you know it, head parts are recursive!

So I could run what is effectively a breadth-first search to find the leaf parts, then go up to the parts that reference them, and so on, and keep track of all the mappings from the original parts (across many plugins) to the duplicated part, and copy each successive layer of parts while rewriting dependencies on the fly. Could do that, but you have no idea how big a pain in the ass that would have been with XEditLib - I would have had to construct the records field by field, because XEditLib throws a fit if you try to copy without adding masters first, and there's no option to say "copy everything but these". Mutagen does have that, so it is technically possible to rewrite these fields on the fly without ever accidentally introducing an unwanted master; I was just being lazy because the current algorithm has already been written and tested.

As I understand it, the DeepCopy method (not DeepCopyIn) creates kind of a free-floating record that isn't in any specific plugin until I add it to that plugin's respective container. So that would be the way to rewrite on the fly: make the copy, then rewrite its form links, then add to the merge.

The things I will be trying to do with Mutagen are probably going to be anything but typical - in fact I had to make my own changes to XEditLib to get them working before. But I don't want to go too far off-topic on your public GitHub board, so I'll leave it at that.

[Noggog]

they're just very likely to produce results that are surprising to the person who wrote the code

Definitely agree. Shallow copy was too complex to even expose in an intelligent way, so was eventually cut from Mutagen real early on.

probably going to be anything but typical

Sounds good! Yeah, if you've got crazy requests/ideas feel free to ask here or the discord. Always up for talking about adding new stuff for the crazier ideas. A lot of the features are the result of someone flexing the systems and inspiring a new feature (ModContexts, for ex)

[focustense]

After having written and tested this, I now realize why CleanMasters makes no sense: because Mutagen doesn't add the masters until the file is actually saved. Those masters literally aren't there to clean until it's already too late.

So I had to take a different tack - but because of the above, I didn't need to rewrite the records on the fly. I can just copy the records as normal, overwrite their head parts and extra parts, and as long as I get a clean sweep, the original plugins will never get added as masters in the first place.

It took several tries to get this exactly right - for some reason, AsSetter().SetTo(...) wasn't actually updating the reference, so I had to clear the old head parts and add new ones - but after working through those issues, I was able to get the expected result.

Seems quite a bit faster than XEdit was, as well, so that's a bonus.

[Noggog]

Yeah, Lists of FormLinks are get only, because that discourages shallow-esque patterns. If they were List<IFormLink<ISomeRecord>> then you could list[1].SetTo(someOtherRecord). And since they're lists, you might have put the same formlink instance on several lists, and changing it there would've affected all of them -> dangerous.

Having them be getters means you're forced to replace:
list[1] = someOtherRecord.AsLink()
which prevents many subtle shallow bugs from being able to occur in the first place

Same reason for having FormLink members not able to be swapped on records themselves

// cant do this:
npc.Race = new FormLink<IRaceGetter>(someRecord);

// because then you could do this:
npc2.Race = npc.Race;
npc2.Race.SetTo(someThing); // I just set it for both npcs, oops

// Instead:
npc.Race.SetTo(someRecord);
npc2.Race.SetTo(npc.Race);

// Each npc has its own formlink object, so you can't get it into a shallow-shared state in the first place