Exception occurred while processing related to a filepath: Folon_HumanRaceStat

[joric]

Fails at about 2 gigs of JSON combined, processing LondonWorldSpace.esm, full log entry:

2024-08-30 20:28:13.353 +05:00 [ERR] Error syncing to Git
Mutagen.Bethesda.Serialization.Exceptions.FilePathedException: Exception occurred while processing related to a filepath: D:\Projects\github\spriggit-project\trunk\LondonWorldSpace\RecordData.yaml
 ---> SubrecordException LondonWorldSpace.esm => Folon_HumanRaceStat (31A873:LondonWorldSpace.esm<Race>) => AM� : Unexpected subrecord    at Mutagen.Bethesda.Plugins.Records.GroupMajorRecordCacheWrapper`1.GetEnumerator()+MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at Noggog.EnumerableExt.WithIndex[T](IEnumerable`1 en)+MoveNext() in D:\a\CSharpExt\CSharpExt\Noggog.CSharpExt\Extensions\EnumerableExt.cs:line 223
   at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
   at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
   at Noggog.WorkEngine.InlineWorkDropoff.EnqueueAndWait[T](IEnumerable`1 items, Func`2 action, CancellationToken cancellationToken) in D:\a\CSharpExt\CSharpExt\Noggog.CSharpExt\WorkEngine\InlineWorkDropoff.cs:line 76
   at Mutagen.Bethesda.Serialization.Utility.SerializationHelper.WriteFilePerRecord[TKernel,TWriteObject,TGroup,TObject](StreamPackage streamPackage, TGroup group, String fieldName, SerializationMetaData metaData, MutagenSerializationWriterKernel`2 kernel, WriteAsync`3 groupWriter, HasSerializationItems`1 groupHasSerializationItems, WriteAsync`3 itemWriter, Boolean withNumbering)
   at Mutagen.Bethesda.Fallout4.Fallout4Mod_Serialization.SerializeFields[TKernel,TWriteObject](TWriteObject writer, IFallout4ModGetter item, MutagenSerializationWriterKernel`2 kernel, SerializationMetaData metaData) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\obj\Release\net8.0\Mutagen.Bethesda.Serialization.SourceGenerator\Mutagen.Bethesda.Serialization.SourceGenerator.Serialization.SerializationSourceGenerator\Fallout4Mod_Serializations.g.cs:line 1341
   at Mutagen.Bethesda.Fallout4.Fallout4Mod_Serialization.Serialize[TKernel,TWriteObject](TWriteObject writer, IFallout4ModGetter item, MutagenSerializationWriterKernel`2 kernel, IWorkDropoff workDropoff, IFileSystem fileSystem, ICreateStream streamCreator, CancellationToken cancel) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\obj\Release\net8.0\Mutagen.Bethesda.Serialization.SourceGenerator\Mutagen.Bethesda.Serialization.SourceGenerator.Serialization.SerializationSourceGenerator\Fallout4Mod_Serializations.g.cs:line 36
   at Mutagen.Bethesda.Serialization.Yaml.MutagenYamlConverterFallout4ModMixIns.SerializeInternal[TMeta](MutagenYamlConverter converterBootstrap, IFallout4ModGetter item, DirectoryPath path, IWorkDropoff workDropoff, IFileSystem fileSystem, ICreateStream streamCreator, TMeta extraMeta, WriteAsync`3 metaWriter, Nullable`1 cancel) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\obj\Release\net8.0\Mutagen.Bethesda.Serialization.SourceGenerator\Mutagen.Bethesda.Serialization.SourceGenerator.Serialization.SerializationSourceGenerator\MutagenYamlConverter_Fallout4Mod_MixIns.g.cs:line 48
   --- End of inner exception stack trace ---
   at Mutagen.Bethesda.Serialization.Yaml.MutagenYamlConverterFallout4ModMixIns.SerializeInternal[TMeta](MutagenYamlConverter converterBootstrap, IFallout4ModGetter item, DirectoryPath path, IWorkDropoff workDropoff, IFileSystem fileSystem, ICreateStream streamCreator, TMeta extraMeta, WriteAsync`3 metaWriter, Nullable`1 cancel) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\obj\Release\net8.0\Mutagen.Bethesda.Serialization.SourceGenerator\Mutagen.Bethesda.Serialization.SourceGenerator.Serialization.SerializationSourceGenerator\MutagenYamlConverter_Fallout4Mod_MixIns.g.cs:line 53
   at Mutagen.Bethesda.Serialization.Yaml.MutagenYamlConverterFallout4ModMixIns.Serialize(MutagenYamlConverter converterBootstrap, IFallout4ModGetter item, DirectoryPath path, SpriggitSource extraMeta, IWorkDropoff workDropoff, IFileSystem fileSystem, ICreateStream streamCreator, Nullable`1 cancel) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\obj\Release\net8.0\Mutagen.Bethesda.Serialization.SourceGenerator\Mutagen.Bethesda.Serialization.SourceGenerator.Serialization.SerializationSourceGenerator\MutagenYamlConverter_Fallout4Mod_MixIns.g.cs:line 89
   at Spriggit.Yaml.Fallout4.EntryPoint.Serialize(ModPath modPath, DirectoryPath outputDir, Nullable`1 dataPath, GameRelease release, IWorkDropoff workDropoff, IFileSystem fileSystem, ICreateStream streamCreator, SpriggitSource meta, CancellationToken cancel) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\EntryPoint.cs:line 36
   at Spriggit.Yaml.Fallout4.EntryPoint.Serialize(String modPath, String outputDir, String dataPath, Int32 release, String packageName, String version, CancellationToken cancel) in D:\a\Spriggit\Spriggit\Translation Packages\Spriggit.Yaml.Fallout4\EntryPoint.cs:line 108
   at Spriggit.Engine.AssemblyLoadedEntryPoint.Serialize(ModPath modPath, DirectoryPath outputDir, Nullable`1 dataPath, GameRelease release, IWorkDropoff workDropoff, IFileSystem fileSystem, ICreateStream streamCreator, SpriggitSource meta, CancellationToken cancel) in D:\a\Spriggit\Spriggit\Spriggit.Engine\AssemblyLoadedEntryPoint.cs:line 57
   at Spriggit.Engine.Services.Singletons.SpriggitEngine.Serialize(ModPath bethesdaPluginPath, DirectoryPath outputFolder, Nullable`1 dataPath, Boolean postSerializeChecks, IEngineEntryPoint entryPt, SpriggitMeta meta, Nullable`1 cancel) in D:\a\Spriggit\Spriggit\Spriggit.Engine\Services\Singletons\SpriggetEngine.cs:line 64
   at Spriggit.UI.ViewModels.Transient.LinkVm.SyncToGit(CancellationToken cancel) in D:\a\Spriggit\Spriggit\Spriggit.UI\ViewModels\Transient\LinkVm.cs:line 244

There's still plenty of free space on disk and I don't think it really syncs to git, it fails at a parsing stage.
I didn't add other files besides LondonWorldSpace.esm, I don't really know how it processes external references.
It's much smaller in YAML though (just about 600M) but it fails in the same spot. Bad record at "31A873:LondonWorldSpace.esm".

[Noggog]

Should hopefully be able to digest that record better in 0.27.

As mentioned in Discord, though, it might be more applicable to process the mod via Mutagen C# code, rather than translating to 2GB+ of json to then process