[bug] [form] updates break on model changes

[janoskabedis]

When we go to update a form after changing the model, we sometimes run into issues where it either expects a field no longer there or struggles with a comparison between the new and old field when the new field isn't in the old data.

  File "/opt/libreforms-fastapi/libreforms_fastapi/utils/document_database.py", line 532, in update_document
    updated_data_dict[field] != document['data'][field],
                                ~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'a_new_field_not_contained_in_the_previous_one'

This may be somewhat related to #169.

[signebedi]

One of the assumptions of this application is that the form model can change at any time, so we need to make sure that the application can handle three situations: first, when a field that was once in the data model is no longer in the data model; second, when a field that was not originally in the data model, is suddenly added; third, when an existing field is changed to a new field. Each these scenarios are important when a user submits a form under the original model, but then goes to update it under the new model.

[signebedi]

# Here we remove data that has not been changed
dropping_unchanged_data = {}
for key, value in updated_data_dict.items():

    # If the key is in the original document and its value is new
    if key in document['data']:
        if value != document['data'][key] and value is not None:
            dropping_unchanged_data[key] = value
    else:
        # If the key is not in the document (new field) and the value is not None
        if value is not None:
            dropping_unchanged_data[key] = value

I think this logic presents a cleaner approach that does not handle all conditions as part of the same if-statement, which is what seems to have caused the error above. We want to support partial or total data updates from users when they submit /api/form/update payloads over the UI and API, and to drop fields from the /api/form/update payload that present no changes from existing data. We do not necessarily want to delete fields from documents simply because they are not passed as part of the update, because this would effectively change our approach to requiring total data updates.

That said, cleaning out old fields from documents when the data model changes isn't the worst idea; it keeps data small, and the old data is retained in the journal in any event. However, while our journaling system handles changes to the values assigned to existing fields & the addition of new fields just fine, it does not have a straightforward way to denote dropped fields... that is, when an existing field is removed, or changed to a new name.

What this means is that the form data model can handle new fields just fine. It can handle the removal of fields in a somewhat strange way; that is, new data will not be impacted, but old data will retain the old field data. It will handle changes to a field name poorly - because old data will have data under the old field name, while new data will have data under the new field name.

Maybe the answer is to add a straightforward method to coerce old data - eg. an admin endpoint to remove a specified field from existing data, to remove all fields from old data that are no longer in the form model, or change values stored under an old field to a new field name. We can then write a custom note to the journal under a metadata field named something like model_changes to document the changes made in a human and machine readable way... @rl-utility-man any thoughts?