Resolve Key error in veda_dataset_pipeline

[anayeaye]

What

Small task: veda_dataset_pipeline currently removes assets object when mutating payload between tasks but should not fail when no assets object is provided in dataset config.

[2025-01-24, 18:41:33 UTC] {taskinstance.py:2731} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/miniconda3/envs/py11/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 444, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/miniconda3/envs/py11/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 414, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/miniconda3/envs/py11/lib/python3.11/site-packages/airflow/decorators/base.py", line 241, in execute
    return_value = super().execute(context)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/miniconda3/envs/py11/lib/python3.11/site-packages/airflow/operators/python.py", line 200, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/miniconda3/envs/py11/lib/python3.11/site-packages/airflow/operators/python.py", line 217, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/dags/veda_data_pipeline/groups/processing_tasks.py", line 29, in remove_thumbnail_asset
    payload.pop("assets")
KeyError: 'assets'

AC

• veda_dataset_pipeline does not fail when optional assets key not provided in dataset config

[smohiudd]

Options:

• fix discovery objects nested assets (tech debt)
• enforce schema - can't create dataset config without thumbnail asset