TypeError: Object of type Decimal is not JSON serializable

[jpuris]

External ref: Slack thread on Meltano Slack community

meltano.yaml

# ...
  extractors:
  - name: tap-oracle
    pip_url: git+https://github.com/christianelliott/pipelinewise-tap-oracle.git
# ...
  loaders:
  - name: target-snowflake
    variant: meltanolabs
    pip_url: meltanolabs-target-snowflake
# ...

The error

❯ meltano --environment=dev run tap-oracle target-snowflake
...
2023-07-09T15:36:51.661162Z [info     ] 2023-07-09 17:36:51,661 | INFO     | target-snowflake     | Target sink for 'BUSINESSDATA_PROD-FINANCIAL_TRANSACTION' is full. Draining... cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.491003Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.491140Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 8, in <module> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.491266Z [info     ]     sys.exit(TargetSnowflake.cli()) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.491388Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.492633Z [info     ]     return self.main(*args, **kwargs) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.492732Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.492888Z [info     ]     rv = self.invoke(ctx)      cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.492978Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.493094Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.493165Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.493245Z [info     ]     return __callback(*args, **kwargs) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.493310Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 549, in invoke cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.493539Z [info     ]     target.listen(file_input)  cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.493606Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494018Z [info     ]     self._process_lines(file_input) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494089Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 291, in _process_lines cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494297Z [info     ]     counter = super()._process_lines(file_input) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494490Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/io_base.py", line 93, in _process_lines cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494551Z [info     ]     self._process_record_message(line_dict) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494619Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 353, in _process_record_message cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494712Z [info     ]     self.drain_one(sink)       cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.494974Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/target_base.py", line 494, in drain_one cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495049Z [info     ]     sink.process_batch(draining_status) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495120Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/sinks/sql.py", line 267, in process_batch cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495330Z [info     ]     self.bulk_insert_records(  cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495389Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/target_snowflake/sinks.py", line 141, in bulk_insert_records cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495443Z [info     ]     for files in batches:      cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495495Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/batch.py", line 106, in get_batches cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495545Z [info     ]     gz.writelines(             cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495592Z [info     ]   File "/Users/janispuris/projects/meltano_poc/project/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/singer_sdk/batch.py", line 107, in <genexpr> cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495641Z [info     ]     (json.dumps(record) + "\n").encode() for record in chunk cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495825Z [info     ]   File "/Users/janispuris/.pyenv/versions/3.10.11/lib/python3.10/json/__init__.py", line 231, in dumps cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.495973Z [info     ]     return _default_encoder.encode(obj) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496033Z [info     ]   File "/Users/janispuris/.pyenv/versions/3.10.11/lib/python3.10/json/encoder.py", line 199, in encode cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496118Z [info     ]     chunks = self.iterencode(o, _one_shot=True) cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496221Z [info     ]   File "/Users/janispuris/.pyenv/versions/3.10.11/lib/python3.10/json/encoder.py", line 257, in iterencode cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496284Z [info     ]     return _iterencode(o, 0)   cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496358Z [info     ]   File "/Users/janispuris/.pyenv/versions/3.10.11/lib/python3.10/json/encoder.py", line 179, in default cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496420Z [info     ]     raise TypeError(f'Object of type {o.__class__.__name__} ' cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.496562Z [info     ] TypeError: Object of type Decimal is not JSON serializable cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.532959Z [info     ] 2023-07-09 17:36:52,532 | INFO     | snowflake.connector.connection | closed cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.533122Z [info     ] 2023-07-09 17:36:52,532 | INFO     | snowflake.connector.connection | No async queries seem to be running, deleting session cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2023-07-09T15:36:52.606484Z [error    ] [Errno 32] Broken pipe         
...

Might be related to #66

[jpuris]

Was not sure what additional information to add or gather. Please let me know 🙏

If necessary I can also try to produce a MRE on this 🤷

[pnadolny13]

@jpuris thanks for opening this!

Do you have an example of the record thats causing this? That would make it much easier to create a test case for it.

I'm wondering if this is related to meltano/sdk#1809. We deserialize as decimals now so that they can be validated properly in python but its possible that the later serialization part is now failing.

cc @edgarrmondragon

[jpuris]

The stream that the issue is reproducible, has multiple decimal columns. I do not know, if this is the offending row or not (I do not know how I can catch the exact row in the batch that it does not like something in), but here are example of a Decimal field in the stream

    "AMT_EXCL_TAX": 0.8

            "AMT_EXCL_TAX": {
                "multipleOf": 0.0001,
                "type": [
                    "null",
                    "number"
                ]
            }

[edgarrmondragon]

This will be addressed in the next release of the SDK. See meltano/sdk#1826.

[pnadolny13]

@jpuris we created a new 0.4.1 release https://github.com/MeltanoLabs/target-snowflake/releases/tag/v0.4.1 that includes the SDK fix. I'm going to close this issue for now but please reopen if the issue is not resolved for you.