Allow selecting decode errors bahaviour

[stefanor]

Forwarded from https://bugs.launchpad.net/ubuntu/+source/python-html2text/+bug/1318227

Currently it stops conversion on any decode error:

$ html2markdown broken_text
Traceback (most recent call last):
  File "/usr/bin/html2markdown", line 9, in <module>
    load_entry_point('html2text==3.200.3', 'console_scripts', 'html2text')()
  File "/usr/lib/python3/dist-packages/html2text.py", line 781, in main
    data = data.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 4: invalid start byte

But for the files I'm working on it would be perfectly fine just to add

data = data.decode(encoding, errors='ignore')

It can be exposed as an option.

[Alir3z4]

load_entry_point('html2text==3.200.3', 'console_scripts', 'html2text')()

This is possible duplicate of #83 and some others I guess, but the main issue is that this one is not using the latest version.

@stefanor Could you please confirm the latest version throws the same error as well?

[theSage21]

@stefanor That would be a good idea. Am I right in saying that this should be a non boolean value? something like decode_errors = ignore?

[int-ua]

@Alir3z4 I'm afraid it's not fixed, just check the code here:
https://github.com/Alir3z4/html2text/blob/master/html2text/cli.py#L204

No option to ignore errors. Have you seen one anywhere?

[theSage21]

@stefanor @int-ua I am working on this. Should the default be 'ignore' or 'strict'?

[int-ua]

Looks like backward compatibility needs 'strict'. Also, losing some characters should be acknowledged IMO.

[theSage21]

@int-ua @stefanor Does this fix it?

[int-ua]

Yes, thanks. Also, looks like it would be nice catching this UnicodeDecodeError and recommending to use the new option.

[theSage21]

@int-ua Something like Warning: this is set for deprication. use --decode-errors=ignore?

[theSage21]

@int-ua It is done. @Alir3z4 Consider the issue closed?

[theSage21]

@Alir3z4 you merged #88 . This issue is closed.

[Alir3z4]

@theSage21 Thanks for the heads up.
The issue is closed now.