New versions of pdfminer.six cannot extract chinese characters from pdf

[yadavsandip32]

Describe the bug:

I have observed that latest version of pdfminer.six i.e. 20200124 is not able to read Chinese character.

The version appropriately working right now is 20181108. After that all the later versions are having that problem.

To Reproduce:

Simply run pdf2txt.py input.pdf

Current output:

Ysj Lvf
Tel:
Fax:
Email:
...
...
Abc Lvf
28H abcdefg Road#06-90 Tower 9 The Abcdef@AbcdefGhijklmnop 8336661

Expected Output:

Ysj退Lvf送
Tel:
Fax:
Email:
...
...
Abc退Lvf送
28H abcdefg Road#06-90 Tower 9 The Abcdef@AbcdefGhijklmnop 8336661

Reference PDF:
Sample pdf for chinese character.pdf

P.S. The acrobat may show error while opening this pdf, you can ignore that error. It is prompting this error because some part of the pdf had customer data, so I have manually changed it.

[pietermarsman]

I can replicate this issue.

I did a git bisect and the commit that introduced the error is fa40043.

I tested with

python tools/pdf2txt.py uncompressedunbroken.pdf | head -n 1

[yadavsandip32]

hi @pietermarsman , you or some else is working on this issue ?

[pietermarsman]

Not that I am aware of.

[fakabbir]

@yadavsandip32 Looking into this

[dwalton76]

Post fa40043 CMapDB.get_cmap(cmap_name) will only be called if cmap_name in IDENTITY_ENCODER. For the Chinese characters in the PDF in this issue the cmap_name is UniGB-UCS2-H. If get_cmap is called for UniGB-UCS2-H it has the info needed to create the cmap.

        if cmap_name in IDENTITY_ENCODER:
            return CMapDB.get_cmap(IDENTITY_ENCODER[cmap_name])
        else:
            try:
                return CMapDB.get_cmap(cmap_name)
            except Exception as e:
                log.info(f"get_cmap for {cmap_name} failed due to '{e}'")
                return CMap()

I have not tested on other PDFs but for the PDF in this issue the code block above (at the end of get_cmap_from_spec restores the previous behavior of Ysj退Lvf送

[pietermarsman]

Thanks @dwalton76. I've used your analysis as a starting point.

Someone wants to review my PR: #438?