[FEATURE] Need to detokenize a BertTokenizer output

[daden-ms]

Description

Currently the output of the NER prediction contains the subword, but the end user doesn't care about subword but the original word

For example , 'call Qingxiong Daisy'
tokenizer.tokenize([text]) -> [['call', 'Qing', '##xi', '##ong', 'Daisy']]
output label [['O', 'PersonName', 'X', 'X', 'X', 'PersonName', 'X', 'X']]

Expected behavior with the suggested feature

The desired output should be
'Qingxiong Daisy'->PersonName

It can also be helpful to provide the position of the entity

Other Comments

[daden-ms]

There is a similar ask for the pytorch-pretrained-BERT: huggingface/transformers#36

[daden-ms]

This is the file to see an example of how to deal with the problem: https://github.com/huggingface/pytorch-pretrained-BERT/blob/c304593d8fa93f25febe1458c63497a846749c89/examples/run_squad_dataset_utils.py#L179

[hlums]

@daden Can we consider this issue as resolved?

[miguelgfierro]

hey @daden-ms, I think Hong linked another person lol

[daden-ms]

Yes, we can close this issue now. For the record, we changed the input of the tokenize_ner to take in a list of tokens, instead of a string.