关于segment_bert.py

[lyh124]

你好，我在使用segment_bert.py时一直遇到这个bug，请问这可能是什么原因呢

465it [00:00, 32845.81it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/nfs/users/longyuhui/anaconda3/envs/syngec/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "../../utils/segment_bert.py", line 15, in split
tokens = tokenizer.tokenize(line)
File "/nfs/users/longyuhui/anaconda3/envs/syngec/lib/python3.8/site-packages/bert/tokenization.py", line 192, in tokenize
for token in self.basic_tokenizer.tokenize(text):
File "/nfs/users/longyuhui/anaconda3/envs/syngec/lib/python3.8/site-packages/bert/tokenization.py", line 237, in tokenize
if preserve_token(token, self.vocab):
File "/nfs/users/longyuhui/anaconda3/envs/syngec/lib/python3.8/site-packages/bert/tokenization.py", line 41, in preserve_token
if not FLAGS.preserve_unused_tokens:
File "/nfs/users/longyuhui/anaconda3/envs/syngec/lib/python3.8/site-packages/absl/flags/_flagvalues.py", line 479, in getattr
raise _exceptions.UnparsedFlagAccessError(
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --preserve_unused_tokens before flags were parsed.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "../../utils/segment_bert.py", line 19, in
for ret in pool.imap(split, tqdm(sys.stdin), chunksize=1):
File "/nfs/users/longyuhui/anaconda3/envs/syngec/lib/python3.8/multiprocessing/pool.py", line 865, in next
raise value
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --preserve_unused_tokens before flags were parsed.

[HillZhang1999]

您好，可以参考这个issue
google-research/bert#1133

[lyh124]

感谢您的回复。参考您给出的issue将bert-tensorflow和tensorflow降级至1.0.1和1.15.0后即解决问题。