Integer overflow during FastText training with corpus_file

[joelkuiper]

Description

model = FastText(corpus_file="sentences_norm.txt.gz", workers=14, iter=5, size=200, sg=1, hs=1)

with the following sizes

2018-11-05 16:57:52,809 : INFO : collected 6532860 word types from a corpus of 4728738902 raw words and 238627116 sentences
2018-11-05 16:57:52,809 : INFO : Loading a fresh vocabulary
2018-11-05 16:58:00,788 : INFO : effective_min_count=5 retains 1887156 unique words (28% of original 6532860, drops 4645704)
2018-11-05 16:58:00,788 : INFO : effective_min_count=5 leaves 4721157112 word corpus (99% of original 4728738902, drops 7581790)
2018-11-05 16:58:07,437 : INFO : deleting the raw counts dictionary of 6532860 items
2018-11-05 16:58:07,615 : INFO : sample=0.001 downsamples 26 most-common words
2018-11-05 16:58:07,615 : INFO : downsampling leaves estimated 3749158657 word corpus (79.4% of prior 4721157112)
2018-11-05 16:58:11,281 : INFO : constructing a huffman tree from 1887156 words
2018-11-05 16:59:36,077 : INFO : built huffman tree with maximum node depth 30
2018-11-05 17:00:17,300 : INFO : estimated required memory for 1887156 words, 1929637 buckets and 200 dimensions: 7871448352 bytes
2018-11-05 17:00:17,398 : INFO : resetting layer weights
2018-11-05 17:01:43,333 : INFO : Total number of ngrams is 1929637
2018-11-05 17:02:11,990 : INFO : training model with 14 workers on 1887156 vocabulary and 200 features, using sg=1 hs=1 sample=0.001 negative=5 window=5

yields

Exception in thread Thread-2120:
Traceback (most recent call last):
  File "/home/joelkuiper/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/joelkuiper/anaconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/joelkuiper/anaconda3/lib/python3.6/site-packages/gensim/models/base_any2vec.py", line 175, in _worker_loop_corpusfile
    total_examples=total_examples, total_words=total_words, **kwargs)
  File "/home/joelkuiper/anaconda3/lib/python3.6/site-packages/gensim/models/fasttext.py", line 561, in _do_train_epoch
    total_examples, total_words, work, neu1)
  File "gensim/models/fasttext_corpusfile.pyx", line 126, in gensim.models.fasttext_corpusfile.train_epoch_sg
OverflowError: value too large to convert to int

on all workers. Note that the sg and hs parameters seem to have no relation to this, also happens without them.

Steps to reproduce

model = FastText(corpus_file="sentences_norm.txt.gz", workers=14, iter=5,size=200)

Expected Results

Should train the model

Actual Results

Exception thrown, no further output.

Traceback (most recent call last):
  File "/home/joelkuiper/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/joelkuiper/anaconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/joelkuiper/anaconda3/lib/python3.6/site-packages/gensim/models/base_any2vec.py", line 175, in _worker_loop_corpusfile
    total_examples=total_examples, total_words=total_words, **kwargs)
  File "/home/joelkuiper/anaconda3/lib/python3.6/site-packages/gensim/models/fasttext.py", line 561, in _do_train_epoch
    total_examples, total_words, work, neu1)
  File "gensim/models/fasttext_corpusfile.pyx", line 126, in gensim.models.fasttext_corpusfile.train_epoch_sg
OverflowError: value too large to convert to int

Versions

Python 3.6.6 |Anaconda, Inc.| (default, Oct 9 2018, 12:34:16)
[GCC 7.3.0]
NumPy 1.15.3
SciPy 1.1.0
gensim 3.6.0

On Ubuntu 16.04

edit seems to work fine when passing in a LineSentence object

[CuriousG102]

I see a similar error in Doc2Vec. I an verify that total_words is larger than a 32 bit integer. There's not an easy solution to this since training on a corpus_file will throw a different exception if total_words isn't present.

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/Volumes/Backblaze_MacEx1TB50506065/cs221/project/CS221/venv/lib/python3.5/site-packages/gensim/models/base_any2vec.py", line 175, in _worker_loop_corpusfile
    total_examples=total_examples, total_words=total_words, **kwargs)
  File "/Volumes/Backblaze_MacEx1TB50506065/cs221/project/CS221/venv/lib/python3.5/site-packages/gensim/models/doc2vec.py", line 686, in _do_train_epoch
    doctag_vectors=doctag_vectors, doctag_locks=doctag_locks)
  File "gensim/models/doc2vec_corpusfile.pyx", line 280, in gensim.models.doc2vec_corpusfile.d2v_train_epoch_dm

[menshikh-iv]

Thanks for the report @joelkuiper!

[mpenkov]

@menshikh-iv Since this is tagged "easy", I'm guessing the fix is to replace the int declaration here with something like a long?

[menshikh-iv]

@mpenkov yes, something like this (int -> longest_int_type for all variables that can be "too large") in all *_corpusfile.pyx files

[lifengjin]

I am experiencing this same bug as well when training Word2Vec with a large corpus. There has been a pull request for this bug here for a couple of months. Would you please fix this one? Thanks.