Fix overflow error for *Vec corpusfile-based training

[bm371613]

Fix #2258

Corpus size is currently limited in terms of the total word count to what fits a 32-bit signed integer. This can result in a OverflowError: value too large to convert to int for a big corpus. This PR replaces int with a long long to allow for bigger corpora.

[piskvorky]

Oh wow, how did that slip through. Thanks a lot for spotting & reporting!

@menshikh-iv @persiyanov Any other places where we use counters or data structures that might overflow with large corpora?

[piskvorky]

How can we test this works as expected? I'd like to avoid regressions in the future.

[piskvorky]

How about changing these to uint64 too (I prefer explicit sizes to long long, but it's not a big deal), to be on the safe side? Especially total_effective_words. Any reason not to?

[bm371613]

There are long longs in many other places. Do you want explicit sizes everywhere?

[piskvorky]

I think it makes sense, but is not critical.

More important to me is that any variable that holds a potentially large count (token frequency, number of tokens in a corpus, number of sentences in a corpus, …) is at least 64 bit.

[bm371613]

@piskvorky If "as expected" means accepting bigger values, a test could pass a big number and assert there is no OverflowError. Is that what you meant?

[bm371613]

Note that CI failed due to flake8 errors on files not modified in this PR. A new version of flake8 was released a few days ago and it's likely that the CI would also fail on the target branch.

[menshikh-iv]

Big thanks @bm371613, can you please allow me to push into your branch (https://help.github.com/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork/#enabling-repository-maintainer-permissions-on-existing-pull-requests (4)), I need that for fix CI

[bm371613]

Big thanks @bm371613, can you please allow me to push into your branch (https://help.github.com/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork/#enabling-repository-maintainer-permissions-on-existing-pull-requests (4)), I need that for fix CI

@menshikh-iv It seems to be already enabled, are your push attempts rejected?

[menshikh-iv]

@bm371613 yes :(

(asdadsf) ivan@P50:~/release/bm371613/gensim$ git push
ERROR: Permission to bm371613/gensim.git denied to menshikh-iv.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[menshikh-iv]

@bm371613 can you give me permissions manually to commit into your fork repo?

[bm371613]

@menshikh-iv ok done

[menshikh-iv]

@bm371613 it works, thank you!

[menshikh-iv]

@bm371613 awesome, thanks for the fix, congratz with the first contribution 🥇