Phraser requires unnecessary memory

[piskvorky]

Currently, Phraser objects (= the trimmed-down version of the full bigram finder Phrases) contains the actual bigrams in an internal attribute called phrasegrams. This is the biggest and most memory-intense part of a Phraser object.

phrasegrams is a dict of {tuple of strings => (frequency [int], score [float])}. But the int (the frequency count of that particular bigram) is unused. This means we're constructing that int, plus the wrapping tuple, for no good reason, inflating the necessary RAM. See also mailing list discussion.

Task:

• Drop the int from Phraser values, leaving only the float.
• And while at it, rename (deprecate) the .vocab attribute of Phrases to something more appropriate, for example bigram_counts.

[gojomo]

Note the vocab dict of a Phrases also contains unigram counts. So if renamed, a name like frequencies might make sense.

But since renaming could break user code (or necessitate more compatibility-maintaining cruft), I wouldn't rename it unless part of a major refactoring/cleanup. The name vocab isn't that bad in its context-of-initial-assignement (where it has a helpful comment), and is roughly in line with the naming of similar frequency-dicts elsewhere in gensim (such as the word2vec-and-related spaghetti-mountains). And there's a bunch of other variables/methods with _vocab in them in phrases.py, all somewhat related to the vocab dict's role, which may equally need better-names – but those better-names aren't necessarily just search-and-replacing vocab with whatever new name is chosen for the dict-property.

[jenishah]

I would like to take this up. Should I rename the .vocab attribute ?

[menshikh-iv]

I agree with @gojomo: .vocab are really ok. If you only want to rename it - just add an getter/setter with new name: this don't break anything and allows to use both (backward compatible). I'm also +1 for variant if we don't rename / add getters to don't increase "compatibility-maintaining cruft".