Implement __contains__ for vocab.

[sebpuetz]

Python performs linear search over sequences if __contains__ is
not explicitly implemented. This is painfully slow over large
vocabularies, therefore implement __contains__ explicitly.

Fixes #74

thanks @twuebi and @ketxd for pointing this out.

[danieldk]

Yep, Python philosophy: why not require a proper implementation if you can do it incredibly slowly ;).

[danieldk]

Reminder, we should implement word(&self) -> Option<...> and subword(&self) -> Option<...> on WordIndex, so that these cases can be simplified to:

.vocab().idx(&word).and_then(WordIndex::word).is_some()

[sebpuetz]

Do you want to open the issue at finalfusion-rust?

[danieldk]

finalfusion/finalfusion-rust#76

[danieldk]

is_some()

[sebpuetz]

Good catch

[sebpuetz]

Actually, is_some() conveys that the word is in-vocab if there's a WordIndex::Subword returned. We'd only return false if no subword indices can be generated for a word. I'd like to return false for any word that isn't part of the actual vocabulary

[danieldk]

Ah right. But this can be made nice with is_some() with the new WordIndices changes in finalfusion-rust.

[sebpuetz]

Only with a bumped dependency ;)

[sebpuetz]

I was playing around with some alternative vocab implementations last night. I thought it'd be nice to index the vocab both with String and int, since we could return the word of the given index and also the index for a given word. This is doable through PyMappingProtocol and an enum, where extract() constructs the correct variant depending on the type coming from python.

Then I tried to implement an iterator on the vocabulary since we don't get that for free on PyMappingProtocol by implementing __getitem__ and __len__, which is when things got weird:

Implementing PyMappingProtocol::__iter__ for the vocab worked fine, the method returns some iterator struct, implementing PyIterProtocol - similar to what we do in PyEmbeddings. Although, for some reason in Python that doesn't make the vocab iterable, e.g. for _ in vocab fails, but vocab.__iter__() works.

If I go ahead and implement PyIteratorProtocol for the vocab, the vocab becomes iterable (for _ in vocab works) but ignores both PyMappingProtocol::__iter__ and PyMappingProtocol::__contains__ and uses PyIteratorProtocol::__iter__ for a linear search for something like "a" in vocab. vocab.__contains__("a") on the other hand calls PyMappingProtocol::__contains__.

I opened an issue at PyO3/pyo3#611 regarding this. Do you have more experience with this or any idea what's going on here?