predict should return label/tag plus confidence value

[tabergma]

Currently, the predict() method of a trained text classification model will just return the label name, but not the confidence. Same holds true for the sequence labeling model.

However, depending on the use case you maybe only want to use labels with a confidence value higher than 0.9. Thus, we should add the confidence to the return value of the predict() method.

[kilian-gebhardt]

I'd be interested in confidences for the assigned labels in the sequence labeling tasks (and may implement it, if you are not already working on this).
In case of a linear output layer one can apply an additional softmax to obtain a confidence value. In case of a CRF output layer, it seems as if a forward-backward algorithm is required (which is perhaps a bit too costy).

[alanakbik]

Yes, we agree :) We are currently implementing this and will push soon into release-0.3 branch for testing, and then (hopefully soon) we will release the next version that will contain this feature!

[alanakbik]

Added to release-03 branch - will be part of next release!

[Rajat-Mehta]

In the case of two label classification, can we somehow get the probability of given text document for both classes?

For example, For a given text document D1, I need output like this:
Class 1: probability 0.7
Class 2: probability 0.3
(both probabilities should be summing to 1)
If yes, how to achieve that? It would be really helpful for me.

Thanks in advance,
Rajat

[alanakbik]

There is a PR pending right now that will allow you to do this, see #693. Once its merged you will have this functionality in the master branch and eventually in the next release of Flair.

[haozturk]

Hi guys, in which versions and how can I use this feature?

Thanks :)

[alanakbik]

Hi @haozturk you already can with the current version:

from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load('ner')

sentence = Sentence('I live in Berlin')

tagger.predict(sentence)

for entity in sentence.get_spans('ner'):
    print(entity)
    print(entity.tag)
    print(entity.score)