Move retriever probability calculations to document_store

[tanaysoni]

This PR moves the retrieval probability(pseudo probability by scaling query scores) calculations to the respective document stores.

It simplifies the similarity matching method(get_answers_via_similar_questions()) in the Finder. Additionally, now the retrieved documents have an explicit probability field that could be useful in the future.

[tholor]

Looking good.
Only comment: Do we want to keep the naming like this (probability and query_score) or make it more consistent: e.g. probability and score

[tanaysoni]

score sounds more appropriate. I'll change it.

[guillim]

Hi, I read your PR and I liked the renaming for more simplicity, however I end up a bit confused.

In haystack/schema.py I can read :

        :param score: Retriever's query score for a retrieved document
        :param probability: a psuedo probability by scaling score in the range 0 to 1

So score is the retriever score.
And probability is the retriever score, scaled from 0 to 1

So... where is the Reader score ? I thought it was the probability property but not sure any longer after reading this. Also, if no score for the reader is returned, then it means the results are sorted by their retriever score only. Do I get it wrong ?

[guillim]

Ok my bad : we are talking Document schema here : the Reader results won't appear here. I think I was confused because the exact same property name is used in the Reader like here : haystack/reader/transformers.py in the predict() function

"answer": pred["answer"],
"context": doc.text[context_start:context_end],
"offset_start": pred["start"],
"offset_end": pred["end"],
"probability": pred["score"],
"score": None,
"document_id": doc.id,
"meta": doc.meta

[guillim]

But it means, at the moment, there are no information about the retriever score passed to the answers yet. Is that correct @tholor ?

[tholor]

@guillim Yes, that's correct.