Relying on pySBD is not feasible- package hasn't been updated in 3 years (sentence segmentation)

[diegodebrito]

Describe the Feature
One of RAGAS dependencies is the package pySBD. This package has several bugs but it seems to be largely unmaintained at this point. The last release was 3 years ago. This will expose RAGAS to bugs, security vulnerabilities and more.

Why is the feature important for you?
I found a bug related to pySBD and opened an issue there. I noticed that the repo seems to have no activity anymore. It's an extremely bad idea to have that as one of the main dependencies in my opinion.

Additional context
Switching to a better known and more well-maintained package will avoid headaches for RAGAS maintainers in the future.

[shahules786]

Hey @diegodebrito thanks for reporting this. Any alternative package you can recommend to do just segmentation?

[diegodebrito]

Hi @shahules786, sorry for taking so long to answer. I am not a specialist on that, but maybe just standard Spacy would work for that purpose?

Something along these lines

import spacy

    split_sentences = []
    for answer in answers:
        doc = nlp(answer)
        sentences_with_index = {i: sent.text for i, sent in enumerate(doc.sents)}
        split_sentences.append(sentences_with_index)

What was the reason you went with pySBD in the first place?

[shahules786]

Hey @diegodebrito We have removed the need for segmenting from ragas with the latest PR. Thank you.