Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce penalty weight #114

Closed
steffencruz opened this issue Feb 21, 2024 · 1 comment · Fixed by #115
Closed

Reduce penalty weight #114

steffencruz opened this issue Feb 21, 2024 · 1 comment · Fixed by #115
Assignees

Comments

@steffencruz
Copy link
Collaborator

steffencruz commented Feb 21, 2024

The current penalty weight of 1.0 effectively limits the max. score to 0.8 because there is always overlap between the challenge and completion texts. This was desirable at initial launch because we wanted to disincentivize echo mining, however such strategies are not viable at all any more.

Also, we don't want 'gaming' the penalty model to become profitable at the expense of producing high quality reference answers.

My recommendation is that we reduce the penalty to 0.5 for qa and summarization

@steffencruz steffencruz self-assigned this Feb 21, 2024
@steffencruz steffencruz linked a pull request Feb 21, 2024 that will close this issue
@steffencruz
Copy link
Collaborator Author

Closed in #115

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant