Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

Evaluator #5445

Merged
merged 19 commits into from
Jan 27, 2022
Merged

Evaluator #5445

merged 19 commits into from
Jan 27, 2022

Conversation

gabeorlanski
Copy link
Contributor

@gabeorlanski gabeorlanski commented Oct 23, 2021

Fixes #5405.

Changes proposed in this pull request:

  • New Evaluator Base class that serves as a way to evaluate and post-process a single data set.
    The new Postprocessor Base Class will post-process both batch inputs and output predictions into a human-readable string format.
  • For both, SimpleEvaluator and SimplePostprocessor are implemented to replicate the original evaluation behavior.
  • Integrated it with the Evaluate class.
    • Users will have a new top-level key in their configs called "evaluation" that maps directly to an Evaluator` sub-class. (Placeholder poping of this from the config was added to the train function so that it does not break).
    • For the time being, it is not required by evaluate to have this key and will default to the original behavior.
  • Added better handling for multiple file evaluation so that users need only pass the input files and then use a CLI arg to automatically generate predictions and output files.
  • Fixed the tests for multi-file evaluation to properly test multi-file evaluation.
  • Probably a few others I am missing.

Before submitting

  • I've read and followed all steps in the Making a pull request
    section of the CONTRIBUTING docs.
  • I've updated or added any relevant docstrings following the syntax described in the
    Writing docstrings section of the CONTRIBUTING docs.
  • If this PR fixes a bug, I've added a test that will fail without my fix.
  • If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

  • All GitHub Actions jobs for my pull request have passed.
  • codecov/patch reports high test coverage (at least 90%).
    You can find this under the "Actions" tab of the pull request once the other checks have finished.

@gabeorlanski
Copy link
Contributor Author

@AkshitaB Sorry for the delay but here is the draft. Handling extra metrics is not yet implemented, but I felt like this was already big enough to warrant a PR

@gabeorlanski gabeorlanski marked this pull request as ready for review October 31, 2021 16:40
Copy link
Contributor

@AkshitaB AkshitaB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gabeorlanski Looks great, overall! I left some comments/suggestions.

@gabeorlanski gabeorlanski requested a review from AkshitaB January 4, 2022 14:24
@AkshitaB AkshitaB enabled auto-merge (squash) January 27, 2022 12:19
@AkshitaB AkshitaB merged commit dd5a010 into allenai:main Jan 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Evaluator Class to allow more metrics and saving of input tokens during evaluation
3 participants