Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: eval_method cannot catch run factor error #260

Merged
merged 4 commits into from
Sep 14, 2024
Merged

fix: eval_method cannot catch run factor error #260

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9caf644
Update eval_method.py
qew21 Sep 12, 2024
b8aa7f3
add AttributeError
qew21 Sep 12, 2024
b14e20f
add raise_exception for eval_method
qew21 Sep 12, 2024
a8ddd59
remove custom exception catch
qew21 Sep 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Update eval_method.py 
abort subsequent evaluators for invalid gen_df
  • Loading branch information
@qew21
qew21 authored Sep 12, 2024
commit 9caf6443cc467e59a61060c9b36dde75644c2322
10 changes: 10 additions & 0 deletions rdagent/components/benchmark/eval_method.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,16 @@ def eval_case(
for each item
If the evaluation run successfully, return the evaluate results. Otherwise, return the exception.
"""
if not case_gen:
return Exception("invalid implementation")
execution_feedback, gen_df = case_gen.execute()
if (isinstance(gen_df, pd.DataFrame) and gen_df.empty) or (not isinstance(gen_df, pd.DataFrame) and not gen_df):
execution_feedback = re.sub(r"(?<=\D)(,\s+-?\d+\.\d+){50,}(?=\D)", ", ", execution_feedback)
execution_feedback = "\n".join(
[line for line in execution_feedback.split("\n") if "warning" not in line.lower()]
)
return Exception(execution_feedback)

eval_res = []
for ev in self.evaluator_l:
try:
Expand Down
Loading