Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apply "symflower fix" to a "write-test" result of a model when it errors, so model responses can possibly be fixed #229

Merged
merged 2 commits into from
Jul 2, 2024
Merged

Apply "symflower fix" to a "write-test" result of a model when it errors, so model responses can possibly be fixed #229

merged 2 commits into from
Jul 2, 2024

Conversation

ruiAzevedo19
Copy link
Contributor

@ruiAzevedo19 ruiAzevedo19 commented Jun 28, 2024 •
edited
Loading

Part of #213

@ruiAzevedo19 ruiAzevedo19 added the enhancement New feature or request label Jun 28, 2024
@ruiAzevedo19 ruiAzevedo19 added this to the v0.6.0 milestone Jun 28, 2024
@ruiAzevedo19 ruiAzevedo19 self-assigned this Jun 28, 2024
@ruiAzevedo19 ruiAzevedo19 marked this pull request as draft June 28, 2024 13:12
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch from b9deae5 to 1dff749 Compare June 28, 2024 13:45
bauersimon
bauersimon requested changes Jul 1, 2024
View reviewed changes
evaluate/report/collection.go Show resolved Hide resolved
evaluate/metrics/testing/assessments.go Outdated Show resolved Hide resolved
evaluate/metrics/testing/assessments.go Outdated Show resolved Hide resolved
evaluate/metrics/testing/assessments.go Outdated Show resolved Hide resolved
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch from 1dff749 to dc10753 Compare July 1, 2024 07:54
@ruiAzevedo19 ruiAzevedo19 changed the title Apply "symflower repair" to a "write-test" result of a model when it errors, so model responses can possibly be fixed Apply "symflower fix" to a "write-test" result of a model when it errors, so model responses can possibly be fixed Jul 1, 2024
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch 2 times, most recently from 5eed690 to 475a3c1 Compare July 1, 2024 09:09
@ruiAzevedo19 ruiAzevedo19 requested a review from bauersimon July 1, 2024 09:10
bauersimon
bauersimon requested changes Jul 1, 2024
View reviewed changes
evaluate/metrics/testing/assessments.go Outdated Show resolved Hide resolved
evaluate/metrics/testing/assessments.go Outdated Show resolved Hide resolved
evaluate/task/symflower-fix.go Outdated Show resolved Hide resolved
evaluate/task/symflower-fix.go Outdated Show resolved Hide resolved
evaluate/task/symflower-fix.go Outdated Show resolved Hide resolved
evaluate/task/task-write-test.go Outdated Show resolved Hide resolved
@ruiAzevedo19
refactor, Let the tasks return assessments per task identifiers, so e…
fcf95c4 
…ach task can run the "symflower fix" and store its assessments separately

Part of #213
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch 2 times, most recently from a3ab62b to 4a7909b Compare July 1, 2024 16:57
@ruiAzevedo19 ruiAzevedo19 requested a review from bauersimon July 1, 2024 16:57
@bauersimon bauersimon marked this pull request as ready for review July 2, 2024 07:25
@bauersimon bauersimon force-pushed the 213-symflower-repair branch from 4a7909b to 313fef3 Compare July 2, 2024 07:28
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch 4 times, most recently from ebee196 to 3df1a0d Compare July 2, 2024 11:17
@bauersimon bauersimon force-pushed the 213-symflower-repair branch from 3df1a0d to 0390ff0 Compare July 2, 2024 11:20
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch from 0390ff0 to c341983 Compare July 2, 2024 11:36
@ruiAzevedo19
Copy link
Contributor Author

Follow-up: Don't run symflower fix if there is a timeout on the LLM response

@ruiAzevedo19 ruiAzevedo19 mentioned this pull request Jul 2, 2024
Closed
1 task
bauersimon
bauersimon reviewed Jul 2, 2024
View reviewed changes
evaluate/evaluate.go Outdated Show resolved Hide resolved
@bauersimon
Copy link
Member

@ruiAzevedo19 LGTM plz take a look as well at my changes

@ruiAzevedo19
Apply "symflower fix" to a "write-test" result of a model when it err…
eb9e3f0 
…ors, so model responses can possibly be fixed

Closes of #213
@ruiAzevedo19 ruiAzevedo19 force-pushed the 213-symflower-repair branch from c341983 to eb9e3f0 Compare July 2, 2024 11:45
@ruiAzevedo19
Copy link
Contributor Author

@bauersimon Also LGTM

@bauersimon bauersimon self-requested a review July 2, 2024 12:01
@bauersimon bauersimon merged commit fb5ff24 into main Jul 2, 2024
4 checks passed
@bauersimon bauersimon deleted the 213-symflower-repair branch July 2, 2024 12:02
@bauersimon bauersimon mentioned this pull request Jul 31, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahumenberger ahumenberger ahumenberger left review comments

@bauersimon bauersimon bauersimon approved these changes

Assignees

@ruiAzevedo19 ruiAzevedo19

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.6.0
Development

Successfully merging this pull request may close these issues.
3 participants
@ruiAzevedo19 @bauersimon @ahumenberger