Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updated endpoints #33

Open
surcyf123 opened this issue Sep 10, 2023 · 4 comments
Open

updated endpoints #33

surcyf123 opened this issue Sep 10, 2023 · 4 comments
Assignees
Labels
wontfix This will not be worked on

Comments

@surcyf123
Copy link
Owner

surcyf123 commented Sep 10, 2023

new endpoints:
urls = [
"http://172.218.204.83:2701",
"http://37.27.2.44:60102",
"http://184.67.78.114:42098",
]
I also won't be normalizing the scores within the reward endpoint anymore. We can always iterate through the scores in the csv and append a normalized scores after the fact. Will link a commit with updated reward endpoint response format once it is completed

endpoint response format is as follows:
{'Bert': -0.029786773025989532, 'DPO': -11.46875, 'MPNet': 0.029283182695508003, 'RLHF': -2.9505324363708496, 'Reciprocate': -5.0703125}
@surcyf123 surcyf123 added the wontfix This will not be worked on label Sep 10, 2023
@surcyf123
Copy link
Owner Author

changed again to

    "http://142.182.6.112:55469", "http://142.182.6.112:55467", "http://142.182.6.112:55430", "http://142.182.6.112:55430", #"vast2",
    "http://184.67.78.114:42036", "http://184.67.78.114:42091", "http://184.67.78.114:42082", "http://184.67.78.114:42093", # "vast3",
    "http://37.27.2.44:60113", "http://37.27.2.44:60180", "http://37.27.2.44:60151", "http://37.27.2.44:60181", # "vast4",
]

fixed normalization to be a constant and not a moving average, so output format is reverted to

[{'Bert': [-0.014279717579483986, 1.0], 'DPO': [-14.8203125, 0.0726899802684784], 'MPNet': [0.5161169767379761, 1.0], 'RLHF': [5.964920997619629, 0.9999698400497437], 'Reciprocate': [6.05078125, 0.8049955368041992], 'Total Reward': 0.6632935911417007}, {'Bert': [-0.014279717579483986, 1.0], 'DPO': [-14.8203125, 0.0726899802684784], 'MPNet': [0.5161169767379761, 1.0], 'RLHF': [5.964920997619629, 0.9999698400497437], 'Reciprocate': [6.05078125, 0.8049955368041992], 'Total Reward': 0.6632935911417007}]

@surcyf123
Copy link
Owner Author

this should help us avoid postporcessing on the testing data as well.

@surcyf123
Copy link
Owner Author

we need a plan of a place where I should keep the updated endpoints so you know where to look if they changed.

@gee842
Copy link
Collaborator

gee842 commented Sep 11, 2023

hm so from my experience the best practice for this would be to have another flask server which will have an endpoint that returns the endpoints, which the testing code can directly call before running any experiments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

2 participants