Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproduced results different from paper #11

Open
ncoop57 opened this issue Feb 15, 2021 · 1 comment
Open

Reproduced results different from paper #11

ncoop57 opened this issue Feb 15, 2021 · 1 comment
Assignees

Comments

@ncoop57
Copy link
Owner

ncoop57 commented Feb 15, 2021

Some of the reproduced results are different from the paper, specifically tango_vis and tango_txt can vary in performance by 5-10%. However, tango_comb only varies around 1%.

tango_vis differences may be due to stochasticity in the kmeans cluster assignment during inference.

Differences in tango_txt might be explained due to differences in which frames are selected to have text extracted from in current implementation. However, this doesn't seem like it would cause the stark difference.

@ncoop57 ncoop57 self-assigned this Feb 15, 2021
@ncoop57
Copy link
Owner Author

ncoop57 commented Jun 9, 2021

Issue does not appear to be due to stochasticity as different seeds have been tried and no change in performance has occurred. I was able to reproduce the original paper's results using an older version of the code. Therefore, the issue might be due to some change in code or data. Investigations are still underway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant