Question regarding final_beam_acc and BEM

[awasthiabhijeet]

Hi @OhadRubin ,

Could you please let me know whether the final_beam_acc metric in the code is the same as the BEM metric reported in Table 3 of your paper?

I'm finding final_beam_acc it to be much lesser than the EM and BEM scores.
Is this expected?

{
"best_epoch": 204,
"peak_worker_0_memory_MB": 11110.15234375,
"peak_gpu_0_memory_MB": 17076.2431640625,
"training_duration": "1 day, 11:19:15.241460",
"training_start_epoch": 0,
"training_epochs": 304,
"epoch": 304,
"training_final_beam_acc": 0.9714408725602756,
"training_loss": 1.5601123754273762,
"training_worker_0_memory_MB": 11110.15234375,
"training_gpu_0_memory_MB": 17076.2431640625,
"validation_final_beam_acc": 0.6634335596508244,
"validation_spider": 0.7177497575169738,
"validation_reranker": 0.5470417070805044,
"validation_leafs_acc": 0.965082444228904,
"validation_loss": 0.0,
"best_validation_final_beam_acc": 0.6702230843840931,
"best_validation_spider": 0.7381183317167799,
"best_validation_reranker": 0.5606207565470417,
"best_validation_leafs_acc": 0.9515033947623667,
"best_validation_loss": 0.0
}

[awasthiabhijeet]

Also, if final_beam_acc is not the same as BEM, do we currently have a way to compute BEM anywhere in this code?

Thanks,

[awasthiabhijeet]

@OhadRubin