Question About Hidden Space Tokenization Decoding

[Lingy12]

multilingual_analysis/layers/test_layer.py

Line 44 in be2b161

hidden_states, outputs = model.generate(**{'input_ids':inputs.input_ids, 'max_new_tokens':64})

I am trying to reproduce the token analysis of layers' hidden embedding. I have done the following:

1. Install according to requirements.txt
2. Replace the generation and models folder of transformers with the one in the repository

However, the output of the line is just the decoded token ids, no hidden_states (as normal generate).

May I ask is there any thing I am missing? Thank you very much.

[daoyangl]

Hello, did you solve this problem? I also have the same question about test_layer.py code.