You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
According to Analyzing Transformers in Embedding Space using the moore penrose pseudo inverse lead to problems. This is what is used to initialize the decoder invert method. It might be worth it to implement a simpler version that just multiplies by the transpose of the embedding matrix as advocated in that paper.
The text was updated successfully, but these errors were encountered:
Are you saying that we initialize with the transpose of the embedding matrix, or are you saying we should "implement" an entirely separate inverse that consists of nothing more than multiplying by the transpose? I'd be fine with testing the former, but the latter seems sort of pointless since it's so simple.
According to Analyzing Transformers in Embedding Space using the moore penrose pseudo inverse lead to problems. This is what is used to initialize the decoder invert method. It might be worth it to implement a simpler version that just multiplies by the transpose of the embedding matrix as advocated in that paper.
The text was updated successfully, but these errors were encountered: