Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[transformer] add cross attention #2388

Merged
merged 1 commit into from
Mar 6, 2024
Merged

[transformer] add cross attention #2388

merged 1 commit into from
Mar 6, 2024

Conversation

Mddct
Copy link
Collaborator

@Mddct Mddct commented Mar 5, 2024

@Mddct Mddct force-pushed the Mddct-cross-att branch 2 times, most recently from faa8e5a to cca9554 Compare March 5, 2024 17:06
@Mddct Mddct force-pushed the Mddct-cross-att branch from cca9554 to e8a6e6d Compare March 5, 2024 18:00
@Mddct
Copy link
Collaborator Author

Mddct commented Mar 6, 2024

aishell att cer float32/float16
old decoder cache 5.53 / 5.52
new decoder cache 5.53 / 5.52

** 对aishell的解码结果tn+繁体转简体后

whisper att cer A100 bs=16 float16
old decoder cache 5.13 148m
new decoder cache 5.13 68m

@Mddct Mddct requested review from robin1001 and xingchensong March 6, 2024 03:40
@xingchensong xingchensong merged commit ea6ff53 into main Mar 6, 2024
6 checks passed
@xingchensong xingchensong deleted the Mddct-cross-att branch March 6, 2024 05:28
@xingchensong
Copy link
Member

xingchensong commented Mar 6, 2024

整个的思考逻辑是这样:

想用crossattn的cache模式 ->

先尝试在原来那个公用的类的forward里改,发现能提速但是因为selfcache和crosscache处理逻辑耦合,所以不太想要 ( #2377 ) ->

决定分开 (现在的PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants