Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CPU] Optimize PagedAttention performance #27718

Closed
wants to merge 8 commits into from
Closed

[CPU] Optimize PagedAttention performance #27718

wants to merge 8 commits into from

Conversation

luo-cheng2021
Copy link
Contributor

@luo-cheng2021 luo-cheng2021 commented Nov 25, 2024 •
edited
Loading

Details:

  • Common optimization for PagedAttention
    • Generate more work items to avoid thread imbalance
    • Simplify scale subgraph
  • ChatGLM
    • Fix ROPE pattern
    • Fuse (FC)->Split->Reshape/(ROPE)->Reshape/(PageAttention)->Reshape->(MLP) into PagedAttention
  • GPTJ
    • Fix ROPE pattern

Tickets:

@github-actions github-actions bot added category: CPU OpenVINO CPU plugin category: transformations OpenVINO Runtime library - Transformations labels Nov 25, 2024
@luo-cheng2021 luo-cheng2021 added WIP work in progress do_not_review and removed category: CPU OpenVINO CPU plugin category: transformations OpenVINO Runtime library - Transformations labels Nov 25, 2024
@github-actions github-actions bot added category: CPU OpenVINO CPU plugin category: transformations OpenVINO Runtime library - Transformations labels Nov 27, 2024
@luo-cheng2021
Copy link
Contributor Author

CPU plugin part will be merged into #27847.

@itikhono itikhono mentioned this pull request Jan 15, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Jan 16, 2025
@itikhono
Fix RopeFusion transformation after applying SDPA to PagedAttention c…
ed50d51 
…onversion (#28447)

### Details:
After internal discussion, we decided to use the changes from
#27718 as a base line
Fixed Rope pattern detection for ChatGLM.

### Tickets:
 - *CVS-158393*
MirceaDan99 pushed a commit to MirceaDan99/openvino that referenced this pull request Jan 22, 2025
@itikhono @MirceaDan99
Fix RopeFusion transformation after applying SDPA to PagedAttention c…
675bbf3 
…onversion (openvinotoolkit#28447)

### Details:
After internal discussion, we decided to use the changes from
openvinotoolkit#27718 as a base line
Fixed Rope pattern detection for ChatGLM.

### Tickets:
 - *CVS-158393*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
category: CPU OpenVINO CPU plugin category: transformations OpenVINO Runtime library - Transformations do_not_review WIP work in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@luo-cheng2021