Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lookup: add print for drafting performance #5450

Merged

Conversation

JohannesGaessler
Copy link
Collaborator

This PR measures the time needed to create the draft in the lookup example and prints this information at the end. As it turns out the time is negligible:

n_draft   = 4
n_predict = 1025
n_drafted = 712
t_draft   = 2.85 ms, 4.01 us per token, 249387.04 tokens per second
n_accept  = 410
accept    = 57.584%

Copy link
Owner

@ggerganov ggerganov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to measure the entire prompt_lookup() call:

{
    const int64_t t_start_us = ggml_time_us();

    prompt_lookup();

    t_draft_us += ggml_time_us() - t_start_us;
}

@JohannesGaessler JohannesGaessler merged commit e4640d8 into ggerganov:master Feb 11, 2024
48 of 53 checks passed
jordankanter pushed a commit to jordankanter/llama.cpp that referenced this pull request Mar 13, 2024
hodlen pushed a commit to hodlen/llama.cpp that referenced this pull request Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants