Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add API methods for profiling pipeline execution #73

Closed
tdruez opened this issue Jan 22, 2021 · 2 comments
Closed

Add API methods for profiling pipeline execution #73

tdruez opened this issue Jan 22, 2021 · 2 comments
Assignees
Milestone

Comments

@tdruez
Copy link
Contributor

tdruez commented Jan 22, 2021

The goal is to locate the bottlenecks for future optimizations.

@tdruez tdruez added this to the 1.1 milestone Jan 22, 2021
@tdruez tdruez self-assigned this Jan 22, 2021
tdruez added a commit that referenced this issue Jan 22, 2021
@tdruez
Copy link
Contributor Author

tdruez commented Jan 22, 2021

A profile() method is now available on the Run model for profiling pipeline execution:

from scanpipe.models import Run

run = Run.objects.get(project__uuid="6ab92dfe-8623-4753-8fb6-98c9c9ae0a5b")
run.profile()
{'analyze_scanned_files': 1,
 'collect_and_create_codebase_resources': 73,
 'collect_and_create_system_packages': 525,
 'collect_images_information': 1,
 'end': 1,
 'extract_images': 5,
 'extract_layers': 4,
 'find_images_linux_distro': 1,
 'scan_for_application_packages': 1,
 'scan_for_files': 21,
 'start': 1,
 'tag_not_analyzed_codebase_resources': 1,
 'tag_uninteresting_codebase_resources': 4}

Formatted results can also be printed to stdout:

run.profile(print_results=True)

Screenshot 2021-01-22 at 1 26 55 PM

@tdruez
Copy link
Contributor Author

tdruez commented Jan 22, 2021

Once a problematic step is located, https://github.com/joerick/pyinstrument is a nice tool to go deeper in the profiling:

pip install pyinstrument
from pyinstrument import Profiler

with Profiler() as profiler:
    code_to_profile

# Stdout
print(profiler.output_text(unicode=True, color=True))
# HTML
profiler.open_in_browser()

tdruez added a commit that referenced this issue Jan 22, 2021
Signed-off-by: Thomas Druez <[email protected]>
tdruez added a commit that referenced this issue Jan 22, 2021
tdruez added a commit that referenced this issue Feb 16, 2021
@tdruez tdruez closed this as completed Feb 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant