Profiling

[jstac]

At this stage we have no discussion of profiling code to find bottlenecks. Here's the IPython solution:

http://pynash.org/2013/03/06/timing-and-profiling.html

I also hear good things about this visual profiling library:

http://pycallgraph.slowchop.com/en/master/

[sglyon]

One bonus for using nose to run our tests is that they have an option to automatically profile the code as the tests are run. If we can get to 100% coverage, then we should be able to see profiling results for all our code.

Also, if we just want to test a specific module, function, or method, we can just have nose run the specific test and profile only that one.

pandas uses a utility they wrote called vbench that keeps track of profiling results over time so they can track progress and easily find performance regressions (see their results by clicking on various links in the RHS sidebar here. Something similar to that would be cool for our project too.

[jstac]

@spencerlyon2 Thanks for those thoughts. I actually meant to put my comments in the lectures repo issue tracker, since I was thinking more of exposition. I'll move them over but leave this issue open so we can discuss your suggestions for QuantEcon code.

[sglyon]

Ping.

Any interest in writing a suite of performance benchmarks that are tracked via vbench?

[davidrpugh]

Possibly, but will need to read up on vbench first...

[mmcky]

Anyone come across any other performance benchmark approaches other than vbench? I notice that the bench project hasn't had any contributions since 2013 - and pandas benchmarks end in 2012. Just thought I would check before investing time into this.

[mmcky]

I have just come across this project which looks like a good benchmarking tool.
https://github.com/spacetelescope/asv/
It looks like pandas and astropy have started to use this.
Example site: http://mdboom.github.io/astropy-benchmark/

[jstac]

@mmcky It definitely looks useful. You need stable hardware to keep the comparisons consistent but we will have that soon in Singapore...

[mmcky]

Setup and try: https://github.com/spacetelescope/asv

[mmcky]

I have started setting up asv benchmarks for QuantEcon.py.
It currently contains only two benchmarks which is an arma simulation and impulse response but I will use tests to populate some more benchmarks over the coming weeks.

https://github.com/QuantEcon/QuantEcon.py.benchmark

[mmcky]

I think this issue can now be tracked through the new repository -- except the original comments about profiling are still valid. @jstac should we move your early comments to a new lecture issue?

[jstac]

@mmcky Good call, yes please.

[mmcky]

Closing this discussion now. New comments on vbench and profiling can be made in the benchmark repo.