Big performance bottleneck with default comparator

[Delapouite]

Hi

The FilesCollection has a default comparator : [date:-1, name:1]

The issue is that during the first steps of the generation process, the onChange event keeps triggering the sorting of the collection over and over which is a very costly operation over heaps of entries (thousands of models).

By simply removing this comparator, the Parsing phase time was divided by more than 10, going from 50 seconds to only 3. The Contextualizing step went from 15 minutes to only 2min.

Significant improvements that are such a relief to my Docpad experience !

Is the comparator really mandatory in those initial phases or should collections be sorted only once on the contextualizeAfter event, before the rendering ?

[balupton]

Wow. What a fantastic find! Great work!

This makes me think, sorting is important, and it is important for live collections to stay up to date, however not all the time is that important. Perhaps we can pause and resume live collections after we've done processing batches. Kind of combining the best of non-live and live together. This would surely bring huge improvements to everyone that uses queryengine.

For DocPad, that would mean, pausing live collections at the start, then resuming them just before the rendering process. I'll think into this some more. But we can happily wipe that default comparator for any default collections in DocPad, I doubt anyone would use those collections directly.

[balupton]

Just took a look and I couldn't find the default comparator you mention. Is that defined in the docpad configuration file or in docpad itself? Perhaps my eyes are rusty ;)

Also, turns out QueryEngine already does support the pause resume stuff we're after. collection.live(false) to pause and collection.live(true) to resume. This should work well :)

So I think in perhaps generatePrepare we pause all the collections, and then in render event we resume them. Your thoughts? If you're inclined to do a pull, I'll happily accept :)

[balupton]

Did some work on the above suggestion on https://github.com/bevry/docpad/tree/dev-suspendcollections - however there is some major problems with this:

• live must be re-enabled before generateRender
• add events that modify things do not care for the reset event which is what .query() will fire

Not sure how we can optimise this further, as the above requires some nasty hacks.

[Delapouite]

That's definitely a problem not easy to tackle, because we also have to make sure it doesn't break plugin behavior that hook early in the process.

For instance I rely a lot on the relatedPlugin which does its magic on the parseAfter event, setting it's own comparator on sub relatedDocuments collections. By removing the default comparator, and reactivating the live on the generateRender, the order is all shuffled.

[balupton]

@Delapouite the work I've been doing on queryengine should see massive improvements here - http://jsperf.com/queryengine/3

[balupton]

Give DocPad v6.10.0 a try. It handles things a bit differently so you should see good performance boosts.

[balupton]

Released to DocPad v6.12