Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessive GC load related to use of Sqlexpr_sqlite_lwt #13

Closed
mfp opened this issue Feb 28, 2016 · 0 comments
Closed

Excessive GC load related to use of Sqlexpr_sqlite_lwt #13

mfp opened this issue Feb 28, 2016 · 0 comments

Comments

@mfp
Copy link
Owner

mfp commented Feb 28, 2016

The Sqlexpr_sqlite_lwt worker loop uses the Event module to receive tasks across threads boundaries.
As it turns out, Event.sync creates a new Condition.t.

This means in particular that a new condition variable (two in fact, also one for the sending side) is being created per returned row(!).

caml_condition_new allocates the custom blocks with used=1, max = Max_condition_number. The latter was raised to 5000 in 2010, after being set to 1000 back in 1996.

The end effect is that a full major GC run is performed when at most 5000 rows have been read from the SQLite3 DB.

This raises two questions:

  1. are condition variables expensive enough to care about their timely collection instead of using used=0, max = 1
  2. are such static limits reasonable and future-proof, and should they be collected in some visible place so that they can be raised in time? (I ran into a similar limit in the past with ocaml-pcre, which allowed no more than 500 compiled regexps to be unreclaimed)

Now, back to Sqlexpr: time to ditch the Event module and replace it with a single condition variable + manual handling.

Note to self: send issue to ocaml project regarding the performance impact of Event.sync.

Edit: also file Lwt issue, for it also uses Event.sync in Lwt_preemptive (so 1 full major GC run every 2500 Lwt_preemptive.detach at most)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant