Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use mgo.Collection.Bulk to prune docs #20

Merged
merged 1 commit into from
Oct 31, 2016
Merged

Conversation

axw
Copy link
Contributor

@axw axw commented Oct 30, 2016

Use bulk document removal, rather than
a single RemoveAll with a query of "_id $in [...]".
The latter has performance problems in some
configurations of MongoDB. See https://bugs.launchpad.net/juju/+bug/1635311.

Excerpt from the bug:

Today I did the test again on the Azure instance where the issue was reproduced, instead of my local system where I did it first. On there, Mongo does not like the method used at all. In the logs from Comment 6, there's a log message stating that ~9000 transactions need pruning. Using the method from Juju, removing 10000 docs took about 2 minutes in one test. I wrote some more optimised code which does the same thing ~instantly.

(Review request: http://reviews.vapour.ws/r/5955/)

Use bulk document removal, rather than
a single RemoveAll with a query of "_id $in [...]".
The latter has performance problems in some
configurations of MongoDB. See lp:1635311.
Copy link

@mjs mjs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great for 2.0.

Before this gets used with 1.25 (MongoDB 2.4), it would be good to test the performance of mgo's Bulk fallback for older MongoDB versions.

@axw
Copy link
Contributor Author

axw commented Oct 31, 2016

For posterity: I tested with mongo 2.4, and there is no performance penalty for the existing approach. Using Bulk isn't as fast there because 2.4 doesn't support bulk operations, and so mgo just turns it into a series of operations. Nevertheless, it only takes a few seconds to remove the 10000 documents, so we'll stick with just using Bulk across the board for simplicity.

@axw
Copy link
Contributor Author

axw commented Oct 31, 2016

$$merge$$

@axw axw merged commit af2fa20 into juju:master Oct 31, 2016
axw added a commit to axw/mgopurge that referenced this pull request Feb 7, 2017
Apply juju/txn#20,
fixing Issue juju#2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants