point store size fix #265
Merged
point store size fix #265
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kaituo
approved these changes
Aug 6, 2021
wnbts
approved these changes
Aug 6, 2021
kaituo
added a commit
to kaituo/anomaly-detection-1
that referenced
this pull request
Aug 9, 2021
We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
kaituo
added a commit
to opensearch-project/anomaly-detection
that referenced
this pull request
Aug 11, 2021
Disable model splitting in single-stream detectors We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
ohltyler
pushed a commit
to ohltyler/anomaly-detection-2
that referenced
this pull request
Sep 1, 2021
…t#162) Disable model splitting in single-stream detectors We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
ohltyler
pushed a commit
to ohltyler/anomaly-detection-2
that referenced
this pull request
Sep 1, 2021
…t#162) Disable model splitting in single-stream detectors We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
ohltyler
pushed a commit
to opensearch-project/anomaly-detection
that referenced
this pull request
Sep 1, 2021
Disable model splitting in single-stream detectors We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
hamersu9t
added a commit
to hamersu9t/anomaly-detection
that referenced
this pull request
Aug 10, 2024
Disable model splitting in single-stream detectors We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes: The dimensions parameter was multiplied to the size twice.