Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Weighted Routing] Allow Search with preference parameter in weighed in nodes #6296

Closed
wants to merge 9 commits into from
Closed

[Weighted Routing] Allow Search with preference parameter in weighed in nodes #6296

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8dd03c3
Search with preference parameter in weighed in nodes
Feb 13, 2023
7a902f1
Add integ tests
Feb 13, 2023
740b196
Modify integ test to handle strict weighted routing disabled use case
Feb 14, 2023
35728e8
Fix code and add more tests
Feb 14, 2023
355017f
add more integ tests
Feb 14, 2023
fb13bee
Merge branch 'main' into allow-preference-search-weigh-in-nodes
Feb 14, 2023
648f068
refactor code
Feb 14, 2023
0e168ae
Add comments explaining behavioral change
Feb 15, 2023
1a41c46
modify comments
Feb 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Cluster health call to throw decommissioned exception for local decommissioned node([#6008](https://github.com/opensearch-project/OpenSearch/pull/6008))
- [Refactor] core.common to new opensearch-common library ([#5976](https://github.com/opensearch-project/OpenSearch/pull/5976))
- Cluster local health call to throw exception if node is decommissioned or weighed away ([#6198](https://github.com/opensearch-project/OpenSearch/pull/6198))
- Allow search with preference parameter when strict weighted shard routing is enabled ([#6296](https://github.com/opensearch-project/OpenSearch/pull/6296))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now have the following two entries to go in the next release notes:

  • Add support to disallow search request with preference parameter with strict weighted shard routing
  • Allow search with preference parameter when strict weighted shard routing is enabled

I find these two statements confusing! Since the second change is just modifying a subtle behavior of the first change (which is still unreleased), maybe we don't need the second entry?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will modify the first entry itself


### Deprecated

Expand All @@ -125,4 +126,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.5...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.5...2.x
255 changes: 248 additions & 7 deletions server/src/internalClusterTest/java/org/opensearch/search/SearchWeightedRoutingIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -803,9 +803,9 @@ public void testMultiGetWithNetworkDisruption_FailOpenDisabled() throws Exceptio
}

/**
* Assert that preference based search is not allowed with strict weighted shard routing
* Assert that preference search with _only_local fails for weighed away node
*/
public void testStrictWeightedRouting() {
public void testStrictWeightedRoutingWithOnlyLocalNode() {

Settings commonSettings = Settings.builder()
.put("cluster.routing.allocation.awareness.attributes", "zone")
Expand All @@ -818,24 +818,265 @@ public void testStrictWeightedRouting() {
Map<String, List<String>> nodeMap = setupCluster(nodeCountPerAZ, commonSettings);

int numShards = 10;
int numReplicas = 1;
int numReplicas = 2;
setUpIndexing(numShards, numReplicas);

logger.info("--> setting shard routing weights for weighted round robin");
Map<String, Double> weights = Map.of("a", 1.0, "b", 1.0, "c", 0.0);
setShardRoutingWeights(weights);

// throws exception for weighed away node
assertThrows(
PreferenceBasedSearchNotAllowedException.class,
() -> internalCluster().client(nodeMap.get("c").get(0)).prepareSearch().setSize(4).setPreference("_only_local").get()
);

// success for weighed in node
SearchResponse searchResponse = internalCluster().client(nodeMap.get("b").get(0))
.prepareSearch()
.setSize(4)
.setPreference("_only_local")
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());

// disable strict weighed routing
client().admin()
.cluster()
.prepareUpdateSettings()
.setTransientSettings(Settings.builder().put("cluster.routing.weighted.strict", false))
.get();

// make search on weighed away node
searchResponse = internalCluster().client(nodeMap.get("c").get(0)).prepareSearch().setSize(4).setPreference("_only_local").get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());
}

/**
* Assert that preference search with custom string doesn't hit a node in weighed away az
*/
public void testStrictWeightedRoutingWithCustomString() {

Settings commonSettings = Settings.builder()
.put("cluster.routing.allocation.awareness.attributes", "zone")
.put("cluster.routing.allocation.awareness.force.zone.values", "a,b,c")
.put("cluster.routing.weighted.fail_open", true)
.put("cluster.routing.weighted.strict", true)
.build();

int nodeCountPerAZ = 1;
Map<String, List<String>> nodeMap = setupCluster(nodeCountPerAZ, commonSettings);

int numShards = 20;
int numReplicas = 2;
setUpIndexing(numShards, numReplicas);

logger.info("--> setting shard routing weights for weighted round robin");
Map<String, Double> weights = Map.of("a", 1.0, "b", 1.0, "c", 0.0);
setShardRoutingWeights(weights);
String nodeInZoneA = nodeMap.get("a").get(0);
String customPreference = randomAlphaOfLength(10);

SearchResponse searchResponse = internalCluster().client(nodeMap.get("b").get(0))
.prepareSearch()
.setSize(20)
.setPreference(customPreference)
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());
assertNoSearchInAZ("c");
assertSearchInAZ("a");
assertSearchInAZ("b");

// disable strict weighed routing
client().admin()
.cluster()
.prepareUpdateSettings()
.setTransientSettings(Settings.builder().put("cluster.routing.weighted.strict", false))
.get();

// make search requests with custom string
searchResponse = internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(20)
.setPreference(customPreference)
.get();
// assert search on data nodes on az c (weighed away az)
assertSearchInAZ("c");

}

/**
* Assert that preference search with LOCAL doesn't hit a node in weighed away az
*/
public void testStrictWeightedRoutingWithLocal() {

Settings commonSettings = Settings.builder()
.put("cluster.routing.allocation.awareness.attributes", "zone")
.put("cluster.routing.allocation.awareness.force.zone.values", "a,b,c")
.put("cluster.routing.weighted.fail_open", true)
.put("cluster.routing.weighted.strict", true)
.build();

int nodeCountPerAZ = 1;
Map<String, List<String>> nodeMap = setupCluster(nodeCountPerAZ, commonSettings);

int numShards = 20;
int numReplicas = 2;
setUpIndexing(numShards, numReplicas);

logger.info("--> setting shard routing weights for weighted round robin");
Map<String, Double> weights = Map.of("a", 1.0, "b", 1.0, "c", 0.0);
setShardRoutingWeights(weights);

// search request with LOCAL on weighed away az nodes
SearchResponse searchResponse = internalCluster().client(nodeMap.get("c").get(0))
.prepareSearch()
.setSize(20)
.setPreference("_local")
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());

assertNoSearchInAZ("c");

// search request with LOCAL on weighed in az nodes
searchResponse = internalCluster().client(nodeMap.get("b").get(0)).prepareSearch().setSize(20).setPreference("_local").get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());

assertSearchInAZ("b");

// disable strict weighed routing
client().admin()
.cluster()
.prepareUpdateSettings()
.setTransientSettings(Settings.builder().put("cluster.routing.weighted.strict", false))
.get();

// make search requests with custom string
searchResponse = internalCluster().client(nodeMap.get("c").get(0)).prepareSearch().setSize(20).setPreference("_local").get();
// assert search on data nodes in az c (weighed away az)
assertSearchInAZ("c");
}

/**
* Assert that preference search with _prefer_nodes doesn't hit a node in weighed away az
*/
public void testStrictWeightedRoutingWithPreferNodes() {

Settings commonSettings = Settings.builder()
.put("cluster.routing.allocation.awareness.attributes", "zone")
.put("cluster.routing.allocation.awareness.force.zone.values", "a,b,c")
.put("cluster.routing.weighted.fail_open", true)
.put("cluster.routing.weighted.strict", true)
.build();

int nodeCountPerAZ = 1;
Map<String, List<String>> nodeMap = setupCluster(nodeCountPerAZ, commonSettings);

int numShards = 20;
int numReplicas = 2;
setUpIndexing(numShards, numReplicas);

logger.info("--> setting shard routing weights for weighted round robin");
Map<String, Double> weights = Map.of("a", 1.0, "b", 1.0, "c", 0.0);
setShardRoutingWeights(weights);

DiscoveryNodes dataNodes = internalCluster().clusterService().state().nodes();

Map<String, String> nodeIDMap = new HashMap<>();
for (DiscoveryNode node : dataNodes) {
nodeIDMap.put(node.getName(), node.getId());
}

// search request with _prefer_nodes on weighed away az nodes
SearchResponse searchResponse = internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(20)
.setPreference("_prefer_nodes:" + nodeIDMap.get(nodeMap.get("c").get(0)))
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());

assertNoSearchInAZ("c");

// search request with _prefer_nodes on weighed in az nodes
searchResponse = internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(20)
.setPreference("_prefer_nodes:" + nodeIDMap.get(nodeMap.get("b").get(0)))
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());

assertSearchInAZ("b");

// disable strict weighed routing
client().admin()
.cluster()
.prepareUpdateSettings()
.setTransientSettings(Settings.builder().put("cluster.routing.weighted.strict", false))
.get();

// make search requests with _prefer_nodes on weighed away az
searchResponse = internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(20)
.setPreference("_prefer_nodes:" + nodeIDMap.get(nodeMap.get("c").get(0)))
.get();
// assert search on data nodes in az c (weighed away az)
assertSearchInAZ("c");
}

/**
* Assert that preference search with only_nodes doesn't hit a node in weighed away az
*/
public void testStrictWeightedRoutingWithOnlyNodes() {

Settings commonSettings = Settings.builder()
.put("cluster.routing.allocation.awareness.attributes", "zone")
.put("cluster.routing.allocation.awareness.force.zone.values", "a,b,c")
.put("cluster.routing.weighted.fail_open", true)
.put("cluster.routing.weighted.strict", true)
.build();

int nodeCountPerAZ = 1;
Map<String, List<String>> nodeMap = setupCluster(nodeCountPerAZ, commonSettings);

int numShards = 20;
int numReplicas = 2;
setUpIndexing(numShards, numReplicas);

logger.info("--> setting shard routing weights for weighted round robin");
Map<String, Double> weights = Map.of("a", 1.0, "b", 1.0, "c", 0.0);
setShardRoutingWeights(weights);

// search request with LOCAL on weighed away az nodes throws exception
assertThrows(
PreferenceBasedSearchNotAllowedException.class,
() -> internalCluster().client(nodeMap.get("b").get(0))
() -> internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(0)
.setPreference(randomFrom("_local", "_only_nodes:" + nodeInZoneA, "_prefer_nodes:" + nodeInZoneA, customPreference))
.setSize(20)
.setPreference("_only_nodes:" + "zone:c")
.get()
);

// search request with LOCAL on weighed in az nodes
SearchResponse searchResponse = internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(20)
.setPreference("_only_nodes:" + "zone:b")
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());

// disable strict weighed routing
client().admin()
.cluster()
.prepareUpdateSettings()
.setTransientSettings(Settings.builder().put("cluster.routing.weighted.strict", false))
.get();

// search request with LOCAL on weighed away az nodes
searchResponse = internalCluster().client(nodeMap.get("a").get(0))
.prepareSearch()
.setSize(20)
.setPreference("_only_nodes:" + "zone:c")
.get();
assertEquals(RestStatus.OK.getStatus(), searchResponse.status().getStatus());
}

/**
Expand Down
Loading