TemplateUpgraders should be called during rolling restart #25263

imotov · 2017-06-15T18:17:35Z

In #24379 we added ability to upgrade templates on full cluster startup. This PR invokes the same update procedure also when a new node first joins the cluster allowing to update templates on a rolling cluster restart as well.

Closes #24680

@abeyad, @spinscale could you take a look to make sure this will work for you?

In elastic#24379 we added ability to upgrade templates on full cluster startup. This PR invokes the same update procedure also when a new node first joins the cluster allowing to update templates on a full cluster restart. Closes elastic#24680

rjernst · 2017-06-15T18:44:53Z

core/src/main/java/org/elasticsearch/node/Node.java

@@ -460,6 +463,7 @@ protected Node(final Environment environment, Collection<Class<? extends Plugin>
                b.bind(UsageService.class).toInstance(usageService);
                    b.bind(NamedWriteableRegistry.class).toInstance(namedWriteableRegistry);
                    b.bind(MetaDataUpgrader.class).toInstance(metaDataUpgrader);
+                    b.bind(TemplateUpgradeService.class).toInstance(templateUpgradeService);


It seems this is only bound in Guice so that tests can get access to it, so they can check if upgrades are in progress. Is this something that should be exposed via an api? At least let's find another way so we can keep additional stuff out of guice.

Good point! I rewrote the test to get rid of guice there. I don't think we need an api for that. I was just trying to prevent a possible race condition in the test, but I found another way to do that without guice.

abeyad

I left a few minor nits and a question. Otherwise, LGTM

abeyad · 2017-06-15T19:18:57Z

core/src/main/java/org/elasticsearch/cluster/metadata/IndexTemplateMetaData.java

@@ -431,7 +439,6 @@ public static void toXContent(IndexTemplateMetaData indexTemplateMetaData, XCont
            }
            builder.endObject();



nit: extra newline

abeyad · 2017-06-17T03:17:40Z

core/src/main/java/org/elasticsearch/cluster/metadata/TemplateUpgradeService.java

+ * Upgrades Templates on behalf of installed {@link Plugin}s when a node joins the cluster
+ */
+public class TemplateUpgradeService extends AbstractComponent implements ClusterStateListener {
+    private final Tuple<Map<String, BytesReference>, Set<String>> EMPTY = new Tuple<>(Collections.emptyMap(), Collections.emptySet());


EMPTY could have a more descriptive name

abeyad · 2017-06-17T05:06:27Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceIT.java

+        AtomicInteger updateCount = new AtomicInteger();
+        // Make sure all templates are recreated correctly
+        assertBusy(() -> {
+            logger.info("checking....");


a more descriptive log message? or is this leftover?

abeyad · 2017-06-17T05:08:38Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+            assertThat(removedListener.getAndSet((ActionListener) args[1]), nullValue());
+            return null;
+        }).when(mockIndicesAdminClient).deleteTemplate(any(DeleteIndexTemplateRequest.class), any(ActionListener.class));
+


nit: extra newline

abeyad · 2017-06-17T05:10:43Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+
+public class TemplateUpgradeServiceTests extends ESTestCase {
+
+    private ClusterService clusterService = new ClusterService(Settings.EMPTY, new ClusterSettings(Settings.EMPTY, ClusterSettings


abeyad · 2017-06-17T05:15:04Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+            deleteTemplateListeners.add((ActionListener) args[1]);
+            return null;
+        }).when(mockIndicesAdminClient).deleteTemplate(any(DeleteIndexTemplateRequest.class), any(ActionListener.class));
+


nit: extra newline

abeyad · 2017-06-17T05:19:27Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+            Collections.emptyList());
+
+        service.updateTemplates(additions, deletions);
+


Should we add an assert here that putTemplateListeners.size() == additionsCount and deleteTemplateListeners.size() == deletionsCount?

nit^2: assertThat(putTemplateListeners, hasSize(additionsCount));

abeyad · 2017-06-17T05:21:46Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+                putTemplateListeners.get(i).onFailure(new RuntimeException());
+            } else {
+                putTemplateListeners.get(i).onResponse(new PutIndexTemplateResponse(randomBoolean()) {
+


its not clear to me what the purpose of this response listener is? we are not asserting anything when invoking the action listener, so I'm not clear what its role is in the test?

spinscale · 2017-06-19T07:50:24Z

To make sure this is addressed (and I may have just missed this, which I assume). The current cluster state listener implementation of TemplateUpgradeService runs on every node in the cluster, whenever there is a change - this also means, that when two nodes get the same cluster state change, they try to store/delete the same templates?

imotov · 2017-06-19T11:35:49Z

@spinscale you didn't miss anything. We are changing the way templates are upgraded - they are upgraded when a node with a new version joins not when it becomes the master, hence we need to run it on every node in the cluster. As a result, you are right, multiple nodes may try to creating the same template at the same time. Not sure if we can do much about out giving the requirements. Perhaps we can try optimizing MetaDataIndexTemplateService to not cause cluster state update if the template is exactly the same.

imotov · 2017-06-19T11:58:50Z

@spinscale and I discussed this a bit more and we came up with a schema that will allow the new nodes to update the template but will prevent update swarms. Before trying to update the templates

A node will check if the master has the newest version. If it does and the nodes is the master, the master will perform the update. Otherwise the node will don't do anything.
If the master doesn't have the highest version, then the node with the highest id among all nodes with the highest version will perform the upgrade.

imotov · 2017-06-20T10:52:39Z

@spinscale, @abeyad, I pushed the new logic that ensures that only one node can try to make updates at a time. Could you take another look?

spinscale

LGTM. I keep thinking if the explicit master node check is fully necessary or can just be part of the highest version check - but I dont have any particular opinion to be honest. Thanks for doing this and coming up with the much better solution.

spinscale · 2017-06-16T16:45:33Z

core/src/main/java/org/elasticsearch/cluster/metadata/TemplateUpgradeService.java

+
+    private final UnaryOperator<Map<String, IndexTemplateMetaData>> indexTemplateMetaDataUpgraders;
+
+    public final ClusterService clusterService;


intentionally public along with the next two variables?

spinscale · 2017-06-16T16:46:57Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+
+                });
+            }
+        }


you are calling the listeners here, but not doing any further assertions? i.e. for the updatesInProgress variable?

spinscale · 2017-06-16T16:49:45Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+        ExecutorService executorService = mock(ExecutorService.class);
+        when(threadPool.generic()).thenReturn(executorService);
+        doAnswer(invocation -> {
+            Object[] args = invocation.getArguments();


you could just use a EsExecutors.newDirectExecutorService()

oh... didnt see the updateInvocation counter... nvm

spinscale · 2017-06-20T11:35:00Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+            Collections.emptyList());
+
+        service.updateTemplates(additions, deletions);
+


nit^2: assertThat(putTemplateListeners, hasSize(additionsCount));

spinscale · 2017-06-20T11:36:49Z

core/src/test/java/org/elasticsearch/cluster/metadata/TemplateUpgradeServiceTests.java

+        }
+    }
+
+    public void testClientNodeRunsTemplateUpdates() {


that should not be doesNotRunTemplateUpgrade I assume

spinscale · 2017-06-20T11:41:40Z

core/src/main/java/org/elasticsearch/cluster/metadata/TemplateUpgradeService.java

+                }
+                for (ObjectCursor<DiscoveryNode> node : nodes.getMasterAndDataNodes().values()) {
+                    if (node.value.getVersion().equals(maxVersion)) {
+                        if (node.value.getId().compareTo(localNode.getId()) > 0) {


can be combined to a single if-statement?

abeyad

LGTM

* master: testCreateShrinkIndex: removed left over debugging log line that violated linting testCreateShrinkIndex should make sure to use the right source stats when testing shrunk target [Test] Add unit test for XContentParserUtilsTests.parseStoredFieldsValue (elastic#25288) Update percolate-query.asciidoc (elastic#25364) Remove remaining `index.mapping.single_type=false` (elastic#25369) test: Replace OldIndexBackwardsCompatibilityIT#testOldClusterStates with a full cluster restart qa test Fix use of spaces on Windows if JAVA_HOME not set ESIndexLevelReplicationTestCase.ReplicationAction#execute should send exceptions to it's listener rather than bubble them up testRecoveryAfterPrimaryPromotion shouldn't flush the replica with extra operations fix sort and string processor tests around targetField (elastic#25358) Ensure `InternalEngineTests.testConcurrentWritesAndCommits` doesn't pile up commits (elastic#25367) [TEST] Add debug logging if an unexpected exception is thrown Update Painless to Allow Augmentation from Any Class (elastic#25360) TemplateUpgraders should be called during rolling restart (elastic#25263)

In #24379 we added ability to upgrade templates on full cluster startup. This PR invokes the same update procedure also when a new node first joins the cluster allowing to update templates on a rolling cluster restart as well. Closes #24680

imotov added :Upgrade API >feature >upgrade v5.6.0 v6.0.0 labels Jun 15, 2017

imotov requested review from spinscale and abeyad June 15, 2017 18:17

rjernst reviewed Jun 15, 2017

View reviewed changes

Remove TemplateUpgradeService from guice

8d141d0

abeyad approved these changes Jun 17, 2017

View reviewed changes

imotov added 2 commits June 19, 2017 14:49

Address @abeyad's comments

cbe8ca5

Ensure that only one node will try to upgrade the templates at a time

e84c078

spinscale reviewed Jun 20, 2017

View reviewed changes

Address @spinscale's comments

dba5ab1

abeyad approved these changes Jun 20, 2017

View reviewed changes

imotov merged commit e6e5ae6 into elastic:master Jun 22, 2017

imotov added the backport pending label Jun 22, 2017

imotov removed the backport pending label Jun 23, 2017

clintongormley added v6.0.0-beta1 and removed v6.0.0 labels Jul 25, 2017

imotov deleted the issue-24680-rolling-restart-upgrades branch May 1, 2020 22:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TemplateUpgraders should be called during rolling restart #25263

TemplateUpgraders should be called during rolling restart #25263

imotov commented Jun 15, 2017

rjernst Jun 15, 2017

imotov Jun 15, 2017

abeyad left a comment

abeyad Jun 15, 2017

abeyad Jun 17, 2017

abeyad Jun 17, 2017

abeyad Jun 17, 2017

abeyad Jun 17, 2017

abeyad Jun 17, 2017

abeyad Jun 17, 2017

spinscale Jun 20, 2017

abeyad Jun 17, 2017

spinscale commented Jun 19, 2017

imotov commented Jun 19, 2017

imotov commented Jun 19, 2017

imotov commented Jun 20, 2017

spinscale left a comment

spinscale Jun 16, 2017

spinscale Jun 16, 2017

spinscale Jun 16, 2017

spinscale Jun 16, 2017

spinscale Jun 20, 2017

spinscale Jun 20, 2017

spinscale Jun 20, 2017

abeyad left a comment

		@@ -431,7 +439,6 @@ public static void toXContent(IndexTemplateMetaData indexTemplateMetaData, XCont
		}
		builder.endObject();


		public class TemplateUpgradeServiceTests extends ESTestCase {

		private ClusterService clusterService = new ClusterService(Settings.EMPTY, new ClusterSettings(Settings.EMPTY, ClusterSettings

		Collections.emptyList());

		service.updateTemplates(additions, deletions);


		private final UnaryOperator<Map<String, IndexTemplateMetaData>> indexTemplateMetaDataUpgraders;

		public final ClusterService clusterService;

TemplateUpgraders should be called during rolling restart #25263

TemplateUpgraders should be called during rolling restart #25263

Conversation

imotov commented Jun 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abeyad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spinscale commented Jun 19, 2017

imotov commented Jun 19, 2017

imotov commented Jun 19, 2017

imotov commented Jun 20, 2017

spinscale left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abeyad left a comment

Choose a reason for hiding this comment