Use openshift-master-api and openshift-master-controllers setup always #4832

smarterclayton · 2017-07-22T18:12:13Z

The installer now always configures two services on the master for the api server and the controllers. Native clustering is the default configuration mode, even when only one master is configured. A warning is added if there are no etcd groups configured. Finally, the new client based leader election code in the controllers is used as the default, which removes the need for the controllers to have access to etcd.

This will set the stage for running controllers and apiservers as static pods.

@sdodson

smarterclayton · 2017-07-23T17:58:09Z

aos-ci-test

smarterclayton · 2017-07-23T18:23:48Z

Changes required to support this in upgrades:

stop and remove the old origin-master.service and sysconfig files if they exist
error / warn if the user had those files that scripts and other automation may need to change.

smarterclayton · 2017-07-23T18:25:48Z

This does not include the change to move controllers to static pods because I realized we need to deal with some fundamental ordering problems before that (the node needs to be configured and ready ahead of time, instead of masters first). Requires the work @kwoodson will be doing to split out node preparation from node config - once that is set up, we may actually want to run node prep on masters, then remove the service units for both api and controller and replace them with static pods, and use the kubelet in run-once mode as well. I don't see much value in system containers for api or controllers, but there is a lot of value in those as static pods, especially with crio present

openshift-bot · 2017-07-23T18:37:35Z

error: aos-ci-jenkins/OS_3.6_NOT_containerized for 78cb4f8 (logs)

openshift-bot · 2017-07-23T18:39:15Z

error: aos-ci-jenkins/OS_3.6_containerized for 78cb4f8 (logs)

smarterclayton · 2017-07-23T23:18:40Z

aos-ci-test

openshift-bot · 2017-07-23T23:52:16Z

error: aos-ci-jenkins/OS_3.6_NOT_containerized for 8961e7f (logs)

openshift-bot · 2017-07-23T23:53:55Z

error: aos-ci-jenkins/OS_3.6_containerized for 8961e7f (logs)

smarterclayton · 2017-07-24T02:03:31Z

Tsk tsk, jobs not setting etcd hosts

tbielawa

multi-line WHEN statements

tbielawa · 2017-07-24T14:34:14Z

playbooks/common/openshift-cluster/config.yml

+  tasks:
+  - fail:
+      msg: No etcd hosts defined. Running an all-in-one master is deprecated and will no longer be supported in a future upgrade.
+    when: groups.oo_etcd_to_config | default([]) | length == 0 and not openshift_master_unsupported_all_in_one | default(False)


Please put ANDd when statements on multiple lines in a list format, e.g. groups.oo_etcd_to_config | default([]) | length == 0 and not openshift_master_unsupported_all_in_one | default(False) should be more like this:

when: - groups.oo_etcd_to_config | default([]) | length == 0 - not openshift_master_unsupported_all_in_one | default(False)

tbielawa · 2017-07-24T14:34:27Z

roles/openshift_cfme/handlers/main.yml

 - name: restart master api
  systemd: name={{ openshift.common.service_type }}-master-api state=restarted
-  when: (openshift.master.ha is defined and openshift.master.ha | bool) and (not (master_api_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'
+  when: (not (master_api_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'


multi-line WHEN

tbielawa · 2017-07-24T14:34:31Z

roles/openshift_cfme/handlers/main.yml

  notify: Verify API Server

 - name: restart master controllers
  systemd: name={{ openshift.common.service_type }}-master-controllers state=restarted
-  when: (openshift.master.ha is defined and openshift.master.ha | bool) and (not (master_controllers_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'
+  when: (not (master_controllers_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'


multi-line WHEN

tbielawa · 2017-07-24T14:34:39Z

roles/openshift_hosted_metrics/handlers/main.yml

 - name: restart master api
  systemd: name={{ openshift.common.service_type }}-master-api state=restarted
-  when: (openshift.master.ha is defined and openshift.master.ha | bool) and (not (master_api_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'
+  when: (not (master_api_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'


multi-line WHEN

tbielawa · 2017-07-24T14:34:42Z

roles/openshift_hosted_metrics/handlers/main.yml

  notify: Verify API Server

 - name: restart master controllers
  systemd: name={{ openshift.common.service_type }}-master-controllers state=restarted
-  when: (openshift.master.ha is defined and openshift.master.ha | bool) and (not (master_controllers_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'
+  when: (not (master_controllers_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'


multi-line WHEN

tbielawa · 2017-07-24T14:35:41Z

roles/openshift_master/tasks/main.yml

@@ -275,14 +228,14 @@
  delay: 1
  run_once: true
  changed_when: false
-  when: openshift_master_ha | bool and openshift.master.cluster_method == 'native' and master_api_service_status_changed | bool
+  when: openshift.master.cluster_method == 'native' and master_api_service_status_changed | bool


multi-line WHEN

tbielawa · 2017-07-24T14:35:44Z

roles/openshift_master/tasks/main.yml


 - name: Start and enable master controller on first master
  systemd:
    name: "{{ openshift.common.service_type }}-master-controllers"
    enabled: yes
    state: started
-  when: openshift_master_ha | bool and openshift.master.cluster_method == 'native' and inventory_hostname == openshift_master_hosts[0]
+  when: openshift.master.cluster_method == 'native' and inventory_hostname == openshift_master_hosts[0]


multi-line WHEN

tbielawa · 2017-07-24T14:35:49Z

roles/openshift_master/tasks/main.yml


 - name: Start and enable master controller on all masters
  systemd:
    name: "{{ openshift.common.service_type }}-master-controllers"
    enabled: yes
    state: started
-  when: openshift_master_ha | bool and openshift.master.cluster_method == 'native' and inventory_hostname != openshift_master_hosts[0]
+  when: openshift.master.cluster_method == 'native' and inventory_hostname != openshift_master_hosts[0]


multi-line WHEN

tbielawa · 2017-07-24T14:36:05Z

roles/openshift_metrics/handlers/main.yml

  notify: Verify API Server

 - name: restart master controllers
  systemd: name={{ openshift.common.service_type }}-master-controllers state=restarted
-  when: (openshift.master.ha is defined and openshift.master.ha | bool) and (not (master_controllers_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'
+  when: (not (master_controllers_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'


multi-line WHEN

I'd like to do the multi-line when in a follow up, it would make this much harder to review and the re are way more references in these files than just my changes

I can do all the multi-line when in #4947 (comment)

tbielawa · 2017-07-24T14:36:10Z

roles/openshift_metrics/handlers/main.yml

 - name: restart master api
  systemd: name={{ openshift.common.service_type }}-master-api state=restarted
-  when: (openshift.master.ha is defined and openshift.master.ha | bool) and (not (master_api_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'
+  when: (not (master_api_service_status_changed | default(false) | bool)) and openshift.master.cluster_method == 'native'


multi-line WHEN

smarterclayton · 2017-07-25T20:24:33Z

@sdodson @tbielawa where are the job definitions for the failing tests? The inventories need to be updated with an etcd group.

sdodson · 2017-07-27T22:51:11Z

@smarterclayton they're in gerrit, i've updated them so that the master is an etcd host.

smarterclayton · 2017-08-01T01:35:40Z

aos-ci-test

openshift-bot · 2017-08-01T02:11:50Z

error: aos-ci-jenkins/OS_3.6_NOT_containerized for 8738627 (logs)

openshift-bot · 2017-08-01T02:13:34Z

error: aos-ci-jenkins/OS_3.6_containerized for 8738627 (logs)

smarterclayton · 2017-08-01T03:07:48Z

aos-ci-test

openshift-bot · 2017-08-01T03:56:50Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for 4be1576 (logs)

openshift-bot · 2017-08-01T04:00:21Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for 4be1576 (logs)

ingvagabund · 2017-08-01T08:16:48Z

roles/openshift_master_facts/tasks/main.yml

@@ -32,7 +32,7 @@
  openshift_facts:
    role: master
    local_facts:
-      cluster_method: "{{ openshift_master_cluster_method | default(None) }}"
+      cluster_method: "{{ openshift_master_cluster_method | default('native') }}"


Thanks for that, no more openshift_master_cluster_method variable not set after 30 minutes of installation.

ingvagabund · 2017-08-01T08:38:04Z

Changes required to support this in upgrades:

stop and remove the old origin-master.service and sysconfig files if they exist

error / warn if the user had those files that scripts and other automation may need to change.

We should not merge this PR until we have a code that supports both changes. Since if anyone tries to upgrade from an existing (e.g. 3.5 or 3.6) cluster to 3.7 with openshift-ansible-3.7.*, the master service will keep running and may cause disruptions during the upgrade.

Prevents playbooks from accidentally restarting the master service.

smarterclayton · 2017-08-08T16:50:32Z

Updated with the upgrade script (which simply disables and masks origin-master) and also removes all additional places that start origin-master.

The fedora job failure may be because the origin-master service is still started. Rerunning in case it picked up the older change.

smarterclayton · 2017-08-08T16:50:38Z

aos-ci-test

smarterclayton · 2017-08-08T16:53:36Z

[test]

openshift-bot · 2017-08-08T17:44:30Z

error: aos-ci-jenkins/OS_3.6_containerized for c71bed7 (logs)

openshift-bot · 2017-08-08T17:59:10Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for c71bed7 (logs)

sdodson · 2017-08-08T18:15:03Z

registry flake?

sdodson · 2017-08-08T18:15:09Z

aos-ci-test

smarterclayton · 2017-08-08T19:56:49Z

Looks like the tests passed in the one job. [test] again to see if it works this time.

openshift-bot · 2017-08-08T20:07:35Z

Evaluated for openshift ansible test up to c71bed7

openshift-bot · 2017-08-08T20:36:35Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for c71bed7 (logs)

openshift-bot · 2017-08-08T20:42:05Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for c71bed7 (logs)

openshift-bot · 2017-08-08T22:23:40Z

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_openshift_ansible/424/) (Base Commit: 566731d) (PR Branch Commit: c71bed7)

smarterclayton · 2017-08-09T01:25:02Z

This went green on install_update. The fedora error seems to be happening on other PRs. The status_check failure on test was a flake in the job. Seems to be ok from a test perspective.

sdodson · 2017-08-09T02:51:00Z

[merge]

openshift-bot · 2017-08-09T02:55:28Z

Evaluated for openshift ansible merge up to c71bed7

openshift-bot · 2017-08-09T07:47:25Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_openshift_ansible/819/) (Base Commit: 5fe4c8c) (PR Branch Commit: c71bed7)

ingvagabund · 2017-08-09T12:26:57Z

\O/ congratz @smarterclayton

smarterclayton · 2017-08-09T15:47:46Z

Will do follow up for the other changes tomorrow or Friday - specifically the when changes

mffiedler · 2017-08-09T19:29:03Z

This breaks things like the metrics deployer that assume non-ha uses the atomic-openshift-master system unit. Will start filing BZs for other areas to adopt this change.

smarterclayton force-pushed the stop_using_old_mode branch 2 times, most recently from 8096eaa to 78cb4f8 Compare July 23, 2017 01:16

smarterclayton closed this Jul 23, 2017

smarterclayton reopened this Jul 23, 2017

smarterclayton force-pushed the stop_using_old_mode branch from 78cb4f8 to 8961e7f Compare July 23, 2017 19:54

tbielawa previously requested changes Jul 24, 2017

View reviewed changes

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 27, 2017

smarterclayton force-pushed the stop_using_old_mode branch from 8961e7f to 8738627 Compare July 28, 2017 19:33

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2017

smarterclayton force-pushed the stop_using_old_mode branch from 8738627 to 4be1576 Compare August 1, 2017 03:05

smarterclayton mentioned this pull request Aug 1, 2017

polish openshift-master role #4947

Merged

ingvagabund reviewed Aug 1, 2017

View reviewed changes

smarterclayton added 3 commits August 8, 2017 12:50

Use the new election mode (client based) instead of direct etcd access

0d633e9

Disable old openshift-master.service on upgrade

263035e

Remove remaining references to openshift-master.service

c71bed7

Prevents playbooks from accidentally restarting the master service.

smarterclayton force-pushed the stop_using_old_mode branch from 8e28bbc to c71bed7 Compare August 8, 2017 16:50

openshift-bot merged commit 57db372 into openshift:master Aug 9, 2017

This was referenced Aug 9, 2017

Remove additional 'restart master' handler references. #5046

Merged

Unable to restart service origin-master-api #5076

Closed

Nuage changes for Atomic hosts OSE Integration #4991

Merged

mtnbikenc mentioned this pull request Aug 15, 2017

Remove orphan files #5096

Merged

ganhuang mentioned this pull request Nov 22, 2017

Updates for 3.7 install/upgrade openshift/openshift-docs#6434

Merged

ganhuang mentioned this pull request Feb 1, 2018

Added day2 guide openshift/openshift-docs#7258

Merged

Use openshift-master-api and openshift-master-controllers setup always #4832

Use openshift-master-api and openshift-master-controllers setup always #4832

Conversation

smarterclayton commented Jul 22, 2017

smarterclayton commented Jul 23, 2017

smarterclayton commented Jul 23, 2017

smarterclayton commented Jul 23, 2017

openshift-bot commented Jul 23, 2017

openshift-bot commented Jul 23, 2017

smarterclayton commented Jul 23, 2017

openshift-bot commented Jul 23, 2017

openshift-bot commented Jul 23, 2017

smarterclayton commented Jul 24, 2017

tbielawa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Jul 25, 2017

sdodson commented Jul 27, 2017

smarterclayton commented Aug 1, 2017

openshift-bot commented Aug 1, 2017

openshift-bot commented Aug 1, 2017

smarterclayton commented Aug 1, 2017

openshift-bot commented Aug 1, 2017

openshift-bot commented Aug 1, 2017

Choose a reason for hiding this comment

ingvagabund commented Aug 1, 2017

smarterclayton commented Aug 8, 2017

smarterclayton commented Aug 8, 2017

smarterclayton commented Aug 8, 2017

openshift-bot commented Aug 8, 2017

openshift-bot commented Aug 8, 2017

sdodson commented Aug 8, 2017

sdodson commented Aug 8, 2017

smarterclayton commented Aug 8, 2017

openshift-bot commented Aug 8, 2017

openshift-bot commented Aug 8, 2017

openshift-bot commented Aug 8, 2017

openshift-bot commented Aug 8, 2017

smarterclayton commented Aug 9, 2017

sdodson commented Aug 9, 2017

openshift-bot commented Aug 9, 2017

openshift-bot commented Aug 9, 2017 • edited Loading

ingvagabund commented Aug 9, 2017 • edited Loading

smarterclayton commented Aug 9, 2017

mffiedler commented Aug 9, 2017 • edited Loading

openshift-bot commented Aug 9, 2017 •

edited

Loading

ingvagabund commented Aug 9, 2017 •

edited

Loading

mffiedler commented Aug 9, 2017 •

edited

Loading