upgrade: Verify required network items are set #5256

ashcrow · 2017-08-29T15:48:06Z

When upgrading osm_cluster_network_cidr, osm_host_subnet_length, and
openshift_portal_net must be set to avoid SDN initialization errors.
This was found when the default parameters were changed between Openshift
versions. This meant users who upgraded and did not specify either
mentioned variable at install/upgrade time ended up getting SDN errors
post upgrade.

When osm_cluster_network_cidr, osm_host_subnet_length, and
openshift_portal_net are not set the upgrade will fail telling the user
that the variables must be set and how to find the current values in the
current install.

References: b50b4ea
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1451023

ashcrow · 2017-09-06T18:54:57Z

aos-ci-test

openshift-bot · 2017-09-06T23:18:17Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for 957856c (logs)

openshift-bot · 2017-09-06T23:20:00Z

error: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for 957856c (logs)

openshift-bot · 2017-09-06T23:21:55Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for 957856c (logs)

ashcrow · 2017-09-07T13:35:07Z

@sdodson It looks like everything actually passed by looking at the logs ... 😕

tbielawa

lgtm. Error message is helpful, I love when software tells me more than just 'it went bad, something with a variable...'

tbielawa · 2017-09-07T14:15:43Z

playbooks/common/openshift-cluster/upgrades/pre/verify_inventory_vars.yml

+        osm_cluster_network_cidr and openshift_portal_net are required inventory variables when upgrading.
+        These variables should match what is currently used in the cluster. If you don't remember what
+        these values are you can find them in /etc/origin/master/master-config.yaml on a master with the names
+        clusterNetworkCIDR(osm_cluster_network_cidr)  and hostSubnetLength (openshift_portal_net).


I'll approve it because it fixes the bug. I think it could be improved though for extra help-a-user-out value. Another task or play or something that fetched those values from a master so they didn't have to search them out themselves. I won't block on it, that might be excessive work, it's just something I thought of.

ashcrow · 2017-09-07T14:19:04Z

[merge]

openshift-bot · 2017-09-07T14:24:20Z

[test]ing while waiting on the merge queue

ashcrow · 2017-09-07T16:37:18Z

[INFO] Logging test suite test-upgrade started at Thu Sep  7 11:25:18 EDT 2017
No resources found.
------------------------------------------
creating index with old pattern
error: invalid resource name "pods/logging-es-data-master-wlfhx9va-1-zkshl": [may not contain '/']

Failing on elastic search.

ashcrow · 2017-09-07T16:37:24Z

aos-ci-test

openshift-bot · 2017-09-07T19:10:05Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for 957856c (logs)

openshift-bot · 2017-09-07T19:12:45Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for 957856c (logs)

ashcrow · 2017-09-07T19:21:20Z

Hrm, hitting the same thing again...

error: invalid resource name "pods/logging-es-data-master-m2hsfj46-1-bpzsl": [may not contain '/']
....
NAME                    READY     STATUS              RESTARTS   AGE
logging-fluentd-ztrtn   0/1       ContainerCreating   0          56s
... repeated 2 times
NAME                    READY     STATUS              RESTARTS   AGE
logging-fluentd-ztrtn   0/1       ContainerCreating   0          57s
... repeated 3 times
NAME                    READY     STATUS              RESTARTS   AGE
logging-fluentd-ztrtn   0/1       ContainerCreating   0          58s
... repeated 2 times
NAME                    READY     STATUS              RESTARTS   AGE
logging-fluentd-ztrtn   0/1       ContainerCreating   0          59s

ashcrow · 2017-09-07T19:27:28Z

Rebasing just in case ...

ashcrow · 2017-09-07T19:27:49Z

aos-ci-test

ashcrow · 2017-09-07T21:21:47Z

👍 good failure. Now to figure out how to get CI to add these variables for upgrade testing ...

  1. Hosts:    localhost
     Play:     Verify upgrade can proceed on first master
     Task:     assert
     Message:  osm_cluster_network_cidr and openshift_portal_net are required inventory variables when upgrading. These variables should match what is currently used in the cluster. If you don't remember what these values are you can find them in /etc/origin/master/master-config.yaml on a master with the names clusterNetworkCIDR(osm_cluster_network_cidr)  and hostSubnetLength (openshift_portal_net).

openshift-bot · 2017-09-08T00:21:17Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for ec76d09 (logs)

openshift-bot · 2017-09-08T00:23:57Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for ec76d09 (logs)

When upgrading osm_cluster_network_cidr, osm_host_subnet_length, and openshift_portal_net must be set to avoid SDN initialization errors. This was found when the default parameters were changed between Openshift versions. This meant users who upgraded and did not specify either mentioned variable at install/upgrade time ended up getting SDN errors post upgrade. When osm_cluster_network_cidr, osm_host_subnet_length, and openshift_portal_net are not set the upgrade will fail telling the user that the variables must be set and how to find the current values in the current install. References: openshift@b50b4ea Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1451023

openshift-bot · 2017-09-08T14:24:14Z

Evaluated for openshift ansible merge up to 10ecef4

sdodson · 2017-09-08T17:06:32Z

[test]

openshift-bot · 2017-09-08T17:16:04Z

continuous-integration/openshift-jenkins/merge FAILURE (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_openshift_ansible/1003/) (Base Commit: 756ab33) (PR Branch Commit: 10ecef4)

ashcrow · 2017-09-08T18:17:26Z

     Message:  osm_cluster_network_cidr, osm_host_subnet_length, and openshift_portal_net are required inventory variables when upgrading. These variables should match what is currently used in the cluster. If you don't remember what these values are you can find them in /etc/origin/master/master-config.yaml on a master with the names clusterNetworkCIDR (osm_cluster_network_cidr), osm_host_subnet_length (hostSubnetLength), and openshift_portal_net (hostSubnetLength).

I'm guessing either the change didn't make it to the CI servers yet OR the test kicked off before the change became available.

ashcrow · 2017-09-08T18:17:30Z

aos-ci-test

sdodson · 2017-09-08T19:00:43Z

Test only failed on job ordering, when aos-ci-test comes back green i'll merge.

openshift-bot · 2017-09-08T21:02:38Z

error: aos-ci-jenkins/OS_3.6_containerized for 10ecef4 (logs)

openshift-bot · 2017-09-08T21:14:52Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for 10ecef4 (logs)

ashcrow · 2017-09-11T13:33:56Z

Unrelated to this change:

     Play:     Verify Requirements
     Task:     openshift_health_check
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required Docker images are not available:
                   XXXX/openshift3/ose-docker-registry:v3.6
               Configured registries: XXXXXXXXXXXXXXXXXXXXXXXXXXX,
               Checked by: timeout 10 skopeo inspect --tls-verify=false

  2. Hosts:    10.8.170.225
     Play:     Configure nodes
     Task:     openshift_node_certificates : fail
     Message:  CA certificate /etc/origin/master/ca.crt doesn't exist on CA host 10.8.170.230. Apply 'openshift_ca' role to 10.8.170.230.

ashcrow · 2017-09-11T13:34:10Z

aos-ci-test

openshift-bot · 2017-09-11T14:14:28Z

error: aos-ci-jenkins/OS_3.6_NOT_containerized for 10ecef4 (logs)

openshift-bot · 2017-09-11T14:22:22Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for 10ecef4 (logs)

ashcrow · 2017-09-11T15:19:06Z

Different missing image 😦

  1. Hosts:    10.8.169.203
     Play:     Verify Requirements
     Task:     openshift_health_check
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required Docker images are not available:
                   XXXX/openshift3/ose-haproxy-router:v3.6

sdodson · 2017-09-11T16:07:10Z

lets assume that's a temporary issue, lets see what [test] says

openshift-bot · 2017-09-11T16:08:29Z

Evaluated for openshift ansible test up to 10ecef4

openshift-bot · 2017-09-11T17:56:19Z

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_openshift_ansible/636/) (Base Commit: 69f0384) (PR Branch Commit: 10ecef4)

ashcrow added the kind/bug Categorizes issue or PR as related to a bug. label Aug 29, 2017

ashcrow requested review from sdodson and dgoodwin August 29, 2017 15:48

ashcrow force-pushed the 1451023 branch from 8c21c43 to 957856c Compare September 6, 2017 18:12

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Sep 6, 2017

ashcrow requested a review from tbielawa September 7, 2017 14:06

tbielawa previously approved these changes Sep 7, 2017

View reviewed changes

sdodson previously approved these changes Sep 7, 2017

View reviewed changes

ashcrow dismissed stale reviews from sdodson and tbielawa via ec76d09 September 7, 2017 19:27

ashcrow force-pushed the 1451023 branch from 957856c to ec76d09 Compare September 7, 2017 19:27

ashcrow requested a review from mtnbikenc September 7, 2017 19:28

sdodson mentioned this pull request Sep 8, 2017

Define osm_cluster_network_cidr and openshift_portal_net openshift-eng/aos-cd-jobs#616

Merged

ashcrow added 2 commits September 8, 2017 10:16

inventory: Denote new required upgrade variables

10ecef4

ashcrow force-pushed the 1451023 branch from ec76d09 to 10ecef4 Compare September 8, 2017 14:19

ashcrow changed the title ~~upgrade: Verify osm_cluster_network_cidr and openshift_portal_net are set~~ upgrade: Verify required network items are set Sep 8, 2017

sdodson merged commit f95e3c4 into openshift:master Sep 11, 2017

upgrade: Verify required network items are set #5256

upgrade: Verify required network items are set #5256

Conversation

ashcrow commented Aug 29, 2017 • edited Loading

ashcrow commented Sep 6, 2017

openshift-bot commented Sep 6, 2017

openshift-bot commented Sep 6, 2017

openshift-bot commented Sep 6, 2017

ashcrow commented Sep 7, 2017 • edited Loading

tbielawa left a comment

Choose a reason for hiding this comment

tbielawa Sep 7, 2017

Choose a reason for hiding this comment

ashcrow commented Sep 7, 2017

openshift-bot commented Sep 7, 2017

ashcrow commented Sep 7, 2017

ashcrow commented Sep 7, 2017

openshift-bot commented Sep 7, 2017

openshift-bot commented Sep 7, 2017

ashcrow commented Sep 7, 2017

ashcrow commented Sep 7, 2017

ashcrow commented Sep 7, 2017

ashcrow commented Sep 7, 2017

openshift-bot commented Sep 8, 2017

openshift-bot commented Sep 8, 2017

openshift-bot commented Sep 8, 2017

sdodson commented Sep 8, 2017

openshift-bot commented Sep 8, 2017

ashcrow commented Sep 8, 2017

ashcrow commented Sep 8, 2017

sdodson commented Sep 8, 2017

openshift-bot commented Sep 8, 2017

openshift-bot commented Sep 8, 2017

ashcrow commented Sep 11, 2017 • edited Loading

ashcrow commented Sep 11, 2017

openshift-bot commented Sep 11, 2017

openshift-bot commented Sep 11, 2017

ashcrow commented Sep 11, 2017 • edited Loading

sdodson commented Sep 11, 2017

openshift-bot commented Sep 11, 2017

openshift-bot commented Sep 11, 2017

ashcrow commented Aug 29, 2017 •

edited

Loading

ashcrow commented Sep 7, 2017 •

edited

Loading

ashcrow commented Sep 11, 2017 •

edited

Loading

ashcrow commented Sep 11, 2017 •

edited

Loading