Improve Hive deployment to fail fast on fatal error, and fix deprovision logic #1078
Conversation
Signed-off-by: manosnoam <[email protected]>
Signed-off-by: manosnoam <[email protected]>
Signed-off-by: manosnoam <[email protected]>
Signed-off-by: manosnoam <[email protected]>
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -32,7 +32,16 @@ | |||
| ${clustername_exists} = Does ClusterName Exists | |||
| ${template} = Select Provisioner Template ${provider_type} | |||
| IF ${clustername_exists} | |||
| ... FAIL Cluster name '${cluster_name}' already exists. Please choose a different name. | |||
| Log Cluster name '${cluster_name}' already exists in Hive pool '${pool_name}' - Checking if it has a valid web-console console=True | |||
Check warning
Code scanning / Robocop
Line is too long ({{ line_length }}/{{ allowed_length }}) Warning
| ... FAIL Cluster name '${cluster_name}' already exists. Please choose a different name. | ||
| Log Cluster name '${cluster_name}' already exists in Hive pool '${pool_name}' - Checking if it has a valid web-console console=True | ||
| ${pool_namespace} = Get Cluster Pool Namespace ${pool_name} | ||
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json | jq -r '.status.webConsoleURL' --exit-status shell=yes |
Check warning
Code scanning / Robocop
Line is too long ({{ line_length }}/{{ allowed_length }}) Warning
| Log Cluster name '${cluster_name}' already exists in Hive pool '${pool_name}' - Checking if it has a valid web-console console=True | ||
| ${pool_namespace} = Get Cluster Pool Namespace ${pool_name} | ||
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json | jq -r '.status.webConsoleURL' --exit-status shell=yes | ||
| IF ${result.rc} != 0 |
Check notice
Code scanning / Robocop
'{{ block_name }}' condition can be simplified Note
| ${pool_namespace} = Get Cluster Pool Namespace ${pool_name} | ||
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json | jq -r '.status.webConsoleURL' --exit-status shell=yes | ||
| IF ${result.rc} != 0 | ||
| Log Cluster '${cluster_name}' has previously failed to be provisioned - Cleaning Hive resources console=True |
Check warning
Code scanning / Robocop
Line is too long ({{ line_length }}/{{ allowed_length }}) Warning
| @{new_lines} = Split To Lines ${install_log_data} ${last_line_index} | ||
| FOR ${line} IN @{new_lines} | ||
| Log To Console ${line} | ||
| Watch Hive Install Log |
Check warning
Code scanning / Robocop
Missing documentation in '{{ name }}' keyword Warning
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json | jq -r '.status.webConsoleURL' --exit-status shell=yes | ||
| IF ${result.rc} != 0 | ||
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json shell=yes | ||
| Log Cluster '${cluster_name}' install completed, but it is not accesible - Cleaning Hive resources console=True |
Check warning
Code scanning / Robocop
Line is too long ({{ line_length }}/{{ allowed_length }}) Warning
| FAIL Cluster '${cluster_name}' provisioning failed. Please look into the logs for more details. | ||
| END | ||
| Log Cluster ${cluster_name} created successfully. Web Console: ${result.stdout} console=True | ||
|
|
Check warning
Code scanning / Robocop
Trailing whitespace at the end of line Warning
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o jsonpath\='{ .status.webConsoleURL }' shell=yes | ||
| Log Cluster ${cluster_name} Web Console: ${result.stdout} console=True | ||
| Should Be True ${result.rc} == 0 | ||
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json | jq -r '.status.apiURL' --exit-status shell=yes |
Check warning
Code scanning / Robocop
Line is too long ({{ line_length }}/{{ allowed_length }}) Warning
| Log Cluster ${cluster_name} Web Console: ${result.stdout} console=True | ||
| Should Be True ${result.rc} == 0 | ||
| ${result} = Run Process oc -n ${pool_namespace} get cd ${pool_namespace} -o json | jq -r '.status.apiURL' --exit-status shell=yes | ||
| Should Be True ${result.rc} == 0 Hive Cluster deployment '${pool_namespace}' does not have a valid API access |
Check notice
Code scanning / Robocop
'{{ block_name }}' condition can be simplified Note
| ${result} = Run Process KUBECONFIG\=${cluster_kubeconf} oc login --username\=${username} --password\=${password} ${api} --insecure-skip-tls-verify | ||
| ... shell=yes | ||
| # Test the extracted credentials | ||
| ${result} = Run Process KUBECONFIG\=${cluster_kubeconf} oc login --username\=${username} --password\=${password} ${api} --insecure-skip-tls-verify shell=yes |
Check warning
Code scanning / Robocop
Line is too long ({{ line_length }}/{{ allowed_length }}) Warning
Robot Results
|
|
This has been tested on PSI, GCP, and AWS. Output example of the Provision process exiting right after a fatal error (without waiting for the 50 minutes timeout):
|
manosnoam
changed the title
manosnoam
added
the
enhancements
Fail fast on OCP install Fatal error, instead of waiting for 50 minutes timeout.
This is implemented by watching Hive pods completion status.
Fix deprovision that was running always (in Teardown) if provision failed -
even when provision failed since the cluster name already existed (legitimate failure),
which caused deprovision of existing cluster without intention.
'Wait For Cluster To Be Ready' not only watches for Install to be completed,
but also for Hive cluster deployment to include API endpoint.