OpenStack terraform provider tests fail with a Cinder volume attach resources #571

kayrus · 2020-08-11T09:49:28Z

Thank you for reporting bug!

Describe the bug

Both Cinder V2 and V3 resources tests fail due to timeout:

TestAccBlockStorageVolumeAttachV2_basic
TestAccBlockStorageVolumeAttachV2_timeout
TestAccBlockStorageVolumeAttachV3_basic
TestAccBlockStorageVolumeAttachV3_timeout

Related Job information

https://github.com/terraform-providers/terraform-provider-openstack/pull/1033
https://github.com/terraform-providers/terraform-provider-openstack/pull/1031

wangxiyuan · 2020-08-12T01:57:53Z

The job timeout has been update to 6h, please rechek again.

kayrus · 2020-08-12T07:20:40Z

@wangxiyuan I don't think that 6h will help. Previously disk attach worked just fine, but now it timeouts.

wangxiyuan · 2020-08-12T07:44:17Z

The inner test problem should be debug and fixed at terraform-provider-openstack side first. OpenLab doesn't know how it works. If it's the Infra's problem, just let us know then. Do you know the deep reason for the error test?

kayrus · 2020-08-12T07:46:58Z

@wangxiyuan I'll try to increase the attach waiting timeout and get back to you.

kayrus · 2020-08-12T13:04:45Z

I see the problem. It looks like terraform cannot detect server ID of the volume attachment and produces an invalid link:

2020-08-12 09:38:33.855250 | ubuntu-bionic | 2020/08/12 09:38:33 [DEBUG] OpenStack Request URL: DELETE http://192.168.0.24/compute/v2.1/servers//os-volume_attachments/1b9c5ce8-6aa1-4083-9656-701f26cf58de
2020-08-12 09:38:33.855403 | ubuntu-bionic | 2020/08/12 09:38:33 [DEBUG] OpenStack Request Headers:
2020-08-12 09:38:33.855483 | ubuntu-bionic | Accept: application/json
2020-08-12 09:38:33.855561 | ubuntu-bionic | Cache-Control: no-cache
2020-08-12 09:38:33.855864 | ubuntu-bionic | User-Agent: HashiCorp Terraform/0.12.7-sdk (+https://www.terraform.io) Terraform Plugin SDK/1.14.0 gophercloud/2.0.0
2020-08-12 09:38:33.855909 | ubuntu-bionic | X-Auth-Token: ***
2020-08-12 09:38:33.904970 | ubuntu-bionic | 2020/08/12 09:38:33 [DEBUG] OpenStack Response Code: 404
2020-08-12 09:38:33.905206 | ubuntu-bionic | 2020/08/12 09:38:33 [DEBUG] OpenStack Response Headers:
2020-08-12 09:38:33.905312 | ubuntu-bionic | Content-Length: 112
2020-08-12 09:38:33.905413 | ubuntu-bionic | Content-Type: application/json
2020-08-12 09:38:33.905521 | ubuntu-bionic | Date: Wed, 12 Aug 2020 09:38:33 GMT
2020-08-12 09:38:33.905653 | ubuntu-bionic | Server: Apache/2.4.29 (Ubuntu)
2020-08-12 09:38:33.905826 | ubuntu-bionic | X-Compute-Request-Id: req-00355c4f-0137-493f-bd44-dd9165202bd6
2020-08-12 09:38:33.906000 | ubuntu-bionic | X-Openstack-Request-Id: req-00355c4f-0137-493f-bd44-dd9165202bd6
2020-08-12 09:38:33.906150 | ubuntu-bionic | 2020/08/12 09:38:33 [DEBUG] OpenStack Response Body: {
2020-08-12 09:38:33.906236 | ubuntu-bionic |   "code": "404 Not Found",
2020-08-12 09:38:33.906502 | ubuntu-bionic |   "message": "The resource could not be found.\u003cbr /\u003e\u003cbr /\u003e\n\n\n",
2020-08-12 09:38:33.906600 | ubuntu-bionic |   "title": "Not Found"
2020-08-12 09:38:33.906629 | ubuntu-bionic | }

See a null server_id:

2020-08-12 09:38:33.848397 | ubuntu-bionic | 2020/08/12 09:38:33 [DEBUG] OpenStack Response Body: {
2020-08-12 09:38:33.848452 | ubuntu-bionic |   "volume": {
2020-08-12 09:38:33.848539 | ubuntu-bionic |     "attachments": [
2020-08-12 09:38:33.848582 | ubuntu-bionic |       {
2020-08-12 09:38:33.848728 | ubuntu-bionic |         "attached_at": "2020-08-12T09:38:23.000000",
2020-08-12 09:38:33.848900 | ubuntu-bionic |         "attachment_id": "a2dfd6c5-db5c-41c0-b564-eb6faf120fa1",
2020-08-12 09:38:33.848983 | ubuntu-bionic |         "device": "auto",
2020-08-12 09:38:33.849069 | ubuntu-bionic |         "host_name": null,
2020-08-12 09:38:33.849237 | ubuntu-bionic |         "id": "1b9c5ce8-6aa1-4083-9656-701f26cf58de",
2020-08-12 09:38:33.849324 | ubuntu-bionic |         "server_id": null,
2020-08-12 09:38:33.849486 | ubuntu-bionic |         "volume_id": "1b9c5ce8-6aa1-4083-9656-701f26cf58de"
2020-08-12 09:38:33.849528 | ubuntu-bionic |       }
2020-08-12 09:38:33.849605 | ubuntu-bionic |     ],

ozerovandrei · 2020-09-17T08:18:15Z

There is what happening in case of TestAccBlockStorageVolumeAttachV2_timeout:

We're setting host_name = "devstack" as an argument to openstack_blockstorage_volume_attach_v2: resource_openstack_blockstorage_volume_attach_v2_test.go#L156

It can be seen in the request body:

2020-09-17 07:26:47.438263 | ubuntu-bionic | 2020/09/17 07:26:47 [DEBUG] OpenStack Request Body: {
2020-09-17 07:26:47.438289 | ubuntu-bionic |   "os-attach": {
2020-09-17 07:26:47.438320 | ubuntu-bionic |     "host_name": "devstack",
2020-09-17 07:26:47.438350 | ubuntu-bionic |     "mountpoint": "auto"
2020-09-17 07:26:47.438366 | ubuntu-bionic |   }
2020-09-17 07:26:47.438381 | ubuntu-bionic | }

We're getting an emtpy host_name from the attachment body response inside the volume:

2020-09-17 07:26:57.812679 | ubuntu-bionic |     "attachments": [
2020-09-17 07:26:57.812699 | ubuntu-bionic |       {
2020-09-17 07:26:57.812773 | ubuntu-bionic |         "attached_at": "2020-09-17T07:26:48.000000",
2020-09-17 07:26:57.812898 | ubuntu-bionic |         "attachment_id": "1bf4765b-6dbf-4d4b-a296-e6d444a70a16",
2020-09-17 07:26:57.812939 | ubuntu-bionic |         "device": "auto",
2020-09-17 07:26:57.812989 | ubuntu-bionic |         "host_name": null,
2020-09-17 07:26:57.813062 | ubuntu-bionic |         "id": "873d4f20-a728-454a-9aaa-abe6a8c25143",
2020-09-17 07:26:57.813099 | ubuntu-bionic |         "server_id": null,
2020-09-17 07:26:57.813168 | ubuntu-bionic |         "volume_id": "873d4f20-a728-454a-9aaa-abe6a8c25143"
2020-09-17 07:26:57.813194 | ubuntu-bionic |       }
2020-09-17 07:26:57.813242 | ubuntu-bionic |     ],

And we can't find the needed attachment because of the empty host_name since we do the following in resource_openstack_blockstorage_volume_attach_v2.go#L256:

	// Search for the attachmentId
	var attachmentId string
	hostName := d.Get("host_name").(string)
	for _, attachment := range volume.Attachments {
		if hostName != "" && hostName == attachment.HostName {
			attachmentId = attachment.AttachmentID
		}
	}

	if attachmentId == "" {
		return fmt.Errorf(
			"Unable to determine attachment ID for openstack_blockstorage_volume_attach_v2 volume %s.", volumeId)
	}

And test is failing with the error from the code above:

2020-09-17 07:26:57.815386 | ubuntu-bionic | 2020/09/17 07:26:57 [DEBUG] openstack_blockstorage_volume_attach_v2.va_1: apply errored, but we're indicating that via the Error pointer rather than returning it: Unable to determine attachment ID for openstack_blockstorage_volume_attach_v2 volume 873d4f20-a728-454a-9aaa-abe6a8c25143.
2020-09-17 07:26:57.815559 | ubuntu-bionic | 2020/09/17 07:26:57 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: Unable to determine attachment ID for openstack_blockstorage_volume_attach_v2 volume 873d4f20-a728-454a-9aaa-abe6a8c25143.
2020-09-17 07:26:57.815728 | ubuntu-bionic | 2020/09/17 07:26:57 [ERROR] <root>: eval: *terraform.EvalSequence, err: Unable to determine attachment ID for openstack_blockstorage_volume_attach_v2 volume 873d4f20-a728-454a-9aaa-abe6a8c25143.

All logs are from 68499c1e3dd34a52aeb686467e4c713e?logfile=console.log that was triggered by terraform-provider-openstack/pull/1067

@wangxiyuan could you please check why the host_name of the new attachment is empty despite the fact that we set it explicitly?

Edit: same thing is happening in this build with the TestAccBlockStorageVolumeAttachV2_basic , TestAccBlockStorageVolumeAttachV3_basic , TestAccBlockStorageVolumeAttachV3_timeout tests.

ozerovandrei · 2020-09-17T08:35:24Z

Follow-up to my last comment.

I guess this is happening because of those patches:

https://review.opendev.org/#/c/726751/
openstack/cinder@8bf454b

ozerovandrei · 2020-09-17T08:40:33Z

I guess we can add testAccPreCheckAdminOnly condition to attachment tests.
And then they will be run only after we will decide what to do in #365.

wangxiyuan · 2020-09-17T09:33:30Z

@ozerovandrei I'm away from OpenStack Upstream almost 2 years. Sorry that I don't have enough time to debug the error. (Since the problem is not related to OpenLab infrastructure).

But How do you think if you can login to the VM to debug by yourself? I can add a temp admin user and pause the test in the job defination code, then you guys from terraform can login and do what ever you want.

ozerovandrei · 2020-09-18T06:45:21Z

@wangxiyuan thanks but in this particular case it's not needed since host information should be hidden from the regular user: https://review.opendev.org/#/c/726751/.

I guess we can close this issue.

We will enable those tests for admin only and update documentation for those resources in terraform-provider-openstack/terraform-provider-openstack#1068.

wangxiyuan · 2020-09-18T07:09:15Z

cool. Feel free to close the issue if the problem is solved.

kayrus added the bug Something isn't working label Aug 11, 2020

kayrus assigned bzhaoopenstack, liusheng and wangxiyuan Aug 11, 2020

kayrus mentioned this issue Aug 15, 2020

feat: add floating_ip_enabled in containerinfra terraform-provider-openstack/terraform-provider-openstack#1043

Merged

ozerovandrei mentioned this issue Aug 28, 2020

Update Terraform SDK to v1.15.0, use Go 1.14.7 terraform-provider-openstack/terraform-provider-openstack#1051

Merged

ozerovandrei mentioned this issue Sep 17, 2020

Run BlockStorage attach tests as admin only terraform-provider-openstack/terraform-provider-openstack#1068

Merged

kayrus closed this as completed Sep 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenStack terraform provider tests fail with a Cinder volume attach resources #571

OpenStack terraform provider tests fail with a Cinder volume attach resources #571

kayrus commented Aug 11, 2020

wangxiyuan commented Aug 12, 2020

kayrus commented Aug 12, 2020

wangxiyuan commented Aug 12, 2020 •

edited

Loading

kayrus commented Aug 12, 2020

kayrus commented Aug 12, 2020 •

edited

Loading

ozerovandrei commented Sep 17, 2020 •

edited

Loading

ozerovandrei commented Sep 17, 2020

ozerovandrei commented Sep 17, 2020

wangxiyuan commented Sep 17, 2020 •

edited

Loading

ozerovandrei commented Sep 18, 2020

wangxiyuan commented Sep 18, 2020

OpenStack terraform provider tests fail with a Cinder volume attach resources #571

OpenStack terraform provider tests fail with a Cinder volume attach resources #571

Comments

kayrus commented Aug 11, 2020

Describe the bug

Related Job information

wangxiyuan commented Aug 12, 2020

kayrus commented Aug 12, 2020

wangxiyuan commented Aug 12, 2020 • edited Loading

kayrus commented Aug 12, 2020

kayrus commented Aug 12, 2020 • edited Loading

ozerovandrei commented Sep 17, 2020 • edited Loading

ozerovandrei commented Sep 17, 2020

ozerovandrei commented Sep 17, 2020

wangxiyuan commented Sep 17, 2020 • edited Loading

ozerovandrei commented Sep 18, 2020

wangxiyuan commented Sep 18, 2020

wangxiyuan commented Aug 12, 2020 •

edited

Loading

kayrus commented Aug 12, 2020 •

edited

Loading

ozerovandrei commented Sep 17, 2020 •

edited

Loading

wangxiyuan commented Sep 17, 2020 •

edited

Loading