Integration tests sometimes fail with Endpoint request timed out

[lukehoban]

We have started frequently seeing the containers test fail with the following:

aws_test.go:232: GET https://ys6gq6ipi1.execute-api.us-east-2.amazonaws.com/stage/ [504/application/json]: {"message": "Endpoint request timed out"}

The other endpoints on this HttpAPI don't fail, but this one endpoint does frequently fail.

The one difference between this endpoint and the others is that it tries to make an HTTP request to one of the cloud.Services. It seems that this connection must be timing out (or taking more than 30seconds - causing the HttpEndpoint to time out).

My best guess is that the containers are not ready to start accepting requests yet at the time this HttpEndpoint it invoked. I suspect this is because our change to add wait_for_steady_state to ECS service only handled the Update case, not the Create case. And possibly because we are now creating resource not in program order - the ECS service is getting created closer in time to when the ExtraRuntimeValidation happens, leading to occasional test failures.

[mmdriley]

Should be fixed with pulumi/terraform-provider-aws#22 amended with pulumi/terraform-provider-aws#23. Reopen if we see this again.