Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Private SQLInstance and private network cannot be created in the same set of configs. #148

Open
AlexBulankou opened this issue Apr 18, 2020 · 12 comments
Labels
bug Something isn't working

Comments

@AlexBulankou
Copy link
Contributor

Describe the bug
Private SQLInstance and private network cannot be created in the same set of configs.

ConfigConnector Version
1.7.0

To Reproduce
Create private networking and sql instance that depends on this network in the same set of yamls.
Expected: dependencies are resolved and network is created, then instance is created.
Actual: sql instance is stuck in the failed state and cannot be updated.
Workaround: create network, wait for it to be successfully created, then create sql instance.

YAML snippets:

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeNetwork
metadata:
  name: mysql-private-network
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: mysql-private-instance
spec:
  databaseVersion: MYSQL_5_7
  region: us-central1
  settings:
    tier: db-f1-micro
    ipConfiguration:
      ipv4Enabled: false
      privateNetworkRef:
        name: mysql-private-network
@AlexBulankou AlexBulankou added the bug Something isn't working label Apr 18, 2020
@jcanseco
Copy link
Member

Hi @AlexBulankou, are you able to consistently reproduce this issue? I applied your YAML verbatim just now, and both my ComputeNetwork and SQLInstance were created successfully.

@tonybenchsci
Copy link

A related issue we face (also seen in some Terraform forums) is that we cannot create two private CloudSQL instances if it's your first time. This is because after you set up private service networking, the first CloudSQL instance opens up a new peering connection, so any additional instances will "fail to create" if it's done simultaneously.

@caieo
Copy link
Contributor

caieo commented Apr 24, 2020

@tonybenchsci, thanks for bringing this up -- we'll look into this. In the meantime, I will open a separate GitHub issue to track this scenario specifically. Would you happen to have YAML snippets we can use to reproduce your issue? It would be helpful to include in the new GitHub issue.

@AlexBulankou
Copy link
Contributor Author

@jcanseco, if you check the status of SQLInstance, what is it? Mine shows FAILED.

$ gcloud sql instances list


NAME                    DATABASE_VERSION  LOCATION       TIER         PRIMARY_ADDRESS  PRIVATE_ADDRESS  STATUS
mysql-private-instance  MYSQL_5_7         us-central1-a  db-f1-micro  -                -                FAILED

@paulczar
Copy link

paulczar commented Jun 6, 2020

I also see this error when creating the private IP and SQL in the same run:

Failed to create subnetwork. Please create Service Networking connection with service 'servicenetworking.googleapis.com' from consumer project '87697109829' network 'demo-db' again.

the sqlinstance resource in kubernetes shows as ready:

k -n cloudsql get sqlinstances.sql.cnrm.cloud.google.com demo-db3 -o json | jq .status
{
  "conditions": [
    {
      "lastTransitionTime": "2020-06-06T17:34:29Z",
      "message": "The resource is up to date",
      "reason": "UpToDate",
      "status": "True",
      "type": "Ready"
    }
  ],
  "connectionName": "XXX:us-central1:demo-db3",
  "selfLink": "https://sqladmin.googleapis.com/sql/v1beta4/projects/XXX/instances/demo-db3"
}

even though gcloud shows it as failed:

$ gcloud sql instances list
NAME      DATABASE_VERSION  LOCATION       TIER              PRIMARY_ADDRESS  PRIVATE_ADDRESS  STATUS
demo-db3  POSTGRES_9_6      us-central1-a  db-custom-1-3840  -                -                FAILED

@amnk
Copy link

amnk commented Aug 17, 2020

Yep, I have the same in 1.8.0: ready status in Kubernetes but failed - in gcloud. Recreating same instance but with different name several minutes later works fine.

@jcanseco
Copy link
Member

Hi all, thanks, we're looking into this issue. The gist is that if a Cloud SQL instance specifies a private network, an implicit dependency seems to be established between the Cloud SQL instance and the network's Service Networking Connection resource. This implicit dependency imposes a requirement where the Service Networking Connection must be created first before the Cloud SQL instance. This is quite a weird behavior.

No real updates yet, but we're looking into it, and we'll keep you updated.

@caieo
Copy link
Contributor

caieo commented Sep 23, 2020

Update in response to @tonybenchsci's earlier comment -- Cloud SQL is aware of this issue and we do not have an expected timeline for when it might be resolved. Our suggested workaround is, when creating a private SQLInstance on a network for the first time, wait for the creation of one instance first (make sure it's state is RUNNABLE), before attempting to create more than one instance simultaneously.

@jcanseco
Copy link
Member

Hi all, apologies for the late update. The original issue where you couldn't create a private SQLInstance and the referenced ComputeNetwork at the same time has been fixed: the SQLInstance won't erroneously enter an UpToDate state anymore only to enter an UpdateFailed state later on. Instead, the SQLInstance will enter an UpdateFailed state immediately until the ServiceNetworkingConnection (an implicit dependency) is ready.

This will ensure that the SQLInstance is created successfully eventually without letting the instance enter a FAILED state on GCP (which was problematic since this required having to create a whole new instance with a different name).

This fix has been released as part of KCC v1.30.0.

As for the related issue that @tonybenchsci raised up above, however, I regret to say that a proper fix from the Cloud SQL team is still pending. For now, we maintain our previous recommendation.

@rkgcp185151
Copy link

We are trying to use config connector for creating database instances. We need the support for privateIP. What can we do to help prioritize this request? Thank you!

@rkgcp185151
Copy link

rkgcp185151 commented May 4, 2021

Question: does config connector support creating cloudsql instances with private IP only? We do not want public IP.

Thank you!

#99
#55

@jcanseco
Copy link
Member

jcanseco commented May 4, 2021

Hi @rk185151. Please see my comment here. I believe that is what you're looking for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

7 participants