autoprovisioning scopes are wrong with service account

[mtricolici]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.20

• provider.google v3.5.0
• provider.google-beta v3.5.0
• provider.kubernetes v1.10.0

Affected Resource(s)

• google_container_cluster

Terraform Configuration Files

resource "google_container_cluster" "test1" {
  provider = google-beta
...
  node_config {
    service_account = "[email protected]"
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
...
  cluster_autoscaling {
    enabled = true

    resource_limits {
      resource_type = "cpu"
      minimum = 1
      maximum = 1
    }

    resource_limits {
      resource_type = "memory"
      minimum = 20
      maximum = 20
    }
    auto_provisioning_defaults {
      service_account = "[email protected]"
   }
  }

Expected Behavior

A cluster like this should be created (this one was created via gcloud commands):

gcloud container node-pools describe default-pool --cluster=created-from-ui --zone us-central1-c
autoscaling: {}
config:
  diskSizeGb: 100
  diskType: pd-standard
  imageType: COS
  machineType: n1-standard-1
  metadata:
    disable-legacy-endpoints: 'true'
  oauthScopes:
  - https://www.googleapis.com/auth/cloud-platform
  serviceAccount: [email protected]

Actual Behavior

oauthScopes defaults are wrong. Via terraform we can't set/adjust them.

gcloud container node-pools describe default-pool --cluster=created-from-terraform --zone us-central1-c
config:
  diskSizeGb: 100
  diskType: pd-standard
  imageType: COS
  machineType: n1-standard-1
  metadata:
    disable-legacy-endpoints: 'true'
  oauthScopes:
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/monitoring
  serviceAccount: [email protected]

Steps to Reproduce

1. terraform apply

• #0000

[edwardmedia]

@mtricolici the oauthScopes was controlled by GCP API. The values are related to serviceAccount. Below is the reference regarding how to use oauthScopes and serviceAccount. I don't know the details about your zuzu@ account. Take a close look at the API reference to see if this make sense to you. Thank you

oauthScopes[] | stringScopes that are used by NAP when creating node pools. If oauthScopes are specified, serviceAccount should be empty.
-- | --
serviceAccount | stringThe Google Cloud Platform Service Account to be used by the node VMs. If serviceAccount is specified, scopes should be empty.

https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters

[vgirnet]

The issue we've experienced with defaults is that we were unable to access any resources from the cluster using service account, and the error was related to:

OAUTH scope https://www.googleapis.com/auth/cloud-platform is not allowed

We are very confused by the fact that creating a cluster manually or using gcloud is automatically adding https://www.googleapis.com/auth/cloud-platform to OAUTH scopes, while creating the same cluster with same configuration via terraform OAUTH scopes are set to https://www.googleapis.com/auth/logging.write and https://www.googleapis.com/auth/monitoring , but not https://www.googleapis.com/auth/cloud-platform.

Why there is this difference? Why other defaults for OAUTH scopes?

[mtricolici]

So, let me please be more clear:

we have a service account zuzu in project1 (it doesn't have any permissions here) ...
this service account zuzu is used as service account for kubernetes cluster in project1.

we host docker images in GCR in project2 and zuzu has read access to this project.

BUT, API (scopes) are disabled. as a result this cluster cannot read docker images from GCR Project 2 (it has permissions, but API is disabled! scopes). error: insufficient oauth scopes

When we create the kubernetes cluster manually (via UI or gcloud command line), these APIs are correct (scope is https://www.googleapis.com/auth/cloud-platform) and kubernetes pulls without any issues the images from project2 GCR.
When we create the kubernetes via terraform, these APIs are disabled (scope is https://www.googleapis.com/auth/logging.write, https://www.googleapis.com/auth/monitoring)

google documentation says: when service account is used - scopes are not used (i.e. ALL APIs should be enabled, we control the access via permissions, roles)
terraform documentatioin says: you can use either service account or scopes (not both)

[mtricolici]

Update, we have a good working cluster when we create it via this command:

gcloud config set project $PROJECT

gcloud container clusters create \
  mycluster \
  --zone "us-central1-c" \
  --cluster-version "1.15.7-gke.23" \
  --service-account "[email protected]" \
  --enable-stackdriver-kubernetes \
  --metadata disable-legacy-endpoints=true \
  --enable-private-nodes \
  --enable-ip-alias \
  --network=mynetwork \
  --subnetwork=mysubnetwork \
  --enable-master-authorized-networks \
  --master-authorized-networks 111.111.111.111/32,222.222.222.222/32 \
  --master-ipv4-cidr "172.16.254.0/28" \
  --enable-autoupgrade \
  --enable-autorepair \
  --enable-autoprovisioning \
  --autoprovisioning-service-account "[email protected]" \
  --autoprovisioning-scopes=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email \
  --min-cpu=1 \
  --min-memory=1 \
  --max-cpu=30 \
  --max-memory=60 \
  --addons=HttpLoadBalancing \
  --maintenance-window-start "2020-01-21T23:00:00Z" \
  --maintenance-window-end "2020-01-22T23:00:00Z" \
  --maintenance-window-recurrence "FREQ=WEEKLY;BYDAY=TU"

Unfortunately we can't set both autoprovisioning service account and scopes via terraform :(

BTW. If we don't specify 'autoprovisioning-scopes' then we have the same issue: new pools created by autoprovisioning have limited scopes.

[mtricolici]

So, terraform should allow to specify BOTH service account and scopes as google command line!

[edwardmedia]

@mtricolici @vgirnet If you don't provide the oauth_scopes in terraform config, gcp api (not terraform provider) will give you defaults which you have got. If you want to have a specific oauth_scopes, you may set it in the terraform config like below, and those two will not be showed up

  node_config {
    service_account = "[email protected]"
    metadata = {
      disable-legacy-endpoints = "true"
    }
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]

  }

Please let me know if this helps address your issue?

[mtricolici]

@edwardmedia that fixes the oauth_scopes for default-pool but not for new pools created automatically via autoprovisioning:

$ gcloud container node-pools describe nap-n1-highmem-4-1ea68q8e --cluster=zuzu --zone us-central1-c
autoscaling:
  autoprovisioned: true
  enabled: true
  maxNodeCount: 1000
config:
....
  metadata:
    disable-legacy-endpoints: 'true'
  oauthScopes:
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/monitoring

[mtricolici]

The problem is we can't use BOTH service_account and oauth_scopes inside auto_provisioning_defaults block.

  cluster_autoscaling {
    enabled = true
...
    auto_provisioning_defaults {
      service_account = "[email protected]"
      oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
    }
}

Error given by terraform:

Error: "cluster_autoscaling.0.auto_provisioning_defaults.0.oauth_scopes": only one of `cluster_autoscaling.0.auto_provisioning_defaults.0.oauth_scopes,cluster_autoscaling.0.auto_provisioning_defaults.0.service_account` can be specified, but `cluster_autoscaling.0.auto_provisioning_defaults.0.oauth_scopes,cluster_autoscaling.0.auto_provisioning_defaults.0.service_account` were specified.

  on k8s-cluster.tf line 1, in resource "google_container_cluster" "test":
   1: resource "google_container_cluster" "test" {

[mtricolici]

gcloud console allows using both arguments:

  --autoprovisioning-service-account "[email protected]" \
  --autoprovisioning-scopes=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email \

[davidquarles]

We are experiencing the exact same issue. gcloud allows you to specify both flags, as does the google_container_node_pool resource, but the auto_provisioning_defaults block does not.

@mtricolici: FWIW I think the only scope that's truly necessary for cross-project GCR usage (in addition to the two being generated) is https://www.googleapis.com/auth/devstorage.read_only. There does not appear to be any permission that can be bound to the service account to compensate for this scope, for some reason (my understanding is probably lacking). I even tried project owner on both sides, to no avail.

[danawillow]

I sent an email over to some GKE folks that have context on this, and I'll update back when I get a response. I think the answer is going to be that we should allow both to be set and that the API docs should be updated to make it clear that it's actually allowed, but I'm going to hold off on any changes until I find out for sure.

[danawillow]

Cool, that is indeed the answer. Assigning over to the current bug onduty for the terraform-side fix.

[mtricolici]

thank you @danawillow @edwardmedia for help.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!