Server requests credentials, but they are not assumed by pod. #127
Comments
coryodaniel
changed the title
|
The reason I started investigating the above was that I had a simple policy for accessing an s3 bucket and it was getting denied permissions. With kube2iam I was able to query the metadata API to verify the permissions:
But with Kiam it seems to return 500 and 308 messages depending on the particular API call. |
coryodaniel
changed the title
|
Actually upon making calls to the metadata API I am seeing error messages: |
|
@coryodaniel This looks like a transport error- I suspect there are warnings that are being dropped by your log level? #94 (comment) shows an example of adding some environment variables to cause the gRPC lib to output more information. My guess is that it'll be a TLS handshaking error (mismatching hosts/alternate names etc.). |
|
I'm experiencing the same issue. I have enabled log debugging and added the gRPC environment variables. I'm seeing no errors in both the server and agent logs, but I'm seeing the following warnings. And like @coryodaniel mentioned, I can confirm the server is getting the proper information Its just not being sent to the agent and pod. I did think it was odd that the agent was not showing any other logs except for the initial logs from gRPC and health checks. Are there any ideas for troubleshooting the |
|
I've not seen those connection errors before- could you describe more about the networking you're running please? |
|
Our K8s cluster is configure via Rancher using EC2 instances. We are running Rancher v1.6, which has a specific CNI configuration. https://rancher.com/docs/rancher/v1.6/en/rancher-services/networking/ When setting the network interface for the agent, I was wondering about what interface to use. Based on what is in the README, I chose the actually network interface of the host(ens5). |
|
@jamiebuxxx Most CNI solutions creates their own bridge interface that the pods are bound to, https://github.com/uswitch/kiam#typical-cni-interface-names ifconfig or |
|
https://rancher.com/docs/rancher/v1.6/en/rancher-services/networking/
i'd give docker0 a shot if you have that in your system |
|
@roffe Yea, I thought and about yesterday and changed the interface to Here's the manifest of that kiam agent pod: I will look on the host and see if there are any other bridge interfaces available. |
|
Just to be transparent, I'm using the following Helm chart for the deployment. https://github.com/helm/charts/tree/master/stable/kiam. I re-created the server and agents certs as mentioned in the docs, just to be sure I wasn't missing something. Everything looks as expected but I'm still receiving the Unrelated but relevant, we are unable to deploy the latest version of the kiam image( But |
|
@jamiebuxxx sorry it's taken me a while to notice this. I'd suggest using either |
|
@coryodaniel @jamiebuxxx were you able to resolve this? I have the exact same issue but I'm using the |
|
@herikwebb I haven't found a solution to this yet and had put this on the backburner. Hopefully @coryodaniel was able to get this working. To me, it seem like the TLS cert could be causing issues. But I haven't been able to prove that yet. |
|
@herikwebb I'd check the server and agent logs. The server will request credentials irrespective of an agent requesting them- they're prefetched when the server identifies a pod is running that's annotated with a role. I'd check your agent logs- that will confirm that your applications attempting to access |
|
I never got it working. I only tried for about a day or so. We had another initiative at the time to manage credentials for other services with Vault, so I ended up having an init container fetch creds from Vault using the assume_role method. It’s def more work, but was more in line with our “security posture” and wrangling AWS service accounts. The one “issue” we ran into with this approach was of a pod didn’t have the init container it would assume the nodes role. We ended up removing all node permissions and using the same approach above to give permissions to auxiliary services like external-dns, etc. |
|
I'm having the exact same issue with v3.0. I did some searching, and it seems like that log warning might be a red herring. This issue grpc/grpc-go#1062 states that it is just log spam. I am continuing to try and get this working if anyone has more insight. |
|
I seem to have made it work by choosing the correct interface name. In the chart this is set by I am using a Kubernetes 1.11 cluster with Flannel that was created using Kops. |
|
I'm having this exact same issue with If I remove the annotation on the |
|
If you don't see any accesses in the agent log I'd guess that the iptables
config hasn't loaded correctly- I'd check whether you're setting the
interface name correctly to match your cluster's chosen CNI.
…On Wed, 27 Mar 2019 at 18:44, Ryan Langford ***@***.***> wrote:
I'm having this exact same issue with v3.0 as well, on Kubernetes v1.11.
I can see the credentials being retrieved by the server from AWS, but
nothing appears to make it back to the container. The agent logs are empty
except for what appears to be healthcheck or liveness/readiness probes, but
I see the credentials in the server logs. My logs basically look like
@jamiebuxxx <https://github.com/jamiebuxxx>'s logs
{"generation.metadata":0,"level":"debug","msg":"added pod","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"1150198","time":"2019-03-27T18:24:01Z"}
{"generation.metadata":0,"level":"debug","msg":"announced pod","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"1150198","time":"2019-03-27T18:24:01Z"}
{"credentials.access.key":"<REDACTED>","credentials.expiration":"2019-03-27T18:36:29Z","credentials.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","generation.metadata":0,"level":"info","msg":"fetched credentials","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"1150198","time":"2019-03-27T18:24:01Z"}
{"generation.metadata":0,"level":"debug","msg":"updated pod","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"1150199","time":"2019-03-27T18:24:01Z"}
{"generation.metadata":0,"level":"debug","msg":"updated pod","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"1150201","time":"2019-03-27T18:24:01Z"}
INFO: 2019/03/27 18:24:01 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
WARNING: 2019/03/27 18:24:01 grpc: Server.Serve failed to complete security handshake from "[::1]:41878": EOF
WARNING: 2019/03/27 18:24:01 grpc: Server.Serve failed to complete security handshake from "127.0.0.1:49706": EOF
WARNING: 2019/03/27 18:24:01 grpc: Server.Serve failed to complete security handshake from "[::1]:41888": read tcp [::1]:443->[::1]:41888: read: connection reset by peer
{"generation.metadata":0,"level":"debug","msg":"updated pod","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"100.108.0.17","pod.status.phase":"Running","resource.version":"1150205","time":"2019-03-27T18:24:02Z"}
{"generation.metadata":0,"level":"debug","msg":"updated pod","pod.iam.role":"arn:aws:iam::<ACCOUNT ID REDACTED>:role/bosun/kiam-demo","pod.name":"kiam-demo-ml7bs","pod.namespace":"default","pod.status.ip":"100.108.0.17","pod.status.phase":"Failed","resource.version":"1150206","time":"2019-03-27T18:24:03Z"}
If I remove the annotation on the default workspace, and run the demo
app, the app will complete, but will be using the nodes role just like
@coryodaniel <https://github.com/coryodaniel> noted as well.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#127 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAEfuHA6H7bdg9aKPrwK3QDyLFbNx0zks5va7v-gaJpZM4VdId->
.
|
Thanks @pingles. The interface is set to |
|
The entry points changed between V2 and V3 to distribute commands as a
single binary.
The manifests in the repo should show an example of what you need to set
them to.
…On Fri, 29 Mar 2019 at 10:08, RaviHari ***@***.***> wrote:
I am seeing this issue when creating a cluster with kiam version 3.0 or 3.2
Error: failed to start container "kiam": Error response from daemon: OCI
runtime create failed: container_linux.go:348: starting container process
caused "exec: "/agent": stat /agent: no such file or directory": unknown
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#127 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAEft4y9cFAZvkJV0yDNZCy2JJKBAWMks5vbeYRgaJpZM4VdId->
.
|
|
I was stuck at same issue. I am using kops 1.12 with kubenet as CNI. Changed |
|
Deployed EKS 1.12 cluster with eksctl and also can not get credentials on pod because of the following errors on agent: {"addr":"10.110.43.140:33840","level":"error","method":"GET","msg":"error processing request: error fetching credentials: rpc error: code = Unknown desc = AccessDenied: Access denied\n\tstatus code: 403, request id: ac7ce2c3-9454-11e9-9bb6-0570c095a29f","path":"/latest/meta-data/iam/security-credentials/kubernetes-buildbot-worker","status":500,"time":"2019-06-21T18:45:00Z"}
{"addr":"10.110.43.140:33840","duration":1001,"headers":{"Content-Type":["text/plain; charset=utf-8"],"X-Content-Type-Options":["nosniff"]},"level":"info","method":"GET","msg":"processed request","path":"/latest/meta-data/iam/security-credentials/kubernetes-buildbot-worker","status":500,"time":"2019-06-21T18:45:00Z"}on the server: {"generation.metadata":0,"level":"debug","msg":"found 1/1 pods for ip 10.110.43.140","pod.iam.role":"kubernetes-buildbot-worker","pod.name":"aws-iam-tester-7b6b5b7976-77ttc","pod.namespace":"buildbot","pod.status.ip":"10.110.43.140","pod.status.phase":"Running","resource.version":"15723","time":"2019-06-21T18:45:00Z"}
{"generation.metadata":0,"level":"debug","msg":"found 1/1 pods for ip 10.110.43.140","pod.iam.role":"kubernetes-buildbot-worker","pod.name":"aws-iam-tester-7b6b5b7976-77ttc","pod.namespace":"buildbot","pod.status.ip":"10.110.43.140","pod.status.phase":"Running","resource.version":"15723","time":"2019-06-21T18:45:00Z"}
{"generation.metadata":0,"level":"debug","msg":"found 1/1 pods for ip 10.110.43.140","pod.iam.role":"kubernetes-buildbot-worker","pod.name":"aws-iam-tester-7b6b5b7976-77ttc","pod.namespace":"buildbot","pod.status.ip":"10.110.43.140","pod.status.phase":"Running","resource.version":"15723","time":"2019-06-21T18:45:00Z"}
{"level":"error","msg":"error requesting credentials: AccessDenied: Access denied\n\tstatus code: 403, request id: ac7ce2c3-9454-11e9-9bb6-0570c095a29f","pod.iam.role":"kubernetes-buildbot-worker","time":"2019-06-21T18:45:00Z"}
{"level":"debug","msg":"evicted credentials future had error: AccessDenied: Access denied\n\tstatus code: 403, request id: ac7ce2c3-9454-11e9-9bb6-0570c095a29f","pod.iam.role":"kubernetes-buildbot-worker","time":"2019-06-21T18:45:00Z"}
{"generation.metadata":0,"level":"error","msg":"error retrieving credentials: AccessDenied: Access denied\n\tstatus code: 403, request id: ac7ce2c3-9454-11e9-9bb6-0570c095a29f","pod.iam.requestedRole":"kubernetes-buildbot-worker","pod.iam.role":"kubernetes-buildbot-worker","pod.name":"aws-iam-tester-7b6b5b7976-77ttc","pod.namespace":"buildbot","pod.status.ip":"10.110.43.140","pod.status.phase":"Running","resource.version":"15723","time":"2019-06-21T18:45:00Z"}During curling security-credentials on a test pod I can see my role: curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
kubernetes-buildbot-worker/But I can not access it: curl http://169.254.169.254/latest/meta-data/iam/security-credentials/kubernetes-buildbot-worker/
error fetching credentials: rpc error: code = Unknown desc = forbidden by policyAt the same time at the kiam agent: {"addr":"10.110.43.140:34390","level":"error","method":"GET","msg":"error processing request: error fetching credentials: rpc error: code = Unknown desc = forbidden by policy","path":"/latest/meta-data/iam/security-credentials/kubernetes-buildbot-worker/","status":500,"time":"2019-06-21T18:49:10Z"}
{"addr":"10.110.43.140:34390","duration":5000,"headers":{"Content-Type":["text/plain; charset=utf-8"],"X-Content-Type-Options":["nosniff"]},"level":"info","method":"GET","msg":"processed request","path":"/latest/meta-data/iam/security-credentials/kubernetes-buildbot-worker/","status":500,"time":"2019-06-21T18:49:10Z"}kiam-server configuration: ---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
namespace: kube-system
name: kiam-server
spec:
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: kiam
role: server
spec:
serviceAccountName: kiam-server
nodeSelector:
kiam: server
volumes:
- name: ssl-certs
hostPath:
# for AWS linux or RHEL distros
path: /etc/pki/ca-trust/extracted/pem/
- name: tls
secret:
secretName: kiam-server-tls
containers:
- name: kiam
image: quay.io/uswitch/kiam:v3.2
imagePullPolicy: Always
env:
- name: GRPC_GO_LOG_SEVERITY_LEVEL
value: "info"
- name: GRPC_GO_LOG_VERBOSITY_LEVEL
value: "8"
command:
- /kiam
args:
- server
- --json-log
- --level=debug
- --bind=0.0.0.0:443
- --cert=/etc/kiam/tls/tls.crt
- --key=/etc/kiam/tls/tls.key
- --ca=/etc/kiam/tls/ca.crt
- --role-base-arn-autodetect
- --assume-role-arn=arn:aws:iam::408272790494:role/kiam-server
- --sync=1m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ssl-certs
- mountPath: /etc/kiam/tls
name: tls
livenessProbe:
exec:
command:
- /kiam
- health
- --cert=/etc/kiam/tls/tls.crt
- --key=/etc/kiam/tls/tls.key
- --ca=/etc/kiam/tls/ca.crt
- --server-address=localhost:443
- --gateway-timeout-creation=1s
- --timeout=5s
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 10
readinessProbe:
exec:
command:
- /kiam
- health
- --cert=/etc/kiam/tls/tls.crt
- --key=/etc/kiam/tls/tls.key
- --ca=/etc/kiam/tls/ca.crt
- --server-address=localhost:443
- --gateway-timeout-creation=1s
- --timeout=5s
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: kiam-server
namespace: kube-system
spec:
clusterIP: None
selector:
app: kiam
role: server
ports:
- name: grpclb
port: 443
targetPort: 443
protocol: TCPkiam-agent configuration: apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
namespace: kube-system
name: kiam-agent
spec:
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: kiam
role: agent
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
kiam: agent
tolerations:
- key: "kiam"
operator: "Equal"
value: "server"
effect: "NoSchedule"
volumes:
- name: ssl-certs
hostPath:
# for AWS linux or RHEL distros
path: /etc/pki/ca-trust/extracted/pem/
- name: tls
secret:
secretName: kiam-agent-tls
- name: xtables
hostPath:
path: /run/xtables.lock
type: FileOrCreate
containers:
- name: kiam
securityContext:
capabilities:
add: ["NET_ADMIN"]
image: quay.io/uswitch/kiam:v3.2
imagePullPolicy: Always
command:
- /kiam
args:
- agent
- --iptables
- --host-interface=!eth0
- --json-log
- --port=8181
- --cert=/etc/kiam/tls/tls.crt
- --key=/etc/kiam/tls/tls.key
- --ca=/etc/kiam/tls/ca.crt
- --server-address=kiam-server:443
- --gateway-timeout-creation=1s
env:
- name: GRPC_GO_LOG_SEVERITY_LEVEL
value: "info"
- name: GRPC_GO_LOG_VERBOSITY_LEVEL
value: "8"
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- mountPath: /etc/ssl/certs
name: ssl-certs
- mountPath: /etc/kiam/tls
name: tls
- mountPath: /var/run/xtables.lock
name: xtables
livenessProbe:
httpGet:
path: /ping
port: 8181
initialDelaySeconds: 3
periodSeconds: 3 |
ghost
mentioned this issue
|
Was facing error This worked for me - cloudposse/docs#195 |
|
I think this issue has covered a lot of disparate things. To avoid confusion I'm going to close and people can re-open or ask questions on Slack elsewhere when they have problems. Thanks! |
I have kiam configured and I can see that the server is requesting new credentials when a pod is launched, but the pod still appears to have the role of the k8s node.
Logs from the kiam server:
{"level":"info","msg":"starting server","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"started prometheus metric listener 0.0.0.0:9620","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"detecting arn prefix","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"using detected prefix: arn:aws:iam::ACCOUNT_HERE:role/","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"will serve on 0.0.0.0:443","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"starting credential manager process 0","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"started cache controller","time":"2018-07-24T16:59:35Z"} {"level":"info","msg":"started namespace cache controller","time":"2018-07-24T16:59:36Z"} {"credentials.access.key":"KEY_HERE","credentials.expiration":"2018-07-24T17:14:38Z","credentials.role":"kiam-test","level":"info","msg":"requested new credentials","time":"2018-07-24T16:59:38Z"} {"credentials.access.key":"KEY_HERE","credentials.expiration":"2018-07-24T17:14:38Z","credentials.role":"kiam-test","generation.metadata":0,"level":"info","msg":"fetched credentials","pod.iam.role":"kiam-test","pod.name":"kiam-verifier-31879","pod.namespace":"playground","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"7868839","time":"2018-07-24T16:59:38Z"}I am launching a pod that has a single container which has the aws cli tools installed.
Then from the pod if I run
aws iam get-useroraws sts get-caller-identityit appears that it is still running with thenodesrole.The namespace I am running in is annotated with:
I dont see any error messages in the agent or server logs.
The text was updated successfully, but these errors were encountered: