Adding simple dockerd config file to rotate logs from containers

[max-rocket-internet]

Related issue is here: #36

I myself have seen pods evicted due to disk pressure. Similar configuration is in place on other managed Kubernetes services already.

[micahhausler]

Hey thanks for this PR! We'll have to vet this and get back to you

[max-rocket-internet]

We'll have to vet this and get back to you

Please do, I think it's important and there's already other users who have had issues with disk space. And I've had pods evicted due to disk pressure.

I actually tried to build the AMI myself to test it but packer fails to find the source AMI. Not sure how to fix that.

[rdubya16]

This seems like a must have in a default image. +1 from me

[ledor473]

I've tested the PR but as is it's not working for me, I had to add:
sudo mkdir -p /etc/docker/

[max-rocket-internet]

I've tested the PR but as is it's not working for me, I had to add:

OK done.

[max-rocket-internet]

Why can't I build the AMI myself to test it?

$ make all
packer validate eks-worker-al2.json
Template validated successfully.
packer build -var source_ami_id=ami-0abb414482ac85a71 eks-worker-al2.json
amazon-ebs output will be in this color.

==> amazon-ebs: Prevalidating AMI Name: amazon-eks-node-1540976573
==> amazon-ebs: Error querying AMI: InvalidAMIID.NotFound: The image id '[ami-0abb414482ac85a71]' does not exist
==> amazon-ebs: 	status code: 400, request id: 6a22ead4-b494-42fd-a462-4d1071ac1613
Build 'amazon-ebs' errored: Error querying AMI: InvalidAMIID.NotFound: The image id '[ami-0abb414482ac85a71]' does not exist
	status code: 400, request id: 6a22ead4-b494-42fd-a462-4d1071ac1613

[max-rocket-internet]

Ahh, I had to run export AWS_DEFAULT_REGION=us-west-2 first.

[ledor473]

Yeah I ran into that issue too... to use us-east-1 you need to export AWS_DEFAULT_REGION and change this variable:
https://github.com/awslabs/amazon-eks-ami/blob/master/eks-worker-al2.json#L3

The other gotcha is that you can't update this variable to us-east-1 otherwise it don't work...
https://github.com/awslabs/amazon-eks-ami/blob/master/eks-worker-al2.json#L6

[max-rocket-internet]

@micahhausler if you need more convincing 😉
https://techmovers.salemove.com/infrastructure/2018/11/01/Productionproofing+EKS.html#no-docker-log-rotation

[willejs]

@micahhausler would be great to see this released! 😸

[micahhausler]

Thanks! We'll get this in the next release

[gflarity]

Any eta when the image will be released? We're seeing prod issues that would be resolved by this.

[eswarbala]

We are planning a release on Dec 5th with this patch.

[fcervantes2]

@eswarbala is there a new date for the release which includes this patch? I was hoping to use official AMIs with this issue resolved.

[luispabon]

Has this been released yet?

[max-rocket-internet]

Yes, it's in amazon-eks-node-1.10-v20181210 but still no release on this repo 🤔

[jpb]

Ah, it looks like the version/naming convention has changed. I was waiting for an AMI called amazon-eks-node-v26 (to match version 23 to 25, versions 18 to 22 were named slightly differently). It looks like amazon-eks-node-1.10-v20181210 is the one that is referenced in the documentation.

(Unrelated to this change – is it possible to have something like Subscribing to Amazon ECS-Optimized Amazon Linux AMI Update Notifications but for the EKS AMIs?)

[yrsurya]

Still we are running manually docker system prune -a -f to remove unused images and killed containers on each worker node to avoid eviction on pods , can you please confirm if this works and how often it will remove images and containers or do we need to add any flag while creating/attaching nodes to EKS.

we are using latest version of AMI
let me know if need more details

[max-rocket-internet]

@yrsurya this is already done by kubelet: https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/