Elasticsearch fails due to lowered ulimit

[kekoav]

What happened: The issue in #193 is back in the latest 1.12 image.

Elasticsearch is crashing giving the following error:

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

What you expected to happen: Expected Elasticsearch to start.

How to reproduce it (as minimally and precisely as possible):

SSH to instance, and run any container in docker, inspecting the number of files limit:

docker run -it --rm ubuntu bash -c "ulimit -n -H"

It returns 4096 where I expect 65536.

Anything else we need to know?:

PR #206 (reverting previous change) was selected in favor of #205 (modifying explicit limits) but we may need to be explicit about the default limits.

Environment:

• AWS Region: us-east-2
• Instance Type(s): m5.xlarge
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.1
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.12
• AMI Version: v20190327
• Kernel (e.g. uname -a): Linux ip-10-80-5-21.us-east-2.compute.internal 4.14.104-95.84.amzn2.x86_64 Template is missing source_ami_id in the variables section #1 SMP Sat Mar 2 00:40:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /tmp/release on a node):

cat: /tmp/release: No such file or directory

[garethlewin]

Looks like the line OPTIONS="--default-ulimit nofile=1024:4096" is still in /etc/sysconfig/docker

[garethlewin]

amazon-eks-ami/install-worker.sh

Line 66 in 6090f20

sudo sed -i '/OPTIONS/d' /etc/sysconfig/docker

should be removing it, not sure why it isn't.

[garethlewin]

cat: /tmp/release: No such file or directory that file is moved to /etc/eks/release as part of the build, mine says

$ cat /etc/eks/release
BASE_AMI_ID="ami-027c5e2ccf2970def"
BUILD_TIME="Wed Mar 27 23:11:06 UTC 2019"
BUILD_KERNEL="4.14.104-95.84.amzn2.x86_64"
AMI_NAME="amazon-eks-node-1553728183"
ARCH="x86_64"

[antonosmond]

This is frustrating. We're trying to run Artifactory and have the same problem.
I decided to build our own AMI forked from https://github.com/awslabs/amazon-eks-ami and want to explicitly set the ulimits as per #205 but now I'm blocked by #235
😞

[thiago]

We have the same problem! In the old AMI, the file /usr/lib/systemd/system/docker.service does not have the line EnvironmentFile=-/etc/sysconfig/docker. So dockerd does not have limits.

[mcrute]

I'm in the process of building and testing updated AMIs as part of #233 which I anticipate will restore the corrected config from #206. I'm going to de-duplicate this issue into #233, please follow that issue for updates.