Lambda layer with additional software does not work

[SebastianHurubaru]

Describe the bug
Created a custom lambda layer with ImageMagick and ghostscript software, installed with yumda where all the binaries were installed in the opt folder.

When doing though amplify status or amplify push I get the error: An error occurred hashing directory

To Reproduce
Steps to reproduce the behavior:

1. Install ImageMagick and ghostscript with yumda and mounting the /opt to the opt folder of the layer:
   docker run --rm -v /xxx/amplify/backend/function/layer_name/opt:/lambda/opt lambci/yumda:2 yum install -y ghostscript ImageMagick

2. Do amplify status

3. The following error is thrown:
   An error occurred hashing directory /Users/xxx/yyy/zzz/amplify/backend/function/nodejs/opt
   Error: An error occurred hashing directory /Users/xxx/yyy/zzz/amplify/backend/function/nodejs/opt
   at /usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-category-function/lib/provider-utils/awscloudformation/utils/packageLayer.js:139:19
   at async safeHash (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-category-function/lib/provider-utils/awscloudformation/utils/packageLayer.js:138:17)
   at async Promise.all (index 2)
   at async Object.exports.hashLayerVersionContents (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-category-function/lib/provider-utils/awscloudformation/utils/packageLayer.js:130:27)
   at async exports.hashLayerResource (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-category-function/lib/provider-utils/awscloudformation/utils/packageLayer.js:118:17)
   at async isBackendDirModifiedSinceLastPush (/usr/local/lib/node_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/resource-status.js:46:26)
   at async /usr/local/lib/node_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/resource-status.js:165:41
   at async asyncForEach (/usr/local/lib/node_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/resource-status.js:238:9)
   at async /usr/local/lib/node_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/resource-status.js:158:9
   at async asyncForEach (/usr/local/lib/node_modules/@aws-amplify/cli/lib/extensions/amplify-helpers/resource-status.js:238:9

Expected behavior
Amplify should be able to cope with the layer and by pushing the changes it should create the deployment package and upload it.

Smartphone (please complete the following information):

• Device: MacBook Pro
• OS: macOS Big Sur
• Browser Google Chome, Safari

[attilah]

@SebastianHurubaru could you please specify what version of the CLI you are using? Also the amplify related repro steps would help as I don't see how docker run is related to this. As a hunch, you are dunning a docker image locally from the amplify function folder AND issuing the amplify status command and that's when the error happens?

I think that based on the stacktrace is docker is locking some files and when amplify status sees that your files have changed it tries to rebuild your layer but due to locked files it cannot create the hash that we use for comparison hence the failure.

[SebastianHurubaru]

Hi @attilah. I am using amplify cli 4.38.0.

Just add a lambda layer and amplify creates the folder structure for you. As the system I have is based on MacOS, and the lambda function runs on Amazon Linux 2, I am using yumda that is an image based on Amazon Linux 2. With docker I can create a container using the yumda image, and install the software I want in the /opt folder of the created layer.

After installing the software the container gets terminated, but the content in the /opt folder still remains, as the docker command mapped it from my host to the container.

Once this is done, when I try to upload the layer I get the error.

[attilah]

@SebastianHurubaru Thanks we try to reproduce it based on the information you provided.

[attilah]

The root cause of the problem is that native module is installing symlinks in the bin folder and the hashing library did not support symlinks.

[SebastianHurubaru]

@attilah Thanks for the quick reply! Thought that was the reason, but the documentation does not mention anything about this limitation.

So will this be enhanced or what would be the solution in this case?

[attilah]

@sebastiancrossa it was not a known limitation. The updated code will exclude the symbolic links from hashing and packaging can succeed. BTW this install package with docker creates an invalid symbolic link pointing to /opt/bin/true and that is causing the failure, not sure if it is the package's fault or normal docker behavior.

[SebastianHurubaru]

@attilah removed the invalid symlink but the error is still there. Can I see some extra logs, which point me to all files that cannot be hashed?
Also, when would you think that the updated code will be released?

[edwardfoyle]

@SebastianHurubaru even after fixing the issue with the hashing, there's another issue when the CLI tries to copy the function directory to #current-cloud-backend which is a limitation of fs-extra copy: jprichardson/node-fs-extra#638 (comment) They've added this issue to their v 10 milestone, but until that is released there's not much we can do (even attempting to filter the copy based on symLink still fails).

To find all the symlinks in the directory you can run ls -lR | grep '^l' (there are a lot of them).

Unfortunately I can't think of any work around at this point. Removing all of the symlinks will resolve the issue but then the layer probably won't work as expected for you

[SebastianHurubaru]

Hi @edwardfoyle thanks for your reply. Sorry, I meant how to track down the broken links, as @attilah pointed one out and still after deleting it, the error message was there and wondering how did he manage to find it. Deleting all the symlinks results in the software not working properly, so it is not an option.

Guess I will have to wait then until this feature gets available and in the meantime create the layer differently.

Thanks again for your help and looking forward to getting the release with this feature!

[hisham]

side note: after considering the complexity of deploying image magik or sharp via a lambda layer, since they require linux specific builds, I switched to using jimp as our image processing library. Then you don't have to deal with the complexity of building a layer with docker, etc.. (assuming your code is in javascript). Jimp though is slower than image magick and sharp.

[SebastianHurubaru]

Well, there are other way of uploading a layer so went in the end with a simple zip file decoupled from the main project managed over Amplify. As the layer will not change that often, except when changing the runtime, it is a good temporary solution.

[SwaySway]

Changing this to an enhancement. While this does look like a dependency issue, something we can look into is an escape hatch where you can provide a lambda function already zipped with a potential flag that does not build amplify push --no-build.