Error: Could not load the "sharp" module using the linux-x64 runtime on AWS Lambda

[tpamsler]

Possible bug

Is this a possible bug in a feature of sharp, unrelated to installation?

• Running npm install sharp completes without error.
• Running node -e "require('sharp')" completes without error.

If you cannot confirm both of these, please open an installation issue instead.

Are you using the latest version of sharp?

• I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

If you cannot confirm this, please upgrade to the latest version and try again before opening an issue.

If you are using another package which depends on a version of sharp that is not the latest, please open an issue against that package instead.

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

Cannot run above command as it's executed on AWS Lambda nodejs20.x

What are the steps to reproduce?

Execute lambda on AWS Lambfda

What is the expected behaviour?

Lambda function should load "sharp" image library without arrores

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem

const sharp = require('sharp');

Please provide sample image(s) that help explain this problem

{
"errorType": "Error",
"errorMessage": "Could not load the "sharp" module using the linux-x64 runtime\nPossible solutions:\n- Ensure optional dependencies can be installed:\n npm install --include=optional sharp\n yarn add sharp --ignore-engines\n- Ensure your package manager supports multi-platform installation:\n See https://sharp.pixelplumbing.com/install#cross-platform\n- Add platform-specific dependencies:\n npm install --os=linux --cpu=x64 sharp\n- Consult the installation documentation:\n See https://sharp.pixelplumbing.com/install",
"trace": [
"Error: Could not load the "sharp" module using the linux-x64 runtime",
"Possible solutions:",
"- Ensure optional dependencies can be installed:",
" npm install --include=optional sharp",
" yarn add sharp --ignore-engines",
"- Ensure your package manager supports multi-platform installation:",
" See https://sharp.pixelplumbing.com/install#cross-platform",
"- Add platform-specific dependencies:",
" npm install --os=linux --cpu=x64 sharp",
"- Consult the installation documentation:",
" See https://sharp.pixelplumbing.com/install",
" at Object. (/var/task/node_modules/sharp/lib/sharp.js:114:9)",
" at Module._compile (node:internal/modules/cjs/loader:1376:14)",
" at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)",
" at Module.load (node:internal/modules/cjs/loader:1207:32)",
" at Module._load (node:internal/modules/cjs/loader:1023:12)",
" at Module.require (node:internal/modules/cjs/loader:1235:19)",
" at require (node:internal/modules/helpers:176:18)",
" at Object. (/var/task/node_modules/sharp/lib/constructor.js:10:1)",
" at Module._compile (node:internal/modules/cjs/loader:1376:14)",
" at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)"
]
}

[tpamsler]

Update: Following the execution of npm install --cpu=x64 --os=linux sharp on my development system prior to deploying the code to AWS Lambda using Serverless Framework, everything is now functioning correctly.

[lovell]

https://sharp.pixelplumbing.com/install#aws-lambda
https://sharp.pixelplumbing.com/install#cross-platform

If you still require help, please open a new installation issue and answer all of the questions.

[eudoroolivares2016]

I am seeing the same issue @tpamsler you mentioned that as the solution but, for me running the npm install --cpu=x64 --os=linux sharp does not result in any changes to my package-lock file but, my development machine is macOs and I am using "sharp": "^0.33.2" as well.

[eudoroolivares2016]

My result from npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp is
System: OS: macOS 13.6.4 CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz Memory: 142.20 MB / 16.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 18.18.2 - ~/.nvm/versions/node/v18.18.2/bin/node npm: 9.8.1 - ~/.nvm/versions/node/v18.18.2/bin/npm npmPackages: sharp: ^0.33.2 => 0.33.2
So I believe that something is blocking the install from occurring as expected

[tpamsler]

I just ran npm install --cpu=x64 --os=linux sharp before deploying the lambda to AWS via serverless deploy ... and it had the correct binary in node_mudules during the deploy.

[eudoroolivares2016]

@tpamsler if you run that on your local machine or otherwise if you can see what went into the lambda, do you see what hte binaries are in the node_modules. Going into the node_modules/@img I see sharp-darwin-x64 sharp-libvips-darwin-x64 but, I actually do not see the linux values: node_modules/@img/sharp-libvips-linux-arm and others in my package-lock file as I have run that same npm install targeted for linux

[tpamsler]

Here is what I have in my node_modules/@img ``` sharp-darwin-arm64 sharp-libvips-darwin-arm64 sharp-libvips-linux-x64 sharp-libvips-linuxmusl-x64 sharp-linux-x64 sharp-linuxmusl-x64 ```

…

On Fri, Feb 23, 2024 at 6:59 AM Ed Olivares ***@***.***> wrote: @tpamsler <https://github.com/tpamsler> if you run that on your local machine or otherwise if you can see what went into the lambda, do you see what hte binaries are in the node_modules. Going into the node_modules/ @img <https://github.com/img> I see sharp-darwin-x64 sharp-libvips-darwin-x64 but, I actually see the linux values: ***@***.***/sharp-libvips-linux-arm and others in my package-lock file as I have run that same npm install targeted for linux — Reply to this email directly, view it on GitHub <#4001 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ART4UNTLBJG2BGISTSHLPPTYVCVGVAVCNFSM6AAAAABDRUVOEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRGQ4DKMJZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[eudoroolivares2016]

@tpamsler Would you mind posting your package.json and/or package-lock.json file. I'm interested if there are any diffs between. I can get these linux binaries by doing a npm install --force @img/sharp-linux-x64 etc. But, that is not good practice. It also leads to issues because then it'll try to install those on the dev machine and fail.

[tpamsler]

package-lock.json
package.json

[trippingcats]

I hope this helps somebody: For me installing the packages manually was the only way to make it work on AWS, I am using Sharp v0.33.4, this it is probably not a good practice (as mentioned) but at least it works.

npm install @img/sharp-darwin-arm64 @img/sharp-libvips-darwin-arm64 @img/sharp-libvips-linux-x64 @img/sharp-libvips-linuxmusl-x64 @img/sharp-linux-x64 @img/sharp-linuxmusl-x64 --force

[mattiLeBlanc]

@lovell
Hi, I am having trouble using Sharp on a nodejs Lambda.
It runs fine locally on my Apple Silicon but when I add Sharp to my lambda layer for deployment and do a cdk deploy, it is missing the linux runtime:

Could not load the \"sharp\" module using the linux-x64 runtime\nPossible solutions:\n- Ensure optional dependencies can be installed:\n    npm install --include=optional sharp\n- Ensure your package manager supports multi-platform installation:\n    See https://sharp.pixelplumbing.com/install#cross-platform\n- Add platform-specific dependencies:\n    npm install --os=linux --cpu=x64 sharp\n- Consult the installation documentation:\n    See https://sharp.pixelplumbing.com/install",

If I install the linux runtime, I can't build it locally because my npm will compain about these incompatible runtimes.
So I am adding those runtimes to optionalDepedencies and see if that works.
But what is the best practice for my situation?

OptionalDependencies doesn't work. How do I get the correct dependencies installed in my LambdaLayers for so I can deploy from my mac using cdk deploy? I think when I install the lambda layers via bitbucket pipeline, the machine is linux and it probably fetches the correct dependencies.

[amarsyla]

@mattiLeBlanc I've tried every single workaround mentioned here and anywhere else and nothing worked for me. The only thing that worked was adding sharp as a lambda layer.

[mattiLeBlanc]

@amarsyla yeah, I am using lambda layers. But I have 2 ways of installing:

1. deploy locally via CDK deploy on my Mac M1 machine. This will install the arm64 lib and I can run the lambda index locally and it resizes, works like a charm. But when I deploy, it deploys the wrong binaries to the lambda layer.
2. deploy via bitbucket, which I am trying now and that should run on a linux container, so hopefully that deploys the correct binaries via npm.

However, I would like to deploy locally from my mac to Lambda, with the correct binaries for lambda. Maybe that is not possible atm, hence my question to @lovell

[lovell]

If I install the linux runtime, I can't build it locally because my npm will compain about these incompatible runtimes.

https://sharp.pixelplumbing.com/install#cross-platform
https://sharp.pixelplumbing.com/install#aws-lambda

The summary is that if you're attempting to run multiple/cross-platform then it's best to use a package manager that supports it. Most other approaches will involve you fighting against your choice of package manager.

[amarsyla]

@mattiLeBlanc Instead of building the binaries for the lambda layer I am using prebuilt packages:

https://github.com/pH200/sharp-layer

This is working great regardless of the deployment method.

[mattiLeBlanc]

@lovell I am using npm.
I am not really sure how to deal with cross platform. The article in the docs is very limited, I have to choose between 2 platforms it seems.

I guess what I want is to force the lambda layer package.json (npm) to always install the linux runtime because that is what is deployed to the lambda.
In the root of my mono repo, I have one package.json for all depedencies and there I can use the arm64 version because those are my local packages and the sharp would be a devDepedencies only.

Is that possible?

I just confirmed that my Bitbucket deployed version is working fine because that is all Linux, so I am out of the woods for development, staging and production.
Only for my own test env which I deploy locally, the sharp wouldn't work because npm i will install the arm64 dependencies.

[mattiLeBlanc]

@mattiLeBlanc Instead of building the binaries for the lambda layer I am using prebuilt packages:

https://github.com/pH200/sharp-layer

This is working great regardless of the deployment method.

@amarsyla thank you for this.
I am not sure how to implement this atm (it is late at night here and my brain is cooked).
I am creating my lambda layers atm in my CDK project, installing the deps in the nodejs folder so how would I use this prebuild layer? Just upload the zip to my layers in AWS and reference it?

[amarsyla]

@mattiLeBlanc I am not exactly sure as I am using the serverless framework which makes things easier, but I'm assuming uploading the zip layer and attaching it to the function should do it. I am using release-x64.zip. I am also on a Mac M1 and this has been working great for me. I have also locked the sharp version to 0.33.5 in package.json to make sure I'm using the same version both locally and in the lambda function.

[mattiLeBlanc]

@amarsyla
okay, I will have a look. Thanks for the suggestion.

[mattiLeBlanc]

@amarsyla
Okay, I am using the zip file now. I just modified my CDK deployment slightly to support either installed node_modules or a zip archive. It works as a charm. Thanks for the suggestion,

[renanbronchart]

I found an other solution works weel for me.

1. Install latest nodejs@20 version (Change by the same than lambda node version)

   https://nodejs.org/en/download
   Make sure you have the right path to node in you .zshrc or .bash_profile
export PATH=$HOME/bin:/usr/local/bin:$PATH

2. remove package-lock.json

3. remove node_modules

4. remove dist folder in your lambda function folder

5. remove cache:
   npm cache clean --force

6. Add pre-install in scripts in package.json and replace target by lambda version of node and same architecture than lambda architecture.

"preinstall": "npm install --platform=linux --arch=x64 --target=20x [email protected]"

1. Install and force sharp with linux configuration (like lambda configuration)

npm install --platform=linux --arch=x64 --target=20x [email protected] --force

[remithomas]

I did what @amarsyla suggests by downloading this file https://github.com/pH200/sharp-layer/releases/tag/0.33.5 and I moved the file to an other directory node22 for my case. So it is like this /nodejs/node22/node_modules/

[hongminpark]

I did what @amarsyla suggests by downloading this file https://github.com/pH200/sharp-layer/releases/tag/0.33.5 and I moved the file to an other directory node22 for my case. So it is like this /nodejs/node22/node_modules/

Thanks........... Spent whole day and this was the final workaround

[sher85]

I, too, did what @amarsyla suggested and it worked. My process was:

1. Download the release-x64.zip zip file to my local computer.
2. Create a layer named sharp in AWS set to node 18.x and x86_64 and upload the zip file.
3. Tie that layer into my lambda function.
4. This is what my import looks like in my index.js file and I don't have any libraries in my package.json/package-lock.json:

import sharp from 'sharp'