Bug: sam local start-api not working in a docker-out-of-docker setup, due to an empty /var/task directory.

[git-noise]

Description:

When running sam local start-api in a docker-out-of-docker setup, it seems that the /var/task folder mounted into the lambda-executing container is not populated with the zip containing the lambda binary.

Steps to reproduce:

1. Create a simple JS hello-world, zip it and drop to a common path - here /tmp/lambda-bins

2. Create a basic setup with an API Gateway and one lambda endpoint:

Resources:
  HelloWorld:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: HelloWorld
      Handler: index.handler
      Runtime: nodejs16.x
      CodeUri: ./lambda-hello-world.zip

  ApiGateway:
    Type: AWS::Serverless::Api
    Properties:
      Name: api-gateway
      StageName: prod
      DefinitionBody:
        swagger: '2.0'
        paths:
          /hello:
            get:
              produces:
                - application/json
              responses:
                200:
                  description: '200 response'
              x-amazon-apigateway-integration:
                httpMethod: GET
                type: aws_proxy
                uri:
                  Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorld.Arn}/invocations

3. Create a simple docker container with the aws-sam-cli.

FROM python:3.10-alpine

RUN apk add --no-cache --virtual build-deps build-base libffi-dev gcc \
    && pip install --no-cache-dir aws-sam-cli==1.116.0 \
    && pip3 uninstall --yes pip \
    && apk del build-deps

4. Create a compose file to spin everything up:

networks:
  sam_subnet:
    driver: bridge
    name: test-sub

services:
  sam-lambda:
    build: .
    working_dir: /opt/data
    ports:
      - 3000:3000
    volumes:
      # Docker
      - /var/run/docker.sock:/var/run/docker.sock
      # SAM template
      - ./template.yaml:/opt/data/template.yaml
      # Directory on the host with the lambda binaries
      - /tmp/lambda-bins:/tmp/lambda-bins
    networks:
      - sam_subnet
    command:
      [
        "sam",
        "local",
        "start-api",
        "--debug",
        "--template",
        "template.yaml",
        "--host",
        "0.0.0.0",
        "--docker-network",
        "test-sub",
        "--docker-volume-basedir",
        "/tmp/lambda-bins",
        "--container-host-interface",
        "0.0.0.0",
        "--container-host",
        "host.docker.internal",
      ]
    extra_hosts:
      - "host.docker.internal:host-gateway"

A few points of interest:

• the working directory for the aws-sam-cli container is /opt/data. It will only contain the sam template.
• the lambda binaries are shared via another mounted folder /tmp/lambda-bins

5. Call the lambda function using curl:

curl -v http://localhost:3000/hello

Observed result:

The lambda function fails to execute with the following:

sam-lambda-1  | 2024-05-06 17:41:09,702 | Found one Lambda function with name 'HelloWorld'
sam-lambda-1  | 2024-05-06 17:41:09,702 | Invoking index.handler (nodejs16.x)
sam-lambda-1  | 2024-05-06 17:41:09,702 | No environment variables found for function 'HelloWorld'
sam-lambda-1  | 2024-05-06 17:41:09,702 | Loading AWS credentials from session with profile 'None'
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolving code path. Cwd=/tmp/lambda-bins, CodeUri=/tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolved absolute path to code is /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolving code path. Cwd=/opt/data, CodeUri=/tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolved real code path to /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,715 | Decompressing /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:12,404 | Local image is up-to-date
sam-lambda-1  | 2024-05-06 17:41:12,409 | Checking free port on 0.0.0.0:5345
sam-lambda-1  | 2024-05-06 17:41:12,411 | Using local image: public.ecr.aws/lambda/nodejs:16-rapid-x86_64.
sam-lambda-1  | 
sam-lambda-1  | 2024-05-06 17:41:12,411 | Mounting /tmp/tmpqn5dh6cm as /var/task:ro,delegated, inside runtime container
sam-lambda-1  | 2024-05-06 17:41:12,833 | Starting a timer for 3 seconds for function 'HelloWorld'
sam-lambda-1  | 2024-05-06 17:41:12,833 | Getting lock for the key host.docker.internal-5345
sam-lambda-1  | 2024-05-06 17:41:12,833 | Waiting to retrieve the lock (host.docker.internal-5345) to start invocation
sam-lambda-1  | START RequestId: 37f0705f-d4e9-4a29-922f-60307d74dc08 Version: $LATEST
sam-lambda-1  | 2024-05-06T17:41:12.941Z	undefined	ERROR	Uncaught Exception 	{"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'index'\nRequire stack:\n- /var/runtime/index.mjs","stack":["Runtime.ImportModuleError: Error: Cannot find module 'index'","Require stack:","- /var/runtime/index.mjs","    at _loadUserApp (file:///var/runtime/index.mjs:1087:17)","    at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)","    at async start (file:///var/runtime/index.mjs:1282:23)","    at async file:///var/runtime/index.mjs:1288:1"]}

As one can see, although the binary seems properly located as per this line:
sam-lambda-1 | 2024-05-06 17:41:11,715 | Decompressing /tmp/lambda-bins/lambda-hello-world.zip

It seems that /var/task ends up being empty in the lambda-executing container.

Expected result:

We would expect the binary to be mounted into the lambda-executing container, and then the lambda function to properly execute.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux Ubuntu 24.04
2. sam --version: 1.116.0
3. AWS region: None

# Paste the output of `sam --info` here
sam-lambda-1  | {
sam-lambda-1  |   "version": "1.116.0",
sam-lambda-1  |   "system": {
sam-lambda-1  |     "python": "3.10.14",
sam-lambda-1  |     "os": "Linux-6.8.0-31-generic-x86_64-with"
sam-lambda-1  |   },
sam-lambda-1  |   "additional_dependencies": {
sam-lambda-1  |     "docker_engine": "26.1.1",
sam-lambda-1  |     "aws_cdk": "Not available",
sam-lambda-1  |     "terraform": "Not available"
sam-lambda-1  |   },
sam-lambda-1  |   "available_beta_feature_env_vars": [
sam-lambda-1  |     "SAM_CLI_BETA_FEATURES",
sam-lambda-1  |     "SAM_CLI_BETA_BUILD_PERFORMANCE",
sam-lambda-1  |     "SAM_CLI_BETA_TERRAFORM_SUPPORT",
sam-lambda-1  |     "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
sam-lambda-1  |   ]
sam-lambda-1  | }

[hawflau]

Hey @git-noise thanks for raising the issue. After some digging, it seems the limitation is that, in the docker-in-docker scenario, the path (/tmp/tmpqn5dh6cm in your example) mounted to /var/task is relative to the host but not to the container. /tmp/tmpqn5dh6cm does not exist in the host but only in the container. That's why /var/task is empty. (reference: https://forums.docker.com/t/mounting-a-volume-not-working-with-running-docker-in-docker/25775/3)

SAM CLI does not support "Docker-in-Docker" right now. I'll mark this issue as a feature request for support for "Docker-in-Docker" and bring that up to the team.

[git-noise]

Hello, thanks a lot for looking into it @hawflau.

Based on your insight, I was able to confirm that it is indeed an issue relating to volume mounting.
A quick test check that consists in mounting the full /tmp in the docker container, ensures that the temporary directory - here /tmp/tmpqn5dh6cm - is available within both the host and the container, and then /var/task properly populated.

Maybe an "easy" fix would be to adjust the function that creates these temporary directory, to take an optional base directory parameter? It would then allow for these temporary directory to be still created on the host, but in a specific directory that could be mounted on the sam-cli container?

Thanks for your help,

[mattparrilla]

@hawflau did this break recently? There are lots of posts and references for how to do docker-in-docker, including issues in this repo discussing how to get it working. I'm very surprised to see that it is not supported.

FWIW, I have the same exact issue (empty /var/task)