Unknown application error occurred Runtime.Unknown

[TheSupremeBeard]

I am trying to run an SQS Queue using Bref and Laravel. I am currently getting the error "Unknown application error occurred Runtime.Unknown" in the Cloudwatch logs. I'm pretty sure i've followed the instructors correctly in the docs but don't seem to be having a good time.

I have added the following construct to my serverless.yml file

constructs:
  jobs:
    type: queue
    worker:
      handler: Bref\LaravelBridge\Queue\QueueHandler
      runtime: php-81
      timeout: 28 # seconds

and i have the following in my provider: section

environment:
    SQS_QUEUE: ${construct:jobs.queueUrl}

Any help would be nicely appreciated

[mnapoli]

Could you detail which version of Bref, the laravel bridge and PHP you are using?

[TheSupremeBeard]

Thanks for the swift reply, ofc:

PHP 8.1
"bref/bref": "^2.0",
"bref/laravel-bridge": "^2.1",

[mnapoli]

OK that looks good 🤔

Can you post the entire serverless.yml (redacting any sensitive information)?

Also do you have more details on what is logged in CloudWatch? Any extra info?

[TheSupremeBeard]

Sure here is the entire serverless file:

service: connect-api

provider:
  name: aws
  # The AWS region in which to deploy (us-east-1 is the default)
  region: eu-west-2
  # The stage of the application, e.g. dev, production, staging� ('dev' is the default)
  stage: ${env:ENVNAME}
  runtime: provided.al2
  memorySize: 512
  lambdaHashingVersion: 20201221
  environment:
    SQS_QUEUE: ${construct:jobs.queueUrl}

plugins:
  # We need to include the Bref plugin
  - ./vendor/bref/bref
  - serverless-domain-manager
  - serverless-add-api-key
  - serverless-lift

custom:
  customDomain:
    domainName: ${ssm:/api-gateway/domain}
    stage: ${env:ENVNAME}
    basePath: "connect"
    endpointType: "edge"
    security_policy: tls_1_2
    createRoute53Record: false
  apiKeys:
    - name: pmfApiKey
      value: ${ssm:/api-gateway/api-key}

package:
  # Directories to exclude from deployment
  patterns:
    - "!node_modules/**"
    - "!public/storage"
    - "!resources/assets/**"
    - "!storage/**"
    - "!tests/**"

functions:
  # This function runs the Laravel website/API
  web:
    handler: public/index.php
    memorySize: 512
    timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
    layers:
      - ${bref:layer.php-81-fpm}
    events:
      - http:
          path: /
          method: any
          private: true
          cors: true
      - http:
          path: /{proxy+}
          method: any
          private: true
          cors: true
      - schedule:
          rate: rate(5 minutes)
          input:
            warmer: true
    vpc:
      securityGroupIds: ${ssm:/api-gateway/products/security-groups}
      subnetIds: ${ssm:/api-gateway/products/subnets}
  # This function lets us run artisan commands in Lambda
  artisan:
    handler: artisan
    memorySize: 1024
    timeout: 240 # in seconds
    layers:
      - ${bref:layer.php-81} # PHP
      - ${bref:layer.console} # The "console" layer
    vpc:
      securityGroupIds: ${ssm:/api-gateway/products/security-groups}
      subnetIds: ${ssm:/api-gateway/products/subnets}
constructs:
  jobs:
    type: queue
    worker:
      handler: Bref\LaravelBridge\Queue\QueueHandler
      runtime: php-81
      timeout: 28 # seconds

And this is what we see in CloudWatch on the worker:

[mnapoli]

Thanks, I tried your serverless.yml file locally but it seemed to generate something OK.

Could you try running serverless print and pasting the output here as well?

There should be a function with this kind of config:

  jobsWorker:
    handler: Bref\LaravelBridge\Queue\QueueHandler
    runtime: provided.al2
    timeout: 28
    layers:
      - arn:aws:lambda:eu-west-2:534081306603:layer:php-81:44
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - jobsQueueCEDBAE3E
              - Arn
          batchSize: 1
          maximumBatchingWindow: 0
          functionResponseType: ReportBatchItemFailures
    name: connect-api-dev-jobsWorker

[TheSupremeBeard]

Yup here is the output:

service: connect-api
provider:
  name: aws
  region: eu-west-2
  stage: staging
  runtime: provided.al2
  memorySize: 512
  lambdaHashingVersion: '20201221'
  environment:
    SQS_QUEUE:
      Ref: jobsQueueCEDBAE3E
  versionFunctions: true
  deploymentMethod: direct
  iamRoleStatements:
    - Effect: Allow
      Action:
        - sqs:SendMessage
        - sqs:ChangeMessageVisibility
      Resource:
        - Fn::GetAtt:
            - jobsQueueCEDBAE3E
            - Arn
plugins:
  - ./vendor/bref/bref
  - serverless-domain-manager
  - serverless-add-api-key
  - serverless-lift
custom:
  customDomain:
    domainName: REDACTED
    stage: staging
    basePath: connect
    endpointType: edge
    security_policy: tls_1_2
    createRoute53Record: false
  apiKeys:
    - name: REDACTED
      value: REDACTED
package:
  patterns:
    - '!node_modules/**'
    - '!public/storage'
    - '!resources/assets/**'
    - '!storage/**'
    - '!tests/**'
  artifactsS3KeyDirname: serverless/connect-api/staging/code-artifacts
functions:
  web:
    handler: public/index.php
    memorySize: 512
    timeout: 28
    layers:
      - arn:aws:lambda:eu-west-2:534081306603:layer:php-81-fpm:43
    events:
      - http:
          path: /
          method: any
          private: true
          cors: true
      - http:
          path: /{proxy+}
          method: any
          private: true
          cors: true
      - schedule:
          rate:
            - rate(5 minutes)
          input:
            warmer: true
    vpc:
      securityGroupIds:
        - REDACTED
      subnetIds:
        - REDACTED
        - REDACTED
        - REDACTED
    name: connect-api-staging-web
  artisan:
    handler: artisan
    memorySize: 1024
    timeout: 240
    layers:
      - arn:aws:lambda:eu-west-2:534081306603:layer:php-81:44
      - arn:aws:lambda:eu-west-2:534081306603:layer:console:43
    vpc:
      securityGroupIds:
        - REDACTED
      subnetIds:
        - REDACTED
        - REDACTED
        - REDACTED
    events: []
    name: connect-api-staging-artisan
  jobsWorker:
    handler: Bref\LaravelBridge\Queue\QueueHandler
    runtime: provided.al2
    timeout: 28
    layers:
      - arn:aws:lambda:eu-west-2:534081306603:layer:php-81:44
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - jobsQueueCEDBAE3E
              - Arn
          batchSize: 1
          maximumBatchingWindow: 0
          functionResponseType: ReportBatchItemFailures
    name: connect-api-staging-jobsWorker
constructs:
  jobs:
    type: queue
    worker:
      handler: Bref\LaravelBridge\Queue\QueueHandler
      runtime: provided.al2
      timeout: 28
      layers:
        - arn:aws:lambda:eu-west-2:534081306603:layer:php-81:44
      events:
        - sqs:
            arn:
              Fn::GetAtt:
                - jobsQueueCEDBAE3E
                - Arn
            batchSize: 1
            maximumBatchingWindow: 0
            functionResponseType: ReportBatchItemFailures

[ThomasLabstep]

From my experience, the issue is with bref + a SQS consumer (I'm using Symfony messenger, you're using Laravel). So it's really about how bref endpoint is summoned by the SQS trigger.

I'll just share my solution (it's just an idea but it might quickfix that issue if it's blocking you).

Build a NodeJS lambda to process the SQS queue.
And let the NodeJS lambda invoke another lambda.
This lambda can be a simple PHP command with some arguments.

I believe that might work but I'll need a bit of time to refactor in that shape.

[LorenzoRogai]

Also happened to me. A PHP error inside the class caused the problem

[ThomasLabstep]

Could confirm it's working when using Node to consume the SQS queue and then invokeLambda the Symfony command with some data.

Something is definitely broken with the SQS event and bref.

[mnapoli]

@ThomasLabstep if you have a reproducible example that would be great!

[curlysanders]

I am encountering the same issue.
Running the consumer locally by passing an event manually works.

Running:

• PHP: 8.2.7
• bref/bref: 2.1.1
• bref/symfony-bridge: 0.2.2
• bref/symfony-messenger: 0.5.4
• symfony/*: 6.3.3
• serverless framework: 3.34.0

Configs

serverless.yml

frameworkVersion: '3'
service: xxxxxxxxxxxxx
variablesResolutionMode: 20210326

provider:
    lambdaHashingVersion: "20201221"
    name: aws
    region: eu-central-1
    stage: ${opt:stage, 'dev'}
    runtime: provided.al2
    environment:
        # Symfony environment variables
        APP_ENV: prod
        AUTH0_DOMAIN: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_domain}
        AUTH0_AUTH_CLIENT_ID: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_client_id}
        AUTH0_AUTH_CLIENT_SECRET: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_client_secret}
        AUTH0_M2M_CLIENT_ID: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_m2m_client_id}
        AUTH0_M2M_CLIENT_SECRET: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_m2m_client_secret}
        AUTH0_API_AUDIENCE: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_audience}
        AUTH0_CONNECTION: ${ssm:/${self:provider.stage}/shared-infrastructure/auth0_connection}
        DATABASE_URL: ${ssm:/${self:provider.stage}/application/database_url}
        AUTH0_USER_DATABASE_NAME: ${ssm:/${self:provider.stage}/application/auth0_user_database_name}
        BREF_BINARY_RESPONSES: '1'
        SLACK_DSN: ${ssm:/${self:provider.stage}/application/slack_dsn}
        MESSENGER_TRANSPORT_DSN: { Ref: AsyncEventsQueue }
        CORS_ALLOW_ORIGIN: https://${ssm:/${self:provider.stage}/frontend/domain}
    httpApi:
        payload: '2.0'
        cors:
            allowedOrigins:
                - https://${ssm:/${self:provider.stage}/frontend/domain}
            allowedHeaders:
                - Authorization
                - Content-type
            allowedMethods:
                - GET
                - OPTIONS
                - POST
                - PUT
                - PATCH
                - DELETE
            allowCredentials: false
            exposedResponseHeaders: ~
            maxAge: 3600
    apiGateway:
        binaryMediaTypes:
            - '*/*'
    iamRoleStatements:
        -   Effect: Allow
            Action:
                - sqs:*
            Resource:
                Fn::GetAtt: [ AsyncEventsQueue, Arn ]
plugins:
  - ./vendor/bref/bref
  - serverless-ssm-publish
functions:
    web:
        handler: public/index.php
        timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
        runtime: php-82-fpm
        events:
            - httpApi: '*'
        vpc:
            # there is only one AWS account so hardcoded to prod here
            securityGroupIds: ${ssm:/prod/shared-infrastructure/security_group_id}
            subnetIds: ${ssm:/prod/shared-infrastructure/private_subnets}
    console:
        handler: bin/console
        timeout: 120 # in seconds
        layers:
            - ${bref:layer.php-82} # PHP
            - ${bref:layer.console} # The "console" layer
        vpc:
            # there is only one AWS account so hardcoded to prod here
            securityGroupIds: ${ssm:/prod/shared-infrastructure/security_group_id}
            subnetIds: ${ssm:/prod/shared-infrastructure/private_subnets}
    worker:
        handler: bin/consumer.php
        timeout: 20 # in seconds
        reservedConcurrency: 5 # max. 5 messages processed in parallel
        runtime: php-82
        events:
            # Read more at https://www.serverless.com/framework/docs/providers/aws/events/sqs/
            - sqs:
                arn:
                    Fn::GetAtt: [ AsyncEventsQueue, Arn ]
                    # Only 1 item at a time to simplify error handling
                batchSize: 1
package:
    patterns:
        # Excluded files and folders for deployment
        - '!assets/**'
        - '!node_modules/**'
        - '!public/build/**'
        - '!tests/**'
        - '!var/**'
        # If you want to include files and folders that are part of excluded folders,
        # add them at the end
        - 'var/cache/prod/**'
        - 'public/build/entrypoints.json'
        - 'public/build/manifest.json'

custom:
    ssmPublish:
        enabled: true
        params:
            -   path: /${self:provider.stage}/application/uri
                source: HttpApiUrl
                description: Lambda URI
                secure: true

resources:
    Resources:
        AsyncEventsQueue:
            Type: AWS::SQS::Queue
            Properties:
                QueueName: async-events-${opt:stage, 'dev'}.fifo
                FifoQueue: true

services.yaml

    Bref\Symfony\Messenger\Service\Sqs\SqsConsumer:
        public: true
        autowire: true
        arguments:
            # Pass the transport name used in config/packages/messenger.yaml
            $transportName: 'async'
            # true enables partial SQS batch failure
            # Enabling this without proper SQS config will consider all your messages successful
            # See https://bref.sh/docs/function/handlers.html#partial-batch-response for more details.
            $partialBatchFailure: false
            $bus: '@event.bus'
            $logger: '@logger'
            $serializer: '@messenger.transport.symfony_serializer'

messenger.yaml

framework:
    messenger:
        default_bus: command.bus
        buses:
            command.bus:
                ...
            query.bus:
                ...
            event.bus:
                default_middleware: allow_no_handlers
                middleware:
                    - validation
                    - doctrine_transaction

        transports:
            ...
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                serializer: messenger.transport.symfony_serializer
                options:
                   auto_setup: false
            ...
        routing:
            ...
            Application\Event\Async\AsyncEvent: async

[curlysanders]

Ignore my previous post. I have it working now. Didn't include the VPC in the worker lambda 😬

Thx for your excellent work @mnapoli!

[mnapoli]

I finally managed to reproduce and figure it out! Fix coming in #1616

It turns out in the sparse output we get, we actually get the exception message. But Laravel's exception message only contains a class name, so the logs don't make any sense at all.

Anyway, I'm refactoring those logs entirely, we'll have pleeeenty of explanation now!