Replace netlify's zip-it-and-ship-it for serverless deploy #3782
Conversation
Replaced using vercel's nft (node file trace)
| @@ -0,0 +1,63 @@ | |||
| import path from 'path' | |||
There was a problem hiding this comment.
is packages/cli/src/commands/deploy/aws-providers/packing.js a good home? not sure where this file should live.
Or should these functions be rolled into packages/cli/src/commands/deploy/aws-providers/serverless.js?
There was a problem hiding this comment.
My preference: move packing to a separate folder and called in "packaging/nft.js"
There was a problem hiding this comment.
Now in packages/cli/src/commands/deploy/packing/nft.js.
Is that what you intended, or did you want it in packages/cli/src/commands/deploy/aws-providers/packing/nft.js or other?
| @@ -7,25 +7,10 @@ export const preRequisites = [ | |||
| 'Please follow the steps at https://www.serverless.com/framework/docs/providers/aws/guide/installation/ to install Serverless.', | |||
| ], | |||
| }, | |||
| { | |||
| title: 'Checking if @netlify/zip-it-and-ship-it is installed...', | |||
| command: ['yarn', ['zip-it-and-ship-it', '--version']], | |||
There was a problem hiding this comment.
The new deps that have been added are @vercel/nft, archiver and fs-extra. They are not cli tools so can't use --version as a test, is there an easy way to check deps?
There was a problem hiding this comment.
Not sure it is easy, but in the past to verify dependencies like this are there, I have loaded package.json in code and inspected the JSON object to verify that there is the expected key/value under dependencies/devDependencies
| const mapCommandsToListr = ({ title, command, task, errorMessage }) => { | ||
| return { | ||
| title: title, | ||
| task: task |
There was a problem hiding this comment.
In cases where the task is not a cli command a task function can be provided directly.
| await executingCommand | ||
| } catch (error) { | ||
| if (errorMessage) { | ||
| error.message = error.message + '\n' + errorMessage.join(' ') |
There was a problem hiding this comment.
Error message only extended if an errorMessage array is provided.
| const executingCommand = execa(...command, { | ||
| cwd: BASE_DIR, | ||
| }) | ||
| executingCommand.stdout.pipe(process.stdout) |
There was a problem hiding this comment.
I thought this might be a problem in making all of the commands very noise, but does not, only logging out the severless cli stdout, currently this doesn't respect the verbose flag on purpose because I think these details always need to be shown, in particular when the deploy is finished the last thing to be logged out is:
Serverless: Stack update finished...
Service Information
service: app
stage: test
region: us-east-2
stack: app-test
resources: 19
api keys:
None
endpoints:
GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
functions:
exampleFunc: app-test-exampleFunc
graphql: app-test-graphql
layers:
None
Serverless: Removing old service artifacts from S3...
This info always needs to be shown because the urls are generated by aws, without these logs you can't know what they are without digging into aws console.
That's just my opinion, should I make this respect the verbose flag?
for context here's the whole terminal output for my test project.
➜ rw2 git:(master) ✗ yarn rw deploy aws --verbose --stage test
yarn run v1.23.0-20210726.1745
$ /Users/kurthutten/rw2/node_modules/.bin/rw deploy aws --verbose --stage test
[21:06:42] Building and Packaging... [started]
[21:06:42] Building API... [started]
$ /Users/kurthutten/rw2/node_modules/.bin/rw build api
[21:06:46] Generating Prisma Client... [started]
[21:06:49] Generating Prisma Client... [completed]
[21:06:49] Verifying graphql schema... [started]
[21:06:49] Verifying graphql schema... [completed]
[21:06:49] Building API... [started]
[21:06:50] Building API... [completed]
[21:06:50] Building API... [completed]
[21:06:50] Packing Functions... [started]
[21:06:59] Packing Functions... [completed]
[21:06:59] Building and Packaging... [completed]
[21:06:59] Deploying to AWS [started]
[21:06:59] Deploying... [started]
Serverless: Deprecation warning: Detected ".env" files. In the next major release variables from ".env" files will be automatically loaded into the serverless build process. Set "useDotenv: true" to adopt that behavior now.
More Info: https://www.serverless.com/framework/docs/deprecations/#LOAD_VARIABLES_FROM_ENV_FILES
Serverless: DOTENV: Loading environment variables from .env:
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Deprecation warning: Starting with next major version, the provider tags will be applied to Http Api Gateway by default.
Set "provider.httpApi.useProviderTags" to "true" to adapt to the new behavior now.
More Info: https://www.serverless.com/framework/docs/deprecations/#AWS_HTTP_API_USE_PROVIDER_TAGS
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service exampleFunc.zip file to S3 (279.71 KB)...
Serverless: Uploading service graphql.zip file to S3 (29.21 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
...............
Serverless: Stack update finished...
Service Information
service: app
stage: test
region: us-east-2
stack: app-test
resources: 19
api keys:
None
endpoints:
GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
functions:
exampleFunc: app-test-exampleFunc
graphql: app-test-graphql
layers:
None
Serverless: Removing old service artifacts from S3...
[21:09:03] Deploying... [completed]
[21:09:03] Deploying to AWS [completed]
✨ Done in 146.36s.
There was a problem hiding this comment.
My guess is getting this logged out to the console might be why the original author ran it outside of listr, but don't know.
There was a problem hiding this comment.
Hi! A couple of notes:
- in Serverless Framework v3 the
--verboseflag will exist and be supported, not sure if that could be of any use for you here - in v3 we are redesigning the entire CLI output to make it less verbose/more helpful, you can read about that here: https://www.serverless.com/blog/serverless-framework-v3-beta
That v3 is currently in beta (see the blog post above) if you want to give it a try.
|
Hey @dac09, have a look at 9b8192c I think I understood the spirit of what you were after with commiting to
Putting those two together meant I was able to run each stage for the deploy command with something along the lines of Like I said this was my attempt at a tidy solution, open to trying a different direction. |
Irev-Dev
changed the title
|
I removed the draft label as personally happy with this PR now. Happy to make more changes of course. |
|
This looks much cleaner @Irev-Dev than it was, all for it. Will need a bit of time to try it out first, but looking positive! |
|
@mnapoli Are there any "go-to" function bundlers you see being used? And/or is it common practice to need to handle this via a custom implementation as we've done here? Just looking for best practices in case there are any. |
|
@thedavidprice there isn't a built-in default. There are plugins to build with:
By using a plugin like this the whole build is run in |
|
@thedavidprice I thought one perks of "doing it ourselves" is that running It doesn't happen often but every once in a while I'll find a bug that can only be replicated with |
Co-authored-by: Daniel Choudhury <[email protected]>
|
Hey both - so I took this for a spin. Thank you so much for the effort into this - long post coming 💤 ✅ NFT build process working amazingly This is the first time I've used the rw serverless integration - and it seems overdue for an overhaul! I wonder if we should point the PR to a different branch and chip away at making it more complete. The setup commands are meant to get your project deployable, or very close to a deployable state, where you can Here's what I'm thinking:
plugins:
- serverless-plugin-scripts
custom:
scripts:
hooks:
'before:deploy:deploy': yarn rw deploy aws serverless --build-onlyto run the build and package process, when
tags: # Tags for this specific lambda function
endpoint: /.redwood/functions/auth
+params
+ environment:
+ DATABASE_URL: ${param:DATABASE_URL}This will allow us to set the DATABASE_URL in the serverless dashboard and deploy using CI/CD
I tried this plugin with the following config in serverless.yml and it works well. plugins:
- serverless-finch
custom:
client:
bucketName: redwood-sls-finch-deploy
distributionFolder: web/distOnly problems I see here are:
- - httpApi:
- path: /.redwood/functions/auth
- method: GET
- httpApi:
path: /.redwood/functions/auth
method: POSTIncidentally - I don't understand the difference between
Here's what I tried: - http:
path: /.redwood/functions/auth
method: POST
cors:
origin: '*'
headers:
- Content-Type
- X-Amz-Date
- Authorization
- Auth-provider
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-AgentBut unsuccessful... maybe someone more versed in this framework would know? 9. Not doing a prisma migrate during the deploy step |
|
Here's my hacky attempt at getting both sides working: What I have to do is:
End result: http://redwood-sls-finch-deploy.s3-website.us-east-2.amazonaws.com/ |
|
Awesome, thanks @dac09 |
Replaced using vercel's nft (node file trace)
Related to #3686
Labeling WIP as I have a few questions.
@thedavidprice in regards to the following two points:
Deps should only be added on setup. 👍
No it doesn't create a function-manifest.json or nft.json, are these needed (as far as I know they are not needed for SLS but maybe there's context I'm missing)?
I also added a new option to the deploy command, it
--stageand essentially just passes it along to theserverless deploycommand. Imo opinion it's essential otherwise you're force into testing things live. Maybe I shouldn't have snuck it it though, should I remove it and make a seperate PR?I also added a verbose option for @dac09, and will follow up and add this verbose to other deploy commands that are missing it.
Here's the process I used to test the changes:
created a test project
Ran
change
api/src/services/posts/posts.tsposts function to:i.e. remove db access from this call so we can test the function runs when deployed without setting up a live db
Deployed with
yarn rw deploy aws --stage testonce deployed I could test the example function (i.e.
https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc)update redwood.toml apiUrl to the deployed url (should look like
https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions)The test project homepage should have a single blog post with the dumby data we added (note navigating anywhere else in the app will cause errors since it doesn't have db access.)