Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace netlify's zip-it-and-ship-it for serverless deploy #3782

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 19 additions & 20 deletions packages/cli/src/commands/deploy/aws-providers/serverless.js
Original file line number Diff line number Diff line change
@@ -1,34 +1,33 @@
export const preRequisites = [
import ntfPack from '../packing/nft'

export const preRequisites = () => [
{
title: 'Checking if Serverless framework is installed...',
command: ['serverless', ['--version']],
errorMessage: [
'Looks like Serverless is not installed.',
'Please follow the steps at https://www.serverless.com/framework/docs/providers/aws/guide/installation/ to install Serverless.',
],
},
{
title: 'Checking if @netlify/zip-it-and-ship-it is installed...',
command: ['yarn', ['zip-it-and-ship-it', '--version']],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new deps that have been added are @vercel/nft, archiver and fs-extra. They are not cli tools so can't use --version as a test, is there an easy way to check deps?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure it is easy, but in the past to verify dependencies like this are there, I have loaded package.json in code and inspected the JSON object to verify that there is the expected key/value under dependencies/devDependencies

errorMessage: [
'Looks like @netlify/zip-it-and-ship-it is not installed.',
'Either run `yarn rw setup aws-serverless` or add it separately as a dev dependency in the api workspace.',
'Please follow the steps at https://www.serverless.com/framework/docs/getting-started to install Serverless.',
],
},
]

export const buildCommands = [
{ title: 'Building API...', command: ['yarn', ['rw', 'build', 'api']] },
export const buildCommands = () => [
{
title: 'Packaging API...',
command: [
'yarn',
['zip-it-and-ship-it', 'api/dist/functions/', 'api/dist/zipball'],
],
title: 'Building API...',
command: ['yarn', ['rw', 'build', 'api']],
},
{
title: 'Packing Functions...',
task: ntfPack,
},
]

export const deployCommand = {
title: 'Deploying...',
command: ['serverless', ['deploy']],
export const deployCommands = (yargs) => {
const stage = yargs.stage ? ['--stage', yargs.stage] : []
return [
{
title: 'Deploying...',
command: ['serverless', ['deploy', ...stage]],
},
]
}
82 changes: 46 additions & 36 deletions packages/cli/src/commands/deploy/aws.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import path from 'path'

import execa from 'execa'
import Listr from 'listr'
import VerboseRenderer from 'listr-verbose-renderer'
import terminalLink from 'terminal-link'

import { getPaths } from '../../lib'
Expand All @@ -29,6 +30,17 @@ export const builder = (yargs) => {
default: 'api',
type: 'array',
})
.option('verbose', {
describe: 'verbosity of logs',
default: true,
type: 'boolean',
})
.option('stage', {
describe:
'serverless stage pass through param: https://www.serverless.com/blog/stages-and-environments',
default: 'dev',
type: 'string',
})
.epilogue(
`Also see the ${terminalLink(
'Redwood CLI Reference',
Expand All @@ -37,63 +49,61 @@ export const builder = (yargs) => {
)
}

export const handler = async ({ provider }) => {
export const handler = async (yargs) => {
const { provider, verbose } = yargs
const BASE_DIR = getPaths().base
const providerData = await import(`./aws-providers/${provider}`)

const mapCommandsToListr = ({ title, command, task, errorMessage }) => {
return {
title: title,
task: task
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In cases where the task is not a cli command a task function can be provided directly.

? task
: async () => {
try {
const executingCommand = execa(...command, {
cwd: BASE_DIR,
})
executingCommand.stdout.pipe(process.stdout)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought this might be a problem in making all of the commands very noise, but does not, only logging out the severless cli stdout, currently this doesn't respect the verbose flag on purpose because I think these details always need to be shown, in particular when the deploy is finished the last thing to be logged out is:

Serverless: Stack update finished...
Service Information
service: app
stage: test
region: us-east-2
stack: app-test
resources: 19
api keys:
  None
endpoints:
  GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
  POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
  GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
  POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
functions:
  exampleFunc: app-test-exampleFunc
  graphql: app-test-graphql
layers:
  None
Serverless: Removing old service artifacts from S3...

This info always needs to be shown because the urls are generated by aws, without these logs you can't know what they are without digging into aws console.
That's just my opinion, should I make this respect the verbose flag?

for context here's the whole terminal output for my test project.

➜  rw2 git:(master) ✗ yarn rw deploy aws --verbose --stage test
yarn run v1.23.0-20210726.1745
$ /Users/kurthutten/rw2/node_modules/.bin/rw deploy aws --verbose --stage test
[21:06:42] Building and Packaging... [started]
[21:06:42] Building API... [started]
$ /Users/kurthutten/rw2/node_modules/.bin/rw build api
[21:06:46] Generating Prisma Client... [started]
[21:06:49] Generating Prisma Client... [completed]
[21:06:49] Verifying graphql schema... [started]
[21:06:49] Verifying graphql schema... [completed]
[21:06:49] Building API... [started]
[21:06:50] Building API... [completed]
[21:06:50] Building API... [completed]
[21:06:50] Packing Functions... [started]
[21:06:59] Packing Functions... [completed]
[21:06:59] Building and Packaging... [completed]
[21:06:59] Deploying to AWS [started]
[21:06:59] Deploying... [started]
Serverless: Deprecation warning: Detected ".env" files. In the next major release variables from ".env" files will be automatically loaded into the serverless build process. Set "useDotenv: true" to adopt that behavior now.
            More Info: https://www.serverless.com/framework/docs/deprecations/#LOAD_VARIABLES_FROM_ENV_FILES
Serverless: DOTENV: Loading environment variables from .env:
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
            Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
            More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Deprecation warning: Starting with next major version, the provider tags will be applied to Http Api Gateway by default. 
            Set "provider.httpApi.useProviderTags" to "true" to adapt to the new behavior now.
            More Info: https://www.serverless.com/framework/docs/deprecations/#AWS_HTTP_API_USE_PROVIDER_TAGS
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service exampleFunc.zip file to S3 (279.71 KB)...
Serverless: Uploading service graphql.zip file to S3 (29.21 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
...............
Serverless: Stack update finished...
Service Information
service: app
stage: test
region: us-east-2
stack: app-test
resources: 19
api keys:
  None
endpoints:
  GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
  POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/exampleFunc
  GET - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
  POST - https://5mdfm18azg.execute-api.us-east-2.amazonaws.com/.redwood/functions/graphql
functions:
  exampleFunc: app-test-exampleFunc
  graphql: app-test-graphql
layers:
  None
Serverless: Removing old service artifacts from S3...
[21:09:03] Deploying... [completed]
[21:09:03] Deploying to AWS [completed]
✨  Done in 146.36s.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is getting this logged out to the console might be why the original author ran it outside of listr, but don't know.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! A couple of notes:

  • in Serverless Framework v3 the --verbose flag will exist and be supported, not sure if that could be of any use for you here
  • in v3 we are redesigning the entire CLI output to make it less verbose/more helpful, you can read about that here: https://www.serverless.com/blog/serverless-framework-v3-beta

That v3 is currently in beta (see the blog post above) if you want to give it a try.

await executingCommand
} catch (error) {
if (errorMessage) {
error.message = error.message + '\n' + errorMessage.join(' ')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error message only extended if an errorMessage array is provided.

}
throw error
}
},
}
}

const tasks = new Listr(
[
providerData.preRequisites &&
providerData.preRequisites.length > 0 && {
title: 'Checking pre-requisites',
task: () =>
new Listr(
providerData.preRequisites.map((preReq) => {
return {
title: preReq.title,
task: async () => {
try {
await execa(...preReq.command)
} catch (error) {
error.message =
error.message + '\n' + preReq.errorMessage.join(' ')
throw error
}
},
}
})
providerData.preRequisites(yargs).map(mapCommandsToListr)
),
},
{
title: 'Building and Packaging...',
task: () =>
new Listr(
providerData.buildCommands.map((commandDetail) => {
return {
title: commandDetail.title,
task: async () => {
await execa(...commandDetail.command, {
cwd: BASE_DIR,
})
},
}
}),
{ collapse: false }
),
new Listr(providerData.buildCommands(yargs).map(mapCommandsToListr), {
collapse: false,
}),
},
{
title: 'Deploying to AWS',
task: () =>
new Listr(providerData.deployCommands(yargs).map(mapCommandsToListr)),
},
].filter(Boolean),
{ collapse: false }
{ collapse: false, renderer: verbose && VerboseRenderer }
)

try {
await tasks.run()

console.log(c.green(providerData.deployCommand.title))
const deploy = execa(...providerData.deployCommand.command, {
cwd: BASE_DIR,
})
deploy.stdout.pipe(process.stdout)
await deploy
} catch (e) {
console.log(c.error(e.message))
process.exit(1)
Expand Down
60 changes: 60 additions & 0 deletions packages/cli/src/commands/deploy/packing/nft.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import path from 'path'

import { nodeFileTrace } from '@vercel/nft'
import archiver from 'archiver'
import fse from 'fs-extra'

const ZIPBALL_DIR = './api/dist/zipball'

function zipDirectory(source, out) {
const archive = archiver('zip', { zlib: { level: 5 } })
const stream = fse.createWriteStream(out)

return new Promise((resolve, reject) => {
archive
.directory(source, false)
.on('error', (err) => reject(err))
.pipe(stream)

stream.on('close', () => resolve())
archive.finalize()
})
}

async function packageSingleFunction(functionFile) {
const { name: functionName } = path.parse(functionFile)
const { fileList: functionDependencyFileList } = await nodeFileTrace([
functionFile,
])
const copyPromises = []
for (const singleDependencyPath of functionDependencyFileList) {
copyPromises.push(
fse.copy(
'./' + singleDependencyPath,
`${ZIPBALL_DIR}/${functionName}/${singleDependencyPath}`
)
)
}
const functionEntryPromise = fse.outputFile(
`${ZIPBALL_DIR}/${functionName}/${functionName}.js`,
`module.exports = require('${functionFile}')`
)
copyPromises.push(functionEntryPromise)

await Promise.all(copyPromises)
await zipDirectory(
`${ZIPBALL_DIR}/${functionName}`,
`${ZIPBALL_DIR}/${functionName}.zip`
)
await fse.remove(`${ZIPBALL_DIR}/${functionName}`)
return
}

async function ntfPack() {
const filesToBePacked = (await fse.readdir('./api/dist/functions'))
.filter((path) => path.endsWith('.js'))
.map((path) => `./api/dist/functions/${path}`)
return Promise.all(filesToBePacked.map(packageSingleFunction))
}

export default ntfPack
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@ export const preRequisites = [
]

export const apiDevPackages = [
'@netlify/zip-it-and-ship-it',
'@vercel/nft',
'archiver',
'fs-extra',
'serverless-dotenv-plugin',
]

Expand Down