Node, Express, AWS, and Serverless

Tree

Introduction

This tutorial will show you how to deploy a Node Express API to AWS (Amazon Web Services) using the Serverless Framework.

I really like serverless architecture, particularly because it’s cheap! AWS has a free tier for Lambda functions (a central part of serverless infrastructure) that covers the first million requests and 400k GB of compute time. After that, it’s only a tiny amount per transaction. My bill last month was $0.30 US.

What I don’t like are the serverless tools provided by cloud providers. First, they often don’t lend themselves well to a single sharable code base stored in a Git repository. Configuration is spread across several different cloud resources, and versioning is all but impossible. And second, it’s often slow and awkward to run the project locally in a development or test environment.

This tutorial will show you how to combine a Node Express API with serverless. This approach provides all the advantages of both worlds.

What is Serverless Anyway?

It’s a misnomer. There are definitely servers involved here. Serverless is a general term for tools and resources that remove the necessity for configuring server infrastructure for a web or mobile project. You just need to manage a single configuration file, which specifies which resources from the host that your project implements (validation, database, streaming, etc.). Everything server related is automatically taken care of for you, including provisioning, configuration, and load balancing. Sounds nice right?

What is the Serverless Framework?

Yet more confusion. There are several ways to implement serverless architecture from providers such as AWS, Azure, and Google. Third parties have entered the fray and developed tools which improve upon the tools supplied by serverless providers. The Serverless Framework is one such third party tool. It works with most serverless providers, is one or the easiest to use, provides plugins for specific use cases, and has decent documentation. Our tutorial will cover using the Serverless Framework in combination with AWS serverless resources. Learn more about the Serverless Framework here.

Moving forward, I will use the term serverless to refer to generic serverless architecture. I will use Serverless Framework to refer to the third party framework.

Requirements:

  • Node installed.
  • Familiarity with Node, Express, and basic AWS concepts.
  • An AWS account.

You can clone the GitHub repository for this tutorial here.

Step 1: Create an Express Application

First, create a Node Express application (follow the tutorial here for a quick and simple example). You can skip this step if you already have an Express application lined up. However, it’s often best to start with a bare bones application like the one in the tutorial in order to get things working.

This tutorial makes a few assumptions about the application:

  • It is transpiled (Webpack with TypeScript or Babel).
  • Bootstrapped via a single server or app file.
  • All application code lives in a /src directory located at the project root.

Step 2: Install and Configure AWS CLI

The Serverless Framework must have permission to access and deploy to AWS. There are two ways of granting this. You can either use the browser-based Serverless Dashboard, or you can install and configure AWS CLI. The Serverless Framework will automatically implement the AWS CLI permissions.

I recommend installing AWS CLI. You’ll probably need it for other stuff in the future, so you might as well get the installation out of the way. If you already have AWS CLI installed and configured, skip ahead to the next step.

  • Install – To install AWS CLI, simply download the executable installer. In the past, I’ve run into issues with the installation, such as it failing the first time around since the machine was missing certain dependencies. However, the latest installers seem to have fixed these issues and are a little more user friendly.
  • Create AWS User – Now we need to create and configure an AWS user for the CLI to use.
    • Log into AWS and navigate to the IAM console.
    • Choose an existing user, or create a new user.
    • Open the Permissions tab. The user must be granted read, write, and creation permissions for the following AWS services: Lambda, S3, CloudFormation, CloudWatch, and API Gateway.
    • Open the Security Credentials tab. We will create an Access Key for this user. Scroll down to the Access Keys section and create a new key (you cannot use an existing key). Copy down the access key and the secret key that it generates.
  • Create AWS CLI Profile – Finally we’ll provide the AWS CLI with access to AWS services by using the user and keys that you just created.
    • Open a terminal on your machine and enter the following command.
      aws configure
    • You are prompted to enter the access key and secret key that you just obtained in the previous step. For region, enter the AWS region that you’ll be using, for example I entered us-west-2. This is just the default region, it’s not written in stone and can be changed on a per project basis. Lastly you are asked to enter an output format. The default is JSON, which is usually the best way to go, unless you have a specific use case or preference that calls for yaml, text, or table output.
    • Now let’s make sure this thing is connected. Type in the following command. The console should output a JSON object with the profile’s connection credentials.
      aws sts get-caller-identity

You can find detailed instructions for installing and configuring AWS CLI here. Instructions for using the Serverless Dashboard are here.

Step 3: Install the Serverless Framework

The Serverless Framework is a Node package and is installed via npm or yarn. You can install it on a per project basis or globally. I recommend installing it globally since it makes its command line tool easier to use. Enter the following command in your terminal to install the Serverless Framework globally.

npm install serverless -g

After installation, check to see that everything went as planned. Enter the following command. The terminal should print out the versions of the newly installed Serverless Framework packages.

serverless --version

Step 4: Install Plugins

In order for Serverless Framework to work with an Express application, you must add 5 plugins to the Express project.

  • serverless-bundle – Transpiles and bundles the application for Serverless Framework compatibility. Works with both vanilla JavaScript and TypeScript.
  • serverless-offline – Provides emulation for AWS services, allowing the application to run locally in a test or development environment.
  • serverless-dotenv-plugin – Allows the Serverless Framework to better use .env files.
  • serverless-http – Wraps the Express application in a Lambda function for use with AWS serverless architecture.
  • cross-env – A utility that allows environment variables to be set in the command line using Linux syntax, but automatically implements Windows syntax on Windows machines.

Open your terminal in the context of your project directory and enter the following commands.

npm install serverless-http --save

npm install serverless-bundle serverless-offline serverless-dotenv-plugin cross-env --save-dev

You may uninstall the bundling packages you used previously, such as webpack, webpack-cli, babel, ts-loader, and ts-node. However, if it’s a TypeScript project, keep the typescript and ts-jest packages and the tsconfig.json file.

The project’s package.json file should now look something like this.

{
    "name": "my-api",
    "version": "0.1.1",
    "license": "ISC",
    "dependencies": {
        "express": "4.18.2",
        "serverless-http": "3.1.0"
    },
    "devDependencies": {
        "@types/express": "4.17.14",
        "@types/jest": "27.5.1",
        "cross-env": "7.0.3",
        "jest": "28.1.0",
        "serverless-bundle": "5.5.0",
        "serverless-dotenv-plugin": "4.0.2",
        "serverless-offline": "11.5.0",
        "ts-jest": "28.0.3",
        "typescript": "4.6.4"
    }
}

Step 5: Wrap That App

A key component of AWS serverless architecture is the Lambda function. A Lambda function is a function that lives in the cloud. It takes input from an AWS service, does something with the input, then outputs something to other AWS services. Simple but powerful. Here we will use the serverless-http plugin to wrap the Express application in a Lambda function.

First, find the file that you use to bootstrap your application. If you used my tutorial to create the application, it’s the src/server.ts file. Rename this file to src/handler.ts. This is not strictly necessary, but it’s best practice to name Lambda functions and the module that contains them handler.

Now open the file. It should look something like this.

import http from 'http';
import app from './app';

// initialize server with express
const server = http.createServer(app);

// start server
const port = process.env.PORT || '8080';
server.listen(port, () => console.log(`Server started on port ${port}`));

Replace the code with this.

import serverlessHttp from 'serverless-http';
import app from './app';

export const handler = serverlessHttp(app);

Important: I’ve noticed that some Express middleware conflicts with the Serverless Framework (probably due to the middleware and the Serverless Framework both trying to do the same thing). If requests or responses throw errors, comment out the middleware and all should be well. Later on you can figure out which middleware caused the issue and take appropriate action.

Step 6: Add the Serverless Framework Configuration File

This is the most important file in the entire project. It’s a YAML file that contains all of the configuration settings for the Serverless Framework. I won’t go into detail about what each of these settings does, just the highlights. You can find out more information about configuring the Serverless Framework here.

Create a new file at the project root and name it serverless.yml. Add the following to it.

service: my-api
frameworkVersion: '3'

custom:
    bundle:
        linting: false
        excludeFiles: '**/*.test.ts'
    serverless-offline:
        httpPort: 8080
        noPrependStageInUrl: true
        reloadHandler: true

provider:
    name: aws
    runtime: nodejs16.x
    stage: ${opt:stage, 'offline'}
    region: us-west-2
    memorySize: 512
    timeout: 10
    logRetentionInDays: 90
    logs:
        httpApi: true
    httpApi:
        cors: true

functions:
    app:
        handler: src/handler.handler
        events:
          - httpApi:
              path: '/{proxy+}'
              method: '*'

plugins:
    - serverless-dotenv-plugin
    - serverless-bundle
    - serverless-offline

The custom property is used for configuring the serverless-bundle and serverless-offline plugins. The provider property configures the project’s global runtime settings. The functions property tells Serverless Framework the location of the Lambda function in your project, plus it sets up a single root endpoint on AWS for the Express API to use.

The only property you may want to change is provider.region. Specify which AWS region you want the project to deploy in.

Step 7: Add Command Scripts

Last, but not least, we need to add command scripts. These scripts allow us to start, deploy, and destroy our application.

Open the existing package.json file and add the following scripts. 

{
    // previous configuration settings 

    "scripts": {
        "dev": "cross-env NODE_ENV=development sls offline start --stage dev",
        "deploy": "cross-env NODE_ENV=production sls deploy --stage prod",
        "remove": "cross-env NODE_ENV=production sls remove --stage prod"
    }
}

We’ve added 3 scripts, all of which use the Serverless Framework’s CLI. Note that you can use either serverless or the shorthand sls for invoking Serverless Framework CLI commands.

  • dev – Launches the application locally in the development environment.
  • deploy – Deploys the application to the production environment.
  • remove – Destroys the application in the production environment.

A Note On Environment Variables

Most Express applications use environment variables to handle secret data such as secret tokens, usernames, and passwords. Serverless Framework has some basic functionality for handling environment variables built-in, but the serverless-dotenv-plugin package adds the functionality that you probably expect (such as that provided by the dotenv package).

The example application assumes the use of two environment variable files; one named .env.development, and one named .env.production. You can find more information about how Serverless Framework and the plugin handle environment variables here.

Fire it Up!

Let’s use the scripts that we created and see if this thing actually works.

  • First, try the dev script to see if we can run our application locally. Enter the following command in the terminal. It should launch the application in a local dev environment. The application is accessed at http://localhost:8080. It’s relatively quick and automatically reloads on file changes. Perfect!

npm run dev

  • Now comes the moment that we’ve all been waiting for, deploying the application to AWS. We’ll use the deploy script. It will package the application, then deploy it to AWS.

npm run deploy

This process will take at least a few minutes. After it’s finished, let’s see what the Serverless Framework just did in AWS.

  • Log into the AWS console in your browser.
  • Go to AWS Lambda. One function was created.
  • Go to AWS S3. One bucket was created.
  • Go to AWS CloudFormation. One stack was created.
  • Go to AWS CloudWatch. Two log groups were created.
  • Finally, go to AWS API Gateway. One API application was created.

Wow! That’s a lot of stuff. In the past you would have had to set all of that up yourself. Not a fun time. But the Serverless Framework just did all of it for you using only the project’s YAML configuration file.

Now lets test the API. Go back to AWS API Gateway and select the API application that Serverless Framework just created. On the screen below, click the link in the Invoke URL column and presto, there’s your API, live online.

If you want to remove this application and start over, all you need to do is invoke the remove script. All of the AWS services that Serverless Framework created are automatically removed. You won’t have to worry about an AWS service that you forgot about charging you $1 a month for eternity.

npm run remove

Conclusion

You may have noticed that there is a lot of overlap in functionality between Express and the Serverless Framework. Depending on the future of your project, you will have to choose where to put new features. For example, one day you may want to deploy the application to a host other than AWS that does not support serverless architecture. Therefore you should probably keep functionality in Express in order to keep the project as agnostic as possible. However, if you’re convinced that the Serverless Framework and AWS are the ways forward, by all means, start moving more and more functionality into the serverless.yml file, until one day, the Express code is long gone. Happy coding!

Tags

Related Posts