TLDR
A serverless application is more than just a Lambda Function. It is a combination of Lambda functions, event sources, APIs, databases, and other resources that work together to perform tasks.
Creating serverless resources from the AWS Console is quick and easy, but as you know you should only use it for testing purposes when you are learning how it works. After that, and thinking about how to use the resources in a real project, you will need:
- IaC: to create your resources in a way that allows you to recreate them easily
- Version control: to track all the code modifications
- CI/CD: to automate the release process
Does anyone use CloudFormation or Terraform to manage their serverless resources? Possibly but come on, there is a better way!
To manage your serverless resources, there are much better options:
SAM (Serverless Application Model) / Serverless Framework
: declarative option with templates. Specific frameworks for serverless applicationsCDK / Pulumi
: add a level of abstraction and allow you to define the infrastructure with modern programming languages
In this article, we will review the approach to combining both CDK + SAM.
By the way, CDK + SAM is my preferred approach: you get the best of the 2 options!
CDK vs SAM
In the following articles, you will find the basics of CDK and SAM.
- CDK basics: How to create infrastructure with CDK
- SAM basics: How to create serverless applications with SAM
What do CDK and SAM have in common?
Both…
- Are open-source, Apache-licensed software development frameworks
- Provide Infrastructure as Code (IaC)
- Use AWS CloudFormation behind the scenes to deploy resources
- Provide a CLI to build and deploy applications
- Are well integrated with AWS build pipelines
- Support component reuse
What are the main differences between CDK and SAM?
CDK | SAM | |
---|---|---|
To declare resources | Uses familiar programming languages | Uses JSON or YAML |
Dynamic references | Native language capabilities | Pseudo parameters and logical functions |
Testing | Not supported natively (you could use SAM) | Supported (also debug) |
IaC resources | All | Focus on serverless |
Complexity | Very low | High, verbose configuration |
Maintainability | Higher | Lower |
Demo: CDK + SAM
From Jan 6, 2022, AWS announced the general availability of AWS SAM CLI support for local testing of AWS CDK applications. It means that you can use SAM over your CDK project to test your resources!
So… we will use a new CDK project to show the CDK + SAM.
The source code is available here. This repository has several CDK projects but first, we will use the v1-simple
Prepare to test
With CDK, when you run cdk synth
, it will synthesize a stack defined in your app into a CloudFormation template in a json
file in the cdk.out
folder.
However, SAM uses a yaml
template, template.yaml
or template.yml
, in the root folder.
Also, to test locally, you will need this file created or you will receive an error
1 2 3 > sam local invoke Error: Template file not found at /.../aws-cdk-simple-webservice/template.yml
Then, we have to run cdk synth
and store the result in template.yml
file.
use –no-staging to disable the copy of assets which allows local debugging via the SAM CLI to reference the original source files
1
cdk synth --no-staging > template.yml
You have to use --no-staging because it is required for SAM CLI to local debug the source files.
Testing Lambda Functions
You have now a template.yml
file and can run the SAM command to test your lambda function.
1
2
3
4
5
6
7
8
9
10
> sam local invoke
Invoking index.handler (nodejs14.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.46.0-x86_64.
Mounting /Users/alazaroc/Documents/MyProjects/github/aws/cdk/aws-cdk-simple-webservice/v1-simple/functions/simplest-example as /var/task:ro,delegated inside runtime container
START RequestId: 03d4ef7d-47b4-4ad2-a491-d0e5fc797ece Version: $LATEST
2022-04-22T21:00:38.952Z 03d4ef7d-47b4-4ad2-a491-d0e5fc797ece INFO request: {}
END RequestId: 03d4ef7d-47b4-4ad2-a491-d0e5fc797ece
REPORT RequestId: 03d4ef7d-47b4-4ad2-a491-d0e5fc797ece Init Duration: 0.39 ms Duration: 205.98 ms Billed Duration: 206 ms Memory Size: 512 MB Max Memory Used: 512 MB
{"statusCode":200,"headers":{"Content-Type":"text/html"},"body":"You have connected with the Lambda!"}%
The Lambda returns the following body: You have connected with the Lambda!
Testing Lambda Functions with input data
If your Lambda Functions need input data, you can generate it from SAM CLI with the command generate-event
1
sam local generate-event [OPTIONS] COMMAND [ARGS]...
You can use this command to generate sample payloads from different event sources such as S3, API Gateway, SNS, and so on. These payloads contain the information that the event sources send to your Lambda functions.
Or you can add the input data to the option -e
of the command invoke
1
2
3
4
5
6
7
8
> sam local invoke -e test/events/simple-event.json
Invoking index.handler (nodejs14.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.46.0-x86_64.
Mounting /Users/alazaroc/Documents/MyProjects/github/aws/cdk/aws-cdk-simple-webservice/v1-simple/functions/simplest-example as /var/task:ro,delegated inside runtime container
} "rawPath": "/test"706Z 992499c1-83c4-408d-966b-2e13f5955cbc INFO input: {
{"statusCode":200,"headers":{"Content-Type":"text/html"},"body":"You have connected with the Lambda!"}END RequestId: 992499c1-83c4-408d-966b-2e13f5955cbc
REPORT RequestId: 992499c1-83c4-408d-966b-2e13f5955cbc Init Duration: 0.89 ms Duration: 244.32 ms Billed Duration: 245 ms Memory Size: 512 MB Max Memory Used: 512 MB
Testing API Gateway
You have to run sam local start-api
1
2
3
4
> sam local start-api
Mounting simplest-lambda at http://127.0.0.1:3000$default [X-AMAZON-APIGATEWAY-ANY-METHOD]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2022-04-23 00:03:58 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
You can access
http://127.0.0.1:3000/
to connect with your Lambda Function
If you review your previous console, it will be updated when you accessed your API Gateway:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
> sam local start-api
default [X-AMAZON-APIGATEWAY-ANY-METHOD]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2022-04-23 00:03:58 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking index.handler (nodejs14.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.46.0-x86_64.
Mounting /Users/alazaroc/Documents/MyProjects/github/aws/cdk/aws-cdk-simple-webservice/v1-simple/functions/simplest-example as /var/task:ro,delegated inside runtime container
START RequestId: 8ed4a0a8-18bc-43af-b759-ad0668784351 Version: $LATEST
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Ge "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng} "isBase64Encoded": falsehost"01 +0000",-11ba1c012bf1",
END RequestId: 8ed4a0a8-18bc-43af-b759-ad0668784351
REPORT RequestId: 8ed4a0a8-18bc-43af-b759-ad0668784351 Init Duration: 0.51 ms Duration: 230.55 ms Billed Duration: 231 ms Memory Size: 512 MB Max Memory Used: 512 MB
2022-04-23 00:11:05 127.0.0.1 - - [23/Apr/2022 00:11:05] "GET / HTTP/1.1" 200 -
Invoking index.handler (nodejs14.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.46.0-x86_64.
Mounting /Users/alazaroc/Documents/MyProjects/github/aws/cdk/aws-cdk-simple-webservice/v1-simple/functions/simplest-example as /var/task:ro,delegated inside runtime container
START RequestId: 5759a74a-40b5-4a7e-8362-eec719ae44a7 Version: $LATEST
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Ge} "isBase64Encoded": falseco"t"01 +0000",-11ba1c012bf1",g+xml,image/*,*/*;q=0.8",
END RequestId: 5759a74a-40b5-4a7e-8362-eec719ae44a7
REPORT RequestId: 5759a74a-40b5-4a7e-8362-eec719ae44a7 Init Duration: 0.50 ms Duration: 238.45 ms Billed Duration: 239 ms Memory Size: 512 MB Max Memory Used: 512 MB
2022-04-23 00:11:06 127.0.0.1 - - [23/Apr/2022 00:11:06] "GET /favicon.ico HTTP/1.1" 200 -
Bonus: Testing DynamoDB
Ok, testing a mocked Lambda Function is the “hello world” example and not very useful, but what about a Lambda Function that connects to a DynamoDB table?
We will update our Lambda Function to store the data in a DynamoDB table, so we are using the v2-dynamodb example in the repository.
This code is based on the pattern defined in the web cdkpatterns as the simple webservice
Case 1: Testing cloud DynamoDB
When you try to locally test a Lambda Function that stores data in a DynamoDB table, it will automatically attempt to connect to the DynamoDB service of your AWS Account.
So, to test your Account DynamoDB tables, you have to do nothing.
1
2
3
4
5
6
7
8
9
10
> sam local invoke -e test/events/simple-event.json
Invoking index.handler (nodejs14.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.46.0-x86_64.
Mounting /Users/alazaroc/Documents/MyProjects/github/aws/cdk/aws-cdk-simple-webservice/v2-dynamodb/functions/dynamodb-example as /var/task:ro,delegated inside runtime container
} "rawPath": "/test"492Z 69d093d2-083c-46e7-a318-636ed94d7e47 INFO request: {
2022-04-23T06:38:56.797Z 69d093d2-083c-46e7-a318-636ed94d7e47 ERROR Invoke Error {"errorType":"ResourceNotFoundException","errorMessage":"Requested resource not found","code":"ResourceNotFoundException","message":"Requested resource not found","time":"2022-04-23T06:38:56.787Z","requestId":"RCJDUN8DRCPPOS3SR3034ETK8VVV4KQNSO5AEMVJF66Q9ASUAAJG","statusCode":400,"retryable":false,"retryDelay":49.42319019990148,"stack":["ResourceNotFoundException: Requested resource not found"," at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)"," at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)"," at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)"," at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"]}
{"errorType":"ResourceNotFoundException","errorMessage":"Requested resource not found","trace":["ResourceNotFoundException: Requested resource not found"," at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)"," at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)"," at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)"," at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequentialEND RequestId: 69d093d2-083c-46e7-a318-636ed94d7e47
REPORT RequestId: 69d093d2-083c-46e7-a318-636ed94d7e47 Init Duration: 0.98 ms Duration: 928.39 ms Billed Duration: 929 ms Memory Size: 512 MB Max Memory Used: 512 MB
_executor.js:116:18)"]}%
If you don’t deploy your CDK project before attempting to test it, you will get the following ERROR:
"errorType":"ResourceNotFoundException","errorMessage":"Requested resource not found"
Of course, you can set your AWS account in your sam CLI using the profile
command. If you do not specify it, the default value will be applied.
1
2
3
> sam local invoke -e test/events/simple-event.json profile test
Invoking index.handler (nodejs14.x)
...
Case 2: Testing local DynamoDB
You may want to test your Lambda function locally instead of connecting to your DynamoDB account, so do the following:
- Download DynamoDB docker image
- Run the DynamoDB docker image
- Set up DynamoDB: create tables, insert data, and test it
- Change your Lambda Function code
Download the DynamoDB docker image
The first step is to download the DynamoDB Docker image.
1
2
3
4
5
6
7
8
9
> docker pull amazon/dynamodb-local
Using default tag: latest
latest: Pulling from amazon/dynamodb-local
3a461b3ae562: Pull complete
14d349bd5978: Pull complete
3e361eec6409: Pull complete
Digest: sha256:07e740ad576acdcfdc48676f9a153a93a8e35436ea36942d4c14939caeca8851
Status: Downloaded newer image for amazon/dynamodb-local:latest
docker.io/amazon/dynamodb-local:latest
Run the DynamoDB Docker image locally
Now we have to run the locally downloaded docker image.
This terminal tab will be kept running and you will have to open another one.
1
2
3
4
5
6
7
8
> docker run -p 8000:8000 amazon/dynamodb-local
Initializing DynamoDB Local with the following configuration:
Port: 8000
InMemory: true
DbPath: null
SharedDb: false
shouldDelayTransientStatuses: false
CorsParams: *
This command will not persist data in the local DynamoDB.
Create a local DynamoDB table
We are going to create a table with the name hits
, with a partitionKey with the name path
and the String
type.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
> aws dynamodb create-table --table-name hits --attribute-definitions AttributeName=path,AttributeType=S --key-schema AttributeName=path,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 --endpoint-url http://localhost:8000
{
"TableDescription": {
"TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/hits",
"AttributeDefinitions": [
{
"AttributeName": "path",
"AttributeType": "S"
}
],
"ProvisionedThroughput": {
"NumberOfDecreasesToday": 0,
"WriteCapacityUnits": 1,
"LastIncreaseDateTime": 0.0,
"ReadCapacityUnits": 1,
"LastDecreaseDateTime": 0.0
},
"TableSizeBytes": 0,
"TableName": "hits",
"TableStatus": "ACTIVE",
"KeySchema": [
{
"KeyType": "HASH",
"AttributeName": "path"
}
],
"ItemCount": 0,
"CreationDateTime": 1650668228.617
}
}
Add values to our local DynamoDB table
We will add 2 elements:
- path: “/test”
- path: “/hello”
1
2
aws dynamodb put-item --table-name hits --item '{ "path": {"S": "/test"} }' --return-consumed-capacity TOTAL --endpoint-url http://localhost:8000
aws dynamodb put-item --table-name hits --item '{ "path": {"S": "/hello"} }' --return-consumed-capacity TOTAL --endpoint-url http://localhost:8000
Scan your table locally
We check that our table has the created elements:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
> aws dynamodb scan --table-name hits --endpoint-url http://localhost:8000
{
"Count": 2,
"Items": [
{
"path": {
"S": "/test"
}
},
{
"path": {
"S": "/hello"
}
}
],
"ScannedCount": 2,
"ConsumedCapacity": null
}
Change Lambda Function code
We need to update our Lambda Function code to tell DynamoDB to read from our local DynamoDB service:
You have to use your specific docker endpoint
1
2
3
4
5
6
7
8
if (process.env.AWS_SAM_LOCAL) {
// mac
dynamo.endpoint = new AWS.Endpoint("http://docker.for.mac.localhost:8000/");
// windows
// dynamo.endpoint = new AWS.Endpoint("http://docker.for.windows.localhost:8000/");
// linux
// dynamo.endpoint = new AWS.Endpoint("http://127.0.0.1:8000");
}
Test DynamoDB locally
In summary, we have:
- the DynamoDB service locally running
- a table (hits)
- 2 elements
- path=/test
- path=/hello
Now we are going to test our Lambda Function which will insert data into our local DynamoDB table.
1
2
3
4
5
6
7
8
9
10
> sam local invoke -e test/events/simple-event.json
Invoking index.handler (nodejs14.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-nodejs14.x:rapid-1.46.0-x86_64.
Mounting /Users/alazaroc/Documents/MyProjects/github/aws/cdk/aws-cdk-simple-webservice/v2-dynamodb/functions/dynamodb-example as /var/task:ro,delegated inside runtime container
} "rawPath": "/test"036Z eaf85e61-e9a2-4b49-9953-d247f9794fb8 INFO request: {
2022-04-22T23:54:20.130Z eaf85e61-e9a2-4b49-9953-d247f9794fb8 INFO inserted counter for /test
END RequestId: eaf85e61-e9a2-4b49-9953-d247f9794fb8
REPORT RequestId: eaf85e61-e9a2-4b49-9953-d247f9794fb8 Init Duration: 0.48 ms Duration: 697.40 ms Billed Duration: 698 ms Memory Size: 512 MB Max Memory Used: 512 MB
{"statusCode":200,"headers":{"Content-Type":"text/html"},"body":"You have connected with the Lambda and store the data in the DynamoDB table!"}
If we scan the table again, we can review that in “/test” element will be a new hits
column and 2
values:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
> aws dynamodb scan --table-name hits --endpoint-url http://localhost:8000
{
"Count": 2,
"Items": [
{
"path": {
"S": "/test"
},
"hits": {
"N": "2"
}
},
{
"path": {
"S": "/hello"
}
}
],
"ScannedCount": 2,
"ConsumedCapacity": null
}
If you run your function more times, the value of hits
will be updated.
And, of course, you can also test it from API Gateway:
1
2
3
4
> sam local start-api
Mounting dynamodb-lambda at http://127.0.0.1:3000$default [X-AMAZON-APIGATEWAY-ANY-METHOD]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2022-04-25 19:53:01 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)