Deploying an Application to AWS Via serverless.com

TL;DR = If you want to run a Tiddlywiki with the back-end on AWS at low (or maybe no) cost, here's how. If you want to see how to use serverless.com's framework, here's how.  

 This page documents my adventures in using the serverless.com framework to publish an AWS "serverless" application to Amazon. I use Tiddlywiki (TW) as my sample application. This was run in May of 2021.

Tiddlywiki is a wiki application. It runs in several storage modes.  In the mode of interest, it is configured as a JavaScript front-end with a server-side back-end to store the wiki pages.  

I will be implementing [this project from github](https://gitlab.com/vdorneanu/widdly/-/tree/serverless-tiddly). In theory, one would simply clone the project from github, build, and deploy.  In practice, github contains the project owner's point-in-time snapshot of his implementation, and it hasn't been operationalized for use by other people.  In effect, I'll be documenting how to take his prototype and deploy it for use by others.

Reference URLs

• https://github.com/dorneanu/widdly/tree/serverless-tiddly
• https://tiddly.info/serverless

Why serverless.com?

When you engage directly with AWS, you deal with a large number of interacting services. In order to configure AWS you must figure out which services you need and then go and configure each one individually. There is no "Configure My App" -- just many, separately-configure services. Unless you are doing this stuff every day and have been doing it for months, it leads to cognitive overload.  You need a framework for engaging with AWS.

https://serverless.com (mostly) uses a single configuration file to define the environment to run your app.  You provide the app, and the config file says (in one place), "Here is how to deploy it."

Beyond configuration, Serverless can provide a simple, local AWS emulation. You can develop and do initial debugging on your local computer, and deploy to AWS for final testing. This is somewhat similar to using a local Docker container and then deploying to a container on a production server.

N.B. I'm going to try to use "Serverless" to refer to things related to serverless.com, and "serverless" to refer to running things on AWS without defining a virtual server.

But containers and local development are beyond the scope of this document. I'll focus on how to successfully deploy to the AWS serverless environment using Serverless.

The Serverless site has a really good series of lessons on AWS. They are helpful for understanding AWS, not just for using the Serverless framework.

Getting Ready to Deploy

These are the things I had to do before I could start the deploy-and-debug cycle. Note that I am using a Mac.  If you're using Linux or Windows, you'll need to mentally substitute Nautilus, GNOME Files, File Explorer where I say "Finder", and similar substitutions.  If you're using Windows, I recommend that you install WSL and use it as a Linux environment.  This stuff is centered on a Linux-like environment.

1. Install the AWS CLI per instructions here: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
2. Install Serverless's CLI:
1. Mac/Linux: curl -o- -L https://slss.io/install | bash
2. Windows (pick an approach): 
1. Using Chocolatey package manager: choco install serverless 
2. Using node.js's package manager: npm install -g serverless 
3. Use WSL and run the Mac/Linux command line. (This is what I recommend.)
3. After you run the install, run: serverless 
1. Follow the prompts to set up your serverless environment. 
1. I told it not to create a new project. 
2. I did have it install bash command-completion.
4. For reference, some useful Serverless commands (be sure to append "--stage prod" or "--stage dev" as appropriate):
1. serverless --version # What version do I have? 
2. serverless upgrade   # Upgrade to latest version 
3. serverless remove    # Delete AWS resources for this project 
5. Install git, if you don't have it. Details are out of scope for this document.
6. Open a command line in the directory where you store you development projects. e.g. ~/Documents/code for me.  Run these commands:
• git clone https://github.com/dorneanu/widdly.git  
• cd widdly  
• git checkout e96cd1c34acf8a04cafb7ebaaa1a1ded734cacf5 
• cp serverless.yml serverless.yml.bak  
7. Notes: 
• "Victor" is the owner of https://github.com/dorneanu
• I'm working with the app at https://gitlab.com/vdorneanu/widdly/-/tree/serverless-tiddly and Victor may update it. In order to work with the now-current edition of his code, I'm checking out a specific commit.
• The existence of the "serverless.yml" file is evidence you are on the serverless-tiddly branch.  Victor said he'd merge it into main, but until serverless.yml shows up in the main branch, he never got around to it.
• I'm going to mess with serverless.yml a lot. I don't really know what I'm doing.  I'll need that ".bak" file! 
8. Edit serverless.yml
1. Delete the customDomain section
2. Delete the plugins section. 
3. Why?
• Victor is creating a real site with a real domain name. Amazon charges for that stuff. I'm sticking with free, since this is a learning exercise.  
• AWS free URLs all are non-root.  i.e. You can't have http://example.com/, you must have http://example.com/some/other/stuff. This will require some tinkering at a later step because TW assumes it is running on a root folder.
4. Delete the two lines which contain "private: true"
• Victor uses a hack to prevent the public from updating his site. I'm not interested in implementing his hack, and doing a real authentication subsystem would require WAY too much updating of TW.  I'm going to let the general public update my wiki.  It will only be in existence for a few hours, and it will be on an obscure URL, so I expect no one but me will ever know it is there.
9. We're about to deploy to AWS. You're going to need some storage there.  It will be an "S3 storage bucket".  S3 buckets must have globally unique names. i.e. You can't have a foo if someone else has used the name foo.  Victor used tiddlywiki-serverless, so I'm using tiddlywiki-serverless-kpk. You will need a different name:
• Run: aws s3 mb s3://tiddlywiki-serverless-kpk  
• Edit serverless.yml and change "tiddlywiki-serverless" to "tiddlywiki-serverless-kpk" under the "deploymentBucket:" heading.
10. We need to insert a new tiddler (wiki page) to configure TW to run from /prod/serverless. We'll do that by editing index.html.  This is a hack!
1. On your local machine, in the widdly folder, copy index.html to index.html.bak
2. Edit index.html. Immediately after the line that looks like this:
1. <div id="storeArea" style="display:none;">  
3. Insert these lines and save the file:
1. <div created="20210519164950911" modified="20210519165025763" tags="" title="$:/config/tiddlyweb/host">
2. <pre>$protocol$//$host$/prod/serverless/</pre>
3. </div>
11. The delivered project will build for the wrong architecture on my Mac. I think Victor set up the Makefile to run in a variety of environments with support for Serverless local-testing mode, in addition to production. I will configure the build to always run for the AWS environment. I don't need multi-environment deploy, and this is a simplified tutorial, so I hard-wire it to AWS.
1. Create a file named build-me.sh.  Fill it with:

• #!/bin/bash

• GOARCH=amd64;GOOS=linux make build  

2. The Makefile queries the current machine what OS and CPU architecture.  That's fine if you're going to run unit tests locally.  But that's mostly useful for debugging the golang  web server code.  Victor has already debugged it, so I hard-wire build-me.sh to build for deploy to AWS.
3. While you're at it, create a deploy-me.sh and fill it with:
• #!/bin/bash   
• serverless deploy --stage prod | grep -v "Deprecation warn"   
4. chmod your new .sh files to make them executable.
5. Note: We're going to filter out deprecation warnings, to simplify what we have to read.
12. Run: ./build-me.sh   
13. Run: ./deploy-me.sh   
• It takes a few minutes to complete.
• In real life, you probably would deploy to dev, and only later deploy to prod, but I don't have any users and this is just a tutorial.
• It threw a bunch of deprecation warnings. If you're following this tutorial in the future, these may be errors and you might have to fix them. I'm assuming that warnings are inconsequential.
• Some things it reported (excerpts):
• service: tiddlywiki-serverless
• stage: prod
• region: us-east-1
• stack: tiddlywiki-serverless-prod
• api keys:
• API_KEY: (saved in my password safe)
• endpoints:
• PUT - https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/{any+}
• DELETE - https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/{any+}
• GET - https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/{any+}
• OPTIONS - https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/{any+}
• GET - https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/
• functions:
• tiddlywiki-serverless-api: tiddlywiki-serverless-api-prod
• tiddlywiki-serverless-index: tiddlywiki-serverless-index-prod  
14. Save that output. You will need those URLs.  Going forward, I will refer to the URL next to the "GET" endpoint as "the GET URL".  e.g. My GET URL (yours will differ) is "https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/"
15. Test to see if your Lambda function works:
1. AWS Console > API Gateway > APIs > prod-tiddlywiki-services > / GET (click the top GET)
2. Click Test, scroll down and click Test (again)
3. Scroll down to the end of the logs. 
4. If you get this, you succeeded.
• Wed May 19 14:59:25 UTC 2021 : Method completed with status: 404
5. If you get this, you built for the wrong environment. Check the GOARCH and GOOS variables in build-me.sh:
• Wed May 19 14:34:00 UTC 2021 : Lambda execution failed with status 200 due to customer function error: fork/exec /var/task/widdly-lambda: exec format error. Lambda request id: 0b629094-aee1-4fad-a9da-1d12af533dab
• Wed May 19 14:34:00 UTC 2021 : Method completed with status: 502 
6. Take your GET URL and append "/serverless". e.g. my appended URL is: https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/serverless
• Use wget or curl to fetch this appened URL.  e.g. I ran:
• wget https://zgxj1pyv0d.execute-api.us-east-1.amazonaws.com/prod/serverless  
7. This will fetch the HTML and JavaScript for the Tiddlywiki front-end from S3.  (It was deployed by deploy-me.sh.)
16. Open that same URL in your browser. You'll see Tiddlywiki. There should be no error messages.

It takes an hour or so to modify and deploy, now.  Figuring out what to change from Victor's project and the things to install took me a couple of weeks.

Here are some screen shots.

An empty wiki:

Adding a "FrontPage" story:

The FrontPage story saved (view mode - not edit mode), and the sidebar collapsed:

Showing the result of searching for "major heading". Note that it finds the FrontPage story we just created.

And after showing that it works, I ran this command to delete all of it except for the S3 bucket:

• serverless remove --stage prod  

To delete the S3 bucket:

• AWS Console > S3 > select tiddlywiki-serverless-kpk > Delete