AISDatabase

This repo comprises the source files for the AIS Boat Characteristic Database and the associated workflows. It contains the following folders:

• API: Code relating to API Gateway
• ETL: Code relating to the lambda function which decodes the raw AIS data as new files arrive
• HistoricDump: Code relating to the EC2 which decodes the raw AIS data for archived AIS files
• lambda_layers: Code for the "layers" which exist to be able to add libraries to a lambda function
• sql: Code related to setting up and administering the database

This repository is not meant to be installed but rather acts as a centralized code reference for the project.

This repo in context

Code for the AIS Database project could exist in one of three places:

1. This repository
2. s3://abcdatabase bucket
3. In the lambda console for each function

What's the difference?

This repository is intended to be the centralized collection of resources relating to the project. The files present in s3 are only the ones that need to be available in AWS in order for CloudFormation to create new or update existing resources. This means that the s3 bucket contains the v*.*.*.zip files for lambda code, the *.zip files for the lambda layers, and the architecture.yml files for the cloud formation templates. In other words, this repo should contain all the files assocaited with the project, while the s3 bucket only contains a subset.

The lambda function adds another wrinkle of complexity because we can change the code for the lambda directly in the console. This means that the github repo for the lambda functions may not always be up-to-date with what is present in the lambda function. It would be possible to keep things more in sync by editing the lambda files locally on a github branch and then uploading a zip file to s3, pybassing entirely the lambda console. This could be quite laborious, however, and so this may not always be the best development pattern.

A pothole not to trip in

There is no automatic process which syncs the lambda code to the github nor the github code to the s3 bucket. This could cause scenarios where code is lost if cloudformation templates are updated prematurely. To avoid this situation make sure to do the following before updating a CloudFormation template with new lambda code:

• Ensure that the code in the lambda console is the same as the code in the github branch
• Create a zip of this file with a new version number
• Upload the zip file to s3
• Change the version number of the code in the Cloud Formation template file
• Update the stack

Cloud Formation

There are 3 seperate architecture.yml files in this repo. They each correspond to a different stack:

• API/architecture.yml --> abcAPI (API Gateway lambdas)
• ETL/architecture.yml --> abcETL (ETL resources)
• HistoricDump/architecture.yml --> abcHistoric (EC2 and associated resources)

More detail about each stack can be found in the READMEs of the associated folders containing the yamls.

Quick links

API docs

Usage docs for the API can be found in this repository at:

API/docs/README.md