protobufs

Protobufs for brymck.io. This is both something I use both for personal projects and to demonstrate professionally how you can organize service definitions in a robust way.

Table of contents

• Overview
• Building

Overview

This library is a monorepo housing Protobuf definitions. Its goal is to allow myself and contributors to define services that are

• Easy to write and maintain
• Efficient in production
• Easy for others to discover
• Easy for downstream users, both service developers and API consumers, to use

To accomplish these goals, I use Google's Protobufs as the IDL (interface description language) and gRPC as our RPC framework. Protobufs are language-neutral and come with a rich ecosystem of tools for code generation that allow serializing structured data. gRPC is a high performance RPC (remote procedure call) framework that supports multiple languages, using Protobufs for service definitions. To ensure we follow common practices, we lint and compile Protobufs with Prototool.

When the full CI/CD process is run, it will:

• Lint Protobuf definitions
• Generate, build and deploy Go, Java, Node.js and Python libraries for each Protobuf package
• Generate, build and deploy a Protobuf registry web site

The repo is laid out as follows:

• .github/ - GitHub Actions CI/CD configuration
• bin/ - Python helper scripts for the build process
• config/ - Configuration for Google Cloud Endpoints, Maven, etc.
• proto/ - Protobuf service definition files
• templates/ - Templates for code generation not handled by protoc
• Makefile - Directives for building and deploying libraries

Building

Prerequisites:

• Docker
• Go
• Node.js
• OpenJDK or some other Java installation
• Python

To generate all code and build the resulting libraries (in some cases installing them locally), run:

make

This will build the following targets:

• make dependencies - Create Protobuf descriptor sets and dependency lists for each package
• make generate - Use Prototool (and Proto Registry) to generate code
• make package - Reorganize code into directories for each language + package that contain complete package descriptions, using Jinja for templating
• make build - Build each package

Note that package and build can all be appended with -go, -java, -node and -python to only run the language-specific sections of each.

Other targets

make proto/brymck/foo/v1/foo.proto

Create a new Protobuf file with basic information filled in based on the contents of proto/prototool.yaml. Follow the convention of proto/brymck/<package>/v<version>/<service>.proto.

make lint

Validate your Protobuf files.

make deploy

In general this should only run as part of CI/CD. Similar to package and build, you can provide the language-appropriate suffix to this target to only deploy for that language.

make clean

Remove any generated files.