Serverless automation around GitHub
At Trustpilot we like to move fast. We currently have roughly 800 different services and we deploy 200 times a week on average.
The SRE team is partly responsible for making sure that speed remains achievable. One way of accomplishing that is by automating as many tedious and repetitive tasks as possible. A lot of these tasks happen when code is moved from developer machines into production.
We store all our code in GitHub and one of Trustpilot’s Engineering Principles is to Code Review Everything. This places GitHub at the center of a lot of the tasks executed around our services, and makes it an ideal place for automation.
Events fired
We have enabled a Webhook on GitHub that fires on every event happening on the Trustpilot GitHub organization. A simple serverless application receives those events, applies basic validation, and then forwards them to an AWS SNS topic.
This fan-out method allows for many applications to react to the events, and we’ll go through most of them in this article.
But first, let’s take a look at a simplified implementation of the webhook receiver as an AWS Lambda function:
Pull requests
When using git, changes are commonly (and preferably) done by creating a branch off of the origin/master branch. Changes are then pushed and a pull request is created to get the changes reviewed and merged into master.
The function github-pullrequestcreator handles the automatic creation of this pull request. On the push of a new GitHub repository branch, this function opens a pull request with the same name as the branch. It also assigns the developer creating the branch to the pull request.
Our workflow is for developers to track each pull request on a Trello card; some automation is also applied here. On every push of a new branch, the function github-trellobridge will attach the pull request to a Trello card that has a matching name. This spares the developer the tedious task of going to GitHub, getting the link to the pull request and attaching it to the card manually. The function also syncs labels between the Trello card and pull request, and attaches the Trello card to the pull request as a reverse link.
Don’t leak credentials
Rarely, but still too often, we’ve seen connection strings and AWS key pairs being committed to GitHub. Even with most of our repositories being private, we still don’t want to have credentials anywhere else than where they are used. The risk of exposing credentials is already reduced in a variety of ways. For example, we have adopted the use of specific and short lived credentials. Limiting both the duration and the permissions. We use Vault for this.
We still don’t want the credentials exposed and we also work with databases and external systems where short lived credentials are not yet supported or enabled. Those credentials we don’t ever want exposed, neither in public nor private repositories.
For this purpose we have github-secretsnoop, which reacts to all GitHub pushes. Whenever database credentials, API keys or similar instances are found, the specific line of code is reported in a dedicated Slack channel. If the push is part of a pull request, the function also marks that pull request as tainted; the merge will now require the involvement of a GitHub organisation administrator.
This is what it looks like when the line of code is reported on Slack:
And the pull request review:
CI/CD
Travis CI is the Trustpilot Paved-Road™ choice for building applications and we use it for most of our repositories.
Before having the automation in place a developer had to put a .travis.yml file in the repository, add credentials and manually enable a repository on travis.com.
The configuration file is (if opted for) automatically created when defining the service on our self-service portal. A combination of the GitHub app from Travis CI and our function travis-autoconfigure performs the other part.
The creation of a new repository triggers travis-autoconfigure. This function applies environment variables and enables the feature auto_cancel_pull_requests to prevent clogging of the CI pipeline when people are pushing a lot of small commits to a single repository. When pushing to a branch with a pull request build already running, this features cancels the currently running build and starts building the most recent push to the branch.
Small functions
The functions covered so far have very specific purposes. Luckily this correlates nicely with our principle to build smaller things.
We try to make our services as on-point as possible. Besides the covered functions we have a related handful of even smaller functions. Two of these react on the creation of new repositories. One only makes sure that only the right people and groups have the right permissions on these repositories. The other only focuses on Trustpilot naming conventions being followed for repositories.
The architecture of this systems allows for easy changes made to the logic of specific functions. It also allows the use of the language of preference. Both JavaScript and Python are used in the examples of this article.
Should new automation possibilities arrive it will also be easy to further extend the current pool of functions.
Open sourcing
One of our remaining principles is to aim to open source. And we would love to open source these small, but very useful functions.
It has unfortunately turned out to be too big of a task to decouple them from our specific way of using GitHub and Trello. Decoupling from AWS specific technology would also require quite some effort.
We are instead considering moving to Probot. Probot is a framework for building GitHub apps that would allow us to do more or less the same as we’re currently doing. Probot is already an open source project and we’ll instead be able to contribute to this project.
