tl;dr from now on, we release generator in an automated way. We roll-out this setup to the rest when we see it is needed.
Repetitive tasks are tedious. If what you do manually can be automated, then what are you waiting for!
But these tasks take only a couple of minutes from time to time, gimme a break
A couple of minutes here, a couple of minutes there and all of a sudden you do not have time on more important things, on innovation. Automation makes it easier to scale and eliminates errors. Distractions consume time and make you less productive.
We kick ass at AsyncAPI Initiative at the moment. We started to improve our tooling regularly. We are now periodically sharing project status in our newsletter, and host bi-weekly open meetings, but most important is that we just recently updated our roadmap.
Am I just showing off? It sounds like, but that is not my intention. I wish to point out we are productive, and we want to continue this trend and automation helps here a lot. If you have libraries that you want to release regularly and you plan additional ones to come, you need to focus on release automation.
What full automation means
Full automation means that the release process if fully automated with no manual steps. What else did you think?
Your responsibility is just to merge a pull request. The automation handles the rest.
You might say: but I do not want to release on every merge, sometimes I merge changes that are not related to the functionality of the library.
This is a valid point. You need a way to recognize if the given commit should trigger the release and what kind of version, PATCH, or MINOR. The way to do it is to introduce in your project Conventional Commits specification.
Conventional Commits
At AsyncAPI Initiative we use Semantic Versioning. This is why choosing Conventional Commits specification was a natural decision.
Purpose of Conventional Commits is to make commits not only human-readable but also machine-readable. It defines a set of commit prefixes that can be easily parsed and analyzed by tooling.
This is how the version of the library looks like when it follows semantic versioning: MAJOR.MINOR.PATCH. How does the machine know what release you want to bump because of a given commit? Simplest mapping looks like in the following list:
- Commit message prefix
fix:indicatesPATCHrelease, - Commit message prefix
feat:indicatesMINORrelease, - Commit message prefix
{ANY_PREFIX}!:so for examplefeat!:or evenrefactor!:indicateMAJORrelease.
It other words, assume your version was 1.0.0, and you made a commit like feat: add a new parameter to test endpoint. You can have a script that picks up feat: and triggers release that eventually bumps to version 1.1.0.
Workflow design
At AsyncAPI Initiative where we introduced the release pipeline for the very first time, we had to do the following automatically:
- Tag Git repository with a new version
- Create GitHub Release
- Push new version of the package to NPM
- Push new version of Docker image to Docker Hub
- Bump the version of the package in
package.jsonfile and commit the change to the repository
This is how the design looks like:
There are two workflows designed here.
The first workflow reacts to changes in the release branch (master in this case), decides if release should be triggered, and triggers it. The last step of the workflow is a pull request creation with changes in package.json and package-lock.json. Why are changes not committed directly to the release branch? Because we use branch protection rules and do not allow direct commits to release branches.
You can extend this workflow with additional steps, like:
- Integration testing
- Deployment
- Notifications
The second workflow is just for handling changes in package.json. To fulfill branch protection settings, we had to auto-approve the pull request so we can automatically merge it.
GitHub Actions
Even though I have my opinion about GitHub Actions, I still think it is worth investing in it, especially for the release workflows.
We used the GitHub-provided actions and the following awesome actions built by the community:
Release workflow
Release workflow triggers every time there is something new happening in the release branch. In our case, it is the master branch:
1on:
2 push:
3 branches:
4 - masterGitHub and NPM
For releases to GitHub and NPM, the most convenient solution is to integrate semantic release package and related plugins that support Conventional Commits. You can configure plugins in your package.json in the order they should be invoked:
1"plugins": [
2 [
3 "@semantic-release/commit-analyzer",
4 {
5 "preset": "conventionalcommits"
6 }
7 ],
8 [
9 "@semantic-release/release-notes-generator",
10 {
11 "preset": "conventionalcommits"
12 }
13 ],
14 "@semantic-release/npm",
15 "@semantic-release/github"
16]Conveniently, functional automation uses a technical bot rather than a real user. GitHub actions allow you to encrypt the credentials of different systems at the repository level. Referring to them in actions looks as follows:
1- name: Release to NPM and GitHub
2 id: release
3 env:
4 GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
5 NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
6 GIT_AUTHOR_NAME: asyncapi-bot
7 GIT_AUTHOR_EMAIL: info@asyncapi.io
8 GIT_COMMITTER_NAME: asyncapi-bot
9 GIT_COMMITTER_EMAIL: info@asyncapi.io
10 run: npm run releaseAside from automation, the bot also comments on every pull request and issue included in the release notifying subscribed participants that the given topic is part of the release. Isn't it awesome?
Docker
For handling Docker, you can use some community-provided GitHub action that abstracts Docker CLI. I don't think it is needed if you know Docker. You might also want to reuse some commands during local development, like image building, and have them behind an npm script like npm run docker-build.
1- name: Release to Docker
2 if: steps.initversion.outputs.version != steps.extractver.outputs.version
3 run: |
4 echo ${{secrets.DOCKER_PASSWORD}} | docker login -u ${{secrets.DOCKER_USERNAME}} --password-stdin
5 npm run docker-build
6 docker tag asyncapi/generator:latest asyncapi/generator:${{ steps.extractver.outputs.version }}
7 docker push asyncapi/generator:${{ steps.extractver.outputs.version }}
8 docker push asyncapi/generator:latestBump version in package.json
A common practice is to bump the package version in package.json on every release. You should also push the modified file to the release branch. Be aware though that good practices in the project are:
- Do not commit directly to the release branch. All changes should go through pull requests with proper peer review.
- Branches should have basic protection enabled. There should be simple rules that block pull requests before the merge.
Release workflow, instead of pushing directly to the release branch, should commit to a new branch and create a pull request. Seems like an overhead? No, you can also automate it. Just keep on reading.
1- name: Create Pull Request with updated package files
2 if: steps.initversion.outputs.version != steps.extractver.outputs.version
3 uses: peter-evans/create-pull-request@v2.4.4
4 with:
5 token: ${{ secrets.GH_TOKEN }}
6 commit-message: 'chore(release): ${{ steps.extractver.outputs.version }}'
7 committer: asyncapi-bot <info@asyncapi.io>
8 author: asyncapi-bot <info@asyncapi.io>
9 title: 'chore(release): ${{ steps.extractver.outputs.version }}'
10 body: 'Version bump in package.json and package-lock.json for release [${{ steps.extractver.outputs.version }}](https://github.com/${{github.repository}}/releases/tag/v${{ steps.extractver.outputs.version }})'
11 branch: version-bump/${{ steps.extractver.outputs.version }}Conditions and sharing outputs
GitHub Actions has two excellent features:
- You can set conditions for specific steps
- You can share the output of one step with another
These features are used in the release workflow to check the version of the package, before and after the GitHub/NPM release step.
To share the output, you must assign an id to the step and declare a variable and assign any value to it.
1- name: Get version from package.json after release step
2 id: extractver
3 run: echo "::set-output name=version::$(npm run get-version --silent)"You can access the shared value by the id and a variable name like steps.extractver.outputs.version. We use it, for example, in the condition that specifies if further steps of the workflow should be triggered or not. If the version in package.json changed after GitHub and NPM step, this means we should proceed with Docker publishing and pull request creation:
if: steps.initversion.outputs.version != steps.extractver.outputs.versionFull workflow
Below you can find the entire workflow file:
1name: Release
2
3on:
4 push:
5 branches:
6 - master
7
8jobs:
9 release:
10 name: 'Release NPM, GitHub, Docker'
11 runs-on: ubuntu-latest
12 steps:
13 - name: Checkout repo
14 uses: actions/checkout@v2
15 - name: Setup Node.js
16 uses: actions/setup-node@v1
17 with:
18 node-version: 13
19 - name: Install dependencies
20 run: npm ci
21 - name: Get version from package.json before release step
22 id: initversion
23 run: echo "::set-output name=version::$(npm run get-version --silent)"
24 - name: Release to NPM and GitHub
25 id: release
26 env:
27 GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
28 NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
29 GIT_AUTHOR_NAME: asyncapi-bot
30 GIT_AUTHOR_EMAIL: info@asyncapi.io
31 GIT_COMMITTER_NAME: asyncapi-bot
32 GIT_COMMITTER_EMAIL: info@asyncapi.io
33 run: npm run release
34 - name: Get version from package.json after release step
35 id: extractver
36 run: echo "::set-output name=version::$(npm run get-version --silent)"
37 - name: Release to Docker
38 if: steps.initversion.outputs.version != steps.extractver.outputs.version
39 run: |
40 echo ${{secrets.DOCKER_PASSWORD}} | docker login -u ${{secrets.DOCKER_USERNAME}} --password-stdin
41 npm run docker-build
42 docker tag asyncapi/generator:latest asyncapi/generator:${{ steps.extractver.outputs.version }}
43 docker push asyncapi/generator:${{ steps.extractver.outputs.version }}
44 docker push asyncapi/generator:latest
45 - name: Create Pull Request with updated package files
46 if: steps.initversion.outputs.version != steps.extractver.outputs.version
47 uses: peter-evans/create-pull-request@v2.4.4
48 with:
49 token: ${{ secrets.GH_TOKEN }}
50 commit-message: 'chore(release): ${{ steps.extractver.outputs.version }}'
51 committer: asyncapi-bot <info@asyncapi.io>
52 author: asyncapi-bot <info@asyncapi.io>
53 title: 'chore(release): ${{ steps.extractver.outputs.version }}'
54 body: 'Version bump in package.json and package-lock.json for release [${{ steps.extractver.outputs.version }}](https://github.com/${{github.repository}}/releases/tag/v${{ steps.extractver.outputs.version }})'
55 branch: version-bump/${{ steps.extractver.outputs.version }}Automated merging workflow
You may be asking yourself:
Why automated approving and merging is handled in a separate workflow and not as part of release workflow
One reason is that the time between pull request creation and its readiness to be merged is hard to define. Pull requests always include some automated checks, like testing, linting, and others. These are long-running checks. You should not make such an asynchronous step a part of your synchronous release workflow.
Another reason is that you can also extend such an automated merging flow to handle not only pull requests coming from the release-handling bot but also other bots, that, for example, update your dependencies for security reasons.
You should divide automation into separate jobs that enable you to define their dependencies. There is no point to run the automerge job until the autoapprove one ends. GitHub Actions allows you to express this with needs: [autoapprove]
Below you can find the entire workflow file:
1name: Automerge release bump PR
2
3on:
4 pull_request:
5 types:
6 - labeled
7 - unlabeled
8 - synchronize
9 - opened
10 - edited
11 - ready_for_review
12 - reopened
13 - unlocked
14 pull_request_review:
15 types:
16 - submitted
17 check_suite:
18 types:
19 - completed
20 status: {}
21
22jobs:
23
24 autoapprove:
25 runs-on: ubuntu-latest
26 steps:
27 - name: Autoapproving
28 uses: hmarr/auto-approve-action@v2.0.0
29 if: github.actor == 'asyncapi-bot'
30 with:
31 github-token: "${{ secrets.GITHUB_TOKEN }}"
32
33 automerge:
34 needs: [autoapprove]
35 runs-on: ubuntu-latest
36 steps:
37 - name: Automerging
38 uses: pascalgn/automerge-action@v0.7.5
39 if: github.actor == 'asyncapi-bot'
40 env:
41 GITHUB_TOKEN: "${{ secrets.GH_TOKEN }}"
42 GITHUB_LOGIN: asyncapi-bot
43 MERGE_LABELS: ""
44 MERGE_METHOD: "squash"
45 MERGE_COMMIT_MESSAGE: "pull-request-title"
46 MERGE_RETRIES: "10"
47 MERGE_RETRY_SLEEP: "10000"For a detailed reference, you can look into this pull request that introduces the above-described workflow in the generator.
Conclusions
Automate all the things, don't waste time. Automate releases, even if you are a purist that for years followed a rule of using imperative mood in commit subject and now, after looking on prefixes from Conventional Commits you feel pure disgust.
In the end, you can always use something different, custom approach, like reacting to merges from pull requests with the specific label only. If you have time to reinvent the wheel, go for it.
Cover photo by Franck V. taken from Unsplash.