Suppose you contribute to an existing open source project, and are interested in regularly running a custom workflow (e.g. static analysis, or tests with sanitizers, or other checks) against that project. To give a concrete example: I wanted to start running git’s test suite with various sanitizers enabled (e.g. ASAN and LSAN) in CI, without having to go through the process of including this in git’s official CI configuration.
Github Actions offer a nice and (relatively) easy way of doing this, here’s how I configured daily ASAN+LSAN tests for git. For a TL;DR: version I recommend just looking at the workflow I ended up writing – but for further context and reasoning read below.
Although I was working against an existing Github project, this approach should also work for an arbitary git project outside of Github, or even mercurial/svn/tarball based projects. You’ll just need to do a bit more work if you aren’t working against a source of truth hosted on Github. (Fun fact: git isn’t even a purely Github-hosted project, although the Github repo is one of multiple authoritative sources.)
High-Level approach
Actions can be configured to run on a schedule – in my case I decided that running my workflow once a day was sufficient (I probably won’t even look at the outputs daily, but at least I’ll have reasonably fresh data available on days that I do want to check the outputs). If your project is sufficiently big, you might consider running your workflow less frequently (because they’ll get in the way of each other and/or other jobs you might run using your account.).
Once you’ve figured out a schedule, the actual workflow is pretty simple – each time it is run you’ll want to:
- Figure out if the upstream projects has new changes. This isn’t strictly necessary, but reduces resource usage and potentially reduces noise (no point in rerunning the same failing tests if nothing has changed upstream).
- Run the tests or workflow that you want. This is very project specific – fortunately git already has some default Github Actions to build the binary and run the tests, and I was able to adapt those for my needs.
Determining if there are any changes
There are many ways to keep track of which commits have already been tested. I was lazy and decided to create a branch in my fork which tracks the upstream branch that I want to test. My fork’s branch is a snapshot of the upstream branch at the point in time when I last tested it, and is updated every time the workflow is run. In other words: if my fork’s branch tip == upstream branch tip, then there are no new changes needing testing. If upstream has changed, then there are new changes needing testing.
This approach has some obvious failure modes: suppose that the workflow is cancelled at the wrong time, or fails for infrastructural reasons (as opposed to actual test failures). My fork’s branch might be up to date, and suggests that we’ve run our workflow against that tip – and we won’t try re-running our workflow (at least until the upstream branch changes again). That’s a bit ugly, but good enough for me – we’ll rerun the workflow as soon as the upstream branch is changed again anyway (Github also sends an email when a workflow fails, giving you an opportunity to run it again manually).
A superior approach would be to record verified commits using e.g. git notes. Our custom workflow could then start by syncing our forked branch, followed by checking git notes to determine if the current tip has already been tested, followed by adding a git note once testing is completed. This is something I intend to implement in future. That said – we would also need to add logic to determine whether a given job failure was due to test failures (no rerun needed) or infrastructural/intermittent failures (rerun desired).
(My tracking branch approach could be modified to update the tracking branch only AFTER a successful test run, but that still requires adding logic to differentiate test failures vs infrastructure failures – so I haven’t bothered to do that yet.)
Example sync & compare job
If you are using an existing Github project as source of truth, you can reuse some existing Github Actions to do pretty much everything you need – my example is almost a 1:1 copy of the Fork-Sync-With-Upstream action’s example:
jobs:
sync-with-upstream:
runs-on: ubuntu-latest
outputs:
synced_changes: steps.sync.outputs.has_new_commits # So other jobs know if anything changed
steps:
- name: Checkout next
uses: actions/checkout@v2
with:
ref: your_forks_tracking_branch
# token: can be added here if you want actions to run on push (by
# default, this step uses GITHUB_TOKEN, and therefore no actions are run on
# push). Also, using the default token means workflow changes are blocked,
# which simply forces you review and push workflow changes manually. (Yes,
# the latter means you'll need to manually trigger your custom jobs
# after pushing the reviewed workflow changes.)
- name: Pull upstream changes
id: sync
uses: aormsby/Fork-Sync-With-Upstream-action@v2.3
with:
upstream_repository: git/git # The source of truth
upstream_branch: upstream_branch_you_want_to_test
target_branch: your_forks_tracking_branch
git_pull_args: --ff-only # Might not work for all projects
Running your custom tests
This part will be largely specific to the project you are working with – in any case you’ll want to make sure you don’t run unless changes have been found:
regular:
# Only run after sync is complete, and only if changes exist
needs: sync-with-upstream
if: needs.sync-with-upstream.outputs.synced_changes
runs-on: # Whatever you want to run on
steps:
- name: Checkout Branch
uses: actions/checkout@v2
with:
ref: your_forks_tracking_branch
- # Your project-specific steps...
Alternative Approaches
My original idea was to:
- Write a custom workflow to run the tests I want.
- Write a scheduled workflow to sync an updated copy of the upstream branch being tracked to my fork.
My hope was that it would be possible to run the workflows from 1 against the branch being pushed in 2. Unfortunately that isn’t possible right now: Github will only run actions on push if those actions exist in the ref being pushed. If you tracking branch is a clean mirror of the upstream branch, then your custom workflows by definition won’t be available in that branch, and hence won’t run.
This can be worked around by adding your workflows on top of your tracking branch. But that’s messy: instead of a fast forward, you’ll always have to rebase your branch. Also, by default, Github actions run with the GITHUB_TOKEN and therefore don’t trigger workflows on push – this is something that can be overridden with a custom token, but it’s still something that needs to be taken into account.