After getting a very basic docker container up and running it occurred to me, a lazy person, that there is probably some developer Saas out there that can take containers and regression test them against an expected output I provide.
Two weeks ago I ran the “full run” code for the first time in two months and got the nasty surprise that the CFBD project data had turned bad on us. The /teams/fbs endpoint was no longer providing valid info so all runs from the 1970s showed no games. Fortunately after our enquiries they were able to revert and fix it. (Thanks once again to the very complete and nimble cfbd project!)
The lesson of course is that code rots from all directions and you need to run regression tests regularly. Since we have a terminal-based piece of software easily harnessed to command line options there’s really no excuse. A cronjob running diff with output sent to mail would get the job done. Hypothetically…
#!/bin/sh
run_code > diff -u expected_output.txt -
if [ $? ]
then
mail -s 'you broke it' me@my_email.com
fi
I’m not checking that by the way. I don’t want to do that, I want someone else to do it for me. I asked a busy person and he replied “github actions”.
Let me document what I did before I forget: I hadn’t dealt with github actions before but it’s pretty slick. You buy into their custom yaml schema embedded in magic directories and you can trigger all kinds of cloud-hosted actions on their end. (Of course all this only works if you’re using github, but we are. Cool lock-in.) The main interesting triggers for me are push: (did my code break things?) and regular time intervals via standard crontab schedule: (did something else break things?)
I read their quickstart and committed their basic demo into my project directory. My beginner’s understanding of their world is that they’re trying to recreate a container-like ecosystem in their actions, with the yaml files doing some of the work of both Dockerfile and docker compose environments. They have ways to explicitly use containers too. But for now let’s try to play along with their system and not bolt any docker on.
After that demo gets committed I get an “Actions” tab where I can see the full results. Using their magic actions/checkout verb means that the codebase is now “there” and available inside the container that got created server side.
Let’s try to actually run something interesting. Here’s their python guide. Very nice. We can build from these instructions and get our codebase set up with the actual binaries to run it inside the container. One obvious gotcha is the required cfbd API key which we pass through right now via environment variable. Some detours into github action env variable documentation and github’s overall “secrets” system. Seems pretty straightforward, although if the API key were anything really important I’d spend a lot more time figuring out how github manages things.
So here’s my first pass:
name: Install python and run a test
on: [push]
env:
CFBD_API_KEY: ${{ secrets.CFBD_API_KEY }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.x
uses: actions/setup-python@v3
with:
# Semantic version range syntax or exact version of a Python version
python-version: '3.x'
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Install dependencies
run: python -m pip install --upgrade pip cfbd
- name: Run the code
run: python vconf/mcc_schedule.py -s 1970 -e 1980
Seems to work! We see actual MCC output in their container log in the actions console. So far so good. Now where’s the part where I get a message when it all breaks?
I had to laugh at this section in the github action python doc:
Testing your code You can use the same commands that you use locally to build and test your code.
That’s why we’re here! I’m the the lazy person trying desperately to avoid testing my code locally. It looks like I have to understand pytest after all.
So I could take the extremely janky step of just running that crude diff against a canned expected response, or I could try to build the pytest framework.
Some next step questions:
What is github action best practice for raising the alert?There are some third party plugins for arbitrary email and multichannel alerts but if you fail with a clean exit code you get an email to the maintainer address “for free.”- How easy is it to debug complex action files if you have to commit to see them work?
How easy is it to trigger the same whole flow from either date or push? I don’t want to copy-paste just because I like triggering on push and every day.You can layer triggers at the top with multiple subordinates to the “on” level.
Either way even without a formal test this kind of cloud-driven harness would have helped me to discover and trace the regression fail. Any timestamped log with a bunch of runs would be useful.
Addendum: let’s take a full swing at the hacky regression in this one yaml file. Rather than relying on diff we can get a more atomic approach by using md5 on the text output. Ubuntu’s version of md5 is called md5sum, we can verify its output by using our desktop docker to play around in the rainbow-colored mud of linux:
$ docker pull ubuntu $ docker run -i -t ubuntu /bin/bash
That lets us figure out the exact md5sum we can find in the github runtime container. The trick is how to evaluate tests and branch to different exit statuses in the github yaml itself. It’s not pretty but there’s a section in their context doc about using the “steps” context as a key/value store. Basically if you give the step a uid you can do a tortured command line echo that sets keys that can be retrieved later. After many test commits to make the action fire we’ve got something that works:
name: Install python and run a test
on:
push:
schedule:
- cron: '51 0 * * *'
env:
CFBD_API_KEY: ${{ secrets.CFBD_API_KEY }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.x
uses: actions/setup-python@v3
with:
# Semantic version range syntax or exact version of a Python version
python-version: '3.x'
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Install dependencies
run: python -m pip install --upgrade pip cfbd
- name: Run the code
run: python vconf/mcc_schedule.py -s 1970 -e 2000 > output.txt
- name: md5_results
run: /usr/bin/md5sum output.txt
- name: Check md5
id: check_md5
run: echo "::set-output name=md5_out::"`/usr/bin/md5sum output.txt`
- name: Pass or fail
run: |
if [[ "${{ steps.check_md5.outputs.md5_out }}" == "7de220da82eb846c3c806e06f5da209a output.txt" ]]; then exit 0; else exit 1; fi
We run the MCC application code as before but pipe the output to a file on the container filesystem. Then the native ubuntu md5sum output is stored to the key/value store in the step named “Check md5.” One step later it is retrieved to be passed into a classic /bin/sh “if” test. The good news is that using the shell exit codes gives us a free email from github when the test breaks.
Obviously we’ve stored a magic number here for the expected md5 output but we don’t expect this to change. If we do make an approved change to the code that changes results we’ll be reminded soon enough to change this constant.
It’s not “real” testing but it’s a nice exploration of github actions and it gets us over on the bug that bit last month: should some regression either here or in cfbd drastically change our “approved” results we will hear about it on either the next push or the next 24 hours, whichever is sooner.
10 hours spent being “lazy” and roughing out an end-to-end victory in the last war… wouldn’t have it any other way.