The new regression tests actually caught a failure. Today I got an email notification from github:
This is the daily scheduled run on actual data that we documented earlier in response to a silent fail in the data store. Clicking through into the github actions dash we see that it had been working until the most recent one failed:
At least we caught it fresh this time. Clicking through we can actually see the full output. It’s failing on the final md5 check of the results. However the results don’t look obviously corrupt:
This is the limit of what the automation can tell us: It knows that this output, whatever it is, doesn’t match the magic hash it has stored to compare it to. Fortunately we have those other stored runs that got the green light, so we can click through and copy/paste/save one of the good as well as the bad in separate text files.
$ diff -u succeed_output.txt fail_output.txt --- succeed_output.txt 2022-07-06 10:59:06.000000000 -0700 +++ fail_output.txt 2022-07-06 10:58:36.000000000 -0700 @@ -56,33 +56,33 @@ 37 1998, 11, UCLA, 3-0 38 -1999, 10, Stanford, 3-1 +1999, 7, Stanford, 3-0 39 -2000, 11, Fresno State, 2-1 +2000, 6, Stanford, 2-1 40 -2001, 9, Stanford, 4-0 +2001, 6, Stanford, 3-0 41 -2002, 10, USC, 3-0 +2002, 7, USC, 3-0 42 -2003, 9, California, 2-1 +2003, 7, California, 2-1 43 -2004, 9, USC, 3-0 +2004, 7, USC, 3-0 44 -2005, 10, USC, 4-0 +2005, 7, USC, 3-0 45 -2006, 9, San José State, 3-0 +2006, 6, USC, 2-1 46 -2007, 8, Stanford, 3-1 +2007, 6, Stanford, 2-1 47 -2008, 10, USC, 3-0 +2008, 6, USC, 3-0 48 -2009, 10, Stanford, 3-1 +2009, 7, Stanford, 2-1 49 -2010, 7, Stanford, 3-0 +2010, 6, Stanford, 3-0 50 -2011, 11, Stanford, 4-0 +2011, 6, Stanford, 3-0 51 -2012, 10, Stanford, 5-0 +2012, 8, Stanford, 4-0 52 2013, 10, Stanford, 3-1 53
This at least tells us that something is going on with the years 1999-2012. All those years have fewer MCC games than they should. One big clue is that San Jose State was the 2006 winner in a 9 game schedule and now the code is reporting a 6 game schedule with USC the winner. Going to the collegefootballdata.com API test we can plug in 2006 to the /teams/fbs endpoint and check that San Jose State is not listed there any more. So there’s your problem right there: Somehow the San Jose State years of 1999-2012 are not being marked as FBS and thus all those games get dropped out by our FBS check. (It looks like the whole WAC got improperly demoted.)
I was able to report the bug to the maintainers within a day of it being introduced. This was a pretty good outcome. It still took some human intervention to figure out the exact source. One could imagine a cascade of detailed year-by-year runs stored in the tests so that we could have seen the exact diff showing missing San Jose State games in the log. But for now this is a pretty nice payoff considering how easy it was to wire up the one github action and have it run every day.