Currently we only have pytest. This doesn't cover making sure the build targets and final build actually run.
These should be shell scripts that run the make targets, maybe the tests, and somehow the program when run from the zipapp binary and from within the repository. These might just have to be sanity checks.
Last night I scheduled two Jenkins builds to run. It managed to deadlock and hang because one build uses both Linux and Win10, and instead of the first build using both, one build got Linux, one got Win10, and both builds waited for each other to be done.
Somehow this hung. I have no clue why, it would seem to me like once either were finished they could swap builds, but I guess not.
There's also an annoying Jenkins bug where on Windows it won't or can't log Unicode properly. This doesn't amuse me.
I'm going to look at other CI systems just to see if things are better.
I'm starting to need integration tests so it's about time to plan them.
My idea is that we have a client/server architecture or something where one runs programs sent to it and returns the result and the other handles sending tests and checking if the result is correct. This is kind of needed because some targets such as microcontrollers aren't big enough to do all this on their own and might only be able to handle loading and interpreting one small program at a time. Larger systems that run Linux likely won't need this.
So at minimum we'll need:
It kind of makes me wonder what exactly this should be. Starting with something that runs just for tests is a good idea, but such an interface would be useful for debugging later and probably interactive development where you'd like to parse or update code.
So two concrete situations are:
The host would always have test examples and expected results on it.
Writing something like this seems a bit daunting, so it's probably best I split it like this:
Okay, now that I have that grand plan out of system let's try and cut it down to only stuff we need now:
I imagine the search code will be the hardest to test since it touches the filesystem, not sure though.
I've added a Jenkinsfile and similar utilities to build and test the project. So this might actually be done, it depends on whether there's a better CI system. I'll have to investigate: