Run CLI workflow on cron schedule?

[JonJagger]

In a Workflow run for a branch on the CLI I got an unexpected test failure.
See https://github.com/kosli-dev/cli/actions/runs/12690361576/job/35371065255#step:8:534

This appears to be because there is an attempt to cast nil in the get-flow command.
See https://github.com/kosli-dev/cli/blob/main/cmd/kosli/getFlow.go#L92

I was unable to reproduce this error locally.
I've added an if around the case and the tests now all pass.
The branch is called add-more-debug-http-rety-logging

I'm assuming the problem here is change in the server that the CLI tests run against.
This change in the server could have been many many days ago.
Should we run a CLI workflow every day (say) to pick up these kind of errors?
That way when anyone does some work on the CLI they will hopefully only
get errors related to the new work they are doing.

[sami-alajrami]

currently, the scheduled CI runs the all normal build jobs (including kosli trail creation, build, test and docker image building).

I feel that we don't necessarily need a docker image nor the kosli trails/attestation duplication. There is also wasted CI minutes, snyk credits etc.

If we just want to run tests daily, then the scheduled workflow should just do that. Additionally, if it fails, it goes unnoticed, so we need a way to be alerted if tests are broken.

[FayeSGW]

@sami-alajrami
I spoke to Jon about whether or not we should run the snyk tests and he thought we should. We do daily snyk scans of the running server artifacts, so does it not make sense to do it for the CLI too?
I agree about building a new Docker image each time though, that takes a while and we should just use the current image.

I'm a little confused regarding your trails/attestations point - are you suggesting we don't attest the test results to Kosli at all? (Also we don't actually have any trail duplication at the moment - the results get attested to the trail of the latest commit, which is possibly not what we want either now that I think about it).
[Oh, wait - do you mean we shouldn't duplicate e.g. the pr and never-alone attestations for the trail? That makes sense.]

Good point with the notification - I guess a slack notification like we have for the secret check would work?

[sami-alajrami]

@sami-alajrami I spoke to Jon about whether or not we should run the snyk tests and he thought we should. We do daily snyk scans of the running server artifacts, so does it not make sense to do it for the CLI too?

the difference between server artifacts and CLI is that for artifacts we scan what's running, while on CLI we scan the latest commit. Given that we have limits on snyk usage and that snyk scan is done anyway on each commit and on releases, I don't see much value from doing it everyday.

I'm a little confused regarding your trails/attestations point - are you suggesting we don't attest the test results to Kosli at all? (Also we don't actually have any trail duplication at the moment - the results get attested to the trail of the latest commit, which is possibly not what we want either now that I think about it). [Oh, wait - do you mean we shouldn't duplicate e.g. the pr and never-alone attestations for the trail? That makes sense.]

I don't think we need to record anything to Kosli when we run the tests daily since that's not related to any control we need to check as far as I know. If we want to track anything in Kosli, however, that would be the fact that we are doing the daily testing routine (similar to the daily snyk scan trails). But that should be in a separate flow which only tracks these daily re-runs. The trails in that flow could have a junit attestation for the commit that is being tested, and snyk attestation for the same commit if we insist on doing it.

Good point with the notification - I guess a slack notification like we have for the secret check would work?
yes

[FayeSGW]

the difference between server artifacts and CLI is that for artifacts we scan what's running, while on CLI we scan the latest commit. Given that we have limits on snyk usage and that snyk scan is done anyway on each commit and on releases, I don't see much value from doing it everyday.

Hmm. I see your point, but we frequently go for multiple days (and even weeks) with nothing being pushed to the CLI repo. We could easily have a vulnerability sitting there for quite some time before we find it, which seems less than ideal. Might be worth a quick discussion!
@JonJagger , what do you think?

I don't think we need to record anything to Kosli when we run the tests daily since that's not related to any control we need to check as far as I know. If we want to track anything in Kosli, however, that would be the fact that we are doing the daily testing routine (similar to the daily snyk scan trails). But that should be in a separate flow which only tracks these daily re-runs. The trails in that flow could have a junit attestation for the commit that is being tested, and snyk attestation for the same commit if we insist on doing it.

Yeah, this makes sense.

Good point with the notification - I guess a slack notification like we have for the secret check would work?
yes
Turns out we actually already have this - if the tests fail in the CI we get a notification in the channel ci-failures. I see no reason that wouldn't also work for a scheduled run. So that's one thing less to worry about.

[JonJagger]

There have been periods when we've not done commits on the cli repo for quite some time.
(Much more so than on the server.)
So I guess it mostly depends on how painful it is to fix a bunch of accumulated snyk vulns.
But I'm totally happy to go with what Sami decides.