Job stil running after remove Cronhpa CR

[yccharles]

1. delete CR my-test-cronhpa-cn-1 at 12:18:05.095674
2. but job still running at 19:20:01.004750.

why Cronmanager gc resubmit the job back ?

		if !cm.cronExecutor.FindJob(job) {
			log.Warningf("Failed to found job %s of cronHPA %s in %s in cron engine and resubmit the job.", job.Name(), hpa.Name, hpa.Namespace)
			cm.cronExecutor.AddJob(job)
			continue
		} else {

[yccharles]

@ringtail need your help

[ringtail]

@IrisIris Pls look at this issue.

[yccharles]

I can reproduce this problem by add time.Sleep() when get cm.jobQueue after cm.Unlock() in GC function.

// GC will collect all jobs which ref is not exists and recycle.
func (cm *CronManager) GC() {
	log.Infof("Start GC")
	m := make(map[string]CronJob)
	cm.Lock()
	for k, v := range cm.jobQueue {
		m[k] = v
	}
	cm.Unlock()
	log.Infof("Im beging sleeping. ")
	time.Sleep(15 * time.Second)     
	current := len(cm.jobQueue)
	log.V(2).Infof("Current active jobs: %d,try to clean up the abandon ones.", current)

       .........
}

how to reproduce

1. add time sleep.
2. delete Cronhpa CR when gc is sleeping .
3. you will see log " Failed to find job scale-down of cronHPA cronhpa-sample in monitoring in cron engine and resubmit the job."

[IrisIris]

I can reproduce this problem by add time.Sleep() when get cm.jobQueue after cm.Unlock() in GC function.

// GC will collect all jobs which ref is not exists and recycle.
func (cm *CronManager) GC() {
	log.Infof("Start GC")
	m := make(map[string]CronJob)
	cm.Lock()
	for k, v := range cm.jobQueue {
		m[k] = v
	}
	cm.Unlock()
	log.Infof("Im beging sleeping. ")
	time.Sleep(15 * time.Second)     
	current := len(cm.jobQueue)
	log.V(2).Infof("Current active jobs: %d,try to clean up the abandon ones.", current)

       .........
}

how to reproduce

1. add time sleep.
2. delete Cronhpa CR when gc is sleeping .
3. you will see log " Failed to find job scale-down of cronHPA cronhpa-sample in monitoring in cron engine and resubmit the job."

The reproducing process is not ok; u changed the code and made jobQueue is not right in your code. But in the lasted code the removed jod id should be removed in the jobQueue at the same time.

[IrisIris]

1. delete CR my-test-cronhpa-cn-1 at 12:18:05.095674
2. but job still running at 19:20:01.004750.

why Cronmanager gc resubmit the job back ?

		if !cm.cronExecutor.FindJob(job) {
			log.Warningf("Failed to found job %s of cronHPA %s in %s in cron engine and resubmit the job.", job.Name(), hpa.Name, hpa.Namespace)
			cm.cronExecutor.AddJob(job)
			continue
		} else {

The code in the test is not the latest, would u provide the commit id pls?

[yccharles]

1. delete CR my-test-cronhpa-cn-1 at 12:18:05.095674
2. but job still running at 19:20:01.004750.

why Cronmanager gc resubmit the job back ?

		if !cm.cronExecutor.FindJob(job) {
			log.Warningf("Failed to found job %s of cronHPA %s in %s in cron engine and resubmit the job.", job.Name(), hpa.Name, hpa.Namespace)
			cm.cronExecutor.AddJob(job)
			continue
		} else {

The code in the test is not the latest, would u provide the commit id pls?

the cronhpa controller image tag is : v1.4.0-fc4f6060-aliyun

[ringtail]

@IrisIris any update?

[yccharles]

#120

@ringtail @IrisIris

[IrisIris]

@IrisIris any update?

#124
This pr should fix this issue, please review thx.