Harbor is not fetching the latest digest of particular tag when using harbor proxy with kyverno for image verification.

[Thrinadh-Kumpatla]

How can we help you?

Hi,
We have a workflow where we are using harbor proxy proxied with dockerhub. harbor mutating webhook rewrite the image tag in such a way it pulls from harbor instead of docker.
we do image signing via github actions workflow using cosign.
We run kyverno clusterPolicy in order to verify if image has signature or not.

Let's say I have a particular tag with 4 different digests with latest digest in docker and their associated signatures. When harbor and kyverno is present, kyverno is always pulling the old digest instead of new digest even with pullPolicy set to Always and that's not the case when I removed harbor and instead fetch the image directly from dockerhub. Did anyone faced this issue?

Attached kyverno policy for reference

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image-keyless-thrinadh
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: check-image-keyless-thrinadh
      match:
        any:
        - resources:
            kinds:
              - Pod
              - Deployment
      verifyImages:
      - imageReferences:
        - "hub.company.com/dockerhub-proxy/thrinadh8898/sigstore-keyless-max-testing*"  
        mutateDigest: true 
        required: true
        useCache: false
        verifyDigest: true 
        attestors:
        - entries:
          - keyless:
              subjectRegExp: https://github\.com/.+
              issuerRegExp: https://token\.actions\.githubusercontent.+
              rekor:
                url: https://rekor.sigstore.dev

When I set

  mutateDigest: false 
  required: true
  useCache: false
  verifyDigest: false 

It's pulling the latest which is fine, But what I noticed is that kyverno is allowing the same tag with a digest without signature which shouldn't happen.

[stonezdj]

Can you please describe how you We run kyverno clusterPolicy in order to verify if image has signature or not.? maybe the verification based on accessory API, because proxy cache doesn't implement the accessory API.

[MinerYang]

#20808

[Thrinadh-Kumpatla]

I've leverage this and using the above clusterPolicy I've shared in the parent message. Is this what you're looking for?

[Thrinadh-Kumpatla]

harbor-core-775bb4bd86-rj9zq core 2025-01-27T20:47:08Z [WARNING] [/server/middleware/repoproxy/proxy.go:207]: Artifact: dockerhub-proxy/thrinadh8898/sigstore-keyless-max-testing:sha256-ea5e1020cfb702f16aac41be80b79048174efba8052cd66a3c1b8b88b63b128f.sig, digest: is not found in proxy cache, fetch it from remote repo

the above digest in the log is the latest version but I'm still seeing the old digest image on pod like below

    Image:          hub. company.com/dockerhub-proxy/thrinadh8898/sigstore-keyless-max-testing:pr-8@sha256:240f6bb73d937061ee7bae78af542fe93cd413631ee2ffaffb20d7652c52b67b
    Image ID:       hub. company.com/dockerhub-proxy/thrinadh8898/sigstore-keyless-max-testing@sha256:240f6bb73d937061ee7bae78af542fe93cd413631ee2ffaffb20d7652c52b67b

[Thrinadh-Kumpatla]

@stonezdj Did you get a chance to look into this?

[stonezdj]

Can you please try this command to verify the tag is pullable in Harbor?

curl -k -v -u '<username>:<password>'  https://hub. company.com/v2/dockerhub-proxy/thrinadh8898%2Fsigstore-keyless-max-testing/manifests/sha256-ea5e1020cfb702f16aac41be80b79048174efba8052cd66a3c1b8b88b63b128f.sig

Because keyless image sign and verification is based on cosign, cosign can sign and verify the image pulled through Harbor proxy cache.