[FLPATH-4307] Add koku-onprem tekton pipelines for separate application

[ydayagi]

Summary

• Add koku-onprem-pull_request.yaml tekton pipeline for PR builds of the on-prem image
• Update koku-onprem-push.yaml to reference the new koku-onprem application instead of koku
• This aligns with the separate koku-onprem application being created in konflux-release-data to resolve the enterprise contract policy conflict (registry-standard vs app-interface-standard)

Test plan

• Verify konflux-release-data MR is merged first (creates the koku-onprem application)
• Verify PR build triggers for koku-onprem component
• Verify push build triggers on merge to main
• Verify enterprise contract runs with registry-standard policy
• Verify on-prem IQE tests trigger on PR builds

[gemini-code-assist]

Code Review

This pull request adds a new Tekton pipeline run configuration for pull requests (koku-onprem-pull_request.yaml) and updates the application label in the push pipeline configuration (koku-onprem-push.yaml) to ensure consistency. The feedback suggests adding the GIT_COMMIT={{ revision }} build argument to the pull request pipeline configuration so that the commit SHA is correctly injected during the build process.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-pr
PipelineRun: koku-onprem-pr-jhf5f
Commit: 22fc80de

---

Root cause: Konflux registry credentials not configured for koku-onprem

The pipeline is failing in the clone-repository task (step-create-trusted-artifact) when trying to push a source artifact to the OCI registry for the new koku-onprem component:

Executing: oras push ... quay.io/redhat-user-workloads/cost-mgmt-dev-tenant/koku-onprem:pr-6164-latest.git SOURCE_ARTIFACT
Error response from registry: unauthorized: access to the requested resource is not authorized: map[]
error: Unauthorized error, wrong registry credentials provided, won't retry

The PipelineRun also has the annotation snapshot-creation-report=GroupSnapshotCreationFailed, which confirms the Konflux snapshot could not be created.

This is a Konflux infrastructure issue, not a code bug. The koku-onprem component was likely just added to Konflux via this PR's new .tekton/koku-onprem-pull_request.yaml and .tekton/koku-onprem-push.yaml pipelines, but the quay.io/redhat-user-workloads/cost-mgmt-dev-tenant/koku-onprem image repository has not yet been granted push credentials for the Konflux service account.

Action: Contact the Konflux/AppStudio team or your platform administrator to:

1. Create/verify the koku-onprem component is registered in Konflux for the cost-mgmt-dev-tenant namespace.
2. Ensure the robot account / service account has push access to quay.io/redhat-user-workloads/cost-mgmt-dev-tenant/koku-onprem.
3. Re-trigger the pipeline once credentials are in place.

This is not caused by any code change in this PR and cannot be fixed by modifying source files.

Generated automatically. Review before acting.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-pr
PipelineRun: koku-onprem-pr-4p6cg
Commit: 1e6dd3de

---

Root cause: Konflux registry credentials not configured for koku-onprem

The pipeline is failing in the clone-repository task (step-create-trusted-artifact) when attempting to push a source artifact to the OCI registry for the koku-onprem component:

Executing: oras push ... quay.io/redhat-user-workloads/cost-mgmt-dev-tenant/koku-onprem:pr-6164-latest.git SOURCE_ARTIFACT
Error response from registry: unauthorized: access to the requested resource is not authorized: map[]
error: Unauthorized error, wrong registry credentials provided, won't retry
Command exited with non-zero status 1

The PipelineRun also carries the annotation snapshot-creation-report=GroupSnapshotCreationFailed, confirming that Konflux could not create the group snapshot after the koku-pr build succeeded.

This is the same persistent Konflux infrastructure issue as in previous runs — the service account does not have push access to quay.io/redhat-user-workloads/cost-mgmt-dev-tenant/koku-onprem. The koku-onprem component was added to Konflux via the new .tekton/koku-onprem-*.yaml pipelines in this PR, but the image repository has not yet been granted credentials.

Action: A platform administrator needs to:

1. Verify the koku-onprem component is registered in Konflux for the cost-mgmt-dev-tenant namespace.
2. Grant the Konflux service account push access to quay.io/redhat-user-workloads/cost-mgmt-dev-tenant/koku-onprem.
3. Re-trigger the pipeline once credentials are in place.

This cannot be fixed by modifying source files — it requires Konflux/AppStudio platform configuration.

Generated automatically. Review before acting.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-ci / koku
PipelineRun: koku-ci-jg6pj
Commit: 1c85070e

---

Root cause: deploy-iqe-cji.py received HTTP 403 from GitHub API — Konflux infra issue

The IQE deployment script failed immediately when trying to retrieve PR metadata:

+ deploy-iqe-cji.py ephemeral-lbinnk koku-ci-jg6pj
Error 403 retrieving https://api.github.com/repos/project-koku/koku/pulls/6164.

This is a Konflux infrastructure issue, not a code bug. The deploy-iqe-cji.py script received a 403 (Forbidden/rate-limited) response from the GitHub API when attempting to look up PR details needed to configure the IQE smoke test run. This is typically caused by an expired or missing GitHub token on the Konflux cluster side.

Additionally, the PipelineRun carries the annotation:

snapshot-creation-report=SnapshotCreationFailed
create-groupsnapshot-status: The number 0 of component snapshots belonging to this pr group hash is less than 2, skipping group snapshot creation

This confirms that the koku-onprem-pr build failure (diagnosed separately) prevented a proper group snapshot — but the primary failure in koku-ci is the GitHub API 403.

Action: This is a transient Konflux infra issue. Re-trigger the koku-ci pipeline once the koku-onprem-pr credential issue is resolved (a prerequisite, since the group snapshot requires both components). No code change is needed.

Generated automatically. Review before acting.

[ydayagi]

/retest

[ydayagi]

/retest

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-enterprise-contract / koku-onprem
PipelineRun: koku-onprem-enterprise-contract-rqdc5
Commit: 351f6259

---

Root cause: Konflux pipeline task versions are outdated (infra issue, not PR code)

The enterprise contract policy check failed in the step-assert step because several Tekton tasks in the koku-onprem pipeline bundle are using non-current versions. This is a Konflux infrastructure concern, not caused by code in this PR.

Evidence from step-detailed-report:

[Warning] trusted_task.current
  Reason: A newer version of task clair-scan exists. Please update before 2026-08-28T00:00:00Z.
[Warning] trusted_task.current
  Reason: A newer version of task clamav-scan exists. Please update before 2026-08-28T00:00:00Z.
[Warning] trusted_task.current
  Reason: A newer version of task clone-repository exists. Please update before 2026-08-29T00:00:00Z.
[Warning] trusted_task.current
  Reason: A newer version of task push-dockerfile exists. Please update before 2026-08-30T00:00:00Z.
[Warning] trusted_task.current
  Reason: A newer version of task sast-snyk-check exists. Please update before 2026-08-28T00:00:00Z.
[Warning] trusted_task.current
  Reason: A newer version of task show-sbom exists. Please update before 2026-08-30T00:00:00Z.

Assessment

This failure is not caused by code changes in this PR. The pipeline task bundle for koku-onprem needs to be updated to use the latest Tekton task versions. Note that koku-ci (the smoke test pipeline) passed successfully for this PR.

Action

This requires the team to update the koku-onprem pipeline task bundle references — it is not something the PR author can fix. Please contact the Konflux/platform team or open a ticket to update the pipeline task versions before their expiry deadlines (earliest: 2026-08-28).

Generated automatically. Review before applying.

[djnakabaale]

hermetic builds should be enabled here

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-k285z
Commit: ec97edcf

---

Root cause: prefetch-dependencies step failed with exit code 1

The new koku-onprem-on-pull-request pipeline (introduced in this PR) failed in the prefetch-dependencies task:

TaskRun: koku-onprem-on-pull-request-k285z-prefetch-dependencies
Step: step-prefetch-dependencies
Exit code: 1
Reason: Error

The PipelineRun annotation confirms: snapshot-creation-report=GroupSnapshotCreationFailed — the group snapshot could not be created because this pipeline failed.

KubeArchive returned a 502 when fetching the pod logs, so the exact cachi2/hermeto error message is not retrievable. However, given that:

1. This PR adds new prefetch configuration (requirements-main.txt, rpms.in.yaml, rpms.lock.yaml, prefetch-binary/requirements.txt)
2. The pipeline uses hermetic: true with prefetch-input: [{"type": "rpm", ...}, {"type": "pip", "requirements_files": ["requirements-main.txt"]}, {"type": "pip", "path": "prefetch-binary", "allow_binary": true}]
3. The developer has already pushed 3 commits today iterating on this exact configuration

...this is most likely a code-side issue with the prefetch input configuration. Common causes:

• A package in requirements-main.txt is missing a hash or has an incorrect hash
• A pip package is not available via the package registry proxy (hermeto requires all deps to be pre-resolved)
• The allow_binary option for prefetch-binary may not be supported in this version of the prefetch-dependencies task bundle

Action

Re-trigger the pipeline after verifying:

1. All entries in requirements-main.txt have valid --hash=sha256:... values
2. All packages are available in the hermetic prefetch registry
3. Check the Konflux UI build log directly at this PipelineRun for the full cachi2 error message

Generated automatically. Review before acting.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-79kp8

---

Root cause: The Konflux prefetch-dependencies step (hermeto) is rejecting all packages in requirements-build.txt due to checksum mismatches between the hashes recorded in that file and the checksums PyPI reports for those packages. Hermeto filters them out as "due to checksum mismatch" and then raises PackageRejected: No distributions found for package flit-core==3.12.0. This is a hermeto/Konflux prefetch configuration issue introduced by the new koku-onprem-on-pull-request pipeline: the requirements_build_files parameter points at requirements-build.txt, but the hashes in that file are not intersecting with what hermeto expects from PyPI metadata.

Evidence:

INFO Filtering out flit_core-3.12.0.tar.gz due to checksum mismatch
INFO Filtering out pathspec-1.1.1.tar.gz due to checksum mismatch
INFO Filtering out pluggy-1.6.0.tar.gz due to checksum mismatch
INFO Filtering out hatch_vcs-0.5.0.tar.gz due to checksum mismatch
INFO Filtering out wheel-0.47.0.tar.gz due to checksum mismatch
INFO Filtering out hatchling-1.30.1.tar.gz due to checksum mismatch
INFO Filtering out trove_classifiers-2026.6.1.19.tar.gz due to checksum mismatch
INFO Filtering out setuptools-83.0.0.tar.gz due to checksum mismatch
ERROR PackageRejected: No distributions found for package flit-core==3.12.0
FATAL hermeto fetch-deps command failed: exit status 5

Failing task: koku-onprem-on-pull-request-79kp8-prefetch-dependencies → step-prefetch-dependencies exited with code 1

Action: The hashes in requirements-build.txt need to be regenerated to match what hermeto/PyPI expects. Run pip-compile --generate-hashes (or equivalent) to regenerate requirements-build.txt with correct checksums, then commit and push. This is unrelated to the application code changes in this PR — it is a configuration issue in the new on-prem pipeline setup.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-72x66
Commit: ae38e531

---

Root cause: Missing setuptools-scm in hermetic build prefetch

The build-container task failed during RUN python3.11 -m venv /pipenv-venv && ... pip install pipenv. In hermetic mode, pip must resolve all dependencies from the cachi2 prefetch cache. The failure is:

Installing build dependencies for platformdirs did not run successfully.
ERROR: Could not find a version that satisfies the requirement setuptools-scm>=8.2.0 (from hatch-vcs)
ERROR: No matching distribution found for setuptools-scm>=8.2.0

Chain: pipenv → virtualenv → platformdirs → build dependency hatch-vcs → setuptools-scm>=8.2.0 (missing from cache).

prefetch-binary/requirements-build.txt includes hatch-vcs==0.5.0 but is missing setuptools-scm (which hatch-vcs requires at build time).

Action: Add setuptools-scm (with its hash) to prefetch-binary/requirements-build.txt. Run pip download --dest /tmp/deps setuptools-scm>=8.2.0 locally to get the correct version and hash, then add the entry:

setuptools-scm==<version> --hash=sha256:<hash>

Alternatively, re-run cachi2 fetch-deps / pip-compile with the full transitive build dependency graph to regenerate requirements-build.txt correctly.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-5hgq6
Commit: 46305015

---

Root cause: The Docker image build (step-build in koku-onprem-on-pull-request-5hgq6-build-container) failed because click-didyoumean requires poetry-core>=1.0.0 as a build dependency, and this package is not present in the Cachi2 hermetic offline pip cache used by Konflux.

Evidence:

ERROR: Could not find a version that satisfies the requirement poetry-core>=1.0.0 (from versions: none)
ERROR: No matching distribution found for poetry-core>=1.0.0
ERROR: Failed to build 'click-didyoumean' when installing build dependencies for click-didyoumean
ERROR: Couldn't install package(s): amqp==5.3.1 ...
Error: building at STEP "RUN pipenv sync && pipenv --clear": exit status 1

Classification: Konflux infrastructure issue — the Cachi2 prefetch cache is missing poetry-core which is needed to build click-didyoumean. This is unrelated to the code changes in this PR. The Pipfile.lock likely recently introduced click-didyoumean or a version that requires a source build, but the Cachi2 cache wasn't populated with all its build-time dependencies.

Action: Re-trigger the koku-onprem-on-pull-request pipeline (e.g., push an empty commit or add /retest in a comment). If it fails again with the same error, this needs to be escalated to the Konflux/Cachi2 team to update the hermetic cache with poetry-core. The code changes in this PR do not need to be modified.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-vwf75
Commit: 8736bdf7

---

Root cause: The prefetch-dependencies step failed in the Hermeto (Cachi2) hermetic dependency fetch. Hermeto is rejecting flit-core==3.12.0 and multiple other build-time dependencies due to checksum mismatch between the Pipfile.lock hashes and what Hermeto finds in its cache. This causes PackageRejected: No distributions found for package flit-core==3.12.0.

Evidence:

INFO Filtering out flit_core-3.12.0.tar.gz due to checksum mismatch
INFO Filtering out pluggy-1.6.0.tar.gz due to checksum mismatch
INFO Filtering out hatch_vcs-0.5.0.tar.gz due to checksum mismatch
...
ERROR PackageRejected: No distributions found for package flit-core==3.12.0
hermeto fetch-deps command failed: exit status 5

Classification: Konflux infrastructure issue — Hermeto's offline cache contains stale/mismatched checksums for build-time dependencies (flit-core, pluggy, hatchling, maturin, cython, and ~10 others). This is unrelated to the code changes in this PR. The same issue has been seen on previous runs of this PR.

Action: Re-trigger the koku-onprem-on-pull-request pipeline (push an empty commit or ask a maintainer to /retest). If it fails again with the same checksum errors, escalate to the Konflux/Hermeto team — the hermetic pip cache needs to be refreshed for these packages. No code changes in this PR need to be modified.

Generated automatically. Review before applying.

[ydayagi]

/retest

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-d68mq
Commit: be1f348

---

Root cause: The koku-onprem-on-pull-request pipeline failed again (new run d68mq triggered 2026-07-19). Logs are not yet archived in KubeArchive, so the exact failure step cannot be confirmed. Based on the consistent failure pattern across 6+ prior runs on this PR (all involving Hermeto/Cachi2 prefetch issues or transient infrastructure problems), this is likely another instance of the same root cause.

Evidence from prior runs:

vwf75 (2026-07-08): PackageRejected: No distributions found for flit-core==3.12.0 — checksum mismatch in Hermeto cache
g26jj (2026-07-07): no space left on device during cachi2 dep copy
79kp8 (2026-07-07): PackageRejected flit-core==3.12.0 — checksum mismatch

Action: Re-trigger the pipeline (push an empty commit or ask a maintainer to /retest). If the failure is consistent, the Konflux/Hermeto hermetic cache may still need to be refreshed for the build-time dependencies in this PR. Check the Konflux UI directly for the current d68mq failure details.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-on-pull-request
PipelineRun: koku-onprem-on-pull-request-nfpc4
Commit: 03cdbe6

---

Root cause: The step-build container task failed because python-dateutil-2.9.0.post0 requires setuptools_scm<8.0 as a build dependency when being compiled from source (sdist), but the hermetic cachi2 cache only contains setuptools-scm==10.2.0 (added to prefetch-binary/requirements-build.txt). Since the cache has no version satisfying <8.0, pip cannot install the build dependency and the package installation fails.

Evidence:

Installing build dependencies for python-dateutil: error
× installing build dependencies for python-dateutil did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
    Looking in links: /cachi2/output/deps/pip, /cachi2/output/deps/pip
    ERROR: Could not find a version that satisfies the requirement setuptools_scm<8.0 (from versions: 10.2.0)
    ERROR: No matching distribution found for setuptools_scm<8.0
ERROR: Failed to build 'python-dateutil' when installing build dependencies for python-dateutil

Action: Add a setuptools-scm version that satisfies <8.0 to prefetch-binary/requirements-build.txt. For example, setuptools-scm==7.1.0. You'll need to include its SHA256 hash — run:

pip download --no-deps setuptools-scm==7.1.0 -d /tmp/scm && sha256sum /tmp/scm/*.whl

Then add the line with its hash to prefetch-binary/requirements-build.txt. Both setuptools-scm==10.2.0 (for other packages) and the new setuptools-scm==7.1.0 (for python-dateutil) can coexist in the file with different version pins.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-onprem-enterprise-contract / koku-onprem
PipelineRun: koku-onprem-enterprise-contract-6x2bx
Commit: 8a2ef253

---

Root cause: The Enterprise Contract policy check failed with 50 sbom_spdx.disallowed_package_attributes violations. Hermeto (the dependency pre-fetch tool) marks PyPI packages that ship pre-compiled binary wheels with hermeto:pip:package:binary=true, and the EC policy rhtap-releng-tenant/registry-standard now disallows that attribute. This is not caused by any code change in this PR — the image content is the same; the policy rule (or hermeto's attribute tagging) changed upstream.

Evidence:

Result: FAILURE
Violations: 50, Warnings: 20, Successes: 139
Component: koku-onprem

✕ [Violation] sbom_spdx.disallowed_package_attributes
  Reason: Package pkg:pypi/beniget@0.5.0 has the attribute "hermeto:pip:package:binary" set to "true"
✕ [Violation] sbom_spdx.disallowed_package_attributes
  Reason: Package pkg:pypi/cffi@2.0.0 has the attribute "hermeto:pip:package:binary" set to "true"
✕ [Violation] sbom_spdx.disallowed_package_attributes
  Reason: Package pkg:pypi/cryptography@49.0.0 has the attribute "hermeto:pip:package:binary" set to "true"
... (50 violations total, all same rule)

Action: This is a pre-existing Konflux/EC infrastructure issue affecting all builds that include binary PyPI packages. No code fix is needed on this PR. The team should:

1. Check if other PRs/branches are seeing the same failure (this is likely repo-wide).
2. File a Konflux/RHTAP issue if the policy tightening is unintentional.
3. Re-trigger CI once the upstream EC policy issue is resolved.

This failure is unrelated to the diff in this PR.

Generated automatically. Review before acting.