[SPIKE] OSAC sovereign cloud integration — usage line item table

[martinpovolny]

Summary

Spike/PoC for integrating OSAC sovereign cloud cost data into Koku's
on-prem pipeline. Adds a new self-hosted line item table for OSAC data
and wires it into the existing OCP summarization SQL.

This is a spike — not production-ready. The full design, adversarial
review, and test results are in the cost_ai_grid_poc repo:
myersCody/cost_ai_grid_poc#44

Changes

New model: OSACUsageLineItemDaily

koku/reporting/provider/ocp/self_hosted_models.py

A new self-hosted model for daily OSAC usage data (VMs, clusters, bare
metal, MaaS). Written by the koku-sync tool from the cost-event-consumer
pipeline. Columns: resource_type, resource_id, tenant_id, meter_name,
value, unit, cost_type, koku_metric, cost_amount, currency.

SQL template UNION

koku/masu/database/self_hosted_sql/openshift/reporting_ocpusagelineitem_daily_summary.sql

Conditional {% if osac_exists %} UNION block that reads from the OSAC
table and maps meters to Koku's daily summary columns (cpu core-hours,
memory GiB-hours, infrastructure cost).

DB accessor wiring

koku/masu/database/ocp_report_db_accessor.py

Adds osac_exists table check and passes it to the SQL template params.

Verified

Data flows end-to-end:

• cost-event-consumer → koku-sync → OSAC table → daily summary → UI
• Koku report API returns $0.42 infrastructure cost
• Koku UI shows "OSAC Sovereign Cloud" cluster with cost breakdown

Open Design Questions (from rate incorporation research)

Based on Cody's rate incorporation research
(docs):

1. source_type discriminator on PriceList

Recommendation: extend Koku's PriceList model with a source_type
field (operator vs event_watcher) to scope rate definitions.

Why: OSAC event-driven meters don't map 1:1 to Prometheus-based
metrics. 7 Koku metrics (usage, effective-usage, storage, GPU) are
permanently Prometheus-only and should not appear in OSAC price lists.
A source_type discriminator structurally excludes them, gives REQ-13
custom metrics a clean home, and preserves Koku's existing markup/
distribution/UI machinery.

Status: Proposed — needs design review.

2. Double-billing prevention

Problem: when both Prometheus (via koku-metrics-operator) and our
event watcher meter the same resource (VM, cluster), costs are counted
twice. The source_type discriminator alone doesn't prevent this.

Needed: a precedence mechanism that picks one source per resource,
or deduplicates at the summary layer. Not urgent for PoC (single
source), but critical for production where both pipelines may coexist.

3. Node/cluster cost allocation model

Koku's approach: ratio-based allocation — distributes node costs
proportionally across pods/projects based on resource requests.

OSAC's approach: flat per-instance accrual — each VM/cluster gets
a fixed hourly rate based on its instance type.

Decision needed: do we match Koku's allocation model for OSAC
resources, or accept the divergence? Matching would require significant
rework in the cost-event-consumer; accepting means OSAC costs won't
align with Prometheus-based OCP costs at the project level.

4. Metric-level findings (fixed)

• node_cost_per_month rates with hourly divisors → renamed to
  node_cost_per_hour (fixed in cost_ai_grid_poc)
• CPU/memory meters produce $0 when OSAC drops cores/memory_gib
  → enriched from instance_type catalog (fixed in cost_ai_grid_poc)
• vm_cost_per_hour — clean 1:1 mapping, works as-is
• cluster_worker_node_count meter — referenced in design docs but
  never emitted in code. Decision: implement or remove.

Related

• Design + results: [SPIKE] Koku integration: sync cost-event-consumer data into Koku daily summary myersCody/cost_ai_grid_poc#44
• Rate incorporation research: [SPIKE] Extension of koku integration with a focus on osac rate incorporation myersCody/cost_ai_grid_poc#53
• Integration strategy: docs/koku-integration/strategy.md in PR Add basic tox.ini file for running different envs and linting. Closes #30 #44

[gemini-code-assist]

Code Review

This pull request introduces support for OSAC Sovereign Cloud daily usage data in the self-hosted/on-prem pipeline, including a new OSACUsageLineItemDaily model, a table existence check in ocp_report_db_accessor.py, and a UNION clause in the daily summary SQL template. It also adds extensive development documentation and tooling configurations for Claude Code. The reviewer feedback highlights two important issues: first, the OSAC query in the daily summary SQL template lacks a GROUP BY aggregation, which will cause duplicate rows, and discards the cost_amount data; second, the table existence check for the OSAC table should be gated to run only in on-prem mode, as it is a PostgreSQL-only table and does not exist in Trino.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The current OSAC query does not perform any GROUP BY aggregation. Since openshift_osac_usage_line_items_daily stores daily metering data per meter (e.g., separate rows for CPU, memory, and uptime), querying it without aggregation will insert multiple separate rows for the same resource on the same day into the daily summary table. This significantly increases the row count and degrades query performance.

Additionally, the pre-calculated cost_amount from the OSAC table is currently mapped to NULL for infrastructure_usage_cost, which discards the actual cost data.

We should aggregate the metrics using SUM and group by the resource and day, while also mapping cost_amount to infrastructure_usage_cost as a JSON object.

SELECT null as uuid,
    {{report_period_id}} as report_period_id,
    {{cluster_id}} as cluster_id,
    {{cluster_alias}} as cluster_alias,
    'OSAC' as data_source,
    o.usage_start,
    o.usage_start as usage_end,
    COALESCE(o.project_id, o.tenant_id, '') as namespace,
    COALESCE(o.resource_id, '') as node,
    COALESCE(o.resource_id, '') as resource_id,
    '{}'::text as pod_labels,
    -- CPU: map vm_cpu_core_seconds to core-hours
    SUM(CASE WHEN o.meter_name IN ('vm_cpu_core_seconds')
        THEN o.value / 3600.0 END) as pod_usage_cpu_core_hours,
    SUM(CASE WHEN o.meter_name IN ('vm_cpu_core_seconds')
        THEN o.value / 3600.0 END) as pod_request_cpu_core_hours,
    SUM(CASE WHEN o.meter_name IN ('vm_cpu_core_seconds')
        THEN o.value / 3600.0 END) as pod_effective_usage_cpu_core_hours,
    NULL as pod_limit_cpu_core_hours,
    -- Memory: map vm_memory_gib_seconds to gigabyte-hours
    SUM(CASE WHEN o.meter_name IN ('vm_memory_gib_seconds')
        THEN o.value / 3600.0 END) as pod_usage_memory_gigabyte_hours,
    SUM(CASE WHEN o.meter_name IN ('vm_memory_gib_seconds')
        THEN o.value / 3600.0 END) as pod_request_memory_gigabyte_hours,
    SUM(CASE WHEN o.meter_name IN ('vm_memory_gib_seconds')
        THEN o.value / 3600.0 END) as pod_effective_usage_memory_gigabyte_hours,
    NULL as pod_limit_memory_gigabyte_hours,
    -- Node capacity (from uptime meters)
    NULL as node_capacity_cpu_cores,
    SUM(CASE WHEN o.meter_name IN ('vm_uptime_seconds', 'bm_uptime_seconds')
        THEN o.value / 3600.0 END) as node_capacity_cpu_core_hours,
    NULL as node_capacity_memory_gigabytes,
    NULL as node_capacity_memory_gigabyte_hours,
    NULL as cluster_capacity_cpu_core_hours,
    NULL as cluster_capacity_memory_gigabyte_hours,
    -- No storage data from OSAC
    NULL as persistentvolumeclaim,
    NULL as persistentvolume,
    NULL as storageclass,
    '{}'::text as volume_labels,
    NULL as persistentvolumeclaim_capacity_gigabyte,
    NULL as persistentvolumeclaim_capacity_gigabyte_months,
    NULL as volume_request_storage_gigabyte_months,
    NULL as persistentvolumeclaim_usage_gigabyte_months,
    o.source::uuid as source_uuid,
    json_build_object('raw', SUM(COALESCE(o.cost_amount, 0.0)))::text as infrastructure_usage_cost,
    NULL as csi_volume_handle,
    NULL as cost_category_id,
    o.source,
    o.year,
    o.month,
    EXTRACT(DAY FROM o.usage_start)::text as day
FROM {{schema | sqlsafe}}.openshift_osac_usage_line_items_daily o
WHERE o.source = {{source}}
    AND o.year = {{year}}
    AND lpad(o.month, 2, '0') = {{month}}
    AND o.usage_start >= {{start_date}}
    AND o.usage_start < {{end_date}} + INTERVAL '1 day'
GROUP BY o.usage_start,
    COALESCE(o.project_id, o.tenant_id, ''),
    COALESCE(o.resource_id, ''),
    o.source,
    o.year,
    o.month

[gemini-code-assist]

Checking for the existence of openshift_osac_usage_line_items_daily using trino_table_exists is inappropriate and risky because this table is a self-hosted PostgreSQL-only table (defined in self_hosted_models.py) and does not exist in Trino. In SaaS mode (where ONPREM is False), this check will unnecessarily query Trino, causing overhead.

We should gate this check so that it is only performed when running in on-prem/self-hosted mode.

Suggested change

	storage_exists = trino_table_exists(self.schema, "openshift_storage_usage_line_items_daily")
	osac_exists = trino_table_exists(self.schema, "openshift_osac_usage_line_items_daily")
	is_onprem = self.get_sql_folder_name() == "self_hosted_sql"
	storage_exists = trino_table_exists(self.schema, "openshift_storage_usage_line_items_daily")
	osac_exists = is_onprem and trino_table_exists(self.schema, "openshift_osac_usage_line_items_daily")

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Units - 3.11
Root cause: The new OSACUsageLineItemDaily model was added to koku/reporting/provider/ocp/self_hosted_models.py but the corresponding Django migration was not committed. The migration check step detected the unapplied schema change and failed.

Evidence:

+ changed=koku/reporting/migrations/0352_osacusagelineitemdaily.py
+ echo 'Migrations are out of sync with the models. Run make make-migrations to update.'
Migrations are out of sync with the models. Run 'make make-migrations' to update.
+ exit 1

Action: Run make make-migrations locally and commit the generated koku/reporting/migrations/0352_osacusagelineitemdaily.py file to this branch.

pipenv run make make-migrations
git add koku/reporting/migrations/0352_osacusagelineitemdaily.py
git commit -m "Add migration for OSACUsageLineItemDaily model"

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Units - 3.11
Commit: 553ad78

---

Root cause: The OSACUsageLineItemDaily model was added to koku/reporting/provider/ocp/self_hosted_models.py but no Django migration was generated for it. The Check migrations CI step ran make make-migrations and detected an unapplied model change — it would produce koku/reporting/migrations/0352_osacusagelineitemdaily.py.

Evidence:

+ changed=koku/reporting/migrations/0352_osacusagelineitemdaily.py
Migrations are out of sync with the models. Run 'make make-migrations' to update.

Action: Run make make-migrations locally and commit the generated migration file:

pipenv run make make-migrations
git add koku/reporting/migrations/0352_osacusagelineitemdaily.py
git commit -m "Add migration for OSACUsageLineItemDaily model"

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-ci / koku
PipelineRun: koku-ci-mqcvv
Commit: 553ad78

---

Root cause: The PipelineRun failed with SnapshotCreationFailed — a Konflux infrastructure issue. The pipeline completed 1 task and skipped 7, which indicates it failed at snapshot/group-snapshot creation before the IQE smoke tests could run. This is unrelated to the code changes in this PR.

Evidence:

test.appstudio.openshift.io/snapshot-creation-report: SnapshotCreationFailed
test.appstudio.openshift.io/create-groupsnapshot-status: The number 0 of component snapshots belonging to this pr group hash ... is less than 2, skipping group snapshot creation
Tasks Completed: 1 (Failed: 1, Cancelled 0), Skipped: 7

Action: This is a Konflux infrastructure issue. Re-trigger the pipeline by pushing a new commit or asking a maintainer to re-run the check. Note: the Units - 3.11 failure (missing migration) should be fixed first.

Generated automatically. Review before applying.

[myersCody]

Overall I really like the direction you are taking with a new table & a new data type. That feels correct to me.

I don't know much nit picking you want me to do, but I will say this section doesn't really match the cost model feasibility doc.

If you are just trying to showcase we can connect dots, then feel free to ignore this comment.

[myersCody]

We calculated the cost on the POC side, but don't seem to do anything with it on this side of the fence.

Are you just trying to insert metering information on this side and utilize the cost model to populate the costs?

If we want to maintain the cost calculated within the POC we could explore a similar flow to to back populate costs to the ocp daily summary table like what we do with cloud infrastructure costs.

[koku-ci-triager-bot]

🤖 CI Triager Suggestion

Check: Sanity
Root cause: reorder-python-imports and black require from django.db import migrations, models to be split into separate imports.

Accept the suggestion below with one click.

[koku-ci-triager-bot]

reorder-python-imports modified this file because it requires each import to be on its own line.

Suggested change

	from django.db import migrations, models
	from django.db import migrations
	from django.db import models

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Red Hat Konflux / koku-ci / koku
PipelineRun: koku-ci-cwr89
Commit: fef4808

---

Root cause: init-pipeline-context failed because the PR has the smokes-required label but no specific smoke-test label has been added to authorize a smoke test suite.

Evidence:

[INFO] Checking labels for PR: 6178
[INFO] PR Labels: smokes-required
[INFO] Missing required smoke test label with smokes-required.

Action: This PR modifies production OCP code (ocp_report_db_accessor.py, self_hosted_models.py, a new migration, and a new SQL template). Smoke tests are legitimately required. Add the ocp-smoke-tests label to trigger OCP-scoped smoke tests, which is the most appropriate label given the changes affect the OCP processing pipeline and on-prem self-hosted models.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Suggestion

Check: Units - 3.11
Commit: fef4808

---

Root cause: test_postgres_tables_are_partitioned fails because OSACUsageLineItemDaily inherits OCPLineItemBase which defines a PartitionInfo inner class. Django models with PartitionInfo must be created via a partitioned migration — the migration must call set_pg_extended_mode before and unset_pg_extended_mode after CreateModel. The migration was auto-generated without these calls, so PostgreSQL creates a plain (non-partitioned) table.

Evidence:

FAIL: test_postgres_tables_are_partitioned (...) (model=<class 'reporting.provider.ocp.self_hosted_models.OSACUsageLineItemDaily'>)
AssertionError: False is not true : Model {model.__name__} is not partitioned.
Are `set_pg_extended_mode` and `unset_pg_extended_mode` used in the model's migration?

Fix: Add the RunPython calls around CreateModel as in 0351_create_ocp_cost_breakdown_p.py.

Accept the suggestion below to apply the fix.

[koku-ci-triager-bot]

Suggested change

	# Generated by Django 5.2.15 on 2026-07-11 10:46
	import uuid
	from django.db import migrations, models
	class Migration(migrations.Migration):
	dependencies = [
	('reporting', '0351_create_ocp_cost_breakdown_p'),
	]
	operations = [
	migrations.CreateModel(
	name='OSACUsageLineItemDaily',
	fields=[
	('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
	('report_period_start', models.DateTimeField(null=True)),
	('report_period_end', models.DateTimeField(null=True)),
	('interval_start', models.DateTimeField(db_index=True, null=True)),
	('interval_end', models.DateTimeField(null=True)),
	('usage_start', models.DateField(db_index=True, null=True)),
	('source', models.CharField(db_index=True, max_length=64, null=True)),
	('year', models.CharField(max_length=4, null=True)),
	('month', models.CharField(max_length=2, null=True)),
	('manifestid', models.CharField(max_length=256, null=True)),
	('reportnumhours', models.IntegerField(null=True)),
	('resource_type', models.CharField(max_length=64, null=True)),
	('resource_id', models.CharField(max_length=256, null=True)),
	('tenant_id', models.CharField(max_length=256, null=True)),
	('project_id', models.CharField(max_length=256, null=True)),
	('meter_name', models.CharField(max_length=128, null=True)),
	('value', models.FloatField(null=True)),
	('unit', models.CharField(max_length=64, null=True)),
	('cost_type', models.CharField(max_length=32, null=True)),
	('koku_metric', models.CharField(max_length=128, null=True)),
	('cost_amount', models.FloatField(null=True)),
	('currency', models.CharField(max_length=8, null=True)),
	],
	options={
	'db_table': 'openshift_osac_usage_line_items_daily',
	'indexes': [models.Index(fields=['source', 'year', 'month'], name='osac_daily_src_yr_mo_idx')],
	},
	),
	]
	# Generated by Django 5.2.15 on 2026-07-11 10:46
	import uuid
	from django.db import migrations, models
	from koku.database import set_pg_extended_mode
	from koku.database import unset_pg_extended_mode
	class Migration(migrations.Migration):
	dependencies = [
	("reporting", "0351_create_ocp_cost_breakdown_p"),
	]
	operations = [
	migrations.RunPython(code=set_pg_extended_mode, reverse_code=unset_pg_extended_mode),
	migrations.CreateModel(
	name="OSACUsageLineItemDaily",
	fields=[
	("id", models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
	("report_period_start", models.DateTimeField(null=True)),
	("report_period_end", models.DateTimeField(null=True)),
	("interval_start", models.DateTimeField(db_index=True, null=True)),
	("interval_end", models.DateTimeField(null=True)),
	("usage_start", models.DateField(db_index=True, null=True)),
	("source", models.CharField(db_index=True, max_length=64, null=True)),
	("year", models.CharField(max_length=4, null=True)),
	("month", models.CharField(max_length=2, null=True)),
	("manifestid", models.CharField(max_length=256, null=True)),
	("reportnumhours", models.IntegerField(null=True)),
	("resource_type", models.CharField(max_length=64, null=True)),
	("resource_id", models.CharField(max_length=256, null=True)),
	("tenant_id", models.CharField(max_length=256, null=True)),
	("project_id", models.CharField(max_length=256, null=True)),
	("meter_name", models.CharField(max_length=128, null=True)),
	("value", models.FloatField(null=True)),
	("unit", models.CharField(max_length=64, null=True)),
	("cost_type", models.CharField(max_length=32, null=True)),
	("koku_metric", models.CharField(max_length=128, null=True)),
	("cost_amount", models.FloatField(null=True)),
	("currency", models.CharField(max_length=8, null=True)),
	],
	options={
	"db_table": "openshift_osac_usage_line_items_daily",
	"indexes": [models.Index(fields=["source", "year", "month"], name="osac_daily_src_yr_mo_idx")],
	},
	),
	migrations.RunPython(code=unset_pg_extended_mode, reverse_code=set_pg_extended_mode),
	]

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Sanity
Commit: fef4808

---

Root cause: The reorder-python-imports and black pre-commit hooks failed on koku/reporting/migrations/0352_osacusagelineitemdaily.py. The migration file imports from django.db import migrations, models on a single line, but the hook expects them split into separate lines.

Evidence:

Reorder python imports...........................................[41mFailed[m
- hook id: reorder-python-imports
- files were modified by this hook
Reordering imports in koku/reporting/migrations/0352_osacusagelineitemdaily.py

black................................................................[41mFailed[m
- hook id: black
- files were modified by this hook
- import uuid
- from django.db import migrations, models
+ from django.db import migrations
+ from django.db import models

Action: Run pre-commit run --all-files locally and commit the formatting changes. The hook will split the combined import automatically.

Generated automatically. Review before applying.

[koku-ci-triager-bot]

🤖 CI Triager — Diagnosis

Check: Units - 3.11
Commit: fef4808

---

Root cause: The new OSACUsageLineItemDaily model in self_hosted_models.py fails test_postgres_tables_are_partitioned. The test detects that the model inherits from OCPLineItemBase (which marks it as a partitioned-table type) but the underlying DB table openshift_osac_usage_line_items_daily is not created as a partitioned table. The migration 0352_osacusagelineitemdaily.py creates it as a regular table.

Evidence:

FAIL: test_postgres_tables_are_partitioned
  (masu.test.database.test_partitions.TestPartitionCheck.test_postgres_tables_are_partitioned)
  (model=<class 'reporting.provider.ocp.self_hosted_models.OSACUsageLineItemDaily'>)
AssertionError: False is not true : Model OSACUsageLineItemDaily is not partitioned.
FAILED (failures=1, skipped=40)

Action: Choose one of:

1. Make OSACUsageLineItemDaily a partitioned table — add PartitionInfo to its Meta (like other OCPLineItemBase subclasses) and update the migration to use CREATE TABLE ... PARTITION BY RANGE.
2. If the table should not be partitioned, change the base class from OCPLineItemBase to models.Model (or an appropriate non-partitioned base) so the partition check doesn't apply.

Generated automatically. Review before applying.