Conditional Log URL Entry Creation

[maruiz93]

Problem Statement

Currently, KubeArchive creates a log_url table entry for every container defined in a Pod's spec.containers and spec.initContainers. However, not all containers that appear in the spec actually run and produce logs.

Example: Tekton Pipelines

In Tekton TaskRuns, steps execute sequentially but all step containers are created simultaneously by Kubernetes. When a TaskRun is cancelled:

• Containers that were actively running have logs
• Containers that were waiting for their turn to execute do not have logs (they never ran their workload)

From Kubernetes' perspective, all these containers have state.terminated.startedAt because their container processes started, but only some actually executed their workload and produced logs.

This results in log_url entries pointing to non-existent logs, creating a poor user experience when querying archived Pods.

The Core Issue

There is no universal way to determine from Pod YAML alone whether a container produced logs. Different workload types (Tekton, Jobs, regular Pods) have different patterns for indicating "this container actually ran its workload."

Proposed Solution

Add a configurable CEL expression that determines whether to create a log_url entry for each container. This allows KubeArchive to remain generic while supporting customization for specific environments.

Design Principles

1. Generic by default: Provide a sensible default that works for most Kubernetes workloads
2. Customizable: Allow users to define their own logic based on their environment
3. Declarative: Use CEL expressions (already used throughout KubeArchive) for configuration
4. Future-proof: Design with extensibility in mind for selector-based rules

Technical Design

Configuration

Add a new optional key to the kubearchive-logging ConfigMap:

LOG_URL_CONDITION: "cel:<expression>"

The CEL expression:

• Input: Receives a container status object (from status.containerStatuses[i] or status.initContainerStatuses[i])
• Output: Returns a boolean (true = create log_url entry, false = skip)
• Evaluated: Once per container when processing a Pod

Default Behavior

If LOG_URL_CONDITION is not specified, use the current behavior:

LOG_URL_CONDITION: "cel:has(state.running) || has(state.terminated)"

This creates entries for all containers that have started from Kubernetes' perspective (backward compatible).

Container Status Context

The CEL expression has access to the container status object structure:

{
  "name": "container-name",
  "containerID": "cri-o://...",
  "image": "...",
  "imageID": "...",
  "ready": false,
  "restartCount": 0,
  "started": false,
  "state": {
    "terminated": {
      "containerID": "cri-o://...",
      "exitCode": 0,
      "finishedAt": "2026-01-16T14:14:12Z",
      "message": "...",
      "reason": "Completed",
      "startedAt": "2026-01-16T14:12:30Z"
    }
  }
}

Configuration Examples

Example 1: Tekton Environment

Only create log entries for containers that emitted Tekton's termination message with StartedAt:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubearchive-logging
  namespace: kubearchive
data:
  LOG_URL_CONDITION: "cel:has(state.terminated.message) && state.terminated.message.contains('StartedAt')"
  # ... other logging config

Example 2: Runtime Threshold

Only create entries for containers that ran for more than 5 seconds:

LOG_URL_CONDITION: "cel:(has(state.terminated) && (timestamp(state.terminated.finishedAt) - timestamp(state.terminated.startedAt)) > duration('5s')) || has(state.running)"

Example 3: Success Only

Only create entries for containers that completed successfully:

LOG_URL_CONDITION: "cel:has(state.terminated) && state.terminated.exitCode == 0"

Example 4: Non-waiting Containers

Skip containers that never started (still in waiting state):

LOG_URL_CONDITION: "cel:!has(state.waiting)"

Implementation Plan

Phase 1: Basic Implementation

1. Add LOG_URL_CONDITION support to cmd/sink/logs/logs.go

   • Read and compile the CEL expression from logging config
   • If not specified, use default behavior

2. Modify logurls.GenerateLogURLs() to accept an optional condition

   • Evaluate condition for each container status before generating log tuple
   • Skip containers where condition evaluates to false

3. Update CEL evaluation context

   • Currently evaluates against full Pod object
   • Need to also evaluate against individual container status objects

Phase 2: Selector-Based Rules (Future Enhancement)

Support different conditions for different Pods based on labels/annotations:

LOG_URL_CONDITIONS:
  - selector: "cel:has(metadata.labels['tekton.dev/task'])"
    condition: "cel:has(state.terminated.message) && state.terminated.message.contains('StartedAt')"
  - selector: "cel:metadata.namespace == 'batch-jobs'"
    condition: "cel:has(state.terminated) && state.terminated.exitCode == 0"
  - selector: "cel:true"  # default/fallback
    condition: "cel:has(state.running) || has(state.terminated)"

This would be evaluated in order, using the first matching selector's condition.

Implementation Considerations

CEL Context Requirements

The current implementation passes the full Pod (*unstructured.Unstructured) to CEL evaluation. We need to:

1. Extract individual container statuses from status.containerStatuses and status.initContainerStatuses
2. Convert each container status to a format CEL can evaluate
3. Combine container name generation with condition evaluation

Modified Flow

Current flow:

1. Evaluate CONTAINER_NAME CEL → Get list of container names
2. For each container name → Generate log URL
3. Insert all log URLs

New flow:

1. Evaluate CONTAINER_NAME CEL → Get list of container names
2. Get corresponding container statuses
3. For each (container name, container status):
   a. Evaluate LOG_URL_CONDITION against container status
   b. If true: Generate log URL
4. Insert filtered log URLs

Backward Compatibility

• If LOG_URL_CONDITION is not specified, maintain current behavior
• Existing logging configurations continue to work without changes
• Users can opt-in to filtering when needed

Error Handling

• If LOG_URL_CONDITION fails to compile: Log error and fall back to default behavior
• If evaluation fails for a container: Log warning and skip that container (safe default)
• Validation: CEL expression must return boolean type

Testing Strategy

1. Unit tests for CEL condition compilation and evaluation
2. Integration tests with various Pod scenarios:
   • Successful pods (all containers completed)
   • Failed pods (some containers failed)
   • Cancelled pods (Tekton-style sequential execution)
   • Init containers vs regular containers
3. Test data: Add test Pods with Tekton-style statuses
4. Backward compatibility: Verify existing behavior when condition not specified

Documentation Updates

1. Update logging integration documentation
2. Add examples for common scenarios (Tekton, Jobs, etc.)
3. Document CEL expression context (available fields)
4. Add troubleshooting guide for condition evaluation

Open Questions

1. Should we provide preset conditions? E.g., a built-in "tekton" mode users can reference instead of writing CEL?

   LOG_URL_CONDITION: "preset:tekton"

2. Should condition failures be silent or visible? Should users be able to see which containers were filtered out?

3. Performance impact: How does per-container CEL evaluation impact Pod processing time at scale?

4. Alternative approach: Should we support checking the logging backend directly to verify logs exist before creating entries? (More accurate but adds latency and coupling)

Benefits

• ✅ Solves the immediate problem: No more log_url entries for containers that didn't run
• ✅ Generic solution: Works for any environment, not just Tekton
• ✅ Consistent with KubeArchive design: Uses CEL expressions like other features
• ✅ Backward compatible: Existing deployments unaffected
• ✅ Extensible: Foundation for future selector-based rules
• ✅ User control: Teams can customize based on their needs

Related Work

• Tekton termination message documentation: https://tekton.dev/docs/pipelines/taskruns/
• Kubernetes container states: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states

[rh-hemartin]

So if understood correctly the problem is the size of the table that holds the logs and KubeArchive failing to get those logs from the associated logging system, so it is kind of pointless and prone to confusing results.

I think this could make sense and help reduce useless information. I would keep it just to the basic implementation, the advanced implementation look just as a series of ANDs/ORs.

Where the examples are from? I don't see those values in the PodStatus: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodStatus

[maruiz93]

I would keep it just to the basic implementation, the advanced implementation look just as a series of ANDs/ORs.

I am not following here, do you mean that just with one LOG_URL_CONDITIONS you would be able to set different conditions for example for tekton containers and other type of containers?
The thing is that in my proposal the CEL expression acts only on the element of the containerStatuses/initContainterStatuses element, so it doesn't have the whole manifest to conditionally know for example if the container belongs to a Pod of a tekton TaskRun

Where the examples are from? I don't see those values in the PodStatus: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodStatus

The examples I pasted are specifically from the values of an element of PodStatus.initContainerStatuses/ PodStatus.initContainerStatuses you can see all the fields inside those fields in the link you shared:

It's quite large, but you can see that there is for example a containerStatuses.state

[rh-hemartin]

I am not following here, do you mean that just with one LOG_URL_CONDITIONS you would be able to set different conditions for example for tekton containers and other type of containers?

I meant that with just LOG_URL_CONDITION users should have enough. I did not read the thing properly so I assumed the conditional was meant for the whole Pod Spec. Now I understand that it evaluates only the containerStatuses.

Given that I read it better now I think it makes more sense to go directly with Phase 2, so users can select better what goes into the URLs and what does not.

[ggallen]

There is no universal way to determine from Pod YAML alone whether a container produced logs. Different workload types (Tekton, Jobs, regular Pods) have different patterns for indicating "this container actually ran its workload."

Why does the workload type even matter here? The container structure itself is independent of any workload.

And why do we want to allow this to be configurable at all? Why not just only generate a log URL for a container if one of the following is true:

• Status is running
• Status is terminated
• Status is waiting AND lastState is terminated WITH a startedAt time

I believe this catches almost everything, but I think it might be safer to reverse the logic. So if none of these conditions are true, then there was definitely no log generated.

I would suggest to just implement a solution and not allow users to configure it at all. We should be able to catch almost all conditions except for a couple of corner cases. In that case the only downs is a few URL for logs that don't exist. This is a better outcome that not creating URLs for those corner cases where logs do exist.

[maruiz93]

Why does the workload type even matter here? The container structure itself is independent of any workload.

I should probably extend the explanation around this topic, but it has to do with this:

steps execute sequentially but all step containers are created simultaneously by Kubernetes.

The thing is that I started to study this topic specifically with tekton resources. Let's say we have a TaskRun with 3 steps, this translates into a Pod with 3 containers. The containers run sequentially, but if the first container has finished, the second one is running and the third one is waiting for the second to finish you will see the following by looking at the containerStatuses:

• Container for step 1 - terminated
• Container for step 2 - running
• Container for step 3 - running

So, it's "lying" to us, because it's not true that the container for step 3 is running, it's just "ready" but the execution it's actually on hold until the step 2 finishes, and this workflow it's controller by the tekton pipelines operator.

Here you have the statusContainers of an example I ran reproducing this exact scenario:

Example of containerStatuses

containerStatuses:
  - containerID: containerd://0882a3a240b0dd1908586462fc546336557b95300e4364e0422023bfbb7de003
    image: docker.io/library/busybox:latest
    imageID: docker.io/library/busybox@sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f
    lastState: {}
    name: step-step-one-quick
    ready: false
    resources: {}
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://0882a3a240b0dd1908586462fc546336557b95300e4364e0422023bfbb7de003
        exitCode: 0
        finishedAt: "2026-02-14T12:51:11Z"
        message: '[{"key":"StartedAt","value":"2026-02-14T12:51:11.149Z","type":3}]'
        reason: Completed
        startedAt: "2026-02-14T12:49:08Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        - 10
        uid: 0
    volumeMounts:
    - mountPath: /tekton/downward
      name: tekton-internal-downward
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/creds
      name: tekton-creds-init-home-0
    - mountPath: /tekton/run/0
      name: tekton-internal-run-0
    - mountPath: /tekton/run/1
      name: tekton-internal-run-1
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/2
      name: tekton-internal-run-2
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/bin
      name: tekton-internal-bin
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/steps
      name: tekton-internal-steps
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/artifacts
      name: tekton-internal-artifacts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jgd8n
      readOnly: true
      recursiveReadOnly: Disabled
  - containerID: containerd://bee8f2f84c6d6ec7d0ae13c0f7ba752619bfd774b453c57a4ab1e317e546b390
    image: docker.io/library/busybox:latest
    imageID: docker.io/library/busybox@sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f
    lastState: {}
    name: step-step-three-pending
    ready: true
    resources: {}
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2026-02-14T12:49:34Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        - 10
        uid: 0
    volumeMounts:
    - mountPath: /tekton/creds
      name: tekton-creds-init-home-2
    - mountPath: /tekton/run/0
      name: tekton-internal-run-0
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/1
      name: tekton-internal-run-1
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/2
      name: tekton-internal-run-2
    - mountPath: /tekton/bin
      name: tekton-internal-bin
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/steps
      name: tekton-internal-steps
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/artifacts
      name: tekton-internal-artifacts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jgd8n
      readOnly: true
      recursiveReadOnly: Disabled
  - containerID: containerd://d80761c7e0074246897b063c673cffef20fcab1d67b5e7b7f3258fe46cc1762d
    image: docker.io/library/busybox:latest
    imageID: docker.io/library/busybox@sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f
    lastState: {}
    name: step-step-two-long-sleep
    ready: true
    resources: {}
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2026-02-14T12:49:25Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        - 10
        uid: 0
    volumeMounts:
    - mountPath: /tekton/creds
      name: tekton-creds-init-home-1
    - mountPath: /tekton/run/0
      name: tekton-internal-run-0
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/1
      name: tekton-internal-run-1
    - mountPath: /tekton/run/2
      name: tekton-internal-run-2
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/bin
      name: tekton-internal-bin
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/steps
      name: tekton-internal-steps
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/artifacts
      name: tekton-internal-artifacts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jgd8n
      readOnly: true
      recursiveReadOnly: Disabled

In this case we don't have a way to distinguish which one is running and which one is not running BUT, usually, what's interesting from KubeArchive perspective is to distinguish, when they have finished (terminated state) if the container ever "actually" ran or if it never actually started, and in this case, specifically for tekton results, it can be distinguished.

In the previous example, I decided to cancel the taskrun while it was running the step 2. Step 1 and 2 did run at some point but step 3 never started so I need a way to not pick step 3 while looking at the containerStatuses:

Example of containerStatuses

  containerStatuses:
  - containerID: containerd://33d891de88b2d2a1f5e99c0c46d603c834fd4de0f93273d9e9a6979b3827fcc4
    image: docker.io/library/busybox:latest
    imageID: docker.io/library/busybox@sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f
    lastState: {}
    name: step-step-one-quick
    ready: false
    resources: {}
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://33d891de88b2d2a1f5e99c0c46d603c834fd4de0f93273d9e9a6979b3827fcc4
        exitCode: 0
        finishedAt: "2026-02-14T13:09:36Z"
        message: '[{"key":"StartedAt","value":"2026-02-14T13:09:36.169Z","type":3}]'
        reason: Completed
        startedAt: "2026-02-14T13:09:32Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        - 10
        uid: 0
    volumeMounts:
    - mountPath: /tekton/downward
      name: tekton-internal-downward
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/creds
      name: tekton-creds-init-home-0
    - mountPath: /tekton/run/0
      name: tekton-internal-run-0
    - mountPath: /tekton/run/1
      name: tekton-internal-run-1
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/2
      name: tekton-internal-run-2
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/bin
      name: tekton-internal-bin
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/steps
      name: tekton-internal-steps
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/artifacts
      name: tekton-internal-artifacts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-fhscg
      readOnly: true
      recursiveReadOnly: Disabled
  - containerID: containerd://908edbf26b0205b363d42573282fa454fa758dcf071c667a2c8061eda8554aa6
    image: docker.io/library/busybox:latest
    imageID: docker.io/library/busybox@sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f
    lastState: {}
    name: step-step-three-pending
    ready: false
    resources: {}
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://908edbf26b0205b363d42573282fa454fa758dcf071c667a2c8061eda8554aa6
        exitCode: 1
        finishedAt: "2026-02-14T13:10:07Z"
        message: '[{"key":"StartedAt","value":"2026-02-14T13:10:07.536Z","type":3},{"key":"Reason","value":"Skipped","type":3}]'
        reason: Error
        startedAt: "2026-02-14T13:09:34Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        - 10
        uid: 0
    volumeMounts:
    - mountPath: /tekton/downward
      name: tekton-internal-downward
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/creds
      name: tekton-creds-init-home-2
    - mountPath: /tekton/run/0
      name: tekton-internal-run-0
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/1
      name: tekton-internal-run-1
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/2
      name: tekton-internal-run-2
    - mountPath: /tekton/bin
      name: tekton-internal-bin
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/steps
      name: tekton-internal-steps
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/artifacts
      name: tekton-internal-artifacts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-fhscg
      readOnly: true
      recursiveReadOnly: Disabled
  - containerID: containerd://d04ccece47796b3a3ade2862a01c28cb78696e1ca2dda04bc6b8014625babecc
    image: docker.io/library/busybox:latest
    imageID: docker.io/library/busybox@sha256:b3255e7dfbcd10cb367af0d409747d511aeb66dfac98cf30e97e87e4207dd76f
    lastState: {}
    name: step-step-two-long-sleep
    ready: false
    resources: {}
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://d04ccece47796b3a3ade2862a01c28cb78696e1ca2dda04bc6b8014625babecc
        exitCode: 9
        finishedAt: "2026-02-14T13:10:07Z"
        message: '[{"key":"StartedAt","value":"2026-02-14T13:09:36.286Z","type":3},{"key":"Reason","value":"Cancelled","type":3}]'
        reason: Error
        startedAt: "2026-02-14T13:09:33Z"
    user:
      linux:
        gid: 0
        supplementalGroups:
        - 0
        - 10
        uid: 0
    volumeMounts:
    - mountPath: /tekton/downward
      name: tekton-internal-downward
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/creds
      name: tekton-creds-init-home-1
    - mountPath: /tekton/run/0
      name: tekton-internal-run-0
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/run/1
      name: tekton-internal-run-1
    - mountPath: /tekton/run/2
      name: tekton-internal-run-2
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/bin
      name: tekton-internal-bin
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /workspace
      name: tekton-internal-workspace
    - mountPath: /tekton/home
      name: tekton-internal-home
    - mountPath: /tekton/results
      name: tekton-internal-results
    - mountPath: /tekton/steps
      name: tekton-internal-steps
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /tekton/artifacts
      name: tekton-internal-artifacts
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-fhscg
      readOnly: true
      recursiveReadOnly: Disabled

To sum up now you can see:

• Container for step 1 - terminated with message startedAt only
• Container for step 2 - running with message with Reason: Cancelled
• Container for step 3 - running with message with Reason: Skipped

so now with the proper CEL expression we should have a way to identify that we want to store the LOG_URL for container for step 1 and 2 but nor for step 3

[ggallen]

I think we should consider this instead:

Loki Integration: If you use a forwarder like Loki, Tekton Results verifies the availability of
the logs by querying the third-party API. You can configure a 
LOGGING_PLUGIN_FORWARDER_DELA Y_DURATION to give the external system time to
index the logs before the verification check is performed.

We could generate the log URL and add a verified flag, defaulting to false. Once the verification succeeds, the flag can be updated. And no logs are returned if the URL isn't verified.

[maruiz93]

correct me if I'm wrong, but are you suggesting calling loki or whatever logging backend API is used alongside kubearchive to check if the logs are there?

If this is what you mean, I see two issues:

• We will be in the same situation that we are now, we wouldn't distinguish if a container has actually ran at some but it didn't produce any logs, if they produced logs but due to an external problem, the logs didn't reach the logging backend system or if the container never actually started, which means that there shouldn't be any logs.

• We do a lot of updates the archival of Pods, if we are thinking of checking the logs each time the archived Pod manifest is updated, this will mean a lot of queries to the logging system, and that will probably make the log retrieval slower or less performant, as well as the Pod archival.

[ggallen]

correct me if I'm wrong, but are you suggesting calling loki or whatever logging backend API is used alongside kubearchive to check if the logs are there?

I am.

We will be in the same situation that we are now

In this case, yes. We can still do this work to see if a log URL should be generated.

But right now we have no idea if the links we are generating point to real logs. Any link. It's possible not a single one works. And this can be fixed if we check for the existence of the log. It could be an additional check.

that will probably make the log retrieval slower or less performant, as well as the Pod archival.

I do not think this has to be true. First, Tekton Results is doing exactly this and I don't thinks it's impacting them negatively. This type of operation is exactly what the HTTP HEAD operation is for. It should not put a strain on the logging system.

How we would do it in KubeArchive is up for debate. I would not add a wait to the archive itself, but rather make it some sort of separate process.

[rh-hemartin]

That process would revisit old URLs? Or just verify once and never again (filter by verified!=NULL)? Definitely I would do this as a separate process, just not to tie archival to an external source that could be failing or overwhelmed by any other reason.

[ggallen]

That process would revisit old URLs? Or just verify once and never again (filter by verified!=NULL)?

Yes. And we could then re-verify very easily if we wanted by simply setting verified to NULL.

[maruiz93]

In this case, yes. We can still do this work to see if a log URL should be generated.

So we are talking about two different initiatives:

1. Enable conditional storage of the log_url
2. Enabling log_url verification.

I'm not against doing the 2, but if you agree that the conditional storing is useful, I think we can address it separately and that we should keep this discussion for it.