Kubearchive Shared Libraries

[adambkaplan]

This is a follow up on my comment in the Tekton/Kubearchive discussion thread (see tektoncd/community#1239).

Does it make sense for Kubearchive to refactor its codebase so that some of its individual features/pieces are importable as Go modules (or equivalent shareable pieces of code)?

I could see this being a potential pivot of the project's mission. Instead of Kubearchive being a "competitor" to more specific solutions like Tekton Results and the Argo Workflow Archive, Kubearchive becomes a foundation that these projects build on top of.

I don't think the current kubearchive/kubearchive repo should be torn down - IMO the full end to end archiver should remain as a "reference implementation" to validate these libraries.

[rh-hemartin]

Hmm... which features do you have in mind that people can import? I don't see anything worth exposing.

[adambkaplan]

github.com/kubearchive/kubearchive/cmd/api/auth for starters. This could easily live as a standalone middleware repo for anything that wants to "abuse" Kubernetes RBAC.

I had scaffolded something similar ages ago, but never got around to making it a real project: https://github.com/adambkaplan/go-k8s-middleware

[adambkaplan]

Frankly most everything in github.com/kubearchive/kubearchive/cmd/api can be spun out, since the DB access layer is an interface.

[adambkaplan]

Maybe a good way to experiment with this idea is to write a couple of pseudo-code lines showing how would we expect "this libraries" to be called.

How does vibe code sound instead? tektoncd/results#1265.

The code works in a practical sense. It uses the Results database as storage, with the adapter extracting the object's JSON.

[adambkaplan]

Demo of the code above: https://youtu.be/g551QRpy7_g

[maruiz93]

ok I took a look to it, and if I understand it correctly you are just using the "reader" part, the KubeArchive API just to offer a Kubernetes compatible API in tekton results.
You are still keeping: results schema database, and results way of getting notified of new resources plus writing them in the database.

I don't think this is a good idea, you are performing a transformation of the "results-like" resources to get them to the API, but that's just a 10% of why is worth using kubeArchive in my opinion, I think that using just the API layer of kubeArchive is not worth it.

KubeArchive's advantages are:

• The way you can configure it for whatever resource with high flexibility of the conditions by using CEL expressions
• The way it monitors and sends events to the sink
• The high performance of the writes given the current schema
• The architecture that allows adding new storage systems
• The way it's architectured with different components so they can be horizontally scalated according to different needs
• The way it's database schema is set up to be performant for the specific queries that we offer
• The way the logs are provided through a generic logging-backend integration with the only requirement of having an API to which you can pass the pod UUID and the container name for filtering
• The way the auth is deferred to kubernetes through TR and SAR (this is actually common between the two projects)
• A Kubernetes-like compatible API

With this PoC I think there's only the last 2 points covered, am I right?

I really don't see how kubearchive can be used as a library given the coupling among the different parts.

I think it would make more sense to replace some pieces of results with kubearchive, but not from a library POV:

• Replace Results' watcher + recorder with kubearchive's CloudEvents sink
• Replace Results' schema with kubearchive's schema (or migrate to it)
• Keep Results' gRPC API for the Results-specific API
• Add kubearchive's kubectl-compatible API

That way you would be:

• Installing kubearchive + results
• Results is an extra layer that provides retro compatibility with what's currently used
• Results can also provide the configuration needed for KubeArchive only for the Tekton resources

I am proposing this because I really don't see how to expand the library thing to cover other advantages like kubearchive's performance on queries based on label retrieval for example.

[adambkaplan]

ok I took a look to it, and if I understand it correctly you are just using the "reader" part, the KubeArchive API just to offer a Kubernetes compatible API in tekton results.
I don't think this is a good idea, you are performing a transformation of the "results-like" resources to get them to the API, but that's just a 10% of why is worth using kubeArchive in my opinion, I think that using just the API layer of kubeArchive is not worth it.

That was the scope of my proof of concept - take the reader bits of Kubearchive, import as go modules, and provide an adapter layer as an interface. The value is I was able to prototype this very quickly - I was able to get the working prototype done in an afternoon.

I agree that Kubearchive is way more than this, and there are more pieces that Tekton Results could pick up if they want to. Those last two points are critical; Tekton Results has a very large end user base and the community strives to provide smooth upgrade experiences. Internal architecture changes carry risk, and they may not be willing to bring in of Kubearchive's capabilities at once. Composable libraries allow adopters to pick and choose the things they want.

The way the logs are provided through a generic logging-backend integration with the only requirement of having an API to which you can pass the pod UUID and the container name for filtering

This is a piece that I'd like to explore as another possible "library" component. Here's a concrete use case: Argo Workflows has a log archive feature that is explicitly marked as "not recommended for production" (docs). Could they use the Kubearchive code/components to connect the Argo Workflows UI to the appropriate logs in Elastic, Splunk, Loki, etc.?

[maruiz93]

and there are more pieces that Tekton Results could pick up if they want to.

But this is where I disagree, you could maybe take some small pieces like the CEL expression parsing, but I don't think you can go with more things if you aren't willing to move to the kubearchive compatible database schema or if you aren't willing to use the archival mechanisms of kubearchive.

And re-reading my message there is something I missed, the vacuums, also not useful with the library approach.

I don't like moving on to the library approach because of this, just maintinging a whole project if you actually just want to deliver an API maybe with some adaptors, that doesn't make too much sense, in fact, if we think that's really something we want to do, I would:

1. Create a separate project under the kubearchive organization with the API routers as a library
2. Use this as a dependency for kubearchive, and results can use it too

but I wouldn't "shift" the projects intent for being a library for the API and maybe a library for the CEL parsing (I really still don't see how could you make a library for the rest of the features)

Here's a concrete use case: Argo Workflows has a log archive feature that is explicitly marked as "not recommended for production" (docs). Could they use the Kubearchive code/components to connect the Argo Workflows UI to the appropriate logs in Elastic, Splunk, Loki, etc.?

From my understanding they actually do the same as kubeArchive, which is linking the resources to how to query them from a backend logging system.
Without diving a lot into this, I currently don't see how this can be exposed as a library, unless, you just expose the logging API middleware, and use a lot of adapters like you did on your PoC, and there's actually the following information stored in the database:

1. The URL or the needed pieces to craft a URL to get the logs of a container in a pod by providing the Pod UUID and the container name
2. If you want to take advantage of the linking that kubeArchive does between the a resource and its owned pods, the ownerReference chain between the resources need to be queried from somewhere (if you are not willing to go with the kubeArchive's database schema)

I think it would make more sense to replace some pieces of results with kubearchive, but not from a library POV

Have you considered this as an alternative?

[ggallen]

Wow. That PR has 2264 changed files.

[adambkaplan]

All but 6 files are vendored go modules - actual logic changes are here: aa396a0cc

[maruiz93]

After talking with @adambkaplan, @skoved and I agreed to go on with this effort for the API (routers and middlewares) and the sink as it's only a matter of refactoring and reorganizing the code.

Diving into this matter, the main goal is to externalize api/ and sink/ so they are their own go modules so the people who import them doesn't have to import all the dependencies in kubearchive.

We have to be careful with circular dependencies, and to prevent that we would need a new external component called core/ so we end up with something like this:

Dependency graph

  core/  ←───────  api/   ←──── cmd/api/main.go                                                                                                                 
    ↑               ↑               │                                                                                                                           
    │               │               │                                                                                                                           
    └───────  sink/  ←──────── cmd/sink/main.go                                                                                                               
                                    │                                                                                                                           
                                root go.mod                                                                                                                     
                                (pkg/, cmd/operator, cmd/vacuum)                                                                                                
                                                                                                                                                                   

No circular dependencies: core depends on nothing internal, api and sink depend only on core, and cmd/ binaries import everything.

The idea I have in mind then is to do something as follows:

Module Structure

  core/                              # github.com/kubearchive/kubearchive/core                                                                                  
  ├── go.mod                         # Lightweight deps: gin, k8s types                                                                                         
  ├── models/                        # Resource, LabelFilters, LogTuple                                                                                         
  ├── database/                                                                                                                                                 
  │   ├── interfaces/                # DBReader, DBWriter, LogRecord                                                                                            
  │   └── errors/                    # ErrResourceNotFound, etc.                                                                                                
  ├── cache/                         # TTL cache                                                                                                                
  ├── abort/                         # Gin error response helper                                                                                                
  └── middleware/                    # Shared gin middleware (request logging)                                                                                  
                                                                                                                                                                
  api/                               # github.com/kubearchive/kubearchive/api                                                                                   
  ├── go.mod                         # depends on core/                                                                                                         
  ├── middleware/                                                                                                                                               
  │   ├── pagination/                # Pagination middleware + pure logic
  │   ├── discovery/                 # K8s API resource resolution                                                                                              
  │   └── auth/                      # AuthN, AuthZ, Impersonation                                                                                              
  ├── handler/                       # Controller (GetResources, GetLogURL, etc.)                                                                               
  ├── logging/                       # Log retrieval handler                                                                                                    
  └── server.go                      # Optional server builder                                                                                                  
                                                                                                                                                                
  sink/                              # github.com/kubearchive/kubearchive/sink                                                                 
  ├── go.mod                         # depends on core/                                 
  └── ...                                    # the exposed stuff for the sink library                                      
                                                                                                                                                                
  go.mod                             # Root module (the binaries)                                                                                               
  cmd/                                                                                                                                                          
  ├── api/main.go                    # Thin wiring: imports api/ and core/                                                                                      
  ├── sink/main.go                   # Imports sink/ and core/                                                                                                  
  ├── operator/                      # Imports pkg/ (root module only)                                                                                          
  └── vacuum/                        # Imports pkg/ (root module only)                                                                                          
  pkg/                               # Stays in root: things NOT needed by libraries                                                                            
  ├── cel/                           # Used by operator, vacuum, sink internals                                                                                 
  ├── cloudevents/                   # Used by operator, vacuum                                                                                                 
  ├── constants/                     # Used by operator, vacuum                                                                                                 
  ├── database/                      # Full implementations (sql/, fake/, factory)                                                                              
  ├── filters/                       # Used by operator, vacuum                                                                                                 
  ├── k8sclient/                     # K8s client factory                                                                                                       
  ├── logging/                       # Structured logging setup                                                                                                 
  ├── logurls/                       # Used by sink                                                                                                             
  ├── observability/                 # OpenTelemetry setup                                                                                                      
  ├── cmd/                           # kubectl-ka                                                                                                               
  └── files/                         # File utilities                                                                                                           
                                                                     

So the core and API library would include the following

What goes where and why

  core/ — Types and utilities that both api/ and sink/ libraries need:                                                                                          
                                                               
  ┌─────────────────────┬─────────────────────────────────────────────────────────────────────────┬─────────────────────┐                                       
  │       Package       │                                Why core                                 │       Used by       │                                       
  ├─────────────────────┼─────────────────────────────────────────────────────────────────────────┼─────────────────────┤                                       
  │ models              │ Foundational data types (Resource, LabelFilters)                        │ api, sink, database │
  ├─────────────────────┼─────────────────────────────────────────────────────────────────────────┼─────────────────────┤                                       
  │ database/interfaces │ DBReader/DBWriter contracts — library consumers provide implementations │ api, sink           │                                       
  ├─────────────────────┼─────────────────────────────────────────────────────────────────────────┼─────────────────────┤                                       
  │ database/errors     │ Error types checked by handlers                                         │ api, sink           │                                       
  ├─────────────────────┼─────────────────────────────────────────────────────────────────────────┼─────────────────────┤                                       
  │ abort               │ Gin error helper — both api and sink are gin-based                      │ api, sink           │
  ├─────────────────────┼─────────────────────────────────────────────────────────────────────────┼─────────────────────┤                                       
  │ middleware          │ Request logging middleware — used by both servers                       │ api, sink           │
  └─────────────────────┴─────────────────────────────────────────────────────────────────────────┴─────────────────────┘     
                                                               
  api/ — Importable by external projects that want kubearchive-style API endpoints:                                                                             
                                                               
  ┌───────────────────────┬───────────────────────────────────────────────┐                                                                                     
  │        Package        │               What it provides                │
  ├───────────────────────┼───────────────────────────────────────────────┤                                                                                     
  │ middleware/pagination │ Limit/continue token validation               │
  ├───────────────────────┼───────────────────────────────────────────────┤                                                                                     
  │ middleware/discovery  │ K8s resource kind resolution                  │                                                                                     
  ├───────────────────────┼───────────────────────────────────────────────┤                                                                                     
  │ middleware/auth       │ TokenReview authn + SAR authz + impersonation │                                                                                     
  ├───────────────────────┼───────────────────────────────────────────────┤                                                                                     
  │ handler               │ Controller with GetResources, GetLogURL, etc. │
  ├───────────────────────┼───────────────────────────────────────────────┤                                                                                     
  │ logging               │ Log retrieval handler chain                   │
  └───────────────────────┴───────────────────────────────────────────────┘                            

With this structure we can conditionally trigger rebuilding things through konflux based on changes in different directories:

Conditional rebuilds

┌───────────────┬─────────────────────────────────────────────┐                                                                                               
│   Change in   │             Triggers rebuild of             │
├───────────────┼─────────────────────────────────────────────┤                                                                                               
│ core/         │ cmd/api, cmd/sink, cmd/operator, cmd/vacuum │                                                                                               
├───────────────┼─────────────────────────────────────────────┤                                                                                               
│ pkg/          │ cmd/api, cmd/sink, cmd/operator, cmd/vacuum │                                                                                               
├───────────────┼─────────────────────────────────────────────┤                                                                                               
│ api/          │ cmd/api                                     │                                                                                               
├───────────────┼─────────────────────────────────────────────┤                                                                                               
│ sink/         │ cmd/sink                                    │
├───────────────┼─────────────────────────────────────────────┤                                                                                               
│ cmd/api/      │ cmd/api                                     │
├───────────────┼─────────────────────────────────────────────┤                                                                                               
│ cmd/operator/ │ cmd/operator                                │
└───────────────┴─────────────────────────────────────────────┘