Skip to content

Store applied config in stage CR status #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
concaf wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Store applied config in stage CR status #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6ed1af4
Store applied config in DocumentProcessor, ChunksGenerator, and Vecto…
concaf Jul 6, 2026
22372bc
Store applied config in DocumentProcessor, ChunksGenerator, and Vecto…
concaf Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions api/v1alpha1/chunksgenerator_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ type ChunksGeneratorSpec struct {

// ChunksGeneratorStatus defines the observed state of ChunksGenerator
type ChunksGeneratorStatus struct {
LastAppliedGeneration int64 `json:"lastAppliedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
FilesProcessed int64 `json:"filesProcessed,omitempty"`
LastAppliedGeneration int64 `json:"lastAppliedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
FilesProcessed int64 `json:"filesProcessed,omitempty"`
AppliedConfig *ChunksGeneratorConfig `json:"appliedConfig,omitempty"`
}

// +kubebuilder:object:root=true
Expand Down
11 changes: 6 additions & 5 deletions api/v1alpha1/documentprocessor_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,12 @@ type DocumentProcessorSpec struct {

// DocumentProcessorStatus defines the observed state of DocumentProcessor
type DocumentProcessorStatus struct {
LastAppliedGeneration int64 `json:"lastAppliedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
Jobs []Job `json:"jobs,omitempty"`
PermanentlyFailingFiles []string `json:"permanentlyFailingFiles,omitempty"`
FilesProcessed int64 `json:"filesProcessed,omitempty"`
LastAppliedGeneration int64 `json:"lastAppliedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
Jobs []Job `json:"jobs,omitempty"`
PermanentlyFailingFiles []string `json:"permanentlyFailingFiles,omitempty"`
FilesProcessed int64 `json:"filesProcessed,omitempty"`
AppliedConfig *DocumentProcessorConfig `json:"appliedConfig,omitempty"`
}

type Job struct {
Expand Down
7 changes: 4 additions & 3 deletions api/v1alpha1/vectorembeddingsgenerator_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,10 @@ type VectorEmbeddingsGeneratorSpec struct {

// VectorEmbeddingsGeneratorStatus defines the observed state of VectorEmbeddingsGenerator.
type VectorEmbeddingsGeneratorStatus struct {
LastAppliedGeneration int64 `json:"lastAppliedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
FilesProcessed int64 `json:"filesProcessed,omitempty"`
LastAppliedGeneration int64 `json:"lastAppliedGeneration,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
FilesProcessed int64 `json:"filesProcessed,omitempty"`
AppliedConfig *VectorEmbeddingsGeneratorConfig `json:"appliedConfig,omitempty"`
}

// +kubebuilder:object:root=true
Expand Down
15 changes: 15 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

54 changes: 54 additions & 0 deletions config/crd/bases/operator.dataverse.redhat.com_chunksgenerators.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,60 @@ spec:
status:
description: ChunksGeneratorStatus defines the observed state of ChunksGenerator
properties:
appliedConfig:

@PuneetPunamiya PuneetPunamiya Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are also planning to add inaccessible items in the status as well in #294, let's say the pipeline has 10 stages the status would be very large and also we need to think of the size of status and I think we should revisit once if we should add the whole config or some minimal thing ?

properties:
markdownSplitterConfig:
properties:
chunkOverlap:
type: integer
chunkSize:
type: integer
codeBlocks:
type: boolean
headingHierarchy:
type: boolean
joinTableRows:
type: boolean
referenceLinks:
type: boolean
type: object
recursiveCharacterSplitterConfig:
properties:
chunkOverlap:
type: integer
chunkSize:
type: integer
keepSeparator:
type: boolean
separators:
items:
type: string
type: array
type: object
strategy:
type: string
tokenSplitterConfig:
properties:
allowedSpecial:
items:
type: string
type: array
chunkOverlap:
type: integer
chunkSize:
type: integer
disallowedSpecial:
items:
type: string
type: array
encodingName:
type: string
modelName:
type: string
type: object
required:
- strategy
type: object
conditions:
items:
description: Condition contains details for one aspect of the current
Expand Down
69 changes: 69 additions & 0 deletions config/crd/bases/operator.dataverse.redhat.com_documentprocessors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,75 @@ spec:
status:
description: DocumentProcessorStatus defines the observed state of DocumentProcessor
properties:
appliedConfig:
properties:
doclingConfig:
properties:
abort_on_error:
description: 'Deprecated: this field is ignored; abort_on_error
is always false.'
type: boolean
do_chart_extraction:
type: boolean
do_code_enrichment:
type: boolean
do_formula_enrichment:
type: boolean
do_ocr:
type: boolean
do_picture_classification:
type: boolean
do_picture_description:
type: boolean
do_table_structure:
type: boolean
document_timeout:
type: string
force_ocr:
type: boolean
from_formats:
items:
type: string
type: array
image_export_mode:
type: string
images_scale:
type: string
include_images:
type: boolean
md_page_break_placeholder:
type: string
ocr_engine:
description: 'Deprecated: use OCRPreset instead.'
type: string
ocr_lang:
items:
type: string
type: array
ocr_preset:
type: string
page_range:
items:
type: integer
type: array
pdf_backend:
type: string
picture_description_area_threshold:
type: string
pipeline:
type: string
table_cell_matching:
type: boolean
table_mode:
type: string
to_formats:
items:
type: string
type: array
type: object
type:
type: string
type: object
conditions:
items:
description: Condition contains details for one aspect of the current
Expand Down
13 changes: 13 additions & 0 deletions config/crd/bases/operator.dataverse.redhat.com_vectorembeddingsgenerators.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,19 @@ spec:
description: VectorEmbeddingsGeneratorStatus defines the observed state
of VectorEmbeddingsGenerator.
properties:
appliedConfig:
properties:
batchSize:
minimum: 1
type: integer
modelName:
type: string
nomicEmbedTextV15Config:
properties:
encodingformat:
type: string
type: object
type: object
conditions:
items:
description: Condition contains details for one aspect of the current
Expand Down
1 change: 1 addition & 0 deletions internal/controller/chunksgenerator_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ func (r *ChunksGeneratorReconciler) Reconcile(ctx context.Context, req ctrl.Requ
successMessage := fmt.Sprintf("successfully reconciled chunks generator: %s", chunksGeneratorCR.Name)
if err := controllerutils.StatusPatch(ctx, r.Client, chunksGeneratorCR, func() {
chunksGeneratorCR.Status.FilesProcessed += filesProcessed
chunksGeneratorCR.Status.AppliedConfig = chunksGeneratorCR.Spec.ChunksGeneratorConfig.DeepCopy()
chunksGeneratorCR.UpdateStatus(successMessage, nil)
}); err != nil {
logger.Error(err, "failed to update ChunksGenerator CR status", "namespace", chunksGeneratorCR.Namespace, "name", chunksGeneratorCR.Name)
Expand Down
1 change: 1 addition & 0 deletions internal/controller/documentprocessor_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ func (r *DocumentProcessorReconciler) Reconcile(ctx context.Context, req ctrl.Re

successMessage := fmt.Sprintf("successfully reconciled document processor: %s", documentProcessorCR.Name)
if err := controllerutils.StatusPatch(ctx, r.Client, documentProcessorCR, func() {
documentProcessorCR.Status.AppliedConfig = documentProcessorCR.Spec.DocumentProcessorConfig.DeepCopy()
documentProcessorCR.UpdateStatus(successMessage, nil)
}); err != nil {
logger.Error(err, "failed to update DocumentProcessor CR status", "namespace", documentProcessorCR.Namespace, "name", documentProcessorCR.Name)
Expand Down
1 change: 1 addition & 0 deletions internal/controller/vectorembeddingsgenerator_controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ func (r *VectorEmbeddingsGeneratorReconciler) Reconcile(ctx context.Context, req
successMessage := fmt.Sprintf("successfully reconciled vector embeddings generator: %s", vectorEmbeddingsGeneratorCR.Name)
if err := controllerutils.StatusPatch(ctx, r.Client, vectorEmbeddingsGeneratorCR, func() {
vectorEmbeddingsGeneratorCR.Status.FilesProcessed += filesProcessed
vectorEmbeddingsGeneratorCR.Status.AppliedConfig = vectorEmbeddingsGeneratorCR.Spec.VectorEmbeddingsGeneratorConfig.DeepCopy()
vectorEmbeddingsGeneratorCR.UpdateStatus(successMessage, nil)
}); err != nil {
logger.Error(err, "failed to update VectorEmbeddingsGenerator CR status", "namespace", vectorEmbeddingsGeneratorCR.Namespace, "name", vectorEmbeddingsGeneratorCR.Name)
Expand Down
Loading