feat(data_drip): add self-documenting backfills + catalog

README.md

@@ -11,6 +11,7 @@ DataDrip is a Rails engine that provides a robust framework for running data bac
 - 🔧 **Flexible Processing**: Choose between batch-level or element-level processing
 - 📈 **Progress Tracking**: Real-time progress updates and batch monitoring
 - 🎯 **Scoped Processing**: Define custom scopes for targeted data processing
+- 📚 **Self-Documenting**: Give backfills a `description` and Markdown `instructions`, browsable in a searchable catalog
 
 ## Installation
 
@@ -358,6 +359,38 @@ DataDrip supports various attribute types that automatically generate appropriat
 - **`:time`** - Time picker
 - **`:datetime`** - Date and time picker
 
+### Documenting Backfills
+
+Backfills can document themselves so operators know what each one does and how to fill in its options — without reading the source.
+
+```ruby
+class AddRoleToEmployee < DataDrip::Backfill
+  # One-line summary, listed in the backfills catalog.
+  description "Assigns the default 'intern' role to employees that don't have one yet."
+
+  # Richer guidance (Markdown), shown in the New Backfill Run form the moment
+  # this backfill is selected.
+  instructions <<~MARKDOWN
+    # Assign default role
+    Sets the **intern** role on all employees that don't have one yet.
+
+    ## Options
+    - `age`: Filter employees by age (optional)
+    - `name`: Filter by exact name match (optional)
+  MARKDOWN
+
+  attribute :age, :integer
+  attribute :name, :string
+
+  # ...
+end
+```
+
+- **`description`** — a one-line summary shown in the [backfills catalog](#web-interface). Optional; defaults to `nil`.
+- **`instructions`** — Markdown rendered as formatted rich text in the New Backfill Run form when the backfill is selected. A small, dependency-free renderer supports headings (`#`/`##`/`###`), `**bold**`, `` `inline code` ``, bullet lists, and fenced code blocks. Optional; defaults to `nil`.
+
+Both are declared with the same setter/getter idiom (a plain `def self.instructions` override also works).
+
 ### Backfill Structure
 
 Every backfill must inherit from `DataDrip::Backfill` and implement:
@@ -397,7 +430,9 @@ Navigate to `/data_drip/backfill_runs` in your application to access the DataDri
 - Stop running backfills
 - Schedule backfills for future execution
 
-When creating a new backfill run, the interface dynamically generates form fields based on the attributes defined in your backfill class, making it easy to customize each run without code changes.
+When creating a new backfill run, the interface dynamically generates form fields based on the attributes defined in your backfill class, making it easy to customize each run without code changes. If the backfill declares [`instructions`](#documenting-backfills), they render as formatted guidance above the options.
+
+You can also browse a searchable **backfills catalog** at `/data_drip/backfills` (the **Catalog** tab in the header): every backfill available in the app, with its description and the configurable fields it accepts — search by name, description, or field name to find, say, every backfill that takes `company_ids`.
 
 The interface supports light and dark mode (following the OS preference) and ships as precompiled CSS inside the gem — host applications need no Node, Tailwind, or any other frontend tooling.
 

app/assets/stylesheets/data_drip/tailwind.css

@@ -44,6 +44,7 @@
     --color-zinc-800: oklch(27.4% 0.006 286.033);
     --color-zinc-900: oklch(21% 0.006 285.885);
     --color-zinc-950: oklch(14.1% 0.005 285.823);
+    --color-black: #000;
     --color-white: #fff;
     --spacing: 0.25rem;
     --container-lg: 32rem;
@@ -69,6 +70,7 @@
     --tracking-tight: -0.025em;
     --tracking-wide: 0.025em;
     --tracking-widest: 0.1em;
+    --leading-relaxed: 1.625;
     --radius-sm: 0.25rem;
     --radius-md: 0.375rem;
     --radius-lg: 0.5rem;
@@ -368,6 +370,9 @@
   .mb-2 {
     margin-bottom: calc(var(--spacing) * 2);
   }
+  .mb-3 {
+    margin-bottom: calc(var(--spacing) * 3);
+  }
   .mb-4 {
     margin-bottom: calc(var(--spacing) * 4);
   }
@@ -447,6 +452,12 @@
   .w-\(--progress\) {
     width: var(--progress);
   }
+  .w-1\/2 {
+    width: calc(1/2 * 100%);
+  }
+  .w-1\/4 {
+    width: calc(1/4 * 100%);
+  }
   .w-9 {
     width: calc(var(--spacing) * 9);
   }
@@ -468,6 +479,9 @@
   .max-w-64 {
     max-width: calc(var(--spacing) * 64);
   }
+  .max-w-80 {
+    max-width: calc(var(--spacing) * 80);
+  }
   .max-w-\[45ch\] {
     max-width: 45ch;
   }
@@ -483,6 +497,9 @@
   .flex-1 {
     flex: 1;
   }
+  .table-fixed {
+    table-layout: fixed;
+  }
   .-translate-1\/2 {
     --tw-translate-x: calc(calc(1/2 * 100%) * -1);
     --tw-translate-y: calc(calc(1/2 * 100%) * -1);
@@ -537,9 +554,19 @@
   .justify-end {
     justify-content: flex-end;
   }
+  .gap-1\.5 {
+    gap: calc(var(--spacing) * 1.5);
+  }
   .gap-3 {
     gap: calc(var(--spacing) * 3);
   }
+  .space-y-0\.5 {
+    :where(& > :not(:last-child)) {
+      --tw-space-y-reverse: 0;
+      margin-block-start: calc(calc(var(--spacing) * 0.5) * var(--tw-space-y-reverse));
+      margin-block-end: calc(calc(var(--spacing) * 0.5) * calc(1 - var(--tw-space-y-reverse)));
+    }
+  }
   .gap-x-1 {
     column-gap: calc(var(--spacing) * 1);
   }
@@ -605,6 +632,9 @@
   .overflow-y-auto {
     overflow-y: auto;
   }
+  .rounded {
+    border-radius: 0.25rem;
+  }
   .rounded-full {
     border-radius: calc(infinity * 1px);
   }
@@ -702,6 +732,9 @@
   .bg-zinc-400 {
     background-color: var(--color-zinc-400);
   }
+  .bg-zinc-900 {
+    background-color: var(--color-zinc-900);
+  }
   .bg-zinc-950\/5 {
     background-color: color-mix(in srgb, oklch(14.1% 0.005 285.823) 5%, transparent);
     @supports (color: color-mix(in lab, red, red)) {
@@ -817,6 +850,9 @@
   .pb-3 {
     padding-bottom: calc(var(--spacing) * 3);
   }
+  .pl-5 {
+    padding-left: calc(var(--spacing) * 5);
+  }
   .pl-6 {
     padding-left: calc(var(--spacing) * 6);
   }
@@ -835,6 +871,9 @@
   .align-middle {
     vertical-align: middle;
   }
+  .align-top {
+    vertical-align: top;
+  }
   .font-mono {
     font-family: var(--font-mono);
   }
@@ -872,6 +911,9 @@
   .text-\[0\.6875rem\] {
     font-size: 0.6875rem;
   }
+  .text-\[0\.8125rem\] {
+    font-size: 0.8125rem;
+  }
   .leading-5 {
     --tw-leading: calc(var(--spacing) * 5);
     line-height: calc(var(--spacing) * 5);
@@ -880,6 +922,10 @@
     --tw-leading: calc(var(--spacing) * 6);
     line-height: calc(var(--spacing) * 6);
   }
+  .leading-relaxed {
+    --tw-leading: var(--leading-relaxed);
+    line-height: var(--leading-relaxed);
+  }
   .font-bold {
     --tw-font-weight: var(--font-weight-bold);
     font-weight: var(--font-weight-bold);
@@ -911,6 +957,9 @@
   .text-pretty {
     text-wrap: pretty;
   }
+  .break-words {
+    overflow-wrap: break-word;
+  }
   .wrap-anywhere {
     overflow-wrap: anywhere;
   }
@@ -953,6 +1002,9 @@
   .text-white {
     color: var(--color-white);
   }
+  .text-zinc-100 {
+    color: var(--color-zinc-100);
+  }
   .text-zinc-300 {
     color: var(--color-zinc-300);
   }
@@ -1099,6 +1151,11 @@
       color: var(--color-zinc-400);
     }
   }
+  .first\:mt-0 {
+    &:first-child {
+      margin-top: calc(var(--spacing) * 0);
+    }
+  }
   .last\:mb-0 {
     &:last-child {
       margin-bottom: calc(var(--spacing) * 0);
@@ -1322,6 +1379,14 @@
       }
     }
   }
+  .dark\:bg-black\/40 {
+    @media (prefers-color-scheme: dark) {
+      background-color: color-mix(in srgb, #000 40%, transparent);
+      @supports (color: color-mix(in lab, red, red)) {
+        background-color: color-mix(in oklab, var(--color-black) 40%, transparent);
+      }
+    }
+  }
   .dark\:bg-blue-400\/10 {
     @media (prefers-color-scheme: dark) {
       background-color: color-mix(in srgb, oklch(70.7% 0.165 254.624) 10%, transparent);
@@ -1715,6 +1780,11 @@
   inherits: false;
   initial-value: 0;
 }
+@property --tw-space-y-reverse {
+  syntax: "*";
+  inherits: false;
+  initial-value: 0;
+}
 @property --tw-divide-y-reverse {
   syntax: "*";
   inherits: false;
@@ -1933,6 +2003,7 @@
       --tw-translate-x: 0;
       --tw-translate-y: 0;
       --tw-translate-z: 0;
+      --tw-space-y-reverse: 0;
       --tw-divide-y-reverse: 0;
       --tw-border-style: solid;
       --tw-gradient-position: initial;

app/controllers/data_drip/backfill_runs_controller.rb

@@ -224,7 +224,9 @@ def backfill_options
           options: {}
         )
 
-      html = helpers.backfill_option_inputs(temp_run)
+      # Instructions + option inputs, so both refresh together when the class
+      # changes (the backfill-options Stimulus controller swaps this container).
+      html = helpers.backfill_form_details(temp_run)
 
       render json: { html: html }
     end

app/controllers/data_drip/backfills_controller.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+module DataDrip
+  # Catalog of the backfill classes available in the host app. Unlike
+  # BackfillRunsController (which lists persisted *runs*), this lists the
+  # backfill *definitions* themselves so users can discover what each one does
+  # and which options it accepts.
+  class BackfillsController < DataDrip.base_controller_class.constantize
+    include DataDrip::Paginatable
+    include DataDrip::BackfillerContext
+
+    layout "data_drip/layouts/application"
+    helper DataDrip::BackfillRunsHelper
+    helper DataDrip::BackfillsHelper
+
+    def index
+      @query = params[:q].to_s.strip
+
+      # Dedupe by name, preferring a copy that carries a description. In
+      # development, Zeitwerk reloading leaves stale class copies in
+      # DataDrip::Backfill.descendants (DataDrip.all): the same backfill would
+      # otherwise be listed several times, and only some copies retain their
+      # `description` (so we can't just pick the current constant). Anonymous
+      # subclasses (nil name, e.g. those defined in tests) drop out here, since
+      # only named backfills belong in the catalog.
+      backfills =
+        DataDrip.all
+                .select { |klass| klass.name.present? }
+                .group_by(&:name)
+                .map { |_name, copies| copies.find { |c| described?(c) } || copies.first }
+                .sort_by(&:name)
+      backfills = filter_backfills(backfills, @query) if @query.present?
+
+      pagination_data = paginate_collection(backfills, per_page: 10)
+      @backfills = pagination_data[:collection]
+      @pagination = pagination_data
+    end
+
+    private
+
+    def described?(backfill_class)
+      backfill_class.respond_to?(:description) &&
+        backfill_class.description.present?
+    end
+
+    # Client asks for a needle; we match it (case-insensitively) against the
+    # class name, the description, and each option name — so searching
+    # "company_ids" surfaces every backfill that accepts it.
+    def filter_backfills(backfills, query)
+      needle = query.downcase
+      backfills.select do |klass|
+        haystack =
+          [
+            klass.name,
+            (klass.description if klass.respond_to?(:description)),
+            *klass.backfill_options_class.attribute_types.keys
+          ].compact.join(" ").downcase
+        haystack.include?(needle)
+      end
+    end
+  end
+end