Skip to content

feat: add Scripts — run arbitrary parameterized code from the UI #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rogercampos merged 1 commit into from
Jul 23, 2026
Merged

feat: add Scripts — run arbitrary parameterized code from the UI #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,82 @@ To play with the UI locally, boot the dummy app and visit `http://localhost:3000
bin/dev
```

## Scripts

Besides backfills, DataDrip can run **scripts**: arbitrary one-shot pieces of code with typed inputs, triggered from the web UI. Unlike backfills there is no collection to iterate — a script runs once, in a background job, and everything it logs is stored and shown live in the UI.

Every script run is recorded permanently in the `data_drip_script_runs` table: who triggered it, the inputs given, the full log output, any error (message and backtrace), and created/started/finished timestamps.

### Installing

Fresh installs of DataDrip get everything through `rails generate data_drip:install`. If you installed DataDrip before scripts existed, run the upgrade generator:

```bash
rails generate data_drip:install_scripts
```

This creates the `app/scripts` directory and the `data_drip_script_runs` migration.

### Creating Scripts

Generate a new script:

```bash
rails generate data_drip:script BackfillCompanyTimezones
```

A script declares its inputs with the `input` DSL and implements a single `call` method:

```ruby
# app/scripts/backfill_company_timezones.rb
class BackfillCompanyTimezones < DataDrip::Script
description "Recomputes the timezone of a company from its address."

input :company_id, :integer, required: true
input :mode, :enum, values: %w[fast thorough], required: true
input :dry_run, :boolean, default: true
input :effective_date, :date

def call
company = Company.find(company_id)
log "Recomputing timezone for #{company.name} (mode: #{mode})"

timezone = TimezoneResolver.call(company.address, mode: mode)

if dry_run
log "[dry run] would set timezone to #{timezone}"
else
company.update!(timezone: timezone)
log "Timezone updated to #{timezone}"
end
end
end
```

- **`description`** is shown in the UI so other developers know what the script does.
- **`input name, type, default:, required:`** declares a typed input. The UI form is generated automatically from these declarations, values are coerced to the declared type, and `required: true` inputs are validated before the run is created (required booleans accept `false` but not missing). Supported types are the same as backfill attributes: `:string`, `:integer`, `:decimal`, `:float`, `:boolean`, `:date`, `:time`, `:datetime` and `:enum`.
- **`log(message)`** appends a timestamped line to the run's output, persisted immediately and streamed to the show page. Note each `log` call issues one database update — log sparsely inside hot loops.

### Running Scripts

Navigate to `/data_drip/script_runs`, pick a script, fill in its inputs and trigger it — immediately or scheduled for later. The show page displays live status, output, and the error with backtrace if the script failed. Runs can only be deleted while they are still enqueued; anything that executed stays in the history.

### Script Hooks

Scripts fire lifecycle hooks with the same precedence rules as backfills (script class first, then the global handler): `on_script_run_pending`, `on_script_run_enqueued`, `on_script_run_running`, `on_script_run_completed`, `on_script_run_failed`. Each receives the `ScriptRun`.

### Script Configuration

```ruby
# The Active Job queue for script runs (default: ENV["DATA_DRIP_SCRIPT_QUEUE"] or :data_drip_script)
DataDrip.script_queue_name = :scripts
```

### Notes

- Datetime inputs are interpreted without timezone conversion (only the run's top-level "start at" field is converted from your browser timezone).
- If your `base_job_class` retries on errors, a failed script will re-run from scratch. Make scripts idempotent, or configure `discard_on` in your base job class.

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/factorialco/data_drip.
Expand Down
39 changes: 39 additions & 0 deletions app/assets/stylesheets/data_drip/tailwind.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@
--radius-lg: 0.5rem;
--radius-xl: 0.75rem;
--animate-pulse: pulse 2s cubic-bezier(0.4, 0, 0.6, 1) infinite;
--default-transition-duration: 150ms;
--default-transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);
--default-font-family: var(--font-sans);
--default-mono-font-family: var(--font-mono);
--color-drip-50: #f9f4fd;
Expand Down Expand Up @@ -261,6 +263,9 @@
.static {
position: static;
}
.sticky {
position: sticky;
}
.inset-x-0 {
inset-inline: calc(var(--spacing) * 0);
}
Expand Down Expand Up @@ -433,6 +438,9 @@
.max-h-72 {
max-height: calc(var(--spacing) * 72);
}
.min-h-24 {
min-height: calc(var(--spacing) * 24);
}
.min-h-dvh {
min-height: 100dvh;
}
Expand Down Expand Up @@ -927,6 +935,12 @@
.text-red-700 {
color: var(--color-red-700);
}
.text-red-700\/90 {
color: color-mix(in srgb, oklch(50.5% 0.213 27.518) 90%, transparent);
@supports (color: color-mix(in lab, red, red)) {
color: color-mix(in oklab, var(--color-red-700) 90%, transparent);
}
}
.text-red-800 {
color: var(--color-red-800);
}
Expand Down Expand Up @@ -1057,6 +1071,11 @@
.filter {
filter: var(--tw-blur,) var(--tw-brightness,) var(--tw-contrast,) var(--tw-grayscale,) var(--tw-hue-rotate,) var(--tw-invert,) var(--tw-saturate,) var(--tw-sepia,) var(--tw-drop-shadow,);
}
.transition-colors {
transition-property: color, background-color, border-color, outline-color, text-decoration-color, fill, stroke, --tw-gradient-from, --tw-gradient-via, --tw-gradient-to;
transition-timing-function: var(--tw-ease, var(--default-transition-timing-function));
transition-duration: var(--tw-duration, var(--default-transition-duration));
}
.select-all {
-webkit-user-select: all;
user-select: all;
Expand Down Expand Up @@ -1151,6 +1170,13 @@
}
}
}
.hover\:text-zinc-900 {
&:hover {
@media (hover: hover) {
color: var(--color-zinc-900);
}
}
}
.focus\:outline-2 {
&:focus {
outline-style: var(--tw-outline-style);
Expand Down Expand Up @@ -1214,6 +1240,11 @@
margin-inline: calc(var(--spacing) * -6);
}
}
.sm\:gap-x-6 {
@media (width >= 40rem) {
column-gap: calc(var(--spacing) * 6);
}
}
.sm\:px-6 {
@media (width >= 40rem) {
padding-inline: calc(var(--spacing) * 6);
Expand Down Expand Up @@ -1396,6 +1427,14 @@
color: var(--color-red-400);
}
}
.dark\:text-red-400\/90 {
@media (prefers-color-scheme: dark) {
color: color-mix(in srgb, oklch(70.4% 0.191 22.216) 90%, transparent);
@supports (color: color-mix(in lab, red, red)) {
color: color-mix(in oklab, var(--color-red-400) 90%, transparent);
}
}
}
.dark\:text-white {
@media (prefers-color-scheme: dark) {
color: var(--color-white);
Expand Down
25 changes: 2 additions & 23 deletions app/controllers/data_drip/backfill_runs_controller.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@
module DataDrip
class BackfillRunsController < DataDrip.base_controller_class.constantize
include DataDrip::Paginatable
include DataDrip::BackfillerContext

layout "data_drip/layouts/application"
helper_method :backfill_class_names, :find_current_backfiller
helper_method :backfill_class_names
helper DataDrip::BackfillRunsHelper

before_action :set_user_timezone

def index
@current_tab = params[:tab] || "my_runs"
@query = params[:q].to_s.strip
Expand Down Expand Up @@ -228,26 +227,6 @@ def backfill_options

private

# Exposed to views via helper_method (see top of class), but not a routable action.
def find_current_backfiller
if DataDrip.current_backfiller_method.blank?
raise "Missing DataDrip.current_backfiller_method, please set it in an initializer (like DataDrip.current_backfiller_method = :current_user"
end
unless respond_to?(DataDrip.current_backfiller_method, true)
raise "Invalid DataDrip.current_backfiller_method: #{DataDrip.current_backfiller_method}. Maybe you need to change the `base_controller_class` for DataDrip (currently: #{DataDrip.base_controller_class})?"
end

send(DataDrip.current_backfiller_method)
end

def set_user_timezone
@user_timezone =
params[:user_timezone].presence || session[:user_timezone] || "UTC"
session[:user_timezone] = @user_timezone if params[
:user_timezone
].present?
end

def backfill_run_params
params.require(:backfill_run).permit(
:backfill_class_name,
Expand Down
169 changes: 169 additions & 0 deletions app/controllers/data_drip/script_runs_controller.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
# frozen_string_literal: true

module DataDrip
class ScriptRunsController < DataDrip.base_controller_class.constantize
include DataDrip::Paginatable
include DataDrip::BackfillerContext

layout "data_drip/layouts/application"
helper_method :script_class_names
helper DataDrip::BackfillRunsHelper
helper DataDrip::ScriptRunsHelper

def index
@current_tab = params[:tab] || "my_runs"

runs = DataDrip::ScriptRun.all
my_runs = runs.where(backfiller: find_current_backfiller)

@my_runs_count = my_runs.count
@all_runs_count = runs.count

@stats = {
running: runs.running.count,
enqueued: runs.enqueued.count,
failed_recently: runs.failed.where(updated_at: 7.days.ago..).count,
completed_recently: runs.completed.where(updated_at: 7.days.ago..).count
}

base_scope = @current_tab == "my_runs" ? my_runs : runs

pagination_data =
paginate_collection(base_scope.order(created_at: :desc), per_page: 10)

@script_runs = pagination_data[:collection]
@pagination = pagination_data
end

def new
@script_run = DataDrip::ScriptRun.new
@recent_script_class_names = recent_script_class_names
end

def create
if params[:script_run][:start_at].present?
user_timezone = params[:user_timezone].presence || @user_timezone

if user_timezone.present?
Time.use_zone(user_timezone) do
local_time = Time.zone.parse(params[:script_run][:start_at])
params[:script_run][:start_at] = local_time.utc if local_time
end
end
end

@script_run =
DataDrip::ScriptRun.new(
script_run_params.merge(backfiller: find_current_backfiller)
)

if @script_run.save
local_time = @script_run.start_at.in_time_zone(@user_timezone)
notice =
if @script_run.start_at <= 1.minute.from_now
"Script run for #{@script_run.script_class_name} has been enqueued and will start shortly."
else
"Script run for #{@script_run.script_class_name} has been enqueued. Will run at #{local_time.strftime("%d-%m-%Y, %H:%M:%S %Z")}."
end

redirect_to script_runs_path(tab: "my_runs"), notice: notice
else
@recent_script_class_names = recent_script_class_names
render :new, status: :unprocessable_entity
end
end

def show
@script_run = DataDrip::ScriptRun.find(params[:id])
end

def destroy
@script_run = DataDrip::ScriptRun.find(params[:id])
if @script_run.enqueued?
@script_run.destroy!
flash[:notice] = "Script run has been deleted."
else
flash[
:alert
] = "Script run cannot be deleted as it is not in an enqueued state."
end
redirect_to script_runs_path(tab: params[:tab] || "my_runs")
end

def updates
@script_run = DataDrip::ScriptRun.find(params[:id])

render json: {
status: @script_run.status,
terminal: @script_run.completed? || @script_run.failed?,
status_html: helpers.status_tag(@script_run.status),
output: @script_run.output.to_s,
error_message: @script_run.error_message.to_s,
error_backtrace: @script_run.error_backtrace.to_s,
started_at:
helpers.format_datetime_in_user_timezone(
@script_run.started_at,
@user_timezone
),
finished_at:
helpers.format_datetime_in_user_timezone(
@script_run.finished_at,
@user_timezone
)
}
end

def script_inputs
script_class_name = params[:script_class_name]

if script_class_name.blank?
render json: { html: "" }
return
end

script_class =
DataDrip.scripts.find { |klass| klass.name == script_class_name }

if script_class.nil?
render json: { html: "" }
return
end

temp_run =
DataDrip::ScriptRun.new(
script_class_name: script_class_name,
inputs: {}
)

render json: { html: helpers.script_input_fields(temp_run) }
end

private

def script_run_params
params.require(:script_run).permit(
:script_class_name,
:start_at,
inputs: {}
)
end

def script_class_names
@script_class_names ||= DataDrip.scripts.map(&:name).compact.uniq.sort
end

# The current user's most-recently-run scripts (that still exist), surfaced
# at the top of the class picker for quick reselection.
def recent_script_class_names(limit: 6)
available = script_class_names
DataDrip::ScriptRun
.where(backfiller: find_current_backfiller)
.group(:script_class_name)
.maximum(:created_at)
.sort_by { |_name, run_at| -run_at.to_i }
.map(&:first)
.select { |name| available.include?(name) }
.first(limit)
end
end
end
Loading
Loading