Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Changelog

## Unreleased

### Added

- New `Drupal\rl\Experiment\VariantArmsTrait` providing reusable arm-id helpers
(`getArmIds`, `getArmText`, `buildVariantExperimentId`) for experiments that
follow the "v0 = original, v1..vN = stored variants" convention.
- New `Drupal\rl\Experiment\VariantParser` static helper for parsing textarea
variant input into normalized lists.
- New `Drupal\rl\Experiment\VariantExperimentInterface` extending
`ContentEntityInterface`. Variant-style experiment entities implement this
to plug into the shared selector / decorator / delete-form base classes.
- New `Drupal\rl\Experiment\VariantSelectorBase`,
`VariantExperimentDecoratorBase`, and `VariantExperimentDeleteFormBase` for
consumer modules to extend.
- New submodule `rl_page_title` for A/B testing page titles on any page (nodes,
Views displays, custom controllers, path-based). **Multilingual: per-language
experiments scoped via the `langcode` entity key, with an "all languages"
fallback. Each language has its own Thompson Sampling state.**
- New submodule `rl_menu_link` for A/B testing menu link labels (works for both
`menu_link_content` entities and YAML-defined links). **Multilingual: same
per-language scoping as rl_page_title.**

### Architecture

- Both new variant submodules use **content entities**, not config entities.
This is a deliberate choice to scale to tens of thousands of experiments
per site without the config-management cliff and the O(N) lookup penalty
of config entities. Lookups are indexed; admin lists use Views; multilingual
is first-class via the `langcode` entity key. Mirrors the Redirect module's
storage approach.

### Changed

- **BC break (minor):** `ExperimentManagerInterface` now declares three new
methods:
- `purgeExperiment(string $experiment_id)` - removes turns, rewards,
totals, snapshots, and registry entry for an experiment in a single
transaction.
- `getTotalTurnsMultiple(array $experiment_ids): array` - batched lookup
of total turns for many experiments in one query, used by list builders
to avoid N+1 query patterns.
- `getAllArmsDataMultiple(array $experiment_ids): array` - batched lookup
of arm data for many experiments in one query, paired with
`getTotalTurnsMultiple()`.

The same three methods are also added to `ExperimentDataStorageInterface`
(the lower-level storage contract).

Any downstream consumer that directly implements either interface (rather
than extending the concrete classes) will need to add these methods to
satisfy the contract. There are no known external implementations of
either interface at the time of this change.

Mitigation for downstream maintainers: copy the implementations from
`ExperimentManager` and `ExperimentDataStorage`. The batch methods are
thin `IN`-clause wrappers around the existing single-row queries; the
purge method uses transactional deletes across `rl_arm_data`,
`rl_experiment_totals`, `rl_arm_snapshots`, and `rl_experiment_registry`.
57 changes: 57 additions & 0 deletions modules/rl_menu_link/.agents/skills/rl_menu_link/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# RL Menu Link — A/B test menu link labels via Drush CLI

A/B test Drupal menu link labels for menu_link_content entities and
YAML-defined links using Thompson Sampling. Per-language scoping.

## Commands

### Discovery
- `drush rl:menu-link:list` — List experiments (`--enabled=yes|no|all`)
- `drush rl:menu-link:get <id>` — Full details + live arm stats + original label

### Lifecycle
- `drush rl:menu-link:create <plugin_id> --variants="A,B,C"` — Create
- `drush rl:menu-link:update <id> --label="X"` — Update
- `drush rl:menu-link:delete <id>` — Delete + purge analytics

### Common options
- `--variants="A,B,C"` or `--variants=A --variants=B` — Alternative labels
- `--label="..."` — Human-readable label
- `--langcode=es` — Per-language scope (default `und` = all languages)
- `--disabled` — Create as disabled
- `--dry-run` — Preview without applying

All commands output YAML. All state-changing commands support `--dry-run`.

## Plugin ID format

- User-created menu links: `menu_link_content:abc-uuid`
- YAML-defined links: machine name like `system.admin_content`,
`user.page`, `system.admin_structure`

## Concepts

- The original label is always tested as variant `v0` (read live from
the menu link manager).
- Stored variants are `v1`, `v2`, etc.
- Each `(plugin_id, langcode)` pair is its own experiment with
independent Thompson Sampling state.
- Reward signal: user clicked the tracked menu link.
- Lookups are indexed; tested at 10K+ experiments per site.

## Example

```bash
# A/B test the core "Content" admin link with alternatives.
drush rl:menu-link:create system.admin_content \
--variants="Content,Manage Content,Site Content"

# Check progress.
drush rl:menu-link:get <id>

# Delete + purge.
drush rl:menu-link:delete <id>
```

The same admin UI is available at /admin/config/services/rl-menu-link
via Views.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
display_name: "RL Menu Link"
description: "A/B test Drupal menu link labels (menu_link_content + YAML-defined) using Thompson Sampling. Per-language scoped, manage via Drush CLI."
default_prompt: "List all menu link experiments and their current status."
allow_implicit_invocation: true
138 changes: 138 additions & 0 deletions modules/rl_menu_link/.claude/skills/rl_menu_link/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
---
name: rl_menu_link
version: 1.0.0
description: >
A/B test Drupal menu link labels using Thompson Sampling. Works for
both menu_link_content entities and YAML-defined links. Per-language
scoping. Manage experiments via Drush CLI.
triggers:
- /rl-menu-link
- menu link test
- menu label test
- menu variant
- a/b test menu
- test menu link
- rl_menu_link
---

# RL Menu Link — Drush CLI

You are managing A/B testing experiments for Drupal menu link labels.
The rl_menu_link module is a content-entity-backed integration on top
of the parent rl module's Thompson Sampling engine.

## Preamble — Auto-discover Current State

```bash
# List all menu link experiments with stats.
drush rl:menu-link:list --format=yaml

# Filter to active experiments only.
drush rl:menu-link:list --enabled=yes --format=yaml
```

## Commands Reference

| Command | Alias | Purpose |
|---|---|---|
| `rl:menu-link:list` | `rl-mll` | List experiments (`--enabled=yes\|no\|all`) |
| `rl:menu-link:get <id>` | `rl-mlg` | Show full details + live arm stats + original label |
| `rl:menu-link:create <plugin>` | `rl-mlc` | Create experiment (`--variants`, `--label`, `--langcode`, `--disabled`, `--dry-run`) |
| `rl:menu-link:update <id>` | `rl-mlu` | Update label / variants / enable / disable (`--dry-run`) |
| `rl:menu-link:delete <id>` | `rl-mld` | Delete experiment AND purge RL analytics (`--dry-run`) |

All state-changing commands support `--dry-run`. All commands output YAML.

## Concepts

- **Plugin ID**: every menu link in Drupal has a plugin ID. For
user-created `menu_link_content` entities it looks like
`menu_link_content:abc-uuid` (the entity UUID). For YAML-defined links
from contrib/custom modules it is the link's machine name (e.g.,
`system.admin_content`, `user.page`).
- **Variant**: an alternative label text. The original label (whatever
the menu link normally renders) is always tested as variant `v0`.
Stored variants are `v1`, `v2`, etc.
- **Langcode**: experiments are scoped per language. Use `und` (the
default, `LANGCODE_NOT_SPECIFIED`) for "all languages". Lookup tries
language-specific match first, falls back to "all languages".
- **Reward**: a click on the tracked menu link.

## Workflow Examples

### Test alternatives for a content menu link

```bash
# Find the plugin ID first via the menu UI or:
drush ev "echo \\Drupal::entityTypeManager()->getStorage('menu_link_content')->load(1)->getPluginId();"

# Then create the experiment.
drush rl:menu-link:create menu_link_content:abc-uuid \
--variants="Services,What We Do,Solutions" \
--label="Services menu link test"
```

### Test a core admin menu link

```bash
drush rl:menu-link:create system.admin_content \
--variants="Content,Manage Content,Edit Site"
```

### Test a Spanish-only variant

```bash
drush rl:menu-link:create menu_link_content:abc-uuid \
--variants="Servicios,Lo Que Hacemos" \
--langcode=es
```

### Use multiple --variants flags

```bash
drush rl:menu-link:create system.admin_structure \
--variants="Structure" \
--variants="Site Structure" \
--variants="Layout"
```

### Preview before applying

```bash
drush rl:menu-link:create system.admin_content \
--variants="One,Two" \
--dry-run
```

### Check current state of an experiment

```bash
drush rl:menu-link:get <id>
# Returns: id, label, plugin id, original_label (read live from menu
# link manager), langcode, enabled, variants list, rl_experiment_id,
# total_turns, per-arm turns/rewards/rate.
```

## How variants reach end users

1. The module's preprocess_menu hook walks the menu tree on every
render and looks up an experiment by each item's plugin ID.
2. Matching items get their `title` swapped with the Thompson Sampling
winner, and `data-rl-ml-experiment-id` / `data-rl-ml-arm-id` data
attributes are injected onto the rendered anchor.
3. The bundled tracking JS uses IntersectionObserver to record an
impression when the link enters the viewport, and a click handler
to record a reward.
4. Tens of thousands of experiments scale via indexed `(plugin_id,
langcode)` lookups (content entities, not config).

## Notes

- The same menu link can have separate experiments per language.
- Vertical tabs on `menu_link_content` edit forms also create menu link
experiments; the Drush commands operate on the same content entities.
- The admin UI is at `/admin/config/services/rl-menu-link` (Views).
- Analytics are managed by the parent `rl` module; use `drush rl:list`
and the `rl:experiment:*` commands to inspect raw turn/reward data.
- Deleting an experiment via this command purges the RL analytics
tables (turns, rewards, snapshots, registry) atomically.
118 changes: 118 additions & 0 deletions modules/rl_menu_link/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# RL Menu Link

A/B test menu link labels using Thompson Sampling. Works for both
`menu_link_content` entities (user-created menu links) and YAML-defined
menu links from contrib/custom modules.

## What it does

You give it alternative labels for a menu link. It rotates them across menu
renders using Thompson Sampling, records impressions when the link is
visible and rewards when the link is clicked, and converges on the
best-performing label.

## How it works

### Storage

A **content entity** per experiment, `rl_menu_link_experiment`, stores:

- `menu_link_plugin_id` - the menu link plugin ID being tested. For
`menu_link_content` entities this looks like
`menu_link_content:abc-uuid`. For YAML-defined links it is the link's
machine name (e.g., `system.admin_content`).
- `langcode` - the language scope
- `variants_data` - JSON-encoded list of alternative labels
- `enabled` - whether the experiment is currently running

Indexed lookups on `(menu_link_plugin_id, langcode)` keep selector latency
constant regardless of how many experiments exist.

The original label is always tested as **arm v0** and is read live from the
menu link manager - it is **not** stored on the experiment entity. Stored
variants are arms v1, v2, ... vN.

The RL experiment ID is a deterministic hash:
`rl_menu_link-{12-char-sha1-of-plugin-id-pipe-langcode}`. The hash includes
the langcode so each language has its own Thompson Sampling state.

### Multilingual

Each (plugin_id, langcode) pair is its own experiment row. Lookup tries
the current request language first, then falls back to "all languages"
(`LANGCODE_NOT_SPECIFIED`) if no language-specific experiment exists.
Same model as the Redirect module.

### Runtime

1. `hook_preprocess_menu()` walks the menu tree, looks up an active
experiment for each item's plugin ID, and swaps `$item['title']` with
the winning variant. It also injects `data-rl-ml-experiment-id` and
`data-rl-ml-arm-id` attributes onto the rendered anchor so the tracking
JavaScript can match anchors precisely.
2. The preprocess hook attaches `rl_menu_link:all` plus per-plugin-id cache
tags so saving an experiment can invalidate the cached menu output
immediately.
3. `js/menu-tracking.js` uses `IntersectionObserver` to record a turn
(impression) when a tracked anchor enters the viewport, and a click
listener to record a reward when the user clicks. Each visit and each
click is its own event - there is no per-session cap, so repeat
visitors do not depress the conversion signal.
4. Both events are POSTed via `navigator.sendBeacon()` to `rl.php`.

### UX flows

- **menu_link_content entities**: vertical tab "Label variants" on the
menu link edit form (advanced tabs group), inline textarea editing.
- **YAML-defined menu links** (e.g., `system.admin_content`): standalone
admin form at `/admin/config/services/rl-menu-link/add` with a
textfield for the plugin ID. The form validates that the plugin ID is
registered with the menu link manager.
- **Editing**: same vertical tab on the menu link edit form, or the
admin list at `/admin/config/services/rl-menu-link`.
- **Deletion**: dedicated delete confirmation form that purges the RL
analytics tables before removing the config entity.

## Configuration

There is no settings form. The page cache TTL override (60 seconds while
an experiment is active) is hardcoded as a class constant. Reward signal
is "user clicked the link", which is the natural conversion signal for
menu links and does not need configuration.

## Permissions

- `administer rl menu link experiments` - create, edit, delete
experiments. Restricted access.

## Cache invalidation

Saving or deleting an experiment invalidates two cache tags:

- `rl_menu_link:all` - attached to every rendered menu, so any first-time
experiment invalidates all cached menu output.
- `rl_menu_link:{plugin_id}` - attached to menus that contain the
affected link, for more targeted invalidation when the experiment
already existed.

## Known limitations

- **Click reward double-counting prevention**: a single click event is
guarded by a per-anchor data attribute that clears immediately after
the event finishes propagating. Subsequent clicks in the same page load
do count, by design - the goal is to track engagement, not session-level
uniqueness.
- **State divergence on the vertical tab path**: identical to the
rl_page_title note - inline experiment writes happen after the parent
menu link save commits, so a write failure is logged but does not roll
back the parent save.
- **YAML link plugin IDs are not autocompleted**: the standalone form
takes the plugin ID as a textfield with examples in the description.
Adding an autocomplete is a Phase 2 enhancement.

## Tests

Coverage is provided by the parent rl module's e2e tests under
`scripts/e2e/`, which exercise install, experiment CRUD, and analytics
end-to-end against a real Drupal site. Run via
`docker compose --profile test run e2e-test` from the rl module root.
Loading