fix: Add cleanup migration for erroneous records in applet_events (M2-9237)

[sultanofcardio]

📝 Description

🔗 Jira Ticket M2-9237

Note

This PR should not be reviewed until this ticket has been prioritised

This PR adds a migration that moves the erroneous entries in the applet_events table to a temporary table called applet_events_cleanup for eventual permanent deletion. This should fix failed data exports in applets with deleted activities/flows before the implementation of M2 9161.

Prior to the implementation of that ticket, rows were created in the applet_events table linking activities/flows (and their events) with versions of the applet they don't exist in. The downstream effect of this is that the API endpoint that returns applet schedule history fails because the query returns NULL values for the result of the JOINs with the activity_historiesand flow_histories tables for these rows

🪤 Peer Testing

Important

Start this test before running the migration

Prepare

1. Create an applet with two activities and two flows. Save it.
2. Remove the first activity and the first flow from the applet. Save it
3. Perform a data export with the schedule history checkbox checked. Confirm the export succeeds.
4. Now we need to simulate the erroneous data by adding extra rows to the applet_events table representing the deleted items linked to the new applet version

To do this, we will first need to identify the IDs of the deleted activity and flow. Run this query to get a rundown of the history of the applet (replace the applet ID)

SELECT applet_histories.id_version applet_history_id,
       coalesce(flow_histories.id_version, activity_histories.id_version) activity_or_flow_history_id,
       coalesce(flow_histories.name, activity_histories.name) activity_or_flow_name,
       event_histories.id_version event_history_id
FROM event_histories
         JOIN applet_events ON event_histories.id_version = applet_events.event_id
         JOIN applet_histories ON applet_events.applet_id = applet_histories.id_version
         LEFT OUTER JOIN activity_histories
                         ON event_histories.activity_id = activity_histories.id AND
                            applet_histories.id_version = activity_histories.applet_id
         LEFT OUTER JOIN flow_histories
                         ON event_histories.activity_flow_id = flow_histories.id AND
                            applet_histories.id_version = flow_histories.applet_id
WHERE applet_events.applet_id ilike '451a9989-a3b2-4844-818f-130dceaf55a0%'
ORDER BY applet_events.created_at;

This should give a result like this

These are the entities are are interested in:

• New Activity 1 (d8b10853-8db6-4366-92df-34aa73430672) and its event history ID 5a00c6cb-f8a0-425a-bbd0-1c45090d6ff7_20250521-1
• Flow 1 (7a75475e-6d9a-4451-a0e9-ad09500f7d57) and its event history ID 422a8b18-a123-4b6e-8569-7583a3c88ff7_20250521-1

Now let's insert new rows into applet_events

INSERT INTO applet_events (is_deleted, created_at, updated_at, migrated_date, migrated_updated, applet_id, event_id)
SELECT is_deleted,
       created_at,
       updated_at,
       migrated_date,
       migrated_updated,
       applet_id,
       '5a00c6cb-f8a0-425a-bbd0-1c45090d6ff7_20250521-1' as event_id
FROM applet_events
WHERE applet_id = '451a9989-a3b2-4844-818f-130dceaf55a0_2.0.0'
AND event_id = 'f920005b-f1c8-424b-9356-78d2c3a4be17_20250521-1';

INSERT INTO applet_events (is_deleted, created_at, updated_at, migrated_date, migrated_updated, applet_id, event_id)
SELECT is_deleted,
       created_at,
       updated_at,
       migrated_date,
       migrated_updated,
       applet_id,
       '422a8b18-a123-4b6e-8569-7583a3c88ff7_20250521-1' as event_id
FROM applet_events
WHERE applet_id = '451a9989-a3b2-4844-818f-130dceaf55a0_2.0.0'
  AND event_id = 'f920005b-f1c8-424b-9356-78d2c3a4be17_20250521-1';

Here we're using the event history ID of New Activity 2 (which already has a row in applet_events for applet version 2) for the insert

Test

1. Log into the admin panel
2. Perform a data export with the schedule history checkbox checked
3. Confirm the data export fails with a 500 error
4. Run the migration with alembic upgrade head
5. Perform the data export again and confirm it works this time

✏️ Notes

N/A

[github-actions]

➡️ Preview environment created: Click Me!