Re-running (cascade)

When you change a step's RiaH or want to re-process a step's output, the orchestrator computes the re-run cascade: the target step plus every sibling step that shares at least one CDM table with it (transitively). All cascade members are wiped + re-run together; nothing else is touched.

The full conceptual story is in Concepts → Re-run cascade. This page is the day-to-day how-to.

Two ways to trigger

From the DAG (right-click)

In Run mode, right-click any etl_map node → ↻ Rerun (cascade).

A confirm dialog tells you exactly what's about to happen:

Re-run cascade for 'synthmas-conditions'?

Will re-run 2 step(s): • synthmas-claims • synthmas-conditions

Will wipe 2 CDM table(s): condition_occurrence, cost

Click through and the orchestrator:

1. Deletes the cascade-touched CSV files (csv_cdm/<env>/<table>) and the on-disk globalSeq/<seq> files for sequences any cascade member writes (so IDs restart at 1, not from the prior max).
2. Starts a new pipeline_run with trigger rerun_cascade and the cascade scope persisted on the row.
3. Marks every non-cascade step skipped with failure_reason='rerun_skip_outside_cascade' — they don't run, but the dispatcher trusts their prior artifacts on disk.
4. Runs the cascade members in topo order; the same shared-globalSeq edges that prevent races during a full run now keep the cascade internally ordered.

Pre/post automation are never cascade members (they don't write data tables, just manage CDM lifecycle). The right-click menu doesn't appear on those nodes — re-run via the full ▶ Run pipeline button if you need to re-init the CDM.

From the step-state modal (left-click)

Same flow, different entry. Click a node → modal opens → ↻ Rerun (cascade) button in the header. Same confirm dialog, same effect.

When the cascade is just one step

For single-writer CDM tables, the cascade is {self} — a true single-step re-run. In the synthmas demo that's location, care-site, provider, person, visits, measurement (clinical), procedures, drugs, devices, cdm-source. The confirm dialog will list just one member.

When the cascade includes others

Three multi-writer pairs in synthmas:

target	cascade	wiped tables
clinical ↔ allergies	both	measurement, observation

The pairing is symmetric: triggering either member pulls in the other. (clinical/allergies is the only shared-table pair left in the demo — claims now writes only cost, and the careplans step was removed; observation_period is produced by post-automation, not a cascade member.)

DAG colours during + after a cascade

While a cascade run is in flight, the DAG strip shows only the cascade members as running. The rest of the nodes stay coloured by their effective state — the latest run in which they actually ran, not the cascade run's "skipped" status. So after re-running {conditions, claims}, clinical / allergies / procedures etc. still show green because their data is unchanged.

The run header / ribbon (run id, timer, total progress) still ties to the literal latest pipeline_run — so "what cascade did I just trigger" and "what's currently in my CDM, per step" don't fight each other.

Re-running after an orchestrator restart

The cascade scope is persisted on pipeline_run.cascade_step_ids_json. If the orchestrator container is restarted mid-cascade (worker reload, host reboot), the orphan-recovery path rebuilds it and resumes correctly — no sliding outside the cascade. A per-pipeline-run file lock (<data>/resume_locks/<id>.lock, atomic via O_EXCL) keeps multiple gunicorn workers from double-dispatching the same orphan.

When to use the full re-run instead

Use ▶ Run pipeline (not cascade) when:

• The CDM should be fully reset (pre-automation wipes every pipeline-written table + the env-scoped indexes/seq state).
• You changed dimension data that everything depends on (person, visits), and a cascade through every event step would re-run most of the pipeline anyway.
• You're not sure what's affected and want a clean baseline.