Re-run cascade — shared-table semantics

The problem

You've changed one step's RiaH (say conditions) and want to re-load condition_occurrence without re-doing everything. Two naive answers are both wrong:

• "Re-run only conditions." Wrong. conditions and claims both write condition_occurrence (claims appends via a named StoreSequence). If we re-run only conditions, its rows replace the original 1..N range, but claims' rows from the prior run still sit at N+1..M referencing the old IDs. The CDM is left silently inconsistent.
• "Re-run conditions + everything topologically after it." Wrong — too greedy. It re-runs clinical, allergies, procedures, drugs, ... — anything appearing later in topo order, even if they don't share any CDM table with conditions and thus aren't affected.

The right answer

The re-run cascade is the connected component of steps you reach from the target by walking the "shares a written CDM table with" relation, transitively. Concretely:

cascade(N) = { N } ∪ { every step S that shares ≥ 1 written CDM table with
                       some step already in cascade(N) }

Why this is exactly right: the engine's per-table cleanup is DELETE FROM T WHERE pk(T) >= captured_seq_before_N, which wipes everything N wrote and everything any later step wrote to the same table (their IDs are higher than N's captured seq). So you can't selectively keep one writer's rows. The downstream set the wipe forces to re-run is, by construction, every other writer of any of N's tables — and transitively their tables' other writers — i.e. the cascade.

For tables with one writer (the common case), the cascade is {N} — a true single-step re-run.

For shared tables, the cascade always contains every writer. In synthmas:

trigger	cascade	tables wiped
clinical ↔ allergies	both	measurement, observation
any single-writer step	self	one table

(clinical and allergies both write observation — the one shared-table pair left in the demo. conditions/claims no longer share a table — claims now writes only cost — and the careplans step was removed.)

Pre/post automation never enter the cascade — they don't write source-driven data tables. Note post-automation now does produce observation_period (one per person, both backends) and, on RDBMS, the eras + preceding-visit-IDs.

CSV CDM cleanup, concretely

The general DELETE-pivot rule:

pivot = max(captured_seq.next_id across NON-cascade writers of T)
      = 0   when every writer of T is in the cascade (the common case)
DELETE FROM T WHERE pk(T) >= pivot

In CSV mode, with the cascade being closed (every writer is included by construction), pivot = 0 always → DELETE FROM T = delete the whole CSV file. The orchestrator's csv_cdm_cleanup does exactly that:

1. /csv_cdm/<env>/<table> — removed for every cascade-touched table.
2. /indexes/<env>/globalSeq/<seq> — removed for every named sequence any cascade member produces (so AutoGenId restarts at 1, not from the prior max).
3. /indexes/<env>/<table>/ — removed for every StoreIndexMap a cascade member produces (the engine rewrites it during the run).

Index dirs produced by steps outside the cascade are left intact — their non-cascade consumers (which we are NOT re-running) still depend on them.

What survives across runs

shared_indexes is env-scoped (/indexes/<env>/), not per-run. Cascade re-runs need this — the cascade target's upstream isn't re-run, but the target still has to read the upstream's StoreIndexMap on disk. Full re-runs reset cleanly because pre_automation wipes the env-scoped indexes dir as part of its CDM-lifecycle work.

Resume semantics

pipeline_run.cascade_step_ids_json persists the cascade scope. If the orchestrator container is restarted mid-cascade (worker reload, host reboot), the resume path:

1. Acquires <data>/resume_locks/<pipeline_run_id>.lock via O_EXCL — only one gunicorn worker can resume a given orphan. Stale-pid sweep handles the case where the prior owner died without releasing.
2. Reads cascade_step_ids_json and re-passes it to _dispatch so the skip-outside-cascade branch keeps firing for the rest of the run.
3. The cleanup phase doesn't re-run on resume (it ran in the original kick-off; resume just continues the topo loop).

Without (2) the resumed dispatcher would treat every step with no prior step_run as a normal dispatch target and slide outside the cascade. This is the bug behind the "why did clinical run when I asked for a conditions cascade?" question that drove the persistence design.

Single-flight, abort, and the cascade

The orchestrator's single-flight guard ("one pipeline_run in flight globally") applies to cascades too: a cascade re-run is just another pipeline_run, and you can't trigger another while one is running. Abort works the same way — pipeline_run.abort_requested propagates across workers.