Skip to content

ETL Orchestrator

A web app that orchestrates ETL Engine pipelines: upload Rabbit-in-a-Hat mapping files (RiaHs), wire them into a DAG by what they read and write, run them against an OMOP CDM, and watch every step unfold live.

The orchestrator replaces the hand-written shell scripts and Excel sheets that glue multi-RiaH ETL loads together today. One web UI for configure + run + history.

Get started Build your first pipeline

Download docs (PDF)

What it does

  • Pipeline configuration. Upload one RiaH per step, attach any Usagi / CSVMap lookup files, set step ordering. The orchestrator auto-wires the DAG by reading each RiaH — GetCDMTableId(@table) becomes a consume edge to whichever sibling step stores index/<table>; multiple writers of the same globalSeq/<seq> get serialized so they don't race.
  • Environments. Each (pipeline, env) pair is a separate target: source DB, CDM DB, license, log level, per-env etl.conf overrides, engine env vars + image overrides. Run the same pipeline against dev and prod with different connections — no copy-paste.
  • Running. Live DAG view with per-step status colours, per-table row counts streaming as the engine emits them, full engine log capture, abort signal that crosses worker boundaries.
  • Selective re-run. Re-run only the steps that share a CDM table with your target — the design-correct re-run cascade. Pre/post automation stay out of scope; unrelated dimension data is untouched.
  • Save / Load. Export a pipeline (or an env) as a single bundle file (.etlopipeline / .etloenv) and load it on another instance — perfect for backup, sharing, or moving an env between machines.

Who it's for

People running OHDSI OMOP CDM loads from one or more source-data extracts, using the Rook IT ETL Engine to do the per-step mapping work. If today you hand-edit run-N.sh scripts and etl.conf files, this gives you a configure-once UI on top.