Skip to content

Pipelines

A pipeline is the recipe: which CDM version to load into, which mapping files (RiaHs) describe the source → CDM transformation, and how the steps wire together. It owns no data of its own — environments attach to it and run it.

Pipelines list

Pipelines index — OMOP version, pinned engine image, step + env counts at a glance.

Pipeline-level settings

Setting What it does
Name Just for you — pick anything unique within the orchestrator.
OMOP version 5.3, 5.4, or 6.0. The engine creates the matching schema during pre-automation. All RiaHs uploaded later must match (the configure page rejects a mismatch).
Engine image Defaults to whatever the global default is in Settings. Override per-pipeline to pin a specific engine release for reproducibility.
Vocabulary Upload an Athena bundle here; every env using this pipeline shares it.
Post-automation toggles create observation periods, add preceding visit IDs, create condition / drug / dose eras — engine features that run after the etl_map steps. Toggle off the ones you don't need. (observation_period is generated on both CSV and RDBMS; the others are RDBMS-only.)
Description Free-text. Shown in the pipelines list.

Steps

A pipeline has three kinds of step:

  • Pre-automation (fixed, one per pipeline) — builds the CDM schema
  • loads vocab. The orchestrator seeds this automatically; you don't delete or reorder it.
  • etl_map (you create these) — the actual mapping work. One RiaH per step, plus optional Usagi / CSVMap lookup files. You can have as few or as many as you want.
  • Post-automation (fixed, one per pipeline) — observation_period, eras + preceding-visit-IDs. Toggled by the pipeline-level flags.

Pre-automation always runs first, post-automation always runs last. Between them, etl_map steps run in topological order derived from the wiring (next section).

Configure mode

The configure-mode view of a pipeline is a Cytoscape DAG:

  • Nodes are steps. Shape encodes kind (rectangle = etl_map, tag-shape = pre-automation, hexagon = post-automation).
  • Solid edges are artifact wiring — one step's StoreIndexMap/StoreSequence output feeds another step's GetCDMTableId/AutoGenId consumer. The orchestrator detects these automatically by parsing the RiaHs.
  • Dashed edges are FK awareness — derived from the OMOP CDM itself, shown as supplemental context. Not gated on; just a "you should know about this" hint.

Configure-mode DAG for Synthmas

Configure mode for the Synthmas pipeline: pre-automation pinned on the left, post-automation on the right, etl_map steps arranged by the topology the orchestrator inferred from RiaH artifact wiring. Drag nodes to reshape; the layout persists.

Drag nodes to lay out the DAG manually — the orchestrator persists the layout per pipeline.

Layout = topology

Step ordering on screen is the dispatcher's actual execution order. Drag a step to a different position and the next run honours it; the orchestrator only refuses an arrangement that violates a hard topo constraint (a consumer can't be moved before its producer).