Skip to content

Your First Pipeline

End-to-end walkthrough — create a pipeline, add a step with a RiaH, define an environment, run.

1. Create a pipeline

Pipelines → + New.

Fields:

Field Purpose
Name Globally unique; appears in the menubar picker.
OMOP version The CDM version every RiaH in this pipeline must target. Uploaded RiaHs are rejected if their cdmDb.dbName (CDMV5.4 etc.) doesn't match.
Engine image Default container image to spawn for steps (e.g. etl-engine:latest). Per-env override available later.

2. Add a step

Open the new pipeline (Configure mode). The DAG canvas shows the fixed Pre-automation and Post-automation nodes. Click Add step, name the step (e.g. location), and you land on its Configure page.

Upload three things (only the RiaH is mandatory):

  • RiaH (.json.gz, exported from Rabbit-in-a-Hat). The orchestrator parses it to auto-populate:
    • expected tables (every CDM table the RiaH writes via tableToTableMaps)
    • produces (named StoreSequence / StoreIndexMap artifacts)
    • consumes (cross-step GetCDMTableId(@<table>) lookups, resolved against sibling steps' produces)
  • Usagi CSV(s) — vocab mapping lookups the RiaH's UsagiMap rules reference. Usagi files are plain CSV in a specific column layout (sourceCode, sourceName, sourceFrequency, targetConceptId, mappingStatus, …) — the same format the OHDSI Usagi tool exports.
  • CSVMap csv(s) — plain CSV lookups the RiaH's CSVMap rules reference.

The DAG view updates the moment the upload succeeds: the new node appears, and any cross-step consumes show up as arrows from the producer to the new node.

3. Define an environment

Each pipeline targets one or more environments (= "where the source data lives + where the CDM lives + what license to use"). Add one:

Pipeline → Environments → + New.

Tabs:

  • Source database — csv (a per-pipeline source set) or one of the RDBMS backends (postgres, mysql/mariadb, sqlserver). For RDBMS, fill in host/port/db/schema/user/password.
  • CDM database — same shape; csv writes per-table CSV files to a per-env directory.
  • Settings — log level, vocab cache size, worker threads, optional engine image override.

When source is csv, you'll need a source set — a named collection of CSV files at the pipeline scope. Upload them via Pipeline → Source sets.

4. Run

Switch the env's URL to Run mode (/run). Press ▶ Run pipeline. The DAG starts colouring per step:

  • white border = not yet run.
  • blue = currently running.
  • green = success (every expected CDM table loaded).
  • amber = partial (engine exited 0 but some expected tables didn't load).
  • red = failed.
  • grey = skipped (intentional — e.g. steps outside a re-run cascade, or post_automation when nothing it would produce is enabled).

The right-side strip lists currently-running steps with per-table progress bars. Each step's full state is one click away — tap the node, the modal opens with status, per-table row counts, the engine log (⬇ Log to download), and the Rerun (cascade) button.

5. Make a change, re-run just the affected steps

This is where the orchestrator earns its keep. Say your conditions RiaH had a bug and you've fixed + re-uploaded it. Right-click conditions in the DAG → Rerun (cascade). A confirm dialog tells you exactly which steps will be re-run (the cascade — every step that shares a CDM table with conditions, transitively) and which CDM tables will be wiped before the re-run. Click through and only those steps run; everything else stays as it was.

→ How the cascade is computed.

6. Save a pipeline you're happy with

Configure → ⤓ Save. Downloads a .etlopipeline bundle (zip) containing the pipeline config, every step file (RiaH/Usagi/CSVMap, sha256-deduped), all wiring (resolved by step name so it re-links cleanly), and optionally the vocabulary zip. Hit Pipelines → ⤒ Import on another instance (or on the same one as a duplicate) and the bundle recreates the whole pipeline in one click.

→ Bundle formats.