Skip to content

DAG wiring

The pipeline DAG you see in Configure mode isn't drawn by hand. The orchestrator derives it from what's in the RiaH files, with a small amount of OMOP-aware enrichment on top. This page is about how it infers those edges and why you can trust the result.

What the engine actually needs

Most multi-RiaH pipelines share state between steps. The engine has exactly two cross-step state mechanisms:

  • StoreSequence(<name>) — write a named global sequence value to disk. Subsequent steps continue from there via AutoGenId(<name>). Used when two RiaHs both append rows to the same CDM table (e.g. claims and conditions both writing into condition_occurrence — they share condition_occurrence_id_seq).
  • StoreIndexMap(<source_key…> → <cdm_key…>) — write a per-key lookup file. Subsequent steps read it via GetCDMTableId(@<table>, …) to resolve a source-side reference into the CDM ID an earlier step assigned.

Together, those two are how the engine's worker threads stay parallel-safe across steps: writes are local to one step, reads are materialized as files between steps.

How the orchestrator parses this

On every RiaH upload, the orchestrator parses the file and records:

  • Produces — the names of every StoreSequence and StoreIndexMap rule the RiaH declares.
  • Consumes — every GetCDMTableId(@X, …) and AutoGenId(<name>) that references a name some other step produces.

These are stored against the step's RiaH version, so updating a RiaH re-derives the edges immediately.

How the edges get drawn

Synthmas configure DAG showing inferred edges

The Synthmas DAG — every edge here was derived from parsing the RiaHs, never declared by hand. `synthmas-person` produces `index/person`; `synthmas-conditions`, `-claims`, `-allergies`, `-immunizations`, `-medications` all consume it (hence the fan-out from the centre-left).

For each pair of steps A, B:

  • If A produces X and B consumes X → solid edge A → B, labeled with the artifact name. That's the artifact wiring layer.
  • If A and B both write into CDM table T (per their expected-tables lists) → they're declared related (no edge — they collaborate via shared StoreSequence — but they share a cascade, see Cascade).

The orchestrator additionally surfaces:

  • FK dependencies between the tables A writes and tables B writes, derived from the OMOP CDM schema for the pipeline's version. These render as dashed edges in the configure DAG — they're informational ("B references rows A creates via foreign key"), not execution-gating.

What you don't need to specify

  • Step order. The dispatcher topo-sorts the DAG before each run.
  • Edge directions. They come straight from "X is produced here, consumed there."
  • Shared-table coordination. The auto-detected shared-writer relationship is what powers cascade re-runs.

If two of your steps SHOULD share state but the orchestrator hasn't drawn an edge, the RiaH side of the wiring is the place to look — chances are the producer is using an unnamed StoreSequence or the consumer's GetCDMTableId is referencing a literal table instead of @<storedIndexName>.

What the orchestrator does NOT auto-infer

  • External data dependencies. If your step's mapping refers to a Usagi or CSVMap file, those aren't artifact-wiring inputs from another step — they're per-step lookup files you upload alongside the RiaH.
  • Vocabulary. Vocab is pipeline-scoped, loaded once per env by pre-automation. There's no per-step "I depend on the vocab" edge — every step does, implicitly.