Skip to content

Run on CSV

The CSV target is the fastest way to see the full pipeline run end to end. No database to set up; the engine writes per-table CSV files to disk and serves the vocab from the Athena bundle directly.

Pin engine 1.4.1 or newer

Engine 1.4.1 made the CSV target semantically identical to the RDBMS targets for CDM-lifecycle management (canonical OMOP headers via create omop cdm: once, truncate-to-canonical via clean omop data: true, virtual-field-aware header validation, append-not- truncate loadWriteTable). Earlier engines have known correctness gaps on multi-writer CDM tables (e.g. cost written by both synthmas-claims and synthmas-cost-drugs). The orchestrator's settings page lets you pin a per-pipeline engine image; CSV demo envs should pin rook-it.com/etl-engine:1.4.1 or later.

Create the environment

Pipelines → Synthmas → Environments → New:

Field Value
Name csv-demo
Source DB backend CSV
Source set the synthea-1k source set you uploaded
CDM DB backend CSV
Log level info
Vocab cache size 10000 (default)
Worker threads 10 (default)

Save. The orchestrator stores the env config; nothing happens yet.

Run it

On the env page, click ▶ Run pipeline. You should see:

  1. Pre-automation turns blue (running), then green. On CSV the engine writes each <cdm-out>/<table> with its canonical OMOP header (so multi-writer tables like cost see the full column set regardless of which step writes first), truncates any existing tables back to that canonical header (SQL TRUNCATE semantics, kept schema), and the orchestrator extracts the Athena vocab CSVs into the same dir. Fast — usually under a second once the vocab is extracted.
  2. Each etl_map step turns blue in topo order — location → care_site → provider → person → visits → conditions → claims → clinical → allergies → procedures → drugs → devices → coverage → cdm-source.
  3. Post-automation turns blue and then green: on CSV it generates observation_period (one row per person, via the engine's driver-agnostic builder). Eras + preceding-visit-IDs stay skipped on CSV — see Limitations of CSV.

Total wall-clock: ~10 minutes on a typical laptop (1,171 patients).

Limitations of CSV

CSV is great for the demo — no DB to set up, file-by-file inspectable output — but it has structural limits you'll hit on real loads.

observation_period: yes; eras + preceding-visit-IDs: still RDBMS-only

observation_period is generated on CSV — the engine ships a driver-agnostic post-automation builder that folds each person's clinical events into one period (MIN start … MAX end). The Synthmas demo relies on this (the old careplans/coverage observation_period writes were removed, and the careplans step with them), so post-automation runs and turns green on CSV.

Eras and preceding-visit-IDs are not produced on the CSV demo:

  • condition_era, drug_era, dose_era are not created — the orchestrator forces the era toggles off for a CSV target (lifting that is a follow-up).
  • visit_occurrence.preceding_visit_occurrence_id stays NULL for every row.

If your downstream analysis needs eras or visit-graph navigation, switch to an RDBMS target — the same pipeline, run against a postgres env, produces both.

No referential integrity, no downstream tooling

The engine can't enforce FK constraints across CSV files at write time — a condition_occurrence.person_id pointing at a non-existent person.person_id will land on disk without complaint. And tools that expect a real OMOP database (Atlas, HADES, the OHDSI Data Quality Dashboard) can't connect to a folder of CSVs. For both reasons CSV is a demo / inspection target, not a production destination.

Where the data lands

CSV CDM tables live under /var/lib/orchestrator/csv_cdm/<env_id>/<table>. One file per CDM table, comma-separated, with the CDM header line at the top.

docker exec etl-orchestrator-orchestrator-1 ls /var/lib/orchestrator/csv_cdm/2/
# care_site  condition_occurrence  cost  death  device_exposure
# drug_exposure  location  measurement  observation  observation_period
# payer_plan_period  person  procedure_occurrence  provider
# visit_detail  visit_occurrence  cdm_source  (plus Athena vocab files)

Spot-check a multi-writer table to confirm the canonical-header behavior is working — cost is written by both synthmas-claims and synthmas-cost-drugs, so its header has to carry every column either writer mentions:

docker exec etl-orchestrator-orchestrator-1 \
  head -1 /var/lib/orchestrator/csv_cdm/2/cost | tr ',' '\n' | nl
#   1 cost_id
#   2 cost_event_id
#   3 cost_domain_id
#   4 cost_type_concept_id
#  ...
#  21 drg_concept_id            ← from synthmas-cost-drugs
#  22 drg_source_value          ← from synthmas-cost-drugs

22 columns, including the cost-drugs-only ones — that's the canonical OMOP 5.4 cost schema, laid by the engine in pre_automation. On older engines (pre-1.4.0) the file's header had only the first writer's columns and synthmas-cost-drugs would abort with "field drg_concept_id is missing in table".

→ Next: Run on PostgreSQL — the same pipeline, RDBMS target, row-for-row identical CDM output on engine 1.4.1+.