Skip to content

First ETL Run

This page walks through the prerequisites and first execution of the ETL Engine. It assumes the engine has already been installed and the folder layout is intact.

Quick path — the Synthmas demo

The Synthmas demo runs on top of a base engine install, layered in via a small overlay tarball that adds the Rabbit-in-a-Hat file, the Usagi files it references, a Synthmas-tuned etl.conf, and a script that pulls the source CSVs from MITRE.

Both tarballs — the latest engine and the Synthmas overlay — are free downloads at rook-it.com/downloads (no login). Older engine versions and debug builds live behind a login on the same site for paid customers and Rook IT consultants.

From a fresh download of both tarballs to a populated CDM:

$ tar xzf etl-engine-X.Y.Z.tar.gz                          # the engine
$ cd etl-engine-X.Y.Z
$ tar xzf ../synthmas-single-RiaH-X.Y.Z.tar.gz --strip-components=1   # overlay the demo
$ ./bin/install-etl-engine.sh                              # load the Docker image
$ ./bin/setup-synthmas.sh                                  # pull source CSVs from MITRE
$ ./bin/check-vocabs.sh                                    # see which vocabularies the RiaH needs
$ ./bin/setup-vocab.sh ~/Downloads/vocabulary_*_v5.zip     # (optional) unpack an Athena vocab
$ ./bin/run-etl-engine.sh                                  # run the engine

Output appears in etc/etl-engine.d/cdm-out/ as a directory of CDM CSV files. The vocab step is optional — without it, VocabMap / VocabSourceId / UsagiMap rules leave their concept fields NULL but the rest of the CDM still populates.

See Vocabulary from Athena for the full walkthrough of selecting the right vocabularies (the Synthmas RiaH needs SNOMED, RxNorm, LOINC, and CVX).

The rest of this page covers the same flow in more detail, plus how to point the engine at your own data and Rabbit-in-a-Hat file.

Prerequisites

Before running the engine, make sure the following are in place:

  1. etl.conf — under etc/etl-engine/. The shipped file is a working Free-tier template (CSV in / CSV out) — see the Configuration Reference for every key, and uncomment the Paid-tier blocks at the bottom of the file when you upgrade.
  2. A Rabbit-in-a-Hat file — alongside etl.conf, named in the rabbit configuration block. The bundled Synthmas RiaH is the default; replace it when you have your own. See Rabbit-in-a-Hat Integration.
  3. (Optional) Usagi files — in etc/etl-engine.d/usagi/, if any rule chain uses UsagiMap.

  4. OMOP vocabulary — only needed if your rule chains do VocabMap / VocabSourceId / UsagiMap lookups. How to load it depends on the CDM driver:

    • CSV target (Free tier) — unzip an Athena vocabulary download directly into the same directory cdm database.connection string: points at (etc/etl-engine.d/cdm-out/ in the shipped config). The engine reads CONCEPT.csv, CONCEPT_RELATIONSHIP.csv, VOCABULARY.csv, etc. in place, alongside the CSV files it writes for the rest of the CDM. No automation step required.
    • RDBMS target (Paid tier) — drop the Athena zip into etc/etl-engine.d/vocab/ and let automation.pre.populate vocabulary: once load it into the database on the next run. See automation.

    For licensed code sets (CPT-4 etc.), run Athena's unpacking scripts before either approach — see Troubleshooting → Athena zips with licensed code sets. 5. Source data — either CSV files in etc/etl-engine.d/db/, or a reachable RDBMS named in the source database block.

Run

$ cd etl-engine-X.Y.Z/bin
$ ./run-etl-engine.sh

The run script:

  1. Clears any previous content from logs/ (so each run starts with a clean log directory).
  2. Starts the etl-engine container in the foreground.
  3. Mounts etc/etl-engine/ (read-only), etc/etl-engine.d/ (read-write), and logs/ (read-write) into the container. The read-only mount on etc/etl-engine/ is intentional — etl.conf and the Rabbit-in-a-Hat file should never change during a run.

The container is started with --privileged, --pid=host, and --security-opt seccomp=unconfined, plus a bind-mount of /sys/kernel/debug. These flags exist so performance-profiling tools (perf, eBPF) can attach to the engine when diagnosing slow runs. They are not load-bearing for an ordinary transformation; if your security policy disallows any of them, edit bin/run-etl-engine.sh and remove the offending flag.

The engine runs in the foreground until it finishes. Press Ctrl-C to abort early — the container is stopped immediately, and your data directories are left as they were at that point.

What gets written

Inside logs/ you will find:

  • etl-engine.log — the engine's own log, controlled by logging.log level in etl.conf.
  • One log per non-vocabulary CDM table — named after the CDM table itself — containing the parsed rule chains for that table and any rows that failed to transform.

If the engine stops with no output on the terminal, etl-engine.log is the first place to check. See Troubleshooting → Log Files for log-level guidance and what to look for.

Re-running

A re-run is just ./run-etl-engine.sh again. The engine clears logs/ at the start of each run, so the log directory only ever reflects the most recent execution. If you need to keep older logs, copy them out of logs/ before the next run.

For larger ETLs split across multiple Rabbit-in-a-Hat files, see Folder Layout → Splitting an ETL.