automation

Toggles the engine's automation steps. Automation runs in two phases:

• pre — runs before the mapping starts. Used to set up the destination database (CDM tables, vocabulary, optional cleaning).
• post — runs after the mapping completes. Used to populate derived tables (era tables, preceding-visit IDs).

Example

automation {
  pre {
    create omop cdm: once
    populate vocabulary: once
    clean omop data: true
    vocabulary directory: /etc/etl-engine.d/vocab/
    vocabulary backup: /etc/etl-engine.d/vocab.backup/
  }

  post {
    add preceding visit ids: true
    create observation periods: true
    create condition eras: true
    create drug eras: true
    create dose eras: true
  }
}

pre block

create omop cdm

Sets up the OMOP CDM tables in the destination CDM — a database or a CSV output directory — using the CDM version declared in the Rabbit-in-a-Hat file. On a database target it issues the CREATE TABLEs; on a CSV target it writes each non-vocabulary CDM table to its own file with the canonical OMOP header. From there the two behave identically, including when a single CDM table is written by several etl_map steps (each appends its columns).

Value	Behaviour
always	Build (or rebuild) the CDM on every run.
once	Build the CDM only if it does not already exist.
never	Do not run this automation. Default.

populate vocabulary

Populates the vocabulary tables from a zip file downloaded from Athena. The zip must live in vocabulary directory.

Value	Behaviour
always	Build (or rebuild) the vocabulary on every run.
once	Build the vocabulary every time a new zip is present in vocabulary directory. After processing, the zip is moved to vocabulary backup.
never	Do not run this automation. Default.

vocabulary directory

Directory the engine looks in for the Athena zip. Default: /etc/etl-engine.d/vocab/.

vocabulary backup

Directory the engine moves processed Athena zips into. Default: /etc/etl-engine.d/vocab.backup/.

clean omop data

If true, empties all non-vocabulary content from the OMOP CDM before the transformation runs, keeping the table schema. Optional; missing values are treated as false. On a database target this is a TRUNCATE; on a CSV target it resets each table file to its canonical OMOP header (empty body) — the same semantics — so a pipeline can clean once and have each subsequent etl_map step append to a still-correctly-shaped table.

Warning

Combined with cdm database.limit tables, only the listed tables are cleaned. See cdm database.

post block

All post steps default to false and may be omitted entirely if not needed.

Key	What it does	CSV target
create observation periods	Populates observation_period from the span of each person's clinical events.	Yes
create condition eras	Populates condition_era from condition_occurrence.	Yes
create drug eras	Populates drug_era from drug_exposure (rolled up to RxNorm ingredients).	Yes ¹
create dose eras	Populates dose_era from drug_exposure and drug_strength.	Yes ¹
add preceding visit ids	Populates the preceding-visit IDs in visit_occurrence and visit_detail.	Yes

Derived tables on a CSV target

observation_period, condition_era, drug_era and dose_era are built by reading the produced CDM tables back and folding them in memory, so they work on a CSV (file) target as well as a database. The whole derived table is rebuilt from current CDM state on every run, so it stays correct under selective re-runs.

add preceding visit ids assigns each visit's predecessor per person (ordered by start datetime/date). On a database target it updates the rows in place; on a CSV target it rewrites the produced visit_occurrence / visit_detail files in place after they are written. Either way only NULL preceding-id cells are filled, so it is safe under selective re-runs.

¹ On a CSV target the vocabulary is never loaded into the CDM, so drug_era and dose_era read the Athena vocabulary CSVs directly: drug_era needs CONCEPT.csv and CONCEPT_ANCESTOR.csv, dose_era needs DRUG_STRENGTH.csv. The engine looks for them in vocabulary directory first, then in the CDM output directory. (On a database target these come from the loaded vocabulary instead.)