Skip to content

Common errors

This page collects the messages most often seen during a run, what they mean, and how to fix them. Errors are grouped by where they appear: terminal output (the engine wouldn't start) versus per-table log files (the engine ran, but a table failed to load).

Engine startup errors

These appear on the terminal when run-etl-engine.sh fails before any log file is written.

Couldn't open file: etl.conf (-2)

The engine could not find or read etl.conf. Check that etc/etl-engine/etl.conf exists and is readable by the user (or container) running the engine. The shipped run-etl-engine.sh mounts etc/etl-engine/ read-only into the container at /etc/etl-engine, so the file must exist on the host before you start the run.

Missing source connection string

The source database block in etl.conf has no connection string key. See the source database reference.

Missing destination connection string

Same as above for the cdm database block. See the cdm database reference.

Couldn't initialize etl-engine

Generic catch-all printed when the engine cannot bring itself up. The preceding line tells you which specific component failed — connection to the source database, parsing of the hat file, license validation, or similar.

Per-table errors

These appear in the per-table log file under logs/<cdm-table>.log and almost always indicate something the user can fix in the Rabbit-in-a-Hat configuration.

Field 'X' is non-nullable and not mapped or defined

note.log:[2023-05-23 15:26:38.950] [note] [error] Field 'note_id'
is non-nullable and not mapped or defined

The CDM table has required fields with no incoming mapping. This is not always a problem — if you do not intend to populate this table at all, the message is benign and the table simply is not loaded.

To populate it: map the listed fields in Rabbit-in-a-Hat. To leave it empty intentionally: ignore the message.

Semantic error: The declared rule 'X' is undefined

[device_exposure] [error] Semantic error: The declared rule `Dates' is undefined.

A rule chain references a rule the engine does not know. Typical causes:

  • A typo (Dates instead of Days).
  • An older Rabbit-in-a-Hat file that uses a rule name that has since been renamed.
  • A version mismatch between the configured engine and the Rabbit-in-a-Hat file (the rule does exist, just not in this engine version).

Cross-reference the Rule Reference for canonical names. Fix the rule name in Rabbit-in-a-Hat and re-run.

Errors break ETL of table

[device_exposure] [critical] Errors break ETL of table

Always the last line of a per-table log when the transformation fails. The real cause is one of the earlier [error] lines in the same file — search backwards. Any CDM tables that depend on the broken table will also fail.

The field 'X' is required, but it is not specified (InsertRecord)

[cdm_source] [error] InsertRecord: The `cdm_release_date' is required,
but it is not specified.

An InsertRecord call omits a non-nullable field. Add the field as a field=value keyword parameter, or change the database schema so the field is nullable.

The field input to <Rule>: ':X' doesn't reference a source field

The named field does not exist on the source table being mapped. This typically means a rule references :patient_id while the source table only has :id. Either rename the source field in Rabbit-in-a-Hat to match, or adjust the rule to reference the correct field name.

The CDM table must exist (DependOn)

[X] [error] The referenced CDM table must exist.

A DependOn rule references a CDM table that is not part of this run — perhaps because of cdm database.limit tables. Either remove the DependOn, or add the referenced table to limit tables.

Vocabulary-loading errors

These appear when automation.pre.populate vocabulary is set but the engine cannot complete the load.

Vocabulary directory empty / no zip found

The directory pointed to by vocabulary directory is empty. Drop the Athena zip in there before the run.

Athena zips with licensed code sets (CPT-4, etc.)

Some vocabularies — most commonly CPT-4 — are not redistributable and Athena ships them as encrypted placeholders. The ETL Engine does not decrypt them automatically. The unpacking process is the same either way; only the last step differs depending on your CDM driver.

  1. Download the vocabulary zip from Athena with the licensed code sets selected.
  2. Unzip it to a working directory.
  3. Run the unpacking scripts that ship inside the Athena zip (cpt.bat / cpt.sh or similar — Athena's own README in the zip has the canonical instructions). These scripts call out to the licensed code set's distribution API and replace the placeholders with real CSV rows.

CSV target (Free tier — no automation needed)

  1. Copy the unpacked CSV files (concept.csv, concept_relationship.csv, vocabulary.csv, etc.) directly into etc/etl-engine.d/cdm-out/ — the same directory the engine writes its CDM output to. The engine reads them in place; no automation step is involved.

RDBMS target (Paid tier — via automation)

  1. Delete the unpacking scripts (cpt.sh, cpt.bat, any *.jar, README, or other non-CSV artefacts) before re-zipping. The engine's vocab loader reads every file in the zip as if it were a CSV, so a leftover shell script becomes a load-time crash.
  2. Repackage the directory back into a zip — only .csv files inside. The engine reads from a zip, not from the unpacked directory.
  3. Drop the new zip into the engine's vocabulary directory (etc/etl-engine.d/vocab/). The populate vocabulary automation will pick it up on the next run and import the rows into the CDM database.

Failure modes

  • If you skip steps 2–3 entirely and let the engine consume the original Athena zip, the licensed concepts simply aren't loaded — no error, just rows missing from the concept table.
  • For the RDBMS path: if you re-zip with cpt.sh / cpt.bat still present, the vocab loader crashes when it tries to parse one as a CSV.

UsagiMap: ConceptID 'X' was not found in the vocabulary database

The Usagi map maps a source value to a concept id that does not exist in the loaded vocabulary. Either re-export the Usagi map against a fresh vocabulary, or add the concept id manually. If the missing concept is in a licensed vocabulary like CPT-4, also check that you ran the unpacking process above before loading.

UsagiMap: ConceptID 'X' does not have the domain id 'Y'

The Usagi map points at a concept that is in a different domain than the rule's AssertVocabDomain requires. Re-curate the Usagi map or relax the domain assertion.

License errors

License file invalid / License expired

The license file the engine found is either malformed or has reached its expiry date. Contact your Rook IT representative to renew or replace it.