Skip to content

Log files

When the engine starts, it accepts at most one argument: a path to a local configuration file. Without one it falls back to /etc/etl-engine. If the configuration file fails to open or parse, the engine prints to stderr:

$ ./etl-engine etl.conf
ETL-Engine version: 1.1.0 Build d612c89
©2023 Rook-IT ApS

  Couldn't open file: etl.conf (-2)

Couldn't initialize etl-engine

or:

$ ./etl-engine etl.conf
ETL-Engine version: 1.1.0 Build d612c89
©2023 Rook-IT ApS

Missing source connection string

Once the configuration loads and the log directory is writable, all subsequent communication is through the log files.

etl-engine.log

The engine's main log. Its verbosity is set by logging.log level in etl.conf. When the engine stops with no output on the terminal, this is the first place to look.

Log levels:

  • TRACE — everything: reading the Rabbit-in-a-Hat file, building the internal mapping, optimisation choices, join planning.
  • DEBUG — same scope minus internal details. Enough to pinpoint where an error originates.
  • INFO — stage transitions only (loading, cleaning, mapping).
  • WARN — recoverable surprises, such as a failed rule-chain parse or a cascading mapping failure.
  • ERROR — fatal errors only. A single line stating the cause of death.

Errors raised inside etl-engine.log during the configuration phase typically indicate an engine-level bug, not a user mapping problem.

Per-table logs

Once the engine begins to configure the transformation, one log file per non-vocabulary CDM table appears in the log directory, named after the table (person.log, condition_occurrence.log, …). These logs always run at INFO level — there is no way to silence them.

Each table log contains:

  • The parsed rule chains for the table — useful when debugging.
  • Any rows that failed to transform, with the cause.
  • A trailing [critical] line if the table failed entirely:
[2023-05-23 15:26:38.962] [device_exposure] [critical] Errors break ETL of table

That line means the table was not mapped, and any other CDM tables that depend on it will also fail.

The good news: errors in per-table logs are almost always caused by a bad Rabbit-in-a-Hat configuration, not by an engine bug. Fix the mapping in Rabbit-in-a-Hat, regenerate the file, and re-run.