Skip to content

Bundle formats

Schema reference for the two save-load file formats. Both are zips with a manifest.json at the root; the orchestrator versions the manifest so forward-compatible loaders can be added later.

.etlopipeline

<pipeline-name>.etlopipeline
├── manifest.json
├── blobs/<sha256>          # one per current StepFile, sha256-deduped
└── vocab/<filename>        # optional — only when exported with "include vocab"

manifest.json (pipeline)

  • Pipeline-level fields: name, description, omop_version, engine_image_ref, post-automation flags, dag_layout_json.
  • steps[]: name, kind, display_order, new_mode, etl_conf_overrides, kind_config + files[]. Each RiaH file carries expected_tables, produces ({kind, name}), and consumes ({kind, name, producer_step: <step name>}).
  • vocab: {vocab_version, blob_sha256, filename} or null.

Wiring is resolved by step name, not id, so the bundle re-links cleanly across id spaces.

Import behaviour

  • Name clash → (2) suffix on the new pipeline.
  • Pre/post automation steps are seeded by ensure_automation_steps() first, then the manifest's pre/post entries update them in place by kind.
  • Step files de-dup via the blob store by sha256 (existing blob = reuse).
  • Wiring is materialised in a second pass after every step exists.

Excluded (by design)

  • Environments, source sets, runs, cycles, licenses.

.etloenv

<pipeline>-<env>.etloenv
└── manifest.json

manifest.json (environment)

  • pipeline_ref: {name, omop_version} — compatibility check on import.
  • environment: name, both *_db_config_json blobs (verbatim — see encryption note below), log_level, vocab_cache_size, cdm_max_connections, worker_threads, use_dedup_cache, etl_conf_overrides_json, engine_image_ref override, engine_env_vars_json (KEY=VALUE bag for the engine container).

Encryption-by-construction

Password fields inside *_db_config_json pass through as the orchestrator stored them. Already-encrypted values stay encrypted in the bundle — the encryption is bound to the originating instance's login password, so the bundle file itself doesn't need an extra encryption layer.

If the target instance has a different login password, you'll need to re-enter the passwords after import. The note_on_passwords field in the manifest is informational.

Import behaviour

  • OMOP version mismatch → hard error before creating the env.
  • Name clash → (2) suffix.
  • csv_source_set_id intentionally NOT restored — pick one in the env config after import.

Excluded (by design)

  • Run history, cycles, CDM state caches.
  • csv_source_set_id (instance-local pointer).