Skip to content

Save / Load

Two single-file bundle formats — one per concept — that round-trip end-to-end:

  • .etlopipeline — a pipeline's config + every step file. Use for backup, sharing across instances, or duplicating inside one instance.
  • .etloenv — an environment's config (small; encryption-by-construction for password fields). Use to move an env between instances or back up the bits you don't want to re-type.

The two are intentionally orthogonal: different scopes, different sensitive surface, separate files.

Pipeline .etlopipeline

Save

Pipelines → (open pipeline) → Configure → ⤓ Save.

Split-button — the dropdown lets you choose whether to embed the vocabulary zip:

  • Save (no vocab) — small (KB–MB). Manifest records the vocab version + sha256; import warns if no matching vocab is installed on the target.
  • Save + include vocabulary (large) — self-contained (potentially ~1 GB). Best for cold installs.

What's inside

<pipeline-name>.etlopipeline       (a zip)
├── manifest.json          pipeline config + steps[] (resolved by step name)
├── blobs/<sha256>         every current step file content, sha256-deduped
└── vocab/<filename>       optional, if you ticked the include-vocab option

The manifest.json:

  • Pipeline-level: name, description, omop_version, engine_image_ref, post-automation flags, dag_layout_json.
  • steps[]: name, kind, display_order, new_mode, etl_conf_overrides, kind_config + files[] (kind, filename, blob_sha256, notes) + per-RiaH expected_tables, produces, consumes. Consumes use the producer step's name, not its id — so the bundle re-links cleanly across id spaces.
  • Schema version for forward compatibility.

What's excluded (by design)

  • Environments (they're per-instance; separate bundle).
  • Source sets (per-pipeline but per-instance; user re-uploads).
  • Runs, cycles, run history.
  • License file (separate concern).

Load

Pipelines → ⤒ Import. Pick the .etlopipeline file.

  • Always creates a new pipeline (name-clash gets a (2) suffix).
  • Pre/post automation steps are seeded by the orchestrator first, then the manifest's pre/post entries update them in place (by kind).
  • Step files de-dup via the blob store by sha256.

Environment .etloenv

Save

Pipeline → Environments → (open env) → Configure → ⤓ Save.

Tiny — typically hundreds of bytes; a few KB at most.

What's inside

<pipeline-name>-<env-name>.etloenv     (a zip)
└── manifest.json
    ├── pipeline_ref: {name, omop_version}     compatibility check at import
    └── environment:
        - source_db_config_json + cdm_db_config_json (verbatim — see below)
        - log_level, vocab_cache_size, cdm_max_connections, worker_threads
        - use_dedup_cache, etl_conf_overrides_json
        - engine_image_ref (override)
        - engine_env_vars_json (KEY=VALUE bag for the engine container)

What about password security?

Password fields inside *_db_config_json are stored AS-IS, which is safe because they're already encrypted with the originating instance's login password (field-level encryption). The bundle file doesn't need its own encryption layer — the sensitive fields are already protected, and they stay protected on import unless the target instance shares the same login password.

The manifest carries a note_on_passwords field flagging:

Encrypted-field values inside *_db_config_json are bound to the ORIGINATING instance's login password. Re-enter passwords on the target instance after import if it has a different login password.

What's excluded

  • csv_source_set_id (instance-local pointer to a per-pipeline source set; user picks one on import).
  • Runs, cycles, CDM state caches.

Load

Pipeline → Environments → ⤒ Import. Pick the .etloenv file. The orchestrator:

  1. Checks pipeline_ref.omop_version matches the target pipeline (hard error on mismatch).
  2. Creates a new env under the target pipeline (name clash → (2) suffix).
  3. Leaves csv_source_set_id null (you pick one in the env config page).