Skip to content

Synthmas — FAQ

Questions that come up when you run the bundled Synthmas demo for the first time and watch logs/run.log or compare row counts in etc/etl-engine.d/cdm-out/. Most of these aren't bugs — they're artefacts of how the engine reports progress versus what it actually persists.

The Records: counter in run.log climbs past 150,000 on the Free tier — isn't there a cap?

There is — and it's holding. What you're seeing in run.log is the counter for source rows the mapper has processed (rowsHandled), not rows committed to the CDM table (rowsCommitted). The Free tier cap is on the committed side: once a CDM table has accepted 150,000 rows, further rows are silently dropped at the persistence step, but the upstream reader keeps pulling source rows and the mapper keeps chewing through them.

The dropped rows show up in two places:

  • The cap was reached. A one-shot warning is written to both run.log and the per-table log (e.g. logs/measurement.log): Row cap reached for measurement (150000) — additional rows dropped.
  • The output CSV stops growing. After the cap, cdm-out/<table>.csv has exactly 150,000 data rows even if Records: ended at 500,000.

Once the cap fires, the engine also stops reading further source rows for that table (in newer builds) — so for unfiltered tables the gap between the final Records: value and 150,000 is small.

For tables whose rule chain filters rows mid-pipeline, the gap can still be large. In the bundled Synthmas demo, measurement illustrates this — its source is observations.csv (~531 K rows), of which ~60 % carry TYPE=numeric and ~40 % carry TYPE=text. The RiaH puts an Assert("numeric") on the chain so text rows are dropped before insert. The cap fires on commit #150,000, but to get 150 K numeric commits, the engine had to chew through roughly 250 K source rows (150 K commits + ~100 K text rejections). The in-flight tail at cap-fire then runs another ~25 K rows before the upstream short-circuit lands. End result in a recent run:

[measurement] Handled 274,442 records
[measurement] Inserted 150,000 records

That's not the cap leaking; it's the cap counting commits while Records: counts everything the mapper has looked at. Tables with no in-chain filter (like procedure_occurrence) land at Handled 150,030 / Inserted 150,000 instead — the small tail is just the in-flight rows at cap-fire moment.

See the next entry for the underlying handled-vs-written distinction that makes this kind of mismatch possible in the first place, and Licensing → Free for the tier overview.

Why doesn't Records: in run.log match the row count in my CDM CSV?

It almost never will, even with no cap in play. Records: is the handled count: the number of source rows the mapper has pulled in and run through the rule chain for the current CDM table. The CDM-side row count is the written count: how many rows actually landed in cdm-out/<table>.csv (or in the target database table on Paid tier). They're two different quantities, and the gap between them is normal — the rules in your RiaH actively change how many output rows each input row produces:

  • Fan-out rules write more than one CDM row per source row. Multi-result mappings like Map(...)[] or unrolled LoopOver rules emit several output values for a single input row. The classic Synthea→OMOP case: a claims.csv row with eight diagnosis columns fans out to up to eight condition_occurrence rows. The written count climbs faster than the handled count.
  • Filtering rules write fewer than one CDM row per source row. Anything that returns "no value" for the row — a Selector that picks a NULL branch, a conditional chain (||) where every branch fails, a LeftJoin row with no match on the join key, a Normalize that deduplicates against an earlier row — drops the row at the persistence step. The written count climbs slower.
  • Both, in the same table. Real mappings combine these. The Records: counter doesn't reflect either.

The Free-tier 150,000 cap applies to the written count, not the handled count — which is why the previous entry's "Records: past 150,000" symptom happens. But the same mismatch happens at any scale, with or without a licence: if you cross-check run.log against wc -l cdm-out/<table>.csv you should expect them to differ.

When a CDM table finishes, the engine writes both numbers to the per-table log (logs/<table>.log):

[procedure_occurrence] [info] Handled 235460 records
[procedure_occurrence] [info] Inserted 150000 records

Handled is what run.log was tracking; Inserted is what landed in cdm-out/<table>.csv. The same per-table log shows skipping row warnings for individual filter / no-mapping / no-match cases while the table is running, which is how you'd track filtering live — though there's no live counter for the cumulative committed total until the summary line at the end.

I tried an RDBMS target, automation, or StoreIndexMap and the run stopped with an error

Those are Paid-tier features, and the Free tier surfaces them loudly rather than ignoring them silently: the engine stops with a clear message at config-parse or table-load time, naming the feature and the licence flag it needs (e.g. OPEN_SOURCE_RDBMSES, AUTO, ETL). To keep running on Free, drop the feature from your etl.conf / RiaH; otherwise add a Paid licence — see Licensing.

Throughput swings from 10 rec/s to 10,000 rec/s mid-table — is that normal?

Yes, and it's usually one of three things, sometimes all at once:

  • Downstream queue backpressure. When the writer can't keep up (e.g. CSV writer holding its mutex with a per-row flush, or RDBMS batch inserts catching their breath), the mapper's output queue fills to its cap and the producer yields. As soon as the writer drains a batch, throughput jumps. The Output queue: 100 line you see in run.log for several seconds in a row is this happening.
  • Per-row cost variance. Multi-result fan-outs, joins, conditional chains, and Selector rules don't all cost the same. A row that hits a fast path runs faster than one that triggers a vocab lookup and a Usagi join and a fan-out.
  • Cold-cache vocab loads. When a row references a vocabulary concept that hasn't been touched yet on this run, the engine fetches it from the loaded CONCEPT.csv. With the CSV CDM driver there's no index — concept lookups scan, which is slow until the in-memory cache is warmed. The first few rows of a new source table mid-run can stall hard if they pull in a cluster of previously-unseen concepts; the same rule on the same table runs 100× faster once the cache is warm.

See Troubleshooting → Performance for the deeper treatment of all three.

Is ~10,000 rec/s the ceiling?

Not at all. Depending on hardware and mapping complexity, the engine will reach 100,000+ rec/s on tables with cheap rules and a warm cache. The Synthmas demo's ~10k rec/s plateau on measurement and observation is a Free-tier, single-thread, CSV-target combination running through joins, vocab lookups, and Usagi joins — slow side of the design space. Real-world ceilings move with:

  • Worker thread count (Paid tier removes the single-thread cap).
  • Target driver (RDBMS batched inserts beat per-row CSV writes).
  • Rule cost (a plain field-to-field copy is much cheaper than a multi-rule chain).
  • Vocabulary cache size (vocabulary cache size: in etl.conf).

Whatever number you see in your run.log, treat it as a property of this mapping on this machine — not a published spec.

Input queue and Output queue show the same number every line — bug?

Cosmetic, and the labelling will improve in the next engine iteration. What run.log currently prints is the producer's output-side queue depth under both labels, because a producer→shared-queue→consumer pipeline sees the same physical queue from both ends. The number itself is meaningful (it's the live backlog the mapper is feeding the writer); the duplication is just the label aliasing.

For tables that use an asymmetric two-queue topology (measurement, visit_detail, the final condition_occurrence pass, cost) the two columns do diverge — typically Input queue: 0 - Output queue: 100, meaning the reader is keeping up and the writer is the bottleneck. That diagnostic does work today; it's the symmetric-queue case where the labels are confusingly tautological.

procedure_occurrence "loaded 205,000 records" but the CSV has 150,000 rows — what happened?

Two effects compounded:

  • Source-row fan-out. One source row can produce more than one CDM row. A single Synthea procedures.csv row that maps via a multi-result chain ends up writing several procedure_occurrence rows. So 205,000 is the source row count the mapper handled; the corresponding CDM row count produced (pre-cap) was higher.
  • Free-tier cap. Of the CDM rows the mapper produced, only the first 150,000 were committed; the rest were dropped. See the first question on this page for the full mechanics.

Both numbers are correct. They measure different things.

I added two source tables to one CDM table and half my rows are missing fields — what happened?

Most likely you registered two table-to-CDM arrows (or relied on both tables already being registered) and assumed the engine would auto-join them on a foreign key. It doesn't — two arrows without an explicit join rule produce two independent row streams. You'll see one cohort of rows with fields populated from source A and the source-B fields all NULL, plus a second cohort with the opposite shape.

Sanity check: count distinct values of a field that only the joined-in source could populate. In the payer_plan_period case, before the fix the output looked like this:

$ awk -F, 'NR>1 {print $1}' cdm-out/payer_plan_period | sort | uniq -c
      1 280       # 1 row per payer name in payers.csv
      1 288
      1 289
      6 327
      1 436
  51794 null      # 51,794 rows from payer_transitions, no payer

That null plateau is the tell: the dimension-table feed (payers.csv) produced exactly one row per source row, while the fact-table feed (payer_transitions.csv) produced ~50K rows with no payer data attached.

Fix: declare the join explicitly on the CDM table's Comments field:

logic: LeftJoin(@payer_transitions.csv, @payers.csv, [:payer], [:id])

The driving table goes first; the lookup table goes second; the shared key is the last two list arguments (@driving.key vs. @lookup.key). Use LeftJoin if you want every driver row to survive even when the lookup miss s; use Join if missing lookups should drop the driver row.

The same trap can hit any composed CDM table — for example a fact table that joins against a code-lookup CSV to enrich a *_source_value. Two arrows alone won't do it; the join rule has to be there.

payer_concept_id is 0 / 100% unmapped in payer_plan_period — what's missing?

If you're running the bundled demo as shipped, this shouldn't happen — the bundle includes a starter etc/etl-engine.d/usagi/payers.csv covering the ten payer names Synthea generates (Aetna, Medicare, Dual Eligible, NO_INSURANCE, …) and the RiaH wires payers.name → payer_concept_id through UsagiMap("payers.csv"). Check three things, in order:

  1. The Usagi file is present. ls etc/etl-engine.d/usagi/ should list payers.csv alongside units-exported.csv and specialities.csv. If payers.csv is missing, the RiaH still runs but the lookup returns 0 for every row.
  2. etl.conf points at the Usagi directory. The rabbit configuration block in etl.conf should set usagi directory: "/etc/etl-engine.d/usagi/". Without it the engine ignores the directory.
  3. The Usagi file has every payer name in your data. Synthea is deterministic about the ten names it emits, but if you replaced the source CSVs with your own Synthea run that uses a custom payer module, names outside the starter map will resolve to 0. The fix is to add rows to payers.csv.

For payer concept choices in the starter map (which SOPT concept maps to which Synthea name), see the derived-tables walkthrough.