Skip to content

LSMS Library v0.8.0

Release range: v0.7.4 → v0.8.0 (60+ merged PRs).

Headline — Automatic content-hash cache invalidation

The L2 parquet cache now detects its own staleness and rebuilds only what changed. Before v0.8.0 the cache read was unconditional ("read if present"), so editing a survey config or a transform could leave a stale parquet silently served; the only recourse was a manual cache clear / LSMS_NO_CACHE.

Each cached parquet now embeds a content hash (pyarrow metadata key lsms_cache_hash) covering its build inputs. On read: hash matches → serve; mismatches → rebuild; no embedded hash (a pre-v0.8.0 parquet) → trust once and re-stamp (no rebuild-all on upgrade). Recompute is config/sidecar reads only (no .dta hashing, no S3, no tree walk), so the v0.7.0 warm fast path is preserved.

What the hash versions (built up across the release):

  • Survey inputs (#454) — data_info.yml / data_scheme.yml, the wave + country modules, _/{table}.py scripts, the _/Makefile, build-time .org inputs, and the DVC sidecar md5 of each declared source (.dta never hashed or downloaded).
  • Script-path wave parquets (#482, GH #479) — these are written hashless by their scripts, so they can't self-invalidate; they are now evicted before every rebuild descent so a source-script fix actually re-runs.
  • Framework transform code (#534 → #539 → #543) — the build/read transform boundary was made structural (build_transforms.py vs read-path transformations.py), then the build-path transform closure was folded into the hash. Editing a build-path transform (e.g. food_acquired_to_canonical, _normalize_dataframe_index's collapse, df_data_grabber, a country's _age_helpers.py, an ihs3_conversions.csv) now auto-invalidates the affected table — closing the GH #522 blind spot. Read-path transforms (kinship, spellings, _finalize_result) are deliberately excluded (they re-apply on every read), so read-path edits never spuriously rebuild. The fingerprint computes on the read gate at ~ms (memoized). See SkunkWorks/build_transform_cache_hash.org.

LSMS_CACHE_SCHEMA remains as a manual belt-and-suspenders lever; the manual overrides (LSMS_NO_CACHE, lsms-library cache clear, pytest --rebuild-caches, LSMS_BUILD_BACKEND=make) are unchanged. Full picture: docs/guide/caching.md.

New data coverage & API features

  • Code/config separation (#452, #455, GH #436) — LSMS_COUNTRIES_ROOT / countries_dir overrides where the config tree is read from, independent of LSMS_DATA_DIR (caches). The v-from-sample() join is now declarative (opt out via data_info.yml, no country.py edit).
  • Education harmonization foundation (#487, #488, #489, #490, #532) — global ordinal vocabulary + row-additive hierarchy; Uganda pilot, Mali/Ethiopia, and Iraq individual-education rollout.
  • Asset harmonization, additive (#485, GH #168).
  • EHCVM / EthiopiaRHS feature expansion (#462, #459, #465, #467, #470) — label foundation, livestock pilot, anthropometry z-scores, crop production, community prices.
  • Currency (#478, #481) — currency labeling and conversion.
  • interview_date (#473, #477) — visit-level interview dates, incl. EHCVM.
  • Grain / unit policy (#471, #472) — u carried in the index; no implicit aggregation in core.
  • labels=X contract (#550) — requesting a label a country never curated (e.g. labels='Aggregate' on an EHCVM country) now degrades cleanly: Country(...) raises a typed LabelUnavailableError (a KeyError subclass, back-compatible), and Feature(...) drops only the unsupported countries with one aggregated warning + a df.attrs['labels_unavailable'] marker — instead of silently returning a partial table.
  • GhanaLSS food_acquired (#457, phase 1).

Cross-country Feature() & audit tooling

  • Feature() mirrors the Country(...).<table> interface (#508, #509) — per-feature kwargs (market/labels/units/age_cuts) forward by signature.
  • Cross-country assembly fixes (#520, GH #511/#512) — index collapse under market= and reduced-index countries.
  • Feature() audit harness (#521, #505, #507, #523, #549) — a deterministic cross-country sweep (bench/feature_audit/) that builds every feature, checks sanity + assembly invariants, and triages findings; the scanner parallelizes one worker per feature to saturate a warm cache.
  • Backlog workflow (#524, #529) — multi-agent triage/fix of the GitHub backlog, building on the audit harness.

Notable bug fixes

  • food_acquired additive collapse (#533, GH #514) — _normalize_dataframe_index sums additive measures instead of .first()-dropping rows.
  • plot_features index collisions (#542, #544, #541, GH #513/#535) — China/Mali (i, plot_id) collisions; Nigeria plot doubling.
  • Harmonization sweep across 13 countries (#494).
  • Kinship (#525, GH #516) — Albania labels + South Africa sentinel.
  • Derived-table & visit-level fixes (#527 food_prices visit; #528 derived household_characteristics market=; #526 cluster_features geo).
  • EHCVM sanity (#523, #480) and region fixes (#540 Azerbaijan, #545 Guatemala — surface a recoverable Region so market='Region' works; GH #530).
  • Panel-id household collisions (#547, #546, GH #504/#536/#513) — update_id no longer mints a suffix that re-uses a live household id (Malawi: 60 collisions → 0; anthropometry +37 / sample +60 / plot_features +23 rows), and Mali's id_walk no longer renames a moved household onto an occupied non-panel slot (sample +8 / plot_features +6). Healthy panels byte-identical.
  • Robustness — S3 fetch retry/backoff (#483), u-code NaN-safe leak fix (#451), missing-orgfile-table handling (#464), Malawi floor casing (#484), GhanaLSS food units (#486).

Release & packaging

  • Automated PyPI publishing (#554, #556) — publishing a GitHub Release now builds and uploads to PyPI in CI via Trusted Publishing (OIDC): no stored token, no local poetry build/keyring hang, version derived from the tag. A workflow_dispatch(tag) fallback ships a release manually if the release event doesn't fire. See the release skill for the flow and gotchas.

Upgrade notes

  • No action required for existing caches. Pre-v0.8.0 parquets lack the embedded hash and are trusted-once then re-stamped; the cache is not wholesale-invalidated on upgrade.
  • Stale data now self-heals for config/script/transform edits — a manual cache clear is no longer needed in the common cases.
  • Known residual: a framework helper reached by purely string-keyed dynamic dispatch is not statically resolvable and is backstopped by the manual LSMS_CACHE_SCHEMA bump (documented in lsms_library/_build_registry.py).

Acknowledgements

The build-transform cache versioning (#543) was developed and hardened through an 8-round adversarial red-team before merge.