Skip to content

v0.9.0

A data-correctness release. Several tables were silently returning fewer rows — or different rows — than the surveys contain. If you have cached data, this release rebuilds it once (see below).

⚠️ Your cache rebuilds once on first use

Two changes each invalidate every cached parquet, and they ship together deliberately so the rebuild happens once rather than across two releases:

  • LSMS_CACHE_SCHEMA 4 → 5 (#655)
  • Wave.grab_data's @build_transform fingerprint changes (#627)

No action needed; the first build after upgrading is a cold one.

Silent data loss, fixed

to_parquet was deleting real people (#655, closes #645). It stringified object-dtype columns and then tried to recover nulls by string matching — converting the literal values 'None', 'nan' and '<NA>' to NULL. None is this library's canonical label for "no education", and _finalize_result's dropna(how='all') then deleted the row entirely.

country before after recovered
Guatemala 20,678 29,527 +8,849 (30%)
Tajikistan 54,462 59,790 +5,328
Ethiopia 62,939 63,181 +242

Guatemala returned the same wrong answer cold and warm, so no cache-clearing would have revealed it. The canonical label is renamed None → No education, with the old spelling retained as an accepted variant.

assets.Value is additive (#629). Nigeria's wave-2 asset module is a per-unit roster — a household owning five hoes gets five rows — and the collapse kept only the first. ₦294,594,970 recovered, 25.6% of that wave's reported asset value. Registering a table as additive also used to silence its grain audit wholesale; it now re-audits the columns the sum does not reconcile, so the residual Age loss is still reported loudly.

The dfs: merge was manufacturing duplicate rows (#627, GH #323 Site 4). An outer merge with no cardinality guard produced rows that every downstream collapse then destroyed. Fixing the merges cleared 4.68M phantom rows — Mali 4,324,668 (#641), Malawi 355,042 and Guinea-Bissau 59 (#653).

Cluster identity and grain (GH #323)

Sites 1, 2 and 4 of the duplicate-index class are addressed, along with country fixes for Uganda, Tanzania, Malawi, Benin, Togo, Ethiopia, Niger, Nigeria, Albania, Pakistan, India and Guinea-Bissau. Notable:

  • Uganda people_last7days was serving visitor counts to about half of households in 2018-19/2019-20 — the long-form category key was undeclared, so first() picked whichever row the file listed first. 6,185 destroyed rows.
  • Tanzania cluster_features read Region/District/Rural from the household cover page — the address where the household was interviewed — while the NPS tracks movers and carries the original cluster forward. Contested cells fall 2,106 → 385.
  • Malawi 2004-05's cluster key was an EA sequence number within its Traditional Authority: 110 codes for 564 real EAs. Fixing it restored 20 → 26 districts and recovered IHS2's urban stratum, which had been erased entirely.
  • Uganda crop_production gains a condition index level, so dry and fresh harvests of the same crop are no longer summed together.

Schema enforcement

Declared spellings are now enforced (#605, closes #602); six countries were shipping non-canonical values. GhanaLSS split households were silently merged (#604, closes #548). discover_waves() searched only collection='lsms' (#599, closes #597).

Documentation

  • A survey of 111 waves' population/universe statements, quoted verbatim with locators and with gaps recorded as gaps (#654), plus per-country universe/exclusion/oversample records in 38 CONTENTS.org files (#658). 32 countries carry documented exclusions; 34 carry oversamples or sub-populations.
  • The Tanzania NPS back-casting is now recorded as a country-level fact: split-off households are written back into earlier waves, so one round-1 household id can carry up to 11 panel ids (#652).
  • The DVC rule is stated in its general form — never invoke the dvc CLI, for any operation — after three places in the repo were found advising otherwise (#631).

Known open

GH #323 remains open: Nigeria's 2012Q3/2013Q1 cartesian cells are unowned, and the groupby().first() key-soundness inventory (#637) is partially reviewed. GH #603 (how the library should represent a sample's population) is a design question with evidence now gathered but no proposal.