Skip to content

v0.11.1

A release about kilograms, and about what the library will and will not decide for you. Two other harmonisations of the same LSMS-ISA files -- the World Bank's LSMS-ISA_Ag panel and the Evans School's EPAR curation -- were surveyed as points of comparison, never as reference answers, and the survey paid for itself several times over: a per-row conversion factor Nigeria had been shipping unread, two World Bank conversion tables Ethiopia had held in DVC for years without opening, a fertiliser-label mismatch that had given every Malawi plot zero nitrogen, and a Uganda sentinel that inflated one wave's harvest three-hundred-fold. Underneath, harvest_kg gains a layer for externally shipped factor tables, food expenditure gains an opt-in valuation of own production, and a fourth read-path audit names the harvest quantities that cannot be true.

Behaviour changes you will see in returned data

  • Uganda 2009-10 crop_production: the 99999 sentinel is gone. 3,097 harvest quantities carried the survey's missing-value code as a number; the wave's Quantity summed to 3.1e8 native units, 321x its neighbours. They are now missing, and the 3,077 rows that carried nothing else leave the table as every other measure-less row does; the wave serves 22,408 rows and 2,323 households. (#861)
  • Uganda crop_production carries the SOLD unit. Unit_sold (and Condition_sold where asked) are reported columns beside the harvest unit u, in every wave. They agree with u on 96% of sales. Where they disagree the row is a data-quality flag, not an identification: measured two ways, Unit_sold is the better price denominator only 40-45% of the time, and the largest disagreement cell (harvest in 100 kg sacks, sale in "kg") is priced like a sack. Consumers forming a per-unit price must treat those rows as ambiguous. (#824, Uganda half)
  • Nigeria crop_production carries KgFactor. The GHS-Panel W4 and W5 harvest sections ask a per-row kg conversion factor; four columns are now carried, so harvest_kg's reported layer serves 19,667 rows that had no factor at all (total 9.3e6 -> 22.1e6 kg). W1-W3 ask none. Eleven factors on kilogram rows are shelling ratios keyed into a container field; they are kept as reported and refused at read time, never stripped. (#859)
  • Ethiopia plot_features.Area for local units, and AreaUnit at last. The World Bank's woreda-level table converts Timad, Kert, Boy and the rest to hectares: 2,273 fields gain an area, NaN 9,727 -> 7,454, no GPS area overridden, 33 implausible conversions counted and served. And AreaUnit, which had been <NA> on every non-GPS field since the feature landed (a wave-keyed decode handed to a bare-code mapper), now carries the unit. (#853)
  • nitrogen_kg on Malawi is no longer zero. The product-to-nitrogen table matched labels by exact lower-case equality; Malawi writes Urea Fertilizer where the key was urea, so only Other Fertilizer (no nitrogen) ever matched. Labels are now normalised; Malawi moves from 331 plots at 0 kg N to 33,113 plots and 835,012 kg N. A coverage tally and a warning name whatever still does not match. The content half -- NPS, D Compound, organics, Portuguese labels -- is #867.

New read-time machinery

  • harvest_kg(..., shipped_factors=): a fifth layer for externally supplied kg-per-unit tables, ranked after the row's own reported factor and before the survey median. It refuses an ambiguous table rather than averaging, reuses the plausibility screen, counts what it matched (shipped_matched) and warns on a zero match, and joins on whichever of (country, t, j, u, condition, region) both sides carry. Nothing is stored; loaders are explicit. Ethiopia's is the first: ethiopia.crop_conversion_factors() reads the four Crop_CF tables (Wave-5's duplicate row resolved by a stated rule) and lifts Ethiopia's kilogram-bearing crop rows from 22,672 to 66,049. (#852)
  • food_expenditures(basis='total', valuation=...): opt-in valuation of produced and in-kind food at the household's own purchase price for the same item and unit ('own_price'), or through the geographic median-price ladder ('median_price'), in the order you give. Provenance rides along: a ValuationSource column on the item frame and a counts dict in attrs. Default behaviour is byte-identical. The docstring states the bias: every rung is a purchase price, and in Uganda own consumption valued that way sits about 23% above what a sale actually fetched. Malawi, which records no such value, moves +52%; Uganda, which does, +0.5%. (#585)
  • median_price_valuation(weight_col=): a weighted median that reproduces the unweighted one exactly under equal weights. The docstring now records the convergence with the WB and EPAR ladders (finest cell with ten or more observations) and the one real delta: theirs are weighted by population-raked weights.
  • Site Q, quantity_audit.py: the read-path audits asked whether a column is present, unique or non-null; none asked whether it is possible. Tanzania 2020-21 reports 7,500,000 kg of coconuts from one plot and Benin 12,500,000 kg of cotton, both graded sane. Site Q names any crop_production.Quantity above 100 times the 90th percentile of its comparison cell (floored at one native unit), counts, and never clips: 151 of 626,687 rows across 15 countries, 65 ms on Uganda. Its own lever, LSMS_QUANTITY_STRICT; its own accessor, quantity_reports(). (#857)
  • Transforms: rcsi and rcsi_phase (label-keyed WFP weights; phases that partition), gross_crop_revenue, livestock_sales_value, hdds and fcs (you supply the food-group mapping; the library refuses one that is not a partition), fertilizer_rate, crop_diversity. Each names the EPAR construct it corresponds to and where it deliberately differs. (#863)

Fixed on the way to this release

  • import lsms_library no longer raises ConfigError when LSMS_COUNTRIES_ROOT points at a config tree with no .dvc/config. The runtime DVC override supplied credentials for the S3 remote without its url, which DVC validates per level; a bare config tree therefore failed at import. The override is now added only where .dvc/config exists to complete it. Not in v0.11.0; caught by this release's cold gate.
  • valuation= is documented on the generated data-methods page (the discoverability test enforces it).

Documentation

  • slurm_logs/2026-09-09_epar_curation/: the EPAR survey -- what the project is, a three-way table against the WB panel and this library, twelve ranked and red-teamed learnings, and the workplan that produced this release. CLAUDE.md now names both comparison projects and the rule the survey kept re-learning: "No issues in this section" is not clearance for a section whose inputs come from one that has an issue.
  • slurm_logs/gh850_design/DESIGN.org: the measured case for fixing the food-side kg inference (no item axis; a median of expenditure where a price was meant; a metric vocabulary that serves a millilitre at 0.74 kg). Implementation is on a branch, held for a dispersion-gate decision; it ships in the next release.
  • Five CONTENTS.org files gain corrections and notes; the Ethiopia file had said the area-unit factors were "NOT in the repo" -- they were.

Filed, not fixed

  • 866: a country-level script-path table whose Makefile has no rule for

    its var/ target is never rebuilt after a hash move -- make reports an existing rule-less target as up to date, and the stale frame is re-stamped fresh. LSMS_NO_CACHE=1 does not escape it. Twenty tables in thirteen countries. Until fixed, clear the country before a re-warm.

  • 868: reduce_to_agreed warns on a conflict but files no grain report.

  • 869: Malawi Cassava, "other" and tree crops vanish from 2016-17 and

    2019-20 (3,581 rows) because the perennial code namespace changed.

  • 865: Niger 2011-12 carries a 999999 sentinel of its own.

Held for 0.11.2

Malawi's shipped-factor loader (#854) is complete but adds the canonical condition level to Malawi crop_production, and Feature()'s modal-shape rule excludes a country for moving toward the canonical index; the fix to Feature lands first. The food-side kg inference (#850) waits on the dispersion gate.