Skip to content

v0.11.0

A release about relationships and about trust in the cache. Kinship gets an honest cell for "other relative", the Ghana surveys get one relationship vocabulary, and two mis-decoded waves that had graded sane for their whole lives are fixed. Underneath, a cache-poisoning path that let stale parquets masquerade as fresh is closed, and cold builds run their waves in parallel. Alongside those: a notes API, two visualisations, Togo non-food expenditures, GhanaLSS nutrition, GhanaSPS plot labour, and a redrawn coverage page.

Behaviour changes you will see in returned data

  • "Other relative" has its own kinship cell. Every "other / unspecified relative" label used to sit on the household head's cell, [0, 0, consanguineal], so the Kroeber tuple could not tell a head from an unspecified relative. Such labels are now [null, null, consanguineal] -- a blood relative of unrecorded generation and distance -- and labels that name the spouse's relatives too ("relative of head or spouse", EHCVM "autres parents du CM ou du conjoint") are [null, null, null]. About 100,000 roster rows across 28 countries now carry NA Generation / Distance for those people; nothing else moves. Nulls in those columns are data, not defects (#698, #797).
  • Categorical mappings run before kinship expansion in _finalize_result, so a country #+name: relationship table with a Preferred Label column canonicalises survey wording before the kinship lookup sees it. Measured byte-identical on every cached roster before landing; no cache hash moves (#797).
  • One relationship vocabulary for the Ghana surveys. GhanaLSS served five vocabularies across seven waves; it and GhanaSPS now serve one -- Head, Spouse, Child, Adopted/foster/stepchild, Child-in-law, Grandchild, Parent, Parent-in-law, Sibling, Sibling-in-law, Niece/nephew, Other relative, Non-relative, Servant, Tenant -- and a Comparable variant (household_roster(labels={'Relationship': 'Comparable'})) that serves the eight categories every wave asked, for pooling. The shared adopted/foster/step label is [-1, 0, null]. A survey-family target, not a corpus-wide canon: surveys elsewhere understand relationships differently.
  • Wave-parallel cold builds are the default. A cold Country.<table>() builds its waves in a fork pool (one worker per distinct make target; memory-guarded at 4 GiB per worker; LSMS_BUILD_WORKERS=1 restores the serial path). Measured 2.2--2.5x on script-path tables, 1.6--1.9x on YAML rosters, output byte-identical including the stamped grain audit. make -j now reads the cgroup-visible CPU count, not the physical node (#764).
  • Every table's cache hash moves once on upgrade (two edits to the build path: the in-tree parquet fix and the parallel build). Expect one cold rebuild per table on first read.

Data fixes

  • Currency-denominated u labels get no inferred kg factor. Value, dinar, Birr and their kin are amounts of money, not containers; conversion_to_kgs was minting a kg-per-unit factor for them exactly as it does for Heap or Bowl, so food_quantities(units='kgs') carried fabricated kilograms for those rows. They now come back with the native u tag and no kg conversion. Step 1 of #770; step 2 (a source= argument drawing on community_prices) is deferred behind #782 (#784).

  • Six stranded food vocabularies now reach labels=. GhanaLSS, GhanaSPS, Cambodia, Serbia, Guatemala and Panama kept their food label tables in _/food_items.org, where Country.<table>(labels=...) could not see them; each is now the harmonize_food table of the country's _/categorical_mapping.org, carried across byte-identical (#789, refs #783 -- labels='Aggregate Label' on GhanaLSS is deliberately still open).

  • GhanaLSS 1991-92 household_roster: GLSS3's nine-code relationship scheme was decoded through GLSS1's fourteen-code table -- 1,353 people served as siblings, nieces and in-laws who were servants, tenants and unrelated (#799). GhanaLSS's relationship decode is now wave-first with no country fall-through, so a wave without its own code table fails loudly.

  • Tajikistan 2003 household_roster: the inline relationship dict was shifted one slot from code 7 -- 4,047 people (15.5%) served as the wrong relative (#800). 2007 and 2009 verified against their own questionnaires; 1999's code 14 is #804.
  • Six roster cells served encoding artefacts as labels (Mali, Niger, Senegal, Benin, Togo 2018-19; Burkina Faso 2014; 16,989 rows). Fixed at the read in each wave's hook; kinship.yml loses the 14 artefact keys that had been absorbing them (#801). The reader-level cause is #805.
  • GhanaLSS food_acquired.j is canonical: 35 label-collision groups over 27.9% of rows (Egg/Eggs/eggs) reduced to zero (#782, #785).
  • Uganda earnings delivers Earnings and no longer fabricates 82.6% of its rows (#772); Uganda enterprise_income was 82% phantom rows (#788).
  • Togo: nonfood_expenditures from the EHCVM s09b--s09f modules; five unrecoverable non-food sidecars retired (#750).
  • GhanaLSS nutrition from the West Africa Food Composition Table (#563), graded as the country-level cell it is (#776); GhanaSPS plot_labor for all three waves.

Cache correctness

  • A parquet inside the source tree is never read. Wave.grab_data and run_make_target preferred an in-tree {wave}/_/{table}.parquet to running the wave script; such files are hashless, graded legacy, and were stamped into the country parquet with a fresh hash -- the shared cache served pre-#785 GhanaLSS food_acquired as fresh, and 109 such artefacts sat in one checkout across 14 countries. They are now ignored with a warning, make -B is forced when one exists, and cache clear lists them (#803). The canonical-shape test no longer rewrites whatever data_root() happens to be.
  • GhanaLSS food_acquired has one builder. Building nutrition ran a country-level _/food_acquired.py through a Makefile prerequisite; it re-keyed the GLSS1/GLSS2 split households with a _0 suffix convention the framework's updated_ids does not use, and its hashless output was graded legacy, re-stamped and served as fresh (101332_0 where the API contract says 101332 / 101332_1). The script and the prerequisite are gone; the wave scripts plus load_from_waves are the only path (#808). The #479 required-column guard now recognises a table built by wave-level scripts alone as script-path, which moves every cache hash once (an upgrade from 0.10.0 already rebuilds once, so no extra cost there). Trusting a hashless var parquet written by a recipe as legacy is the structural gap and is filed separately.
  • The build fingerprint no longer folds the contents of the grain and null-read ledgers, which made a stamped hash depend on which reports had been filed first in the process (#806).
  • DVC writes: the timeout is tunable and no longer below this repository's floor; a timed-out write no longer leaves the repo locked; S3 reader credentials are explicit. Uganda's fct_usda.csv is tracked -- an untracked file was being hashed.

New surface

  • PopulationRecord on df.attrs, with attribute access. The record a frame carries under attrs['population'][country][wave] is now an immutable dict subclass: rec.universe_tag and rec['universe_tag'] both work, df.to_parquet() keeps working (a frozen dataclass there made pandas 3's json.dumps of attrs raise), and records_from_attrs accepts the plain dicts a parquet round-trip gives back. GhanaLSS 2016-17's record is re-graded from the GLSS7 Main Report itself -- institutional population excluded from the frame, 20 region x urban/rural strata, a 15,000-household design -- with page references (#796, refs #603).

  • coordinate_map(where=...) restricts the map to a subset of clusters -- one region, one stratum, rural only -- as a {column: value} / {column: [values]} dict or a callable on the assembled frame (cluster_features columns plus the summed size). A condition that matches nothing raises and lists the values the column holds, the filter is named in the title, and the no-coordinates count is for the selection.

  • Country.notes() / note_topics: CONTENTS.org through the API. Country.features / Wave.features as synonyms for data_scheme.

  • ll.population_pyramid (with ghost= overlays and max_age='auto') and ll.coordinate_map (points on a basemap, sized by a variable), exported at the package top level; matplotlib and folium are now ordinary dependencies (the optional viz group is gone).
  • sampling.yml, a sibling of population.yml recording how each sample was drawn, with a derived design fingerprint (draft; Uganda as the exemplar, #793).

  • The coverage page: cells are per-wave colour strips (no text), a survey timeline (thickness = features graded, colour = share sane) sits above the grid, and one stdlib renderer draws both the docs site and bench/matrix.py -- the two had drifted (the site was blind to five tiers). Snapshot refreshed 2026-09-07 (Nepal and Albania 1996 now blocked rather than broken; Mali interview_date sane).

  • kinship.yml sheds 14 Ghana-only spellings. With GhanaLSS and GhanaSPS now translating their raw Relationship labels through one shared vocabulary before kinship decomposition, the raw spellings those surveys used to emit (Parent/Parentlaw, Son/Daughterlaw, Adopted, Foster/Stepchild, ...) reach the kinship lookup from nowhere. bench/kinship_reach.py measured that over every country with data (34) before they were removed; 131 other never-reached keys are the deliberate English and French reference vocabulary and stay, listed in a follow-up.

Documentation

  • .dta string encoding, measured. A sweep harness over all 3,745 .dta files in the corpus (slurm_logs/dta_encoding/), and CONTENTS.org entries for Senegal and Burkina Faso recording that harmonize_food's Original Label column is keyed on mojibake in six EHCVM countries -- load-bearing in Senegal, a live 12,933-row fragmentation of beef in Burkina Faso. No read path changes here on purpose; the fix must land together with a crosswalk re-key (#795, investigation #794).

Credentials and publishing; the contributing page sent people to the wrong branch; the GLSS7 sample design from the Main Report, verified; the UEMOA PHMECV catalogue inventoried; the GitNexus mandate scoped to when GitNexus actually works; CONTENTS.org TODO states declared everywhere.