Skip to content

Cross-Country Analysis with Feature

The Feature class answers "what do we know about a given table across the developing world?" It assembles a single harmonized DataFrame for a table across all countries that provide it.

Basic Usage

import lsms_library as ll

roster = ll.Feature('household_roster')

# Which countries provide this table?
roster.countries
# ['Albania', 'Armenia', 'Azerbaijan', 'Benin', 'Burkina_Faso', 'Cambodia', ...]

# What are the guaranteed columns?
roster.columns
# ['Sex', 'Age', 'Relationship', 'Generation', 'Distance', 'Affinity']

Loading Data

Feature is callable. Invoke it to load and concatenate data:

# Load all available countries
df = roster()

# Load specific countries
df = roster(['Mali', 'Uganda'])

The returned DataFrame has a country index level prepended:

                                    Sex  Age  Generation  Distance       Affinity
country  t        i     pid
Mali     2014-15  1003  1           M    45   0           0         consanguineal
                        2           F    38   0           0         affinal
Uganda   2019-20  4001  1           M    52   0           0         consanguineal

transformations.seed_kg and transformations.nitrogen_kg retain this country identity when reducing pooled plot_inputs to plots. Their output index is (country, t, i, plot_id) whenever the input carries a country index level or column, including one selected country and empty results. Country labels must be non-null. Ordinary country-less inputs return (t, i, plot_id); attrs['country'] alone does not add an index level. transformations.fertilizer_rate also retains country in its numerator, area grouping, join and output when both plot_inputs and plot_features carry it. Its output is (country, t, i, plot_id) for on='plot' and (country, t, i, parcel) for the existing default on='parcel', including one labeled country and empty results. Country must be non-null on both inputs; labels on only one input are refused, with no inference or broadcasting. An index level takes precedence over a redundant country column. Area keys may still be columns. Country-less calls keep their existing contract, and attrs alone do not add country. A nonempty fertilizer-rate result warns when an entire eligible (country,t) cell has fertilizer and positive-area rows but no matching keys. Anonymous warnings use no country from one-sided or conflicting metadata. The rates and join policy remain unchanged; unmatched plots within a surviving cell are outside this whole-cell check. Where both full plot vocabularies are known to agree, callers can explicitly choose on='plot'. The documented Ethiopia/Malawi literal route also uses known country provenance retained on both inputs. The warning discloses omissions and does not recover missing rates. The crop reducers harvest_kg, gross_crop_revenue and crop_diversity likewise prepend supplied country identity, including singleton and empty results. Crop conversion estimates are computed separately within each country; identical unit labels and household keys do not establish a shared conversion factor. yield_kg requires non-null country on both crop and area operands or neither, retains country through the land join, and selects each row's existing country-specific land route. Labels outrank scalar attrs; country-less inputs keep their existing behavior. This preserves each instrument's land keys and definitions, without making crop occurrence entropy or differently recorded harvest conditions comparable across countries. Other transforms have their own documented grouping and joining contracts.

Design Decisions

Lazy Loading

Construction is cheap -- Feature('household_roster') discovers which countries declare the table but loads nothing from disk. Data is fetched on demand when you call the instance.

Union of Columns

The returned DataFrame takes the union of all columns across countries, not the intersection. Required columns from data_info.yml guarantee a common core; country-specific extras appear as NA where absent.

This makes gaps visible (e.g. District present for Ethiopia but NA for Mali) rather than silently dropping information.

Harmonization Flows Through

All per-country harmonization (categorical mappings, kinship decomposition, canonical spellings, dtype coercion) is applied by each Country before concatenation. Feature delegates to each country's existing table method.

strata Is Not One Vocabulary

Feature('sample') pools a column that means different things in different countries, and sometimes in different waves of one country. A region, a region crossed with urban and rural, a survey's own design code and a district are all served as strata, because that is what each survey's design or files offer:

strata holds Where
Region x milieu, as a label EHCVM waves of Senegal, Mali, Niger, Burkina Faso; Ethiopia
Region alone Benin, Togo, Guinea-Bissau, Cote d'Ivoire 2018-19; GhanaLSS 1998-99 onward
The file's own stratum, a label Niger 2011-12 and 2014-15 (region x agro-ecological zone)
The file's own stratum, a code Mali 2017-18, Burkina Faso 2014
Nothing Mali 2014-15

This table covers the cases measured for GH #812, not every country. So strata identifies a stratum within one wave of one country. Do not compare or group on it across countries, and check before doing so across waves. Spelling differs too: the EHCVM labels end in Urbain or Rural, as the surveys write them, and Ethiopia's in Urban or Rural. Where the label is a construction it is registered: df.attrs['derivations'] names those countries, and Country(c).derivations() gives the rule and its evidence. Each country's _/CONTENTS.org records what its strata holds.

Error Handling

If a country's data fails to load, Feature emits a warning and continues with the countries that succeeded:

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    df = roster()
    # Inspect w for any per-country failures

household_characteristics and food_expenditures validate required provider index levels before comparing country shapes. A provider missing a required level is excluded with a warning; df.attrs['index_unavailable'] records its missing and required levels, returned index names, and row count. A present level with null values is retained. The excluded frame remains available through the Country API.

These requirements are declared separately under Index Info > required_index_levels: (t, v, i) for household characteristics and (t, v, i, j) for food expenditures. A requested market replaces required v with m; monetary index/numeraire modes require currency. Ordinary household characteristics may omit m, and legacy food expenditures may omit s. Their existing shape selection and source detail are preserved: this does not register a new canonical food-expenditure shape or combine incompatible source shapes. Other tables keep their existing reduced-index contracts. If every provider is excluded, Feature returns the existing empty assembly with the exclusion record.

Available Tables

Most tables declared in a country's data_scheme.yml can be used with Feature. Common ones include:

Table Description
household_roster Demographics, kinship decomposition
cluster_features Region, rural/urban classification
food_acquired Food acquisition with units
food_expenditures Derived food spending
shocks Household shocks and coping
individual_education Educational attainment

Country-level-only tables are not Feature tables

panel_ids and updated_ids appear in data_scheme but have no cross-country assembly: Feature('panel_ids')(['Malawi']) returns an empty DataFrame rather than raising. They are graded n/a in the coverage matrix and are reached per country, as Country(...).panel_ids.