Skip to content

Caching and Performance

How Caching Works

When you call a table method (e.g. uga.food_expenditures()), the library reads the relevant raw data files, harmonizes them, and writes the result as a Parquet file under data_root(). Subsequent calls -- in this process or a fresh one -- read that Parquet instead of rebuilding from source. There is no in-memory memoization: each call re-reads the Parquet and re-runs _finalize_result (categorical mappings, kinship expansion, canonical spellings, dtype coercion, the v join, weight normalisation), so repeat calls are much cheaper than the first but not free, and they return a new object each time. The raw .dta blobs themselves are also cached locally, so cold rebuilds (e.g. after editing a wave's data_info.yml) don't have to re-download from S3.

import lsms_library as ll
uga = ll.Country('Uganda')

# First call: builds from source (10s of seconds to minutes depending on country)
hh = uga.household_characteristics()

# Subsequent calls: served from the Parquet, but finalization still runs
# (measured on Uganda `sample`: 0.44s cold -> 0.08s warm, new object each time)
hh = uga.household_characteristics()

Three Cache Tiers

The library maintains three independent caches, all rooted under data_root():

Tier Path Content Populated by Read by
L2-country (parquet) ~/.local/share/lsms_library/{Country}/var/{table}.parquet Per-country, per-table harmonized output (waves concatenated, ready for analysis) Country.{table}() after building from waves The v0.7.0 top-of-function cache read in load_dataframe_with_dvc, delegated since GH #908 to the read gate Country._read_l2_country_cache (hash-gated since v0.8.0)
L2-wave (parquet) ~/.local/share/lsms_library/{Country}/{wave}/_/{table}.parquet Per-wave harmonized output (one wave's portion of the table) Wave.grab_data() on first call (YAML path, since 2026-04-15) or _/{table}.py scripts via to_parquet() (script path, always) Wave.grab_data() on subsequent calls, through the read gates Wave._read_l2_wave_cache (YAML path) and Wave._fresh_script_parquet (script path), since GH #1040; run_make_target for script-path tables
L1 (DVC blobs) ~/.local/share/lsms_library/dvc-cache/{md5[:2]}/{md5[2:]} Content-addressed copies of the raw .dta files (and other DVC-tracked sources) _ensure_dvc_pulled() in local_tools.py downloads the blob directly from S3 (sidecar md5 + the active DVC remote's fsspec backend) on first read, bypassing the DVC CLI / Repo.fetch (v0.7.3) DVCFS.open() via DataFileSystem._get_fs_path's typ == "cache" branch

All three tiers live under the same root, so a single LSMS_DATA_DIR environment variable moves them together. On a shared cluster, set LSMS_DATA_DIR=/shared/... and every user gets the same warm caches.

How they interact

The fast path on warm reads is L2-country alone: the v0.7.0 top-of-function read in load_dataframe_with_dvc (the read gate Country._read_l2_country_cache, since GH #908) returns the per-country aggregated parquet without ever touching the lower tiers. All-warm calls finish in ~0.5 s regardless of how many raw files the country has.

When L2-country is cold (e.g. after editing a data_info.yml and clearing the parquet cache), the library re-runs wave aggregation. For each wave in turn, Wave.grab_data() checks L2-wave first (the read gate Wave._read_l2_wave_cache, since GH #1040) and reads it directly on a hit. On an L2-wave miss the wave script calls get_dataframe() for its raw .dta files; on an L1 hit the file is served from the local blob cache via DVCFS.open(), on an L1 miss _ensure_dvc_pulled downloads the blob directly from S3 (via the sidecar md5, bypassing Repo.fetch) and then DVCFS.open reads from it. After the first cold rebuild, L1 and L2-wave both stay warm across sessions, so the next L2-country rebuild doesn't need any S3 traffic or repeated DVC SQLite lookups.

Empirical numbers on a Linux workstation, Niger household_roster (~10 raw .dta files):

Scenario Time What's running
Truly cold (no caches) ~600 s First-ever fetch, every blob from S3, full DVC index build
Cold-cold (L1 + L2-country both empty) 460-510 s Per-file blob fetch + harmonization + parquet write
L2-country cold, L1 warm ~70 s Sidecar pre-check finds blobs in cache, no S3 traffic, harmonization + parquet write only
All-warm (L2-country hit) ~0.5 s v0.7.0 top read returns the parquet directly

Note (v0.7.3+): the two cold rows were measured in the Repo.fetch era (~11 s/blob). The direct-S3 bypass now fetches each blob in ~0.1 s and lock-free, so cold builds are dominated by harmonization, not fetching — the absolute cold numbers are lower than shown. The relative ordering still holds.

The interesting row is "L2-country cold, L1 warm": much faster than cold-cold, because the per-file fetch overhead and the S3 round-trips are both gone. Adding the L2-wave tier (2026-04-15) further removes the per-wave DVC SQLite metadata lookup on cold L2-country rebuilds (Uganda cluster_features: 408 s → 0.02 s per wave).

Cross-Session Behavior

As of v0.7.0, the library reads {data_root}/{Country}/var/{table}.parquet at the top of load_dataframe_with_dvc -- via the read gate Country._read_l2_country_cache since GH #908 -- when the file exists and its embedded content hash is fresh (or unverifiable; a hashless country parquet is graded stale since GH #809) — the v0.8.0 staleness check described next; before v0.8.0 this read was unconditional. Every country benefits from this; there is no longer a special-case for the seven DVC-stage countries.

Automatic content-hash staleness (v0.8.0). Each cached parquet carries an embedded content hash (in its pyarrow schema metadata) that covers everything determining its pre-finalize contents: the LSMS_CACHE_SCHEMA version, the wave's data_info.yml, the country's data_scheme.yml and Makefile, the wave and country modules ({wave_folder}.py, {country}.py, mapping.py), any _/{table}.py script, the build-time *.org inputs in _/ (food_items.org, categorical_mapping.org, … — CONTENTS.org excluded), and the DVC sidecar md5 of each declared source file (the .dta itself is never hashed or downloaded). On read, the hash is recomputed from the current sources and compared:

  • match → the cached parquet is served (fast path preserved);
  • mismatch → the cache is treated as stale and rebuilt from source;
  • no embedded hash (a parquet built before v0.8.0) → trusted once and re-stamped, so the next read is guarded (no rebuild-all on upgrade).

So editing a data_info.yml, a wave/country _/{table}.py script, the country module, or a source .dta (via its sidecar) now invalidates the cache automatically — no manual step required. The recompute is cheap (~1–4 ms; small config/sidecar reads only, no .dta hashing, no S3, no directory walk), preserving the v0.7.0 fast-path win.

Known limitations. (1) Script-path tables whose sources are referenced via dynamically-constructed paths (not string literals in the _/{table}.py) are covered only by hashing the script text, not the source content — an out-of-band edit to such a source still needs a manual cache clear. (2) assume_cache_fresh=True deliberately skips the hash check ("I promise the cache is current"). (3) LSMS_NO_CACHE=1 still forces a full rebuild.

You can still invalidate manually:

  • Set LSMS_NO_CACHE=1 for the session to force a rebuild from source. Only 1, true or yes (case-insensitive) bypass the cache reads and skip the YAML-path L2-wave write during a rebuild; all other values leave those gates on their ordinary routes. The rebuild rewrites the L2-country parquet in the data root you are pointed at, with the frame an ordinary build would write.
  • Run lsms-library cache clear --country {Country} (optionally with --method {table}) to evict the affected L2-country and L2-wave parquets before the next call.

Either way, the rebuild reads from L1 (the DVC blob cache) where possible, so you only pay the S3 cost once per blob per machine.

Direct country builders with the Make backend

For a declared materialize: make table with a country builder, a current country parquet with sufficient declared wave coverage can be reused. Otherwise the library builds in a temporary data root: current registered prerequisites are copied, missing declared wave contributions use the existing loaders or scripts, and the selected country target starts absent. The actual country builder still supplies the table and its transformations. A zero-exit producer that leaves no output cannot borrow an old wave or country parquet.

Finalization and required-column validation inspect a copy. After checking that declared inputs stayed stable, the library stamps the original pre-finalize parquet with its input hash and full builder coverage, then atomically replaces only the selected country output. Live wave files, other tables, reviewed JSON sources and grouped-rule companions retain their original bytes and availability. The optional lsms_make_inputs footer records declared Other-table hashes and literal source fingerprints not already covered by the selected wave hashes. Direct Make reuse checks this witness as well as the normal hash/coverage gate. Available declared L1 blobs are copied into the temporary root; there is no shared writable L1 link. Cold builds therefore need extra temporary disk space.

Literal registered Other-table references can use their existing builders in the private root when reuse is unproven. Explicit Other-wave references run the physical wave producer; a native country build is not assumed to retain its temporary wave companions. Declared raw/DVC inputs keep their source provenance and are eligible without a harmonized-cache hash. A hashless input alone is not evidence of staleness. Dependencies computed outside the existing declarations/literal scanner remain a boundary: a missing rule or unknown prerequisite keeps its native failure, and an input with no current producer/provenance is not copied and certified merely to keep the build running. Root-honoring, sanctioned IO is required; arbitrary outside-root reads are not certified by this mechanism. This can refuse a legacy build whose computed, hashless, rule-less prerequisite happens to be correct but has no established current producer/provenance. The live file is retained. This availability tradeoff is an explicit residual; broader country exposure and other backend failure paths remain unverified. The native output is copied to a parent-owned private snapshot for validation and promotion, so a later producer write cannot replace the bytes being validated. Prepared selected-wave bytes and declared source fingerprints must remain stable through the country build. Wave grain/date/drop evidence is retained in the country footer; grain and date replay describe the requested waves, using the existing read-gate policies. Finalization never writes back. Default/JSON/legacy failure policies, explicit assume_cache_fresh, and the existing wave-rebuild and guarded country-fallback evictions retain their scope.

Degraded sources with a read-only L1

The once-per-source heal probe normally uses local files or cached DVC blobs, fetching an absent blob through the configured backend. If a local cache mkdir, temporary write or rename fails with EACCES/EROFS, that probe alone may check metadata existence on the same two configured remote md5 keys. It makes at most two existence calls, with no streaming, parsing, World Bank fallback or additional retry loop. A positive result is availability evidence, not proof that a rebuild can decode the source. The existing per-process probe memo and once-per-key rebuild limit still apply.

Cached blobs and successful writable-cache fetches make no extra existence calls. Remote access failures, ambiguous error locations and unrelated IO errors do not trigger this fallback. Ordinary readers and cache warmers keep their best-effort behavior and their existing streaming fallback; there is no new user setting or shared-cache write policy.

Repeated advisory connection failures

After both remote layouts finish their existing retry sequence with uniform, positively identified connection-establishment failures, the fast heal probe can defer other uncached sources on the same eligible S3 route for five seconds. The deadline does not slide. It retains the backend already used, including a default-endpoint route with demonstrably nonrefreshing profile authentication; unknown or refreshing authentication and ambiguous routes keep probing normally. Local files and either L1 layout still win immediately. A deferred False is transient and is not added to the completed per-source memo. Completed actual negative probes keep their existing memo behavior.

Only advisory GET probes participate. The local-write-denial metadata branch still makes its original at-most-two existence calls, even during this window; HEAD-only failures never start it. Ordinary readers, warmers and streaming keep their full paths. State is process-local, reset on owner/configuration changes, and bounded to 16 retained routes. Expiry releases references lazily on a later probe without closing another caller's client. Savings concern sequential eligible wrapper calls after the first sequence, not SDK HTTP attempts, measured field latency or universal outage detection. There is no user setting.

Dropped subframes with multiple source files

A dropped multi-file subframe records every declared tracked source whose L1 blob is absent. It becomes eligible for the existing once-per-key retry only when all recorded sources are fetchable; separate dropped subframes can heal independently. Discovery uses sidecar/local-cache metadata, not new source reads. A definite no-source path remains a configuration problem even when another listed file is tracked; existing required-column config errors stay loud.

The existing file field and older single-file stamps retain their behavior. New stamps add an ordered files list; malformed lists do not claim healing. Older ambiguous multi-file stamps still contain only one path until a legitimate rebuild writes the richer evidence. There is no automatic migration, source-config lookup, schema bump or promise that metadata availability proves decoding.

Explicit input lineage and unknown coverage

Wave table declarations can describe inputs used by a derived step or a final hook. Names refer to harmonized merged columns or index levels before drop:, not raw source variable names or derivation-registry retrieval functions:

derived:
  v:
    kind: coalesce_coord_bin
    lat: Latitude
    lon: Longitude
    inputs: [v, Latitude, Longitude]
output_inputs:
  Region: [Region, Region_enrichment]

inputs is reserved metadata and is consumed before transformer kwargs are dispatched. Derived steps execute in declaration order, once each. Name the existing target explicitly when a step reads it: self-input means its value before that step. Explicit chains propagate interrupted-provider lineage. output_inputs belongs to the table in wave data_info.yml, for final hooks and direct sibling coverage; its entries all refer to the pre-hook state. Empty mappings/lists declare no dependencies. Malformed declarations raise; input presence is not validated by this metadata, and transform/hook errors retain their existing behavior, including absent coordinate errors.

If a declared input's provider was dropped, the footer adds coverage_unknown_required, coverage_unknown_optional and affected_outputs. Each affected output records the interrupted original input names and whether the final output is present, alongside the drop's wave, subframe and source file/files. Existing missing_required/missing_optional fields continue to mean entirely absent direct columns. Unknown coverage is conservative even if a sibling fills every value: it is not evidence of lost rows or wrong values. Warnings distinguish these states and expose coverage_unknown_required, coverage_unknown_optional and input_drops separately from missing fields.

Unavailable explicit interruptions use the existing bounded degraded probe and rebuild policy, including per-source memo, once per key, just-written suppression and all-source readiness for multi-file drops. Configuration drops keep their existing policy. No declarations means legacy behavior; old footers are not inferred or migrated, and unrelated temporary columns remain ungraded. This metadata does not fill, filter, normalize or otherwise change the frame, its axes or attrs. Existing country configurations are not automatically annotated.

Cache Location

All three tiers are under the user data directory:

  • Linux: ~/.local/share/lsms_library/
  • L2-country: {Country}/var/{dataset}.parquet
  • L2-wave: {Country}/{wave}/_/{dataset}.parquet
  • L1: dvc-cache/{md5[:2]}/{md5[2:]} (DVC 2.x layout) or dvc-cache/files/md5/{md5[:2]}/{md5[2:]} (DVC 3.x layout)
  • macOS: ~/Library/Application Support/lsms_library/...
  • Windows: %LOCALAPPDATA%/lsms_library/...

Override with the LSMS_DATA_DIR environment variable or the data_dir key in ~/.config/lsms_library/config.yml. The env var takes precedence.

This controls only the data root (caches + DVC blobs). The config tree (countries/{C}/_/...) is separately overridable via LSMS_COUNTRIES_ROOT / countries_dir (GH #436) — see CLAUDE.md "Data Access". Keeping the two independent is what lets a git worktree read its own config while sharing the warm cache.

The library handles both DVC cache layouts because legacy .dvc sidecars in the countries/ repo carry md5-dos2unix hashes from before the DVC 3.0 cutover and write to the flat layout, while newer sidecars use the files/md5/ subpath. Users don't need to think about this; the sidecar pre-check in _ensure_dvc_pulled looks at both locations.

The package tree never contains DVC-tracked data

This is a hard architectural rule: the package source tree (lsms_library/countries/) holds code, configs, and .dvc sidecars only. Tracked data files live in the L1 cache under data_root(), not in the workspace.

_ensure_dvc_pulled downloads blobs directly from S3 (via the sidecar md5, bypassing the DVC CLI / Repo.fetch) to populate the cache without checking files out into the package tree. And local_file() inside get_dataframe refuses to use any workspace copy of a file that has a sister .dvc sidecar — if it finds one, it emits a UserWarning with a cleanup command and falls through to the DVC cache path.

If you see warnings like:

Refusing workspace copy of DVC-tracked file ... (sister .dvc sidecar exists). The package tree must not contain DVC-tracked data; falling through to the DVC cache path. Clean up with: find lsms_library/countries -type f -name '*.dta' -execdir test -e '{}.dvc' \; -print -delete

it means your dev checkout has leftover .dta files from a prior dvc pull (or from an older version of the library that checked files out via Repo.pull). Run the cleanup command to remove them. The cache under data_root() already has the same content; the warnings will stop and reads will use the canonical cache path.

Assume Cache Fresh Mode

Stored grain-audit replay (GH1066)

Successful assumed-fresh reads replay stored grain warnings and grain_reports() entries, using the same requested-wave filtering and LSMS_GRAIN_STRICT behavior as the normal cache read. Date replay remains; strict grain errors take precedence just as on the normal path. Historical footer evidence does not establish current freshness or served completeness. The selected policy restores stored diagnostics without checking hashes, probing sources, rebuilding data or writing the cache.

uga = ll.Country('Uganda', assume_cache_fresh=True)

With v0.7.0 in place, assume_cache_fresh=True is mostly redundant with the default behavior. It remains useful as a stricter bypass that skips even the LSMS_NO_CACHE check.

What assume_cache_fresh=True does not skip: _finalize_result. Kinship expansion, canonical spelling normalization, dtype coercion, sampling-weight normalization, and the _join_v_from_sample augmentation all still apply on read. The returned DataFrame is not byte-identical to the on-disk parquet — the parquet is closer to the raw source data, and _finalize_result is the harmonization layer applied at every read.

A concrete example of that gap, worth knowing before you read a parquet directly: sample.weight and sample.panel_weight are stored raw (an expansion weight summing to a national population, or whatever the survey shipped) but are served rescaled to within-wave mean 1. So pd.read_parquet(.../var/sample.parquet)['weight'].mean() can be 630 while Country(...).sample()['weight'] has mean 1.0 in every wave — both are correct, at different layers. See CLAUDE.md §"sample() and Cluster Identity".

Do not use assume_cache_fresh=True after editing source data or a wave's configuration unless you have cleared the affected caches first — you will silently get stale results.

Deprecated: trust_cache=True is a legacy alias for assume_cache_fresh=True. It still works but emits a DeprecationWarning, and is slated for removal in a future release.

Managing Cache

Use the CLI to inspect or clear cached parquets:

# List cached datasets
lsms-library cache list --country Uganda

# Remove a specific cache
lsms-library cache clear --country Uganda --method shocks

# Dry-run removal for everything
lsms-library cache clear --all --dry-run

When contributing changes to a wave's data_info.yml or _/{table}.py, clear the affected country's cache before rebuilding so the next call actually reads your new source data rather than an older cached parquet. lsms-library cache clear evicts both L2-country and L2-wave; the L1 blob cache stays populated either way, so the rebuild won't re-fetch from S3. The command also audits the config tree, which it does not own and never deletes from: it names any in-tree *.parquet (never read, GH

803) and any in-tree raw source file -- .dta, .sav, .csv, .dat,

.dct, .ssp, .tab, .xls, .xlsx -- distinguishing a copy beside its .dvc sidecar (refused by get_dataframe, which reads the DVC blob instead; disk cost only) from one with no sidecar, which get_dataframe also refuses, raising InTreeSourceError, unless the caller opts in with LSMS_ALLOW_INTREE_SOURCES=1 or reads inside the acquisition flow (add_wave / populate_and_push, which run under local_tools.allow_intree_sources()) -- GH #831. It used to be served in preference to the DVC blob, silently.

To clear the L1 blob cache too (rarely needed):

rm -rf ~/.local/share/lsms_library/dvc-cache/

Adding new data files

Two workflows for getting new survey data into the library, both of which write .dta (or other source) files into a Data/ directory under the package tree as a transient step:

  1. Manual (per CONTRIBUTING.org). Download the source files, place them in the appropriate lsms_library/countries/{Country}/{wave}/Data/ directory, and publish them with data_access.push_to_cache() -- or push_to_cache_batch() for more than one file. These generate the .dvc sidecar and upload the blob.

    Do not shell out to dvc

    Never invoke the dvc CLI yourself -- not add, not push, not pull. The CLI takes the global .dvc/tmp/lock and rebuilds the repo index by walking ~7k sidecars; concurrent callers collide (measured 91.7% failure at 12 concurrent). The library's writers take the lock with backoff-and-jitter retry, and its readers bypass it entirely.

    1. Automated via the World Bank Microdata API. The data_access.get_data_file() fallback in local_tools.py can download missing files from the World Bank Microdata Library (requires MICRODATA_API_KEY) and add them to DVC for you.

In both cases the new file lives temporarily in the workspace. Once the .dvc sidecar exists, remove the workspace copy — the file is now in the L1 cache and local_file() will refuse to read the workspace copy on subsequent calls (with the cleanup warning). Removing it keeps the package tree clean and consistent with the architectural rule above.

# After push_to_cache() succeeds:
rm lsms_library/countries/Niger/2024-25/Data/new_section.dta
# The blob is in ~/.local/share/lsms_library/dvc-cache/...
# and DVC.open will serve it from there.

Build Backends

The library supports several build paths; which one runs depends on the country and environment:

Backend When Used Description
Python aggregator Default for all countries Reads wave-level data_info.yml and builds in memory; the L2-country parquet is written and read across sessions via the v0.7.0 top read, and each wave's intermediate is cached as L2-wave
Make / script When data_scheme.yml marks a table materialize: make Per-country/wave Makefiles and standalone Python scripts; the only choice for tables that cannot be expressed purely in YAML (post-planting/post-harvest dual rounds, multi-wave source files, etc.). These also write L2-wave parquets via to_parquet()
# Select the Make backend (this does not bypass the parquet tiers)
export LSMS_BUILD_BACKEND=make

LSMS_BUILD_BACKEND=make changes how _aggregate_wave_data dispatches. A declared materialize: make table with a country builder uses the private transaction described above, including verified warm country reuse. A declared table without that builder uses its waves, whose YAML loaders still read and write L2-wave parquets. Other legacy callers retain their existing runner. L1 still serves raw blobs.

Build Parallelism

Wave-parallel cold builds (LSMS_BUILD_WORKERS)

A cold Country(c).<table>() builds each wave -- the YAML extraction, or for a script-path table one make run of python <table>.py -- in a fork-context process pool, one worker per distinct build target, then post-processes the results in the parent in wave order (id_walk, index augmentation, the grain-audited normalisation, concat) exactly as a serial build would. Waves that share a folder and therefore a make target (Nigeria's PP/PH quarter waves, Tanzania's 2008-15/ four-round folder) are built by ONE worker, serially, so no two processes ever run the same target. The output is byte-identical to a serial build -- the API frame, the L2-country parquet including its embedded lsms_cache_hash / lsms_grain_audit, the warning multiset and grain_reports() / null_read_reports() (measured; see slurm_logs/gh797/parallel_waves/REPORT.org).

export LSMS_BUILD_WORKERS=1      # today's serial path, unchanged
export LSMS_BUILD_WORKERS=4      # at most 4 workers

Unset, the worker count is the cgroup-visible CPU count (sched_getaffinity, capped by SLURM_CPUS_PER_TASK / SLURM_CPUS_ON_NODE when set), capped at the number of targets, and capped by a memory guard: the smallest of MemAvailable, the cgroup memory limit (memory.max on the process's own cgroup node and its ancestors, or the v1 memory.limit_in_bytes) and Slurm's allocation (SLURM_MEM_PER_NODE, else SLURM_MEM_PER_CPU x visible CPUs), divided by a per-worker budget:

export LSMS_BUILD_WORKER_MEM_GB=4   # the default; GiB reserved per worker

The default covers the largest wave script measured (GhanaLSS 2016-17 food_acquired, 3.7 GB peak) plus a worker's own 100-350 MB. Example: a 16 GB / 8-core laptop with ~12 GB available runs GhanaLSS food_acquired (7 targets) with 3 workers, a 24 GB / 4-core htc slice with 4, a 93 GB / 32-core node with 7. An explicit LSMS_BUILD_WORKERS=N is still capped by the CPU count but MAY exceed the memory guard -- you asked for it; a RuntimeWarning says so once, and a worker the OOM killer takes fails the build loudly (BrokenProcessPool) rather than silently. Inside a worker make -j is cut to visible // workers, BLAS/OpenMP pools are pinned to one thread, and LSMS_BUILD_WORKERS=1 is exported so a wave script that itself builds another table does not open a nested pool. Warm reads never enter the pool.

make -j inside a build (LSMS_MAKE_JOBS)

Make-based builds default to half of the CPUs visible to the process -- the cgroup / Slurm allocation, not the physical node (GH #764: a 4-core slice on a 56-core node used to run -j28). Override with:

export LSMS_MAKE_JOBS=4

A country-level make fans out to one python <table>.py per wave, so -jN runs N wave builds — and therefore N concurrent large-blob S3 fetches — in parallel. Under LSMS_BUILD_WORKERS an explicit LSMS_MAKE_JOBS is a ceiling on each worker's -j, never a floor. On most hosts that is pure speedup. On a host where concurrent multipart S3 reads occasionally corrupt a TLS record under load (symptoms: ssl.SSLError: ... DECRYPTION_FAILED_OR_BAD_RECORD_MAC, or aiohttp.ClientPayloadError: Response payload is not completed, mid-build), the fix is not a network change — connectivity is fine and a single fetch succeeds. Two levers:

export LSMS_MAKE_JOBS=1      # serialize wave builds (one S3 fetch at a time)
export LSMS_FETCH_ATTEMPTS=5 # or: keep parallelism, retry transient TLS errors

The blob fetch (local_tools._get_file_with_retry) already retries these transient errors (LSMS_FETCH_ATTEMPTS, default 3) with exponential backoff before falling back to a single-stream read, so most such hiccups self-heal without touching -j. Reach for LSMS_MAKE_JOBS=1 only if they persist.

Unreadable managed caches (#1071, first tranche)

The YAML/script wave gates and ordinary/assumed-fresh country gates treat an existing managed Parquet that fails initial payload decoding with OSError or ArrowInvalid as a cache miss, with an explicit unreadable-cache warning. This is a reader failure, not proof of physically damaged bytes. Saved coverage is retained before rejecting the exact old entry; script rebuilds are forced so Make cannot mtime-serve that entry. One ordinary rebuild is attempted; a fresh invalid output or failed producer still raises. Valid hits, assumed-fresh hash bypass, normal null/index/grain/date policy and strict refusals are unchanged.

This is not generic file healing: raw/user inputs, post-payload schema/index restoration and audits are never reclassified. Other-features caches, legacy JSON/parquet alternatives, retired-stage outputs and private Make prerequisite owner routes remain separate #1071 work. Earlier raise-only tests/receipts characterize the old policy; they are not evidence that unreadability proves physical damage or that the entire issue is resolved.