Skip to content

v0.10.1

A single fix, for a single sentence that was not true: a valid World Bank Microdata API key is now sufficient to reach the S3 read cache. Until this release it was necessary but not sufficient -- the unlock also required the gpg binary, which is not a Python package and ships on neither Windows nor macOS by default.

No cache rebuild. _table_cache_hash is byte-identical before and after -- measured across Uganda/household_roster, Nigeria/livestock, Pakistan/cluster_features, CotedIvoire/cluster_features and Malawi/food_acquired. Unlike get_dataframe, whose edit forced v0.10.0's corpus-wide invalidation, data_access.py is not in the build fingerprint.

The WB API key now suffices on its own

permissions() validated the key and then called _auto_unlock_s3(), whose own comment stated the intent exactly -- "a valid WB key proves ToU acceptance, which is what the GPG passphrase was gating" -- while the function still performed a GPG symmetric decryption. Both branches of _gpg_decrypt need the gpg binary: python-gnupg merely wraps it. So the key authorised an unlock it had no way to perform.

What the user saw was NoCredentialsError from botocore, three layers below the actual problem, pointing at their AWS configuration and their World Bank account -- neither of which was wrong. The library imported, listed countries, and returned Country('GhanaLSS').waves correctly; it failed only at the first byte of real data.

s3_reader_creds.enc now ships alongside the .gpg: the same credentials under Fernet, decrypted in pure Python. cryptography becomes an explicit dependency -- it was already present transitively via dvc[s3], but a credential path should not rest on another package's extras continuing to ship it. The .gpg file and its code path are kept, not deprecated: they still serve wheels built before this release, and cost nothing to retain.

The interactive keyless authenticate() path is converted too. That is beyond what the report asked for -- it is the same one-line key derivation, and it strands precisely the users who have neither an API key nor gpg.

Failure is now legible

When both decryption paths fail the library warns, naming which file it tried, what each one needs, and the issue:

LSMS_Library: your World Bank API key validated, but unlocking the S3 read cache failed.
  Falling back to direct World Bank downloads -- slower, but functional; nothing is blocked.
  Tried: s3_reader_creds.enc (MISSING, needs the `cryptography` package) and
         s3_reader_creds.gpg (present, needs the `gpg` binary).
  If `cryptography` is missing, `pip install cryptography` restores the fast path.
  See https://github.com/ligon/LSMS_Library/issues/741

It is a warnings.warn, not a logger.warning: the people this strands are new users who have not configured logging. It does not raise -- the World Bank NADA download path still works, so this is a degraded mode rather than a failure.

The security model is unchanged

Worth being exact, because a new encrypted file invites the assumption that something was hardened. Nothing was. The Fernet key is derived from the same passphrase the GPG file always used, base64-obfuscated in source: cosmetic anti-grep, never a gate. The World Bank API key check remains the authoritative policy, and these are read-only credentials for a cache of data the World Bank distributes. The obfuscation was deliberately not "improved" -- making it look stronger than it is would be worse than leaving it plainly cosmetic.

Verified against the built wheel

The tests run against the repository tree, but this issue is about what reaches a pip user, so the fix was checked against the artifact:

  • s3_reader_creds.enc is in the wheel (11,652 entries). It needs both a git add -f -- .gitignore carries **/.dvc/ -- and an entry in [tool.poetry] include; missing either would leave the fix working in the repository and reaching nobody.
  • The report's own reproduction, run against the wheel's extracted files with git on PATH and gpg absent: unlock succeeds, credentials written, no warnings.
  • With the .enc removed as well, the degraded path produces the message above.

Not verified: a real pip install into a clean virtualenv on Windows or macOS. Full record in slurm_logs/gh741/VERIFICATION.org.

tests/test_s3_unlock.py needs no data, no API key and no network, so unlike most of this suite it runs in CI -- which is the point, since CI would otherwise never exercise a credential path at all. It includes a rotation guard asserting that .enc and .gpg carry identical plaintext: two encrypted copies of one secret drift the moment someone rotates the credentials and regenerates only one, and nothing else would catch it. scripts/reencrypt_s3_creds.py is the .enc's generator.

A near-miss worth recording

The version comes from the git tag via poetry-dynamic-versioning. To build, the plugin rewrites pyproject.toml into a statically-versioned form and restores it afterwards. A build killed partway never restores -- and one was, on a 4-core Slurm slice.

It makes three changes, and all three must be reverted together:

dynamic = ["version"]  ->  dynamic = []
version = "0.0.0"      ->  version = "<computed>"   (moved into [project])
enable = true          ->  enable = false           ([tool.poetry-dynamic-versioning])

Only the version string was reverted at first, and the guard added alongside it checked only that string -- so it passed on a still-broken file. The tag was cut and publish.yml failed with "Building 0.0.0 from v0.10.1", because enable = false meant the plugin never ran. Nothing reached PyPI: the build job failed and the publish job was skipped.

tests/test_version_placeholder.py now asserts all three invariants, and is pinned against the actual broken file. The two guards cover different directions and both are needed: publish.yml catches a disabled plugin, but only in CI after a tag exists; the test catches the rewritten file before the commit, including the case publish.yml structurally cannot -- a stale but plausible static version, which passes a 0.0.0 check and would publish a mis-versioned artifact to PyPI, where a version can never be reused.

Upgrading

Nothing to do. If you previously installed gpg solely to make this library work, you no longer need it -- though the .gpg path still functions if you have it.