Skip to content

AGC hosted-index default: v0.18.1 verification

Live end-to-end test of the atb agc commands against the v0.18.1 default, which downloads a single published index TSV from OSF instead of crawling the three collection nodes. Both access modes (by species, by accession) were exercised against the real hosted index and real archives.

Result

All checks passed. The runtime downloads the published index by default (proved by the cache .source marker), the index bytes match the published md5, a by-species download's archive md5 matches the index, and a by-accession download extracts exactly the named sample. One transient HTTP 429 was observed on a cold OSF fetch; it is external rate-limiting, not a v0.18.1 defect (see Notes).

Environment

Item Value
Code under test feat/agc-balanced-migration HEAD (the v0.18.1 code; AGCIndexURL flipped to the hosted TSV)
atb binary local go build (reports version dev; no goreleaser ldflags)
agc binary v3.x, installed via atb agc install
Hosted index URL https://osf.io/download/6a719381a2e9e3d202b91f7d/
Data dir isolated temp ATB_DATA_DIR (cold cache per run)

Index integrity

The default path writes <data-dir>/agc/atb_agc_files.tsv plus a .source sidecar recording which source produced it.

Check Expected Observed
Cached index md5 8aea3d79da3e2a0af10c9904d0c3a10f (published) 8aea3d79da3e2a0af10c9904d0c3a10f
Index rows 1269 (1268 batches + header) 1269
.source marker hosted URL, not the crawl marker https://osf.io/download/6a719381a2e9e3d202b91f7d/

The .source value is the definitive proof of routing: a crawl would have stamped agc-collection:4jq8u,jmeqg,kzcnr instead.

Tests

# Path Command (shape) Result
1 Index fetch default atb agc download populates the cache md5 + .source as above
2 By species, dry run atb agc download --species "Mycoplasmoides pneumoniae" --dry-run resolves to batch.0617, no download
3 By species, download atb agc download --species "Mycoplasmoides pneumoniae" --combine -o mpneu.fa 336 MB FASTA, 424 samples, 15,509 contigs
4 Archive integrity md5 of downloaded batch.0617.agc vs index 6a5d5f28330ce6fd87eb5e71189243a3 = index md5
5 Locate atb agc locate SAMN43265078 --format {tsv,json} batch.0617 / Mycoplasmoides pneumoniae; JSON url matches index
6 By accession, download atb agc download SAMN43265078 -o byacc Completed: 1 Failed: 0 Unresolved: 0; 29 contigs

By-species download (Test 3-4)

--species "Mycoplasmoides pneumoniae" matched batch.0617 (its species label in the index), downloaded atb.assembly.202505_all.batch.0617.agc, verified its md5 against the index, and extracted the whole batch:

  • Output mpneu.fa: 351,892,498 bytes (~336 MB)
  • Samples: 424 (agc listset on the archive)
  • Contigs: 15,509 (> headers)
  • Downloaded archive md5 6a5d5f28330ce6fd87eb5e71189243a3 equals the md5 column for that batch in the index.

By-accession download (Test 6)

atb agc download SAMN43265078 resolved the accession to batch.0617 via the cached sample-to-archive map, downloaded that batch, and extracted only the named sample:

  • Output SAMN43265078.fa: 837,304 bytes, 29 contigs
  • Every header is >SAMN43265078.contigNNNNN, i.e. only that sample
  • Its first record (contig00001 len=221773) is byte-identical to the first record of the whole-batch mpneu.fa, confirming the by-accession path pulls the same sample out of the same batch that by-species extracts whole.

A two-accession dry run spanning two batches resolved correctly:

atb agc download SAMN43265078 SAMN36072413 --dry-run
(dry-run) 2 archive(s) for 2 sample(s): batch.0617.agc (1), batch.0008.agc (1)

Notes

  • HTTP 429 on cold OSF fetches. A cold atb agc locate run surfaced fetch archive map: HTTP 429 (set agc.archive_map_url to override). OSF rate-limits the /download/ redirect endpoint per IP; back-to-back cold runs from one host trip it. It is external and transient, not a v0.18.1 change (the archive-map URL was untouched by this release), and the cache-first design makes zero OSF calls in steady state. Retrying after a short pause cleared it.
  • --agc-index still reads a local TSV and bypasses the network entirely, which is the reproducible/offline path used for the dry-run checks.

Documentation check

  • docs/reference/cli/atb_agc_download.md (generated by make docs): in sync - already describes the published index as the default with crawl as the fallback.
  • docs/guides/agc.md (hand-written): was stale, still framing the crawl as the default. Updated to match: the published index TSV is the default; crawling the three collection nodes is the fallback when no published index is configured.