Skip to content

Fetching genomes from AGC archives

AllTheBacteria assemblies are distributed as AGC (Assembled Genome Compressor) archives — each .agc batch holds many genomes with cross-genome delta compression, so a ~1 MB archive can expand to hundreds of MB of FASTA. The atb agc command group works with these archives:

  • atb agc download — the subcommand most people want. It finds the right archive(s) for you, downloads them (cache-first, MD5-verified), and extracts FASTA. Works by accession or by species.
  • atb agc ls / info / get — low-level tools for .agc files you already have on disk (list, inspect, extract).
  • atb agc index — build a searchable index of the OSF-hosted archives (what --species searches).
  • atb agc install — fetch the upstream agc binary the others shell out to.

Availability

The atb agc command group is new in v0.18.0; it is not in v0.17.x. The 0.18.0 line is currently in beta while the by-species workflow settles, so install it from the v0.18.0-beta.2 pre-release (newer betas appear on the releases page). The installer's default stays on the latest stable until 0.18.0 is released.

Which command do I want?

Reach for atb agc download to get genomes by sample accession or by species — it handles resolution, download, and extraction. Drop down to atb agc ls/info/get only when you already have a local .agc file, or to atb agc index when you want to (re)generate the by-species index. All shell out to the upstream agc binary, so install it first.

By-species fetch is a preview

The --species path resolves batches from the AllTheBacteria staging OSF node and is under active development. Archive locations and the index may change.

Install the agc binary

atb calls the upstream agc binary; install it once (Linux/macOS x64+arm64, Windows x64):

atb agc install

It is installed alongside the atb binary. Re-running is a no-op if agc is already installed (found next to atb or on PATH).

Fetch genomes by species

Download every batch of a species and extract it whole — no accession list needed. Batches are matched by the archive naming convention <Species>_global_ordered_*.

# Every Acinetobacter baylyi batch, combined into one FASTA
atb agc download --species "Acinetobacter baylyi" --combine -o baylyi.fa

# Same, gzipped, using 8 agc threads
atb agc download --species "Mycoplasmoides pneumoniae" \
    --combine --gzip 6 -t 8 -o mpneumoniae.fa.gz

# Preview which batches would be downloaded (no download, no extract)
atb agc download --species "Salmonella enterica" --dry-run

The first by-species run builds the batch index (atb_agc_files.tsv) and caches it for 7 days. By default atb crawls the full ATB v202505 collection across its OSF nodes and combines their listings into one index. Pass --osf-node <id> to target a single node instead, which uses that node's pre-built published index (falling back to a live crawl when the published file is unavailable). The cache records which source produced it, so a release that repoints the index refreshes your local copy automatically. Use --refresh to force a re-fetch of the index and re-download archives.

Offline or pinned index

Pass a local index TSV with --agc-index to skip the network fetch entirely - useful for reproducible runs or air-gapped environments. Generate the file once with atb agc index:

atb agc download --species "Acinetobacter baylyi" \
    --agc-index atb_agc_files.tsv --combine -o baylyi.fa

Fetch genomes by accession

The default mode. Accessions resolve to AGC archives through a cached sample→archive map; those archives are then found in the same OSF collection index that --species uses - so --agc-index and --osf-node apply here too - then downloaded and each sample extracted by name. Accessions come from arguments, a --from file, or stdin. A sample whose batch is named but not yet published is reported as not yet available (the collection is still being uploaded).

# One sample to the default per-sample output directory
atb agc download SAMD00000344

# Several samples, each written to ./out/<accession>.fa
atb agc download SAMD00000344 SAMD00000345 -o ./out

# Pipe a query straight into retrieval
atb query --species "Escherichia coli" --hq-only --limit 5 --format tsv | \
    atb agc download --from - -o ./ecoli

# Combine many accessions into one gzipped FASTA
atb agc download --from accessions.txt --combine --gzip 6 -o all.fa.gz

# Preview which archives would be downloaded
atb agc download --from accessions.txt --dry-run

--from accepts a query result with a sample_accession column, or a plain list of one accession per line (- for stdin). By default each sample is written to <output-dir>/<accession>.fa; --combine streams everything to a single file (or stdout when -o is omitted).

Useful flags (both modes):

Flag Effect
--combine One output stream/file instead of per-sample files
-o, --output-dir Output directory (per-sample) or file (--combine); stdout if omitted
--gzip N gzip the output at level N (0 = uncompressed)
--line-length N FASTA line wrap width (default: agc's 80)
-t, --threads N agc extraction threads (default: cores − 1)
-p, --parallel N Parallel archive downloads
--archive-dir DIR Where to cache .agc archives (default <data-dir>/agc)
--agc-index FILE Use a local batch index TSV instead of crawling the collection
--osf-node ID Resolve batches against one OSF node's index, not the full collection
--refresh Re-download the index/map and archives even if cached
--dry-run Resolve and list archives without downloading or extracting
--keep-going Continue past unresolved or failed samples (on by default); still exits non-zero if any

Locate a sample's batch (preview)

atb agc locate answers "which AGC batch holds my sample, and is it available yet?" without downloading anything. It is the search half of atb agc download: the same accession→batch map and batch index, printed instead of fetched. No agc binary is required.

# One accession
atb agc locate SAMEA2247573

# A whole query result, as JSON for a pipeline
atb query --species "Escherichia coli" --limit 5 --format tsv | \
  atb agc locate --from - --format json

The TSV output has three columns - accession, batch, part - where part is the collection tier (major, unknown, or dustbin). JSON output adds the resolved OSF url. An accession absent from the map prints <unresolved>; a sample whose batch is named but not yet published prints <not-yet-available>.

Preview

The full v202505 collection (2.76M genomes across three OSF nodes) is still being published. A <not-yet-available> result means the batch exists in the map but its node has not finished uploading - try again later.

Build the by-species index

atb agc index crawls an OSF node's agc_batches/ folder and writes a separate index TSV — one row per .agc batch with its species, OSF download URL, MD5, and size. This is the file atb agc download --species searches. Generate it once and commit it for offline use, or let atb agc download crawl and cache it on demand.

# Write the index to a file you can commit / pass back via --agc-index
atb agc index -o atb_agc_files.tsv

# Print it to stdout
atb agc index

# Crawl a specific OSF node
atb agc index --osf-node z7q5y -o atb_agc_files.tsv

The index is a 6-column TSV (project, project_id, filename, url, md5, size_mb) — the same layout as the master OSF index, so the standard parser round-trips it.

Inspect and extract local archives

When you already have a .agc file on disk, use the low-level atb agc subcommands directly — no download, no index.

# List the sample names in an archive
atb agc ls genomes.agc

# List the contig names within one sample
atb agc ls genomes.agc SAMD00000344

# Show archive metadata (sample count, reference, etc.)
atb agc info genomes.agc

Extract sequences as FASTA with atb agc get. Three mutually exclusive selections: contig queries (positional), whole samples (--sample), or the entire collection (--all):

# One contig region to stdout: contig[@sample][:from-to]
atb agc get genomes.agc "contig_1@SAMD00000344:1000-2000"

# Whole samples to a file
atb agc get genomes.agc --sample SAMD00000344 --sample SAMD00000345 -o out.fa

# Entire collection, gzip level 6, 8 threads
atb agc get genomes.agc --all --gzip 6 -t 8 -o all.fa.gz

get also accepts -l/--line-length to set the FASTA wrap width and -s/--streaming for lower-memory (slower) extraction.