[gerontec]

Follow-up to discussion #990.

Rebuilding a large database from scratch to add one day of new sequences is expensive: for NCBI nr a full makedb run takes hours, while a daily delta is a few hundred thousand sequences. This PR adds makedb --append and one prerequisite change.

Commits

1. makedb: write the sequence offset table to disk while building

The offset table (16 bytes per sequence) was collected in a std::vector and written only at the end of the build. For nr (1.15 billion sequences) the vector reallocated at 2^30 entries and needed about 51 GB at that moment, so the build was OOM-killed on a 64 GB machine. The table is append-only during the build and read back once, sequentially, so it is now buffered (2^20 entries) and streamed to <database>.pos_tmp, then appended to the database. File format and entry order are unchanged.

2. makedb: add --append to add sequences to an existing database

--append reads the header of the existing database, truncates the file at pos_array_offset and continues writing sequences there. When the trailer is written, the old offset table is copied back first and the new entries follow; the same is done for the taxon id lists, and TaxonList::build() is called for the newly added sequences only.

Everything from pos_array_offset to EOF is copied to <database>.append_backup before truncating. If the append fails at any point, or if a backup from an earlier interrupted run exists, the database is restored to its previous state.

The taxonomy options must match the original build: --taxonmap if and only if the database contains taxon id lists, likewise --taxonnodes / --taxonnames. Protein databases only.

Verification

Rebased onto current master (ef05efd), built with -DCMAKE_BUILD_TYPE=Release.

• 2.5 M sequence subset of nr: makedb over all sequences vs. makedb over the first 2.0 M + makedb --append with the remaining 0.5 M → byte-identical .dmnd, including the database hash.
• 71,699 sequences, split 57,359 + 14,340, on the rebased branch:
  • without taxonomy: byte-identical (md5 1e0eb07f… for both)
  • with --taxonmap/--taxonnodes/--taxonnames: byte-identical (md5 f66d595c… for both)
  • --append without --taxonmap on a database that has taxon lists: rejected with an error, database unchanged, no leftover files.
• Commit 1 alone: .dmnd byte-identical to the unpatched makedb.
• In production since 23.09.2026 on a full nr database (1.156 billion sequences, 489 GB): daily GenBank/RefSeq deltas appended in ~2 min instead of a multi-hour rebuild.

🤖 Generated with Claude Code

[gerontec]

see flowchart to better understand how it works: https://github.com/gerontec/blastDiamond/blob/main/doc/append_workflow.pdf