View indexer advances `_local/lastSeq` past failed batch writes — permanent silent view index corruption

[bebor]

Issue

The map/reduce view indexer (pouchdb-abstract-mapreduce) can permanently and
silently drop documents from a view index. If a batch's row write to the
mrview database fails while a later batch's write succeeds, the later batch
stamps _local/lastSeq past the failed batch. The failed batch's documents
then have no view rows, no _local/doc_<id> tracking record, and a checkpoint
that says they were processed — nothing ever revisits them. db.query()
reports success and simply returns fewer rows than it should, indefinitely,
across restarts.

Any transient write failure suffices: fd exhaustion, disk pressure / quota, a
process kill mid-index. We root-caused a production incident (medical records
system, LevelDB adapter) to this: 219 documents invisible in the application
across 134 patient records, with _local/lastSeq sitting above their update
seqs.

Four stacked causes in updateViewInQueue / saveKeyValues:

1. Write is queued, never awaited — queue.add(processChange(...)) is
   fire-and-forget; processBatch fetches the next changes batch while the
   previous batch's write is still pending (and possibly failing). The
   rejection surfaces as an unhandledRejection, not as a query error.
2. The read pointer advances in memory regardless — currentSeq = change.seq per change, so the next batch reads since: currentSeq whether
   or not anything was persisted.
3. Failed writes are discarded — TaskQueue.add chains
   this.promise.catch(function () { /* just recover */ }), so a rejected
   batch write is swallowed and the next queued write runs anyway, stamping
   its own (higher) lastSeq.
4. bulkDocs per-row results are unchecked — rows and _local/lastSeq
   go into one bulkDocs whose response is ignored, so per-document failures
   (e.g. 409s — which do occur in practice, cf. db.compact() runs much slower than auto_compaction #8525 with
   auto_compaction) are invisible too.

Info

• Environment: Node.js (also affects browsers — the code is shared)
• Adapter: leveldb (adapter-independent; the defect is in abstract-mapreduce)
• Version: reproduced on pouchdb@9.0.0; present in master and at least back
  through 7.3.1

Reproduce

Standalone script (attached / below): 30 docs, view batch size 10, reject the
second batch's mrview bulkDocs once (simulating a transient failure), then
reopen the database with storage healthy again.

Observed output on pouchdb@9.0.0:

unhandledRejection (fire-and-forget write): simulated transient write failure
1st query error surfaced: NO (swallowed)
1st query rows: 20 of 30
after reopen rows: 20 of 30 (PERMANENT SILENT HOLE)

Expected: either the query rejects (view stays stale and recovers on the next
query), or it succeeds with all 30 rows. A view must never be holed —
missing rows below its checkpoint.

// PouchDB 9.0.0 — view indexer advances checkpoint past failed writes
// One transient batch-write failure => permanent silent view holes.
const fs = require('fs')
const os = require('os')
const path = require('path')
const PouchDB = require('pouchdb')

const BATCH = 10
const DOCS = 30
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'pouch-ckpt-repro-'))
const dbPath = path.join(dir, 'db')

let mrviewWrites = 0
let failAtWrite = 0 // 0 = disarmed

// the fire-and-forget write escapes as unhandledRejection
process.on('unhandledRejection', (e) => {
  console.log('unhandledRejection (fire-and-forget write):', e.message)
})

// mrview db is instance-bound; hook its creation
function hook (db) {
  const realReg = db.registerDependentDatabase.bind(db)
  db.registerDependentDatabase = (...args) =>
    realReg(...args).then((res) => {
      const real = res.db.bulkDocs.bind(res.db)
      res.db.bulkDocs = (...a) => {
        const docs = a[0] && a[0].docs
        // count only checkpoint-stamping batch writes
        if (Array.isArray(docs) && docs.some((d) => d._id === '_local/lastSeq')) {
          mrviewWrites++
          if (failAtWrite && mrviewWrites === failAtWrite) {
            // simulates fd exhaustion / disk pressure / crash
            return Promise.reject(new Error('simulated transient write failure'))
          }
        }
        return real(...a)
      }
      return res
    })
  return db
}

async function main () {
  let db = hook(new PouchDB(dbPath, { view_update_changes_batch_size: BATCH }))
  await db.put({
    _id: '_design/test',
    views: { byX: { map: 'function (doc) { emit(doc.x) }' } },
  })
  const docs = []
  for (let i = 0; i < DOCS; i++) {
    docs.push({ _id: 'doc-' + String(i).padStart(4, '0'), x: i })
  }
  await db.bulkDocs(docs)

  failAtWrite = 2 // second batch write fails
  let err = null
  const r1 = await db.query('test/byX').catch((e) => { err = e })
  console.log('1st query error surfaced:', err ? err.message : 'NO (swallowed)')
  if (r1) console.log('1st query rows:', r1.rows.length, 'of', DOCS)

  failAtWrite = 0 // storage healthy again
  await db.close() // = process restart
  db = new PouchDB(dbPath, { view_update_changes_batch_size: BATCH })
  const r2 = await db.query('test/byX')
  console.log('after reopen rows:', r2.rows.length, 'of', DOCS,
    r2.rows.length === DOCS ? '(OK)' : '(PERMANENT SILENT HOLE)')
  await db.close()
  fs.rmSync(dir, { recursive: true, force: true })
}

main().catch((e) => { console.error('FATAL', e); process.exit(1) })

Suggested fix

We patched our vendored copies (8.0.1 inline + abstract-mapreduce 7.3.1) as
follows, validated by regression tests; happy to turn this into a PR:

1. Await each batch's write before fetching the next batch (replace the
   fire-and-forget queue.add(processChange(...)) with awaiting
   processChange(...)()), so a failure aborts the run with _local/lastSeq
   still at the last persisted batch — the view goes stale, not holed.
2. Propagate the failure to the query caller (in v8+/master,
   updateViewInQueue's final catch must rethrow after
   activeTasks.remove(taskId, error)).
3. Check bulkDocs per-row results. Note: naively failing on every row error
   breaks recovery, because getDocsToPersist's isGenOne shortcut writes
   rev-less kv/meta docs when re-indexing an already-indexed generation-1 doc
   — producing 409s that today are silently swallowed by the same unchecked
   bulkDocs. We repair conflicts with a single _rev-refresh retry and fail
   the run on anything else.

Cost: indexing loses read-ahead pipelining (writes are on the critical path),
and previously silent errors become visible — which is the point.

Related: #9177 refactors these exact functions (async/await) without changing
this behavior; if that lands first, the fix becomes a few lines simpler.