Newsletter-Link-Catalog/docs/superpowers/plans/2026-05-17-sqlite-web-app.md

SQLite Web App Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make SQLite the default catalog store and add a functional local web app for browsing imported newsletter data.

Architecture: Use Node's built-in node:sqlite DatabaseSync API for a packaged-executable-friendly SQLite layer. Keep the existing Gmail ingestion pipeline and spreadsheet writers, add a DatabaseWriter as the default writer, and add an Express server that renders read-only catalog pages from SQLite.

Tech Stack: TypeScript, Node.js node:sqlite, Express, Commander, Vitest, existing parser/output modules.

---

Task 1: Config Defaults

Files:

• Modify: src/config/config.ts

• Modify: config.example.yaml

• Test: tests/config.test.ts

• Write a failing test that database.enabled defaults to true and database.path defaults to ./data/newsletter-catalog.sqlite.

• Add the database schema to config validation.

• Add database to config.example.yaml.

• Run npm test -- tests/config.test.ts.

Task 2: SQLite Schema and Store

Files:

• Create: src/database/schema.ts

• Create: src/database/store.ts

• Test: tests/database.test.ts

• Write failing tests that migrations create newsletters, issues, links, link_occurrences, sponsors, dead_links, and runs.

• Write failing tests that saving a catalog payload deduplicates canonical URLs and stores occurrences per issue.

• Implement migration and transactional persistence with node:sqlite.

• Run npm test -- tests/database.test.ts.

Task 3: Default Database Writer

Files:

• Create: src/output/databaseWriter.ts

• Modify: src/run/runCatalog.ts

• Modify: src/cli/program.ts

• Test: tests/run.test.ts

• Write a failing test that runCatalog can write rows to a database writer and still preserves spreadsheet writer behavior.

• Add a DatabaseWriter implementing the existing OutputWriter interface.

• Make nlc run add DatabaseWriter by default when database.enabled is true.

• Run npm test -- tests/run.test.ts tests/database.test.ts.

Task 4: Web App

Files:

• Create: src/web/app.ts

• Create: src/web/views.ts

• Modify: src/cli/program.ts

• Test: tests/web.test.ts

• Write failing tests that dashboard, links, sponsored links, dead links, and runs routes render data from a fixture database.

• Add nlc serve --config --host --port.

• Implement read-only Express routes and simple server-rendered HTML.

• Run npm test -- tests/web.test.ts.

Task 5: Docs and Verification

Files:

• Modify: README.md

• Modify: package.json

• Document SQLite default storage and nlc serve.

• Add Express dependencies and update lockfile.

• Run npm install.

• Run npm run lint.

• Run npm run format:check.

• Run npm run typecheck.

• Run npm test.

• Run npm run build.

• Run npm run smoke.

Self-Review

This plan covers all accepted spec requirements: SQLite defaults, schema, default import persistence, optional spreadsheet output, nlc serve, functional catalog views, docs, and automated tests. No unresolved placeholders remain. Types and paths align with the existing project structure.