Newsletter-Link-Catalog/notes/original SPEC.md

# Newsletter Link Catalog — Specification

## Overview

A CLI tool that extracts links from newsletters in a designated Gmail folder, categorizes them, enriches them with metadata, and compiles them into a spreadsheet. Each newsletter gets its own sheet, links are organized by issue date and category, and sponsor links are tracked separately.

## Architecture

### Language & Runtime
- **TypeScript/Node.js** — compiled to a standalone binary via `pkg` or `tsx-bundle`
- CLI tool invoked as `nlc run [flags]`

### Distribution
- Standalone binary — no Node runtime required on the host machine
- Built and packaged via CI or build script

### Run Modes
- **Manual**: Run `nlc run` on demand with optional date filters
- **Scheduled**: Can be run via cron/Task Scheduler for recurring processing
- Designed for both; no daemon mode required

## Gmail Integration

### Authentication
- **OAuth2 browser flow** — user authorizes via browser, tokens persisted locally
- `nlc init` command walks through OAuth setup interactively

### Scope
- Processes emails from a **single designated Gmail folder/label** (configured in `config.yaml`)
- Does not scan the entire inbox or search by sender patterns

### Email Processing
- **HTML only** — plain-text parts are ignored
- **Image-only emails** (single image, no extractable links) are skipped with a warning logged
- **"View in browser" emails** — fetches the web version's HTML and extracts links from that instead
- Incremental by default: tracks processed Message-IDs in a local state file, only processes new emails
- `--full` flag forces reprocessing of all emails

## Link Extraction & Processing

### Extraction Pipeline
1. Fetch emails from the configured Gmail folder (incremental or full)
2. Parse HTML to extract links, section headers, and surrounding text
3. Filter out noise links: unsubscribe, social footer icons, "share this newsletter" links
4. Unwrap tracking redirects and strip UTM parameters — store only the clean canonical URL
5. Merge "Read more" links with their preceding content (detected by: same URL + "read more" anchor text)
6. Categorize each link (see Categorization section)
7. Write to spreadsheet (see Output section)

### Noise Filtering
The following link types are **excluded** from content sheets:
- Unsubscribe links
- Social media footer links (Twitter, LinkedIn, etc.)
- "Share this newsletter" / "Forward to a friend" links
- "View in browser" mirror links (content is extracted from the web version instead)

Sponsor/ad links are **not filtered** — they go to a separate sheet.

### URL Handling
- Unwrap all tracking redirects (Mailchimp, Substack, etc.)
- Strip UTM parameters and other tracking query params
- Store only the clean canonical URL
- Dead/broken links (4xx/5xx during enrichment) are moved to a separate "Dead Links" sheet

### "Read More" Merging
When two consecutive elements point to the same URL and one has "read more" (or similar) anchor text, they are merged into a single entry combining the preceding description text and the link.

## Categorization

### Strategy: Hybrid
1. **Primary**: Use the newsletter's own section headers (e.g., "Python", "DevOps", "Career") as categories
2. **Fallback**: When section headers aren't available or don't cover a link, use rule-based classification (URL patterns + keywords)
3. **Final fallback**: LLM-based categorization when rules don't match

### Category Taxonomy
- **LLM-generated** by default — the model assigns categories based on link content
- Built-in base taxonomy shipped with the tool for common dev categories (Python, JavaScript, DevOps, Security, etc.)
- User can extend via config with custom categories
- LLM is instructed to prefer existing categories and only create new ones when nothing fits

### LLM Provider Support (BYOK)
All providers supported, configurable in `config.yaml`:
- **Claude/Anthropic** — Anthropic API
- **OpenAI/GPT** — OpenAI API
- **Local models** — Ollama, LM Studio
- **OpenAI-compatible endpoints** — Mistral, Groq, Together, etc.

Provider config includes: API key, base URL, model name, and optional parameters.

### Newsletter Parsing: Plugin System
- Generic HTML parser as the default
- Platform-specific parsers loaded as plugins (detected by URL patterns or email headers)
- **Substack** shipped as the first plugin — uses Substack's predictable HTML structure for more reliable extraction
- Additional parsers can be added as plugins without modifying core logic

## Output: Spreadsheet

### Supported Formats
- **Google Sheets** — via Google Sheets API (live, shareable, auto-updated)
- **Local Excel (.xlsx)** — written to disk, can be uploaded manually

Config selects which output(s) to use; both can be active simultaneously.

### Spreadsheet Name
- Fixed name set in `config.yaml` (e.g., "Newsletter Link Catalog")

### Sheet Naming
- Each newsletter gets its own sheet named after the parsed display name from the email's From header
- Names truncated to fit Google Sheets' 100-character limit
- Special characters replaced as needed for sheet name validity

### Content Sheet Columns
Every row is fully populated (flat table — no blank cells for grouping):

| Column | Description |
|---|---|
| Issue Date | Date from email's Date header (overridable per-newsletter) |
| Category | Assigned category (from newsletter sections, rules, or LLM) |
| Link URL | Clean canonical URL after unwrapping and UTM removal |
| Title | Anchor text / headline from the newsletter |
| Description | 1-2 sentence description from the newsletter (if present) |
| Page Title + Meta | `<title>` and meta description from the destination page (enrichment phase) |
| Source Newsletter | Name of the newsletter this link came from |
| Also In | Cross-reference: other newsletters that also mentioned this link |

### Sponsor Sheet (Consolidated)
Single sheet named "Sponsored Links" containing sponsor/ad links from all newsletters:

| Column | Description |
|---|
| Newsletter | Which newsletter this sponsor link appeared in |
| Sponsor | Sponsor name (parsed from newsletter) |
| Link | Sponsor's link URL |
| Description | Sponsor description from the newsletter |

### Dead Links Sheet
Single sheet named "Dead Links" for links that returned errors during enrichment:

| Column | Description |
|---|
| URL | The clean canonical URL |
| Status | HTTP status or error type (404, 403, timeout, etc.) |
| Source | Newsletter name |
| Date | Issue date |

### Cross-References
- Duplicates across newsletters are kept in their respective sheets (all occurrences preserved)
- The **Also In** column annotates each row with which other newsletters mentioned the same link and when (e.g., "TLDR Web Dev (Mar 5)")
- This enables finding cross-newsletter coverage without a separate consolidated sheet

### No "All Links" Master Sheet
Only per-newsletter content sheets, plus the consolidated Sponsor and Dead Links sheets. No "All Links" aggregation sheet.

## Enrichment

### Two-Phase Approach
1. **Phase 1 (Store)**: Extract links from newsletters, categorize, and write to spreadsheet with all available in-newsletter metadata
2. **Phase 2 (Enrich)**: Separate pass to fetch each link's destination page for `<title>` and meta description

This keeps the initial run fast and allows enrichment to be run independently.

### Enrichment Details
- Configurable concurrency (safe defaults: 3-5 parallel, 1-2s delay between batches)
- Retries on transient failures
- Dead links (4xx/5xx) moved to Dead Links sheet
- Skip paywalled/auth-required pages (detected by login redirects) — mark with "paywall" status
- Progress bar shows enrichment status in real-time

### Link Liveness
- Dead links are **not included** in content sheets — they go to the Dead Links sheet
- Paywalled/unreachable links are included in content sheets but flagged in the Page Title + Meta column

## Processing Model

### Incremental Processing
- Local state file (JSON) tracks processed Message-IDs and enrichment status
- On subsequent runs, only new/unprocessed emails are fetched
- `--full` flag forces reprocessing of all emails
- State file location: `~/.nlc/state.json` (or configured path)

### Date Filtering
- `--from YYYY-MM-DD` and `--to YYYY-MM-DD` — absolute date range
- `--last N` (e.g., `--last 30d`, `--last 7d`) — relative date range
- Can be combined with incremental processing

### Dry Run
- `--dry-run` processes the most recent X emails (default: 5) without writing to the spreadsheet
- Shows what would be extracted, categorized, and written
- Useful for testing config changes and parser tweaks

### Error Handling
- **Critical errors** (Gmail auth failure, spreadsheet write failure, config errors) → stop execution
- **Individual errors** (one link fails to enrich, one email fails to parse) → log and continue
- Summary at end includes error counts and details

### Progress & Logging
- Progress bar during processing (emails fetched, links extracted, enrichment status)
- Summary stats at the end: newsletters processed, links extracted, duplicates found, dead links, sponsors, errors

## CLI Interface

### Commands

```
nlc init          # Interactive setup: OAuth, config file, connectivity test
nlc run [flags]   # Main processing command
```

### `nlc run` Flags

| Flag | Description | Default |
|---|---|---|
| `--full` | Reprocess all emails, not just new ones | false |
| `--dry-run [N]` | Process most recent N emails without writing to sheet | 5 |
| `--from YYYY-MM-DD` | Process emails from this date | (none) |
| `--to YYYY-MM-DD` | Process emails up to this date | (none) |
| `--last N` | Process emails from last N days (e.g., `--last 30d`) | (none) |
| `--skip-enrich` | Skip the enrichment phase (only extract + categorize) | false |
| `--enrich-only` | Only run enrichment on already-extracted links | false |
| `--config PATH` | Path to config file | `./config.yaml` |
| `--verbose` | Detailed per-email and per-link output | false |

## Configuration

### File Format: YAML
Location: `./config.yaml` (overridable with `--config`)

### Sample Structure

```yaml
# Gmail settings
gmail:
  folder: "Newsletters"           # Gmail label/folder to process
  credentials: "~/.nlc/gmail-credentials.json"
  token: "~/.nlc/gmail-token.json"

# Output settings
output:
  name: "Newsletter Link Catalog"  # Spreadsheet name
  sheets_api:
    enabled: true
    credentials: "~/.nlc/sheets-credentials.json"
    token: "~/.nlc/sheets-token.json"
  excel:
    enabled: true
    path: "./output/newsletter-catalog.xlsx"

# Newsletter identification
newsletters:
  # Manual overrides for parsed display names
  # sender_pattern: "display_name"
  "alex@bytebytego.com": "ByteByteGo"
  "dan@techtakesweekly.com": "Tech Takes Weekly"

# Link processing
links:
  unwrap_redirects: true
  strip_utm: true
  filter_unsubscribe: true
  filter_social_footer: true
  filter_share_links: true
  merge_read_more: true

# Categorization
categories:
  # Built-in taxonomy is used by default; extend here
  custom:
    - "AI/ML"
    - "Career"
    - "Rust"
  # LLM settings for category inference
  llm:
    provider: "anthropic"         # anthropic | openai | local | openai-compatible
    model: "claude-sonnet-4-6"
    api_key_env: "ANTHROPIC_API_KEY"  # or set in env
    base_url: null                # for local/openai-compatible
    fallback_to_rules: true       # if LLM fails, use rule-based

# Enrichment
enrichment:
  enabled: true
  concurrency: 3
  delay_ms: 1500
  retries: 2
  timeout_ms: 10000

# Rate limiting (applies to both Gmail API and enrichment)
rate_limit:
  gmail_qps: 5                    # queries per second to Gmail API
  link_concurrency: 3             # parallel link fetches

# State
state_file: "~/.nlc/state.json"

# Parsing plugins
plugins:
  substack:
    enabled: true
```

### Issue Date Override
For newsletters where the email arrival date doesn't match the issue date, overrides can be configured:

```yaml
newsletters:
  "sender@domain.com":
    display_name: "Newsletter Name"
    date_override: "subject"      # Parse date from subject line
    date_format: "%B %d, %Y"      # Expected date format in subject
```

## Data Flow

```
┌─────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Gmail API  │────▶│ Parse HTML   │────▶│  Categorize  │────▶│ Write Sheet  │
│  (fetch)    │     │ + Extract    │     │  (hybrid)    │     │ (Phase 1)    │
└─────────────┘     └──────────────┘     └──────────────┘     └──────────────┘
                           │                                          │
                           ▼                                          ▼
                    ┌──────────────┐                           ┌──────────────┐
                    │  State File  │                           │  Enrichment  │
                    │  (processed  │                           │  (Phase 2)   │
                    │   tracking)  │                           │  Page titles  │
                    └──────────────┘                           └──────────────┘
```

## Edge Cases

| Scenario | Behavior |
|---|---|
| Email is a single image with no links | Skip with warning, log to state |
| "View in browser" link instead of content | Fetch web version HTML, extract links from that |
| Same link in multiple newsletters | Keep all occurrences, cross-reference via "Also In" column |
| Same link multiple times in one issue | Deduplicate per-issue; single row per unique URL |
| Link returns 4xx/5xx during enrichment | Move to Dead Links sheet |
| Link is paywalled/auth-required | Include in content sheet, mark Page Title as "[paywall]" |
| Newsletter name > 100 chars | Truncate for sheet name |
| Sheet already exists for newsletter | Append new rows, don't overwrite existing data |
| Gmail API rate limit | Retry with exponential backoff |
| OAuth token expired | Auto-refresh, re-prompt if refresh fails |
| Newsletter format changes | Parser falls back to generic HTML extraction |

## Setup & First Run

1. **`nlc init`** — Interactive walkthrough:
   - Authenticate with Gmail (OAuth browser flow)
   - Authenticate with Google Sheets (if using Sheets output)
   - Select the Gmail folder/label to process
   - Configure output location
   - Test connectivity
   - Generate `config.yaml`

2. **`nlc run --dry-run`** — Test with 5 most recent emails

3. **`nlc run`** — Full processing run

4. **`nlc run --enrich-only`** — Enrich previously extracted links with page titles

## Future Considerations

These are **not** in scope for v1 but noted for potential future work:
- Search/filter functionality within the spreadsheet
- Web UI for browsing the catalog
- Email forwarding as an alternative to Gmail API access
- Automatic category taxonomy refinement based on accumulated data
- Plugin system for additional newsletter platforms beyond Substack
- Notification on new newsletter processing