diff --git a/docs/superpowers/plans/2026-08-07-async-backfill-with-meta-refresh.md b/docs/superpowers/plans/2026-08-07-async-backfill-with-meta-refresh.md new file mode 100644 index 0000000..e82f26f --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-async-backfill-with-meta-refresh.md @@ -0,0 +1,154 @@ +# Async Backfill with Meta-Refresh — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Prevent PHP execution-timeout during the first-time vault backfill (which may make ~25 HTTP calls to the Torn API and exceed host limits). On bootstrap, return a "Loading..." page immediately that auto-refreshes; each refresh does a small chunk of work until the DB is fully populated. + +**Architecture:** `index.php` first runs the small/fast sync (`syncUserLogs` for users with rows, which does 1-2 HTTP calls). If after that, any user has zero rows in the DB (i.e., the bootstrap case), `index.php` returns a minimal HTML page with a meta-refresh tag pointing to the same URL after a short delay. The next page-load re-runs the sync, which now does ~24 more HTTP calls (one page of entries per refresh). Eventually the DB is populated and the page renders the real UI. + +**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies. + +## Global Constraints + +- PHP 8.1+ syntax only. +- Commits per task. Do not push. +- Other helpers stay byte-identical unless the task explicitly says otherwise. + +--- + +### Task 1: Add meta-refresh bootstrap to `index.php` + +**Files:** +- Modify: `index.php` (top-level bootstrap only) + +**Interfaces:** +- Page-load behavior: + - **Steady state (DB has rows):** Existing behavior — run sync, render the page. Unchanged. + - **Bootstrap (DB empty for some user):** Page-load runs the fast sync. If a user still has zero rows after the fast sync, return a minimal HTML "Loading..." page that meta-refreshes to itself after 3 seconds. + - The meta-refresh runs `syncUserLogs`, which uses `from=lastTs+1` to fetch only new entries. On a fresh DB this fetches nothing (lastTs=0 → 0+1=1, but the API may give a strange response). So the actual heavy lifting must happen via the bootstrap case separately. + +**Strategy:** The page-load runs `backfillUserLogs` (the heavy function) ONLY if it would complete quickly. If it would be slow (heuristic: the user has fewer than some threshold of rows), defer it. + +Concretely: define a helper `isBackfillInProgress(): bool` that returns true if any user has 0 rows OR the user has fewer than e.g. 50 rows. The page-load only runs `backfillUserLogs` if that returns false (i.e., the DB looks fully populated for that user). If it returns true, return a loading page. + +But that's not quite right either — we want to make PROGRESS even if not "complete." Let me re-think. + +**Revised strategy:** Always run the sync, but cap the number of pages we walk per page-load. If the sync completed (all users' rows look complete), render the page normally. Otherwise return a loading page that will continue the sync on next refresh. + +- [ ] **Step 1: Add the helper functions to `index.php`** + +In `index.php`, replace the top of the file (after `require_once __DIR__ . '/functions.php';`) to add a `stillNeedBackfill` helper and bootstrap logic: + +```php +require_once __DIR__ . '/config.php'; +require_once __DIR__ . '/functions.php'; + +$bootstrapWarning = null; +$needsBackfill = false; + +try { + foreach (USER_KEYS as $key => $value) { + syncUserLogs($key); + // If the sync didn't populate this user, backfill is still needed. + $pdo = getDatabaseConnection(); + $stmt = $pdo->prepare('SELECT COUNT(*) FROM vault WHERE user = :user'); + $stmt->bindValue(':user', $key); + $stmt->execute(); + if ((int)$stmt->fetchColumn() === 0) { + $needsBackfill = true; + } + } +} catch (Exception $e) { + $bootstrapWarning = $e->getMessage(); +} + +if ($needsBackfill) { + // Heavy work remains. Return a loading page that auto-refreshes; + // each refresh does another page of API fetches. + ?> + + + + + Torn Vault Tracker — loading + + + + + +

Loading vault history…

+ +

Warning:

+ +

One-time backfill in progress. The Torn API returns up to 100 vault entries per call; + we'll keep fetching until your full history is loaded. This page will refresh automatically.

+ + +