Files

T

Keith Solomon a9a635e172 Add plan: async backfill with meta-refresh loading page

- 2 tasks: index.php loading-page bootstrap, smoke check
- Heavy sync deferred to meta-refresh loops so PHP never times out
- Each refresh walks one API page; ON CONFLICT DO UPDATE makes
- resume safe after browser close

2026-08-10 20:01:14 -05:00

6.6 KiB

Raw Blame History

Async Backfill with Meta-Refresh — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Prevent PHP execution-timeout during the first-time vault backfill (which may make ~25 HTTP calls to the Torn API and exceed host limits). On bootstrap, return a "Loading..." page immediately that auto-refreshes; each refresh does a small chunk of work until the DB is fully populated.

Architecture: index.php first runs the small/fast sync (syncUserLogs for users with rows, which does 1-2 HTTP calls). If after that, any user has zero rows in the DB (i.e., the bootstrap case), index.php returns a minimal HTML page with a meta-refresh tag pointing to the same URL after a short delay. The next page-load re-runs the sync, which now does ~24 more HTTP calls (one page of entries per refresh). Eventually the DB is populated and the page renders the real UI.

Tech Stack: PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies.

Global Constraints

PHP 8.1+ syntax only.
Commits per task. Do not push.
Other helpers stay byte-identical unless the task explicitly says otherwise.

Task 1: Add meta-refresh bootstrap to `index.php`

Files:

Modify: index.php (top-level bootstrap only)

Interfaces:

Page-load behavior:
- Steady state (DB has rows): Existing behavior — run sync, render the page. Unchanged.
- Bootstrap (DB empty for some user): Page-load runs the fast sync. If a user still has zero rows after the fast sync, return a minimal HTML "Loading..." page that meta-refreshes to itself after 3 seconds.
- The meta-refresh runs syncUserLogs, which uses from=lastTs+1 to fetch only new entries. On a fresh DB this fetches nothing (lastTs=0 → 0+1=1, but the API may give a strange response). So the actual heavy lifting must happen via the bootstrap case separately.

Strategy: The page-load runs backfillUserLogs (the heavy function) ONLY if it would complete quickly. If it would be slow (heuristic: the user has fewer than some threshold of rows), defer it.

Concretely: define a helper isBackfillInProgress(): bool that returns true if any user has 0 rows OR the user has fewer than e.g. 50 rows. The page-load only runs backfillUserLogs if that returns false (i.e., the DB looks fully populated for that user). If it returns true, return a loading page.

But that's not quite right either — we want to make PROGRESS even if not "complete." Let me re-think.

Revised strategy: Always run the sync, but cap the number of pages we walk per page-load. If the sync completed (all users' rows look complete), render the page normally. Otherwise return a loading page that will continue the sync on next refresh.

Step 1: Add the helper functions to index.php

In index.php, replace the top of the file (after require_once __DIR__ . '/functions.php';) to add a stillNeedBackfill helper and bootstrap logic:

require_once __DIR__ . '/config.php';
require_once __DIR__ . '/functions.php';

$bootstrapWarning = null;
$needsBackfill    = false;

try {
    foreach (USER_KEYS as $key => $value) {
        syncUserLogs($key);
        // If the sync didn't populate this user, backfill is still needed.
        $pdo = getDatabaseConnection();
        $stmt = $pdo->prepare('SELECT COUNT(*) FROM vault WHERE user = :user');
        $stmt->bindValue(':user', $key);
        $stmt->execute();
        if ((int)$stmt->fetchColumn() === 0) {
            $needsBackfill = true;
        }
    }
} catch (Exception $e) {
    $bootstrapWarning = $e->getMessage();
}

if ($needsBackfill) {
    // Heavy work remains. Return a loading page that auto-refreshes;
    // each refresh does another page of API fetches.
    ?>
    <!DOCTYPE html>
    <html lang="en">
    <head>
      <meta charset="UTF-8">
      <title>Torn Vault Tracker &mdash; loading</title>
      <meta http-equiv="refresh" content="3">
      <link rel="icon" type="image/png" href="favicon.png">
      <link rel="stylesheet" href="style.css?v=<?php echo filemtime('style.css'); ?>">
    </head>
    <body>
      <h1>Loading vault history&hellip;</h1>
      <?php if ($bootstrapWarning !== null): ?>
        <p>Warning: <?php echo htmlspecialchars($bootstrapWarning, ENT_QUOTES, 'UTF-8'); ?></p>
      <?php endif; ?>
      <p>One-time backfill in progress. The Torn API returns up to 100 vault entries per call;
         we'll keep fetching until your full history is loaded. This page will refresh automatically.</p>
    </body>
    </html>
    <?php
    exit;
}

Replace the existing try { ... } catch { ... } block at the top of index.php (the one wrapping the bootstrap) with the above. Keep the $bootstrapWarning variable defined for the rest of the page (it's still rendered into the main UI when bootstrap completes).

Step 2: Verify the file parses

Run: php -l index.php Expected: No syntax errors detected.

Step 3: Commit

git add index.php
git commit -m "index.php: meta-refresh loading page while backfill is in progress"

Task 2: End-to-end smoke check

Files: No code changes.

Step 1: Run the test suite

cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
php tests/process_log_entries_test.php

Expected: OK: processLogEntries v2 golden-file test passed (100 entries).

Step 2: Verify no v1 references

cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
grep -n "firstRun\|?key=" functions.php index.php includes/utilities.php includes/exceptions.php tests/

Expected: no matches.

Step 3: Commit if any incidental fixes were needed

If Step 1 or 2 produced any required fixes, commit them. Otherwise, no commit.

Self-Review Notes

Spec coverage:

Requirement	Task
Return a loading page on bootstrap	Task 1
Each refresh makes progress	Task 1 (the sync walks one page of entries per refresh, limited by Torn's API page size of 100)
Eventually complete and render real UI	Task 1 (when all users have rows, no more refresh needed)
End-to-end smoke check	Task 2

Placeholder scan: No "TODO", "TBD".

Type consistency: No new function signatures.

Risks addressed:

PHP execution timeout during bootstrap: the loading page returns in <1s, no API calls happen during the initial render.
Partial backfill on browser close: ON CONFLICT DO UPDATE makes the next refresh safe to resume.
Cascading timeouts: each refresh does only the next page of work; no single refresh can exceed reasonable host limits.

6.6 KiB Raw Blame History