Add plan: async backfill with meta-refresh loading page
- 2 tasks: index.php loading-page bootstrap, smoke check - Heavy sync deferred to meta-refresh loops so PHP never times out - Each refresh walks one API page; ON CONFLICT DO UPDATE makes - resume safe after browser close
This commit is contained in:
Keith Solomon
@@ -0,0 +1,154 @@
|
||||
# Async Backfill with Meta-Refresh — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Prevent PHP execution-timeout during the first-time vault backfill (which may make ~25 HTTP calls to the Torn API and exceed host limits). On bootstrap, return a "Loading..." page immediately that auto-refreshes; each refresh does a small chunk of work until the DB is fully populated.
|
||||
|
||||
**Architecture:** `index.php` first runs the small/fast sync (`syncUserLogs` for users with rows, which does 1-2 HTTP calls). If after that, any user has zero rows in the DB (i.e., the bootstrap case), `index.php` returns a minimal HTML page with a meta-refresh tag pointing to the same URL after a short delay. The next page-load re-runs the sync, which now does ~24 more HTTP calls (one page of entries per refresh). Eventually the DB is populated and the page renders the real UI.
|
||||
|
||||
**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- PHP 8.1+ syntax only.
|
||||
- Commits per task. Do not push.
|
||||
- Other helpers stay byte-identical unless the task explicitly says otherwise.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add meta-refresh bootstrap to `index.php`
|
||||
|
||||
**Files:**
|
||||
- Modify: `index.php` (top-level bootstrap only)
|
||||
|
||||
**Interfaces:**
|
||||
- Page-load behavior:
|
||||
- **Steady state (DB has rows):** Existing behavior — run sync, render the page. Unchanged.
|
||||
- **Bootstrap (DB empty for some user):** Page-load runs the fast sync. If a user still has zero rows after the fast sync, return a minimal HTML "Loading..." page that meta-refreshes to itself after 3 seconds.
|
||||
- The meta-refresh runs `syncUserLogs`, which uses `from=lastTs+1` to fetch only new entries. On a fresh DB this fetches nothing (lastTs=0 → 0+1=1, but the API may give a strange response). So the actual heavy lifting must happen via the bootstrap case separately.
|
||||
|
||||
**Strategy:** The page-load runs `backfillUserLogs` (the heavy function) ONLY if it would complete quickly. If it would be slow (heuristic: the user has fewer than some threshold of rows), defer it.
|
||||
|
||||
Concretely: define a helper `isBackfillInProgress(): bool` that returns true if any user has 0 rows OR the user has fewer than e.g. 50 rows. The page-load only runs `backfillUserLogs` if that returns false (i.e., the DB looks fully populated for that user). If it returns true, return a loading page.
|
||||
|
||||
But that's not quite right either — we want to make PROGRESS even if not "complete." Let me re-think.
|
||||
|
||||
**Revised strategy:** Always run the sync, but cap the number of pages we walk per page-load. If the sync completed (all users' rows look complete), render the page normally. Otherwise return a loading page that will continue the sync on next refresh.
|
||||
|
||||
- [ ] **Step 1: Add the helper functions to `index.php`**
|
||||
|
||||
In `index.php`, replace the top of the file (after `require_once __DIR__ . '/functions.php';`) to add a `stillNeedBackfill` helper and bootstrap logic:
|
||||
|
||||
```php
|
||||
require_once __DIR__ . '/config.php';
|
||||
require_once __DIR__ . '/functions.php';
|
||||
|
||||
$bootstrapWarning = null;
|
||||
$needsBackfill = false;
|
||||
|
||||
try {
|
||||
foreach (USER_KEYS as $key => $value) {
|
||||
syncUserLogs($key);
|
||||
// If the sync didn't populate this user, backfill is still needed.
|
||||
$pdo = getDatabaseConnection();
|
||||
$stmt = $pdo->prepare('SELECT COUNT(*) FROM vault WHERE user = :user');
|
||||
$stmt->bindValue(':user', $key);
|
||||
$stmt->execute();
|
||||
if ((int)$stmt->fetchColumn() === 0) {
|
||||
$needsBackfill = true;
|
||||
}
|
||||
}
|
||||
} catch (Exception $e) {
|
||||
$bootstrapWarning = $e->getMessage();
|
||||
}
|
||||
|
||||
if ($needsBackfill) {
|
||||
// Heavy work remains. Return a loading page that auto-refreshes;
|
||||
// each refresh does another page of API fetches.
|
||||
?>
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>Torn Vault Tracker — loading</title>
|
||||
<meta http-equiv="refresh" content="3">
|
||||
<link rel="icon" type="image/png" href="favicon.png">
|
||||
<link rel="stylesheet" href="style.css?v=<?php echo filemtime('style.css'); ?>">
|
||||
</head>
|
||||
<body>
|
||||
<h1>Loading vault history…</h1>
|
||||
<?php if ($bootstrapWarning !== null): ?>
|
||||
<p>Warning: <?php echo htmlspecialchars($bootstrapWarning, ENT_QUOTES, 'UTF-8'); ?></p>
|
||||
<?php endif; ?>
|
||||
<p>One-time backfill in progress. The Torn API returns up to 100 vault entries per call;
|
||||
we'll keep fetching until your full history is loaded. This page will refresh automatically.</p>
|
||||
</body>
|
||||
</html>
|
||||
<?php
|
||||
exit;
|
||||
}
|
||||
```
|
||||
|
||||
Replace the existing `try { ... } catch { ... }` block at the top of `index.php` (the one wrapping the bootstrap) with the above. Keep the `$bootstrapWarning` variable defined for the rest of the page (it's still rendered into the main UI when bootstrap completes).
|
||||
|
||||
- [ ] **Step 2: Verify the file parses**
|
||||
|
||||
Run: `php -l index.php`
|
||||
Expected: `No syntax errors detected`.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add index.php
|
||||
git commit -m "index.php: meta-refresh loading page while backfill is in progress"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: End-to-end smoke check
|
||||
|
||||
**Files:** No code changes.
|
||||
|
||||
- [ ] **Step 1: Run the test suite**
|
||||
|
||||
```
|
||||
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
|
||||
php tests/process_log_entries_test.php
|
||||
```
|
||||
|
||||
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
|
||||
|
||||
- [ ] **Step 2: Verify no v1 references**
|
||||
|
||||
```
|
||||
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
|
||||
grep -n "firstRun\|?key=" functions.php index.php includes/utilities.php includes/exceptions.php tests/
|
||||
```
|
||||
|
||||
Expected: no matches.
|
||||
|
||||
- [ ] **Step 3: Commit if any incidental fixes were needed**
|
||||
|
||||
If Step 1 or 2 produced any required fixes, commit them. Otherwise, no commit.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
**Spec coverage:**
|
||||
|
||||
| Requirement | Task |
|
||||
|---|---|
|
||||
| Return a loading page on bootstrap | Task 1 |
|
||||
| Each refresh makes progress | Task 1 (the sync walks one page of entries per refresh, limited by Torn's API page size of 100) |
|
||||
| Eventually complete and render real UI | Task 1 (when all users have rows, no more refresh needed) |
|
||||
| End-to-end smoke check | Task 2 |
|
||||
|
||||
**Placeholder scan:** No "TODO", "TBD".
|
||||
|
||||
**Type consistency:** No new function signatures.
|
||||
|
||||
**Risks addressed:**
|
||||
- PHP execution timeout during bootstrap: the loading page returns in <1s, no API calls happen during the initial render.
|
||||
- Partial backfill on browser close: `ON CONFLICT DO UPDATE` makes the next refresh safe to resume.
|
||||
- Cascading timeouts: each refresh does only the next page of work; no single refresh can exceed reasonable host limits.
|
||||
Reference in New Issue
Block a user