diff --git a/docs/superpowers/plans/2026-08-07-walk-prev-chain-on-backfill.md b/docs/superpowers/plans/2026-08-07-walk-prev-chain-on-backfill.md new file mode 100644 index 0000000..b797e5f --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-walk-prev-chain-on-backfill.md @@ -0,0 +1,145 @@ +# Walk `prev` Chain on Backfill — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `backfillUserLogs` capture the user's complete vault transaction history (not just the most recent 100 entries) by walking the v2 API's `prev` chain in addition to `next`. After this, `SUM(amount)` per user returns the true current share, matching what Torn's website shows. + +**Architecture:** Modify `fetchAndStoreLogPage` so the pagination loop follows BOTH `_metadata.links.next` (forward in time, used when `from` is given) and `_metadata.links.prev` (backward in time, used on the first/oldest page). The two are mutually exclusive in practice — `prev` exists on the most-recent page only, `next` exists on the oldest page only — but following both is defensive and safe. + +**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies. + +## Global Constraints + +- PHP 8.1+ syntax only. +- Idempotency: `INSERT … ON CONFLICT(id) DO NOTHING` (or `DO UPDATE` for re-fetch — see below). +- API authentication: `Authorization: ApiKey ` header. +- All other helpers stay byte-identical unless the task explicitly says otherwise. +- Commits per task. Do not push. + +--- + +### Task 1: Update `fetchAndStoreLogPage` to walk `prev` + +**Files:** +- Modify: `includes/utilities.php` (`fetchAndStoreLogPage` only) + +**Interfaces:** +- `fetchAndStoreLogPage($pdo, $user, $url)`: unchanged signature. The pagination loop additionally follows `_metadata.links.prev` to fetch older pages. + +- [ ] **Step 1: Update the pagination loop** + +In `includes/utilities.php`, find the `fetchAndStoreLogPage` function. Replace the entire function body with: + +```php +function fetchAndStoreLogPage($pdo, $user, $url) { + if (!array_key_exists($user, USER_KEYS)) { + throw new ApiKeyMissingException("User does not have an API key configured."); + } + + $insertStmt = $pdo->prepare( + 'INSERT INTO vault (id, user, timestamp, description, amount, running_balance) ' + . 'VALUES (:id, :user, :timestamp, :description, :amount, :running_balance) ' + . 'ON CONFLICT(id) DO UPDATE SET running_balance = EXCLUDED.running_balance, amount = EXCLUDED.amount' + ); + + do { + $responseData = executeApiCall($url, USER_KEYS[$user]); + validateApiResponse($responseData); + + processLogEntries($responseData['log'], $user, $insertStmt); + + // Follow the older-pages chain so the very first call (which gets the + // most recent page) also retrieves every older entry. After the prev + // chain is exhausted, fall back to the newer-pages chain in case the + // caller passed a `from=` URL and the result has a next link. + $url = $responseData['_metadata']['links']['prev'] + ?? $responseData['_metadata']['links']['next'] + ?? null; + } while ($url !== null); +} +``` + +The change from the previous version: the pagination cursor now prefers `prev` (older pages) over `next` (newer pages). Once the prev chain is exhausted, it falls back to next. + +Note: the INSERT statement now uses `ON CONFLICT(id) DO UPDATE` (instead of `DO NOTHING`) for `running_balance` and `amount`. This allows a re-fetch to refresh those fields if the API ever returns different values for an existing entry id. This is necessary because the backfill may run more than once (e.g., after a re-fetch from a different starting URL). + +- [ ] **Step 2: Verify the file parses** + +Run: `php -l includes/utilities.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 3: Run the golden-file test (no regression)** + +Run: `php tests/process_log_entries_test.php` +Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).` + +- [ ] **Step 4: Commit** + +```bash +git add includes/utilities.php +git commit -m "fetchAndStoreLogPage: walk prev chain to fetch full vault history" +``` + +--- + +### Task 2: End-to-end smoke check + +**Files:** No code changes. + +- [ ] **Step 1: Run the test suite** + +Run: `php tests/process_log_entries_test.php` +Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).` + +- [ ] **Step 2: Verify no v1 references and no stray references remain** + +```bash +cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker" +grep -n "firstRun\|?key=" functions.php index.php includes/utilities.php includes/exceptions.php tests/ +``` + +Expected: no matches (same as before). + +- [ ] **Step 3: Verify the live DB state has Zarathos=$0 and Symos=$152,663,117** + +The DB was manually backfilled earlier in the session to contain the correct values. Re-verify: + +```bash +cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker" +php -r " +\$db = new PDO('sqlite:data/vault.db'); +foreach (\$db->query('SELECT user, COUNT(*) AS n, SUM(amount) AS s FROM vault GROUP BY user') as \$r) { + echo \$r['user'] . ': ' . \$r['n'] . ' entries, SUM(amount)=\$' . number_format(\$r['s']) . PHP_EOL; +} +" +``` + +Expected: Zarathos: 374 entries (or similar), SUM(amount)=$0. Symos: 2461 entries (or similar), SUM(amount)=$152,663,117. (The DB has been manually populated; the new code preserves these values via `DO UPDATE`.) + +- [ ] **Step 4: (Optional) Manual backfill via test instance** + +If the test instance is running and the page is loaded, the new code should be a no-op (DB already populated). If the test instance has been restarted with an empty DB, the new code should populate it fully via the `prev` chain walk. Note: the user may need to clear the DB and reload the page to actually exercise the new code path end-to-end. Mark as not-required for this PR if the DB is already correct. + +- [ ] **Step 5: Commit if any incidental fixes were needed** + +If Step 1, 2, 3, or 4 produced any required fixes, commit them. Otherwise, no commit. + +--- + +## Self-Review Notes + +**Spec coverage:** + +| Requirement | Task | +|---|---| +| Follow `prev` chain in pagination | Task 1 | +| Idempotent re-fetch (re-running updates rather than no-ops) | Task 1 (`ON CONFLICT DO UPDATE`) | +| End-to-end smoke check | Task 2 | + +**Placeholder scan:** No "TODO", "TBD", "implement later". + +**Type consistency:** `fetchAndStoreLogPage(PDO $pdo, string $user, string $url): void` signature unchanged. + +**Risks addressed:** +- The `prev` walk adds up to N HTTP calls per user per backfill (N = ceil(entries / 100)). For Symos that's ~25 calls. Acceptable since backfill only runs once on first sync. +- `ON CONFLICT DO UPDATE` means re-running the backfill will overwrite existing rows. This is correct because the API data is the source of truth — re-fetches may have updated values (e.g., if Torn fixes a stale entry). \ No newline at end of file