Torn-Vault-Tracker/docs/superpowers/plans/2026-08-07-walk-prev-chain-on-backfill.md

Walk prev Chain on Backfill — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make backfillUserLogs capture the user's complete vault transaction history (not just the most recent 100 entries) by walking the v2 API's prev chain in addition to next. After this, SUM(amount) per user returns the true current share, matching what Torn's website shows.

Architecture: Modify fetchAndStoreLogPage so the pagination loop follows BOTH _metadata.links.next (forward in time, used when from is given) and _metadata.links.prev (backward in time, used on the first/oldest page). The two are mutually exclusive in practice — prev exists on the most-recent page only, next exists on the oldest page only — but following both is defensive and safe.

Tech Stack: PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies.

Global Constraints

• PHP 8.1+ syntax only.
• Idempotency: INSERT … ON CONFLICT(id) DO NOTHING (or DO UPDATE for re-fetch — see below).
• API authentication: Authorization: ApiKey <key> header.
• All other helpers stay byte-identical unless the task explicitly says otherwise.
• Commits per task. Do not push.

---

Task 1: Update fetchAndStoreLogPage to walk prev

Files:

• Modify: includes/utilities.php (fetchAndStoreLogPage only)

Interfaces:

• fetchAndStoreLogPage($pdo, $user, $url): unchanged signature. The pagination loop additionally follows _metadata.links.prev to fetch older pages.

• Step 1: Update the pagination loop

In includes/utilities.php, find the fetchAndStoreLogPage function. Replace the entire function body with:

function fetchAndStoreLogPage($pdo, $user, $url) {
    if (!array_key_exists($user, USER_KEYS)) {
        throw new ApiKeyMissingException("User does not have an API key configured.");
    }

    $insertStmt = $pdo->prepare(
        'INSERT INTO vault (id, user, timestamp, description, amount, running_balance) '
        . 'VALUES (:id, :user, :timestamp, :description, :amount, :running_balance) '
        . 'ON CONFLICT(id) DO UPDATE SET running_balance = EXCLUDED.running_balance, amount = EXCLUDED.amount'
    );

    do {
        $responseData = executeApiCall($url, USER_KEYS[$user]);
        validateApiResponse($responseData);

        processLogEntries($responseData['log'], $user, $insertStmt);

        // Follow the older-pages chain so the very first call (which gets the
        // most recent page) also retrieves every older entry. After the prev
        // chain is exhausted, fall back to the newer-pages chain in case the
        // caller passed a `from=` URL and the result has a next link.
        $url = $responseData['_metadata']['links']['prev']
            ?? $responseData['_metadata']['links']['next']
            ?? null;
    } while ($url !== null);
}

The change from the previous version: the pagination cursor now prefers prev (older pages) over next (newer pages). Once the prev chain is exhausted, it falls back to next.

Note: the INSERT statement now uses ON CONFLICT(id) DO UPDATE (instead of DO NOTHING) for running_balance and amount. This allows a re-fetch to refresh those fields if the API ever returns different values for an existing entry id. This is necessary because the backfill may run more than once (e.g., after a re-fetch from a different starting URL).

• Step 2: Verify the file parses

Run: php -l includes/utilities.php Expected: No syntax errors detected.

• Step 3: Run the golden-file test (no regression)

Run: php tests/process_log_entries_test.php Expected: OK: processLogEntries v2 golden-file test passed (100 entries).

• Step 4: Commit

git add includes/utilities.php
git commit -m "fetchAndStoreLogPage: walk prev chain to fetch full vault history"

---

Task 2: End-to-end smoke check

Files: No code changes.

• Step 1: Run the test suite

Run: php tests/process_log_entries_test.php Expected: OK: processLogEntries v2 golden-file test passed (100 entries).

• Step 2: Verify no v1 references and no stray references remain

cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
grep -n "firstRun\|?key=" functions.php index.php includes/utilities.php includes/exceptions.php tests/

Expected: no matches (same as before).

• Step 3: Verify the live DB state has Zarathos=$0 and Symos=$152,663,117

The DB was manually backfilled earlier in the session to contain the correct values. Re-verify:

cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
php -r "
\$db = new PDO('sqlite:data/vault.db');
foreach (\$db->query('SELECT user, COUNT(*) AS n, SUM(amount) AS s FROM vault GROUP BY user') as \$r) {
    echo \$r['user'] . ': ' . \$r['n'] . ' entries, SUM(amount)=\$' . number_format(\$r['s']) . PHP_EOL;
}
"

Expected: Zarathos: 374 entries (or similar), SUM(amount)=$0. Symos: 2461 entries (or similar), SUM(amount)=$152,663,117. (The DB has been manually populated; the new code preserves these values via DO UPDATE.)

• Step 4: (Optional) Manual backfill via test instance

If the test instance is running and the page is loaded, the new code should be a no-op (DB already populated). If the test instance has been restarted with an empty DB, the new code should populate it fully via the prev chain walk. Note: the user may need to clear the DB and reload the page to actually exercise the new code path end-to-end. Mark as not-required for this PR if the DB is already correct.

• Step 5: Commit if any incidental fixes were needed

If Step 1, 2, 3, or 4 produced any required fixes, commit them. Otherwise, no commit.

---

Self-Review Notes

Spec coverage:

Requirement	Task
Follow prev chain in pagination	Task 1
Idempotent re-fetch (re-running updates rather than no-ops)	Task 1 (ON CONFLICT DO UPDATE)
End-to-end smoke check	Task 2

Placeholder scan: No "TODO", "TBD", "implement later".

Type consistency: fetchAndStoreLogPage(PDO $pdo, string $user, string $url): void signature unchanged.

Risks addressed:

• The prev walk adds up to N HTTP calls per user per backfill (N = ceil(entries / 100)). For Symos that's ~25 calls. Acceptable since backfill only runs once on first sync.
• ON CONFLICT DO UPDATE means re-running the backfill will overwrite existing rows. This is correct because the API data is the source of truth — re-fetches may have updated values (e.g., if Torn fixes a stale entry).