Add plan: walk prev chain on backfill for full vault history
- 2 tasks: update fetchAndStoreLogPage pagination loop, smoke check - Loop prefers prev (older pages) over next (newer pages) so a single fetchAndStoreLogPage call captures the full history - INSERT uses ON CONFLICT DO UPDATE for running_balance and amount so re-runs refresh data
This commit is contained in:
Keith Solomon
@@ -0,0 +1,145 @@
|
||||
# Walk `prev` Chain on Backfill — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make `backfillUserLogs` capture the user's complete vault transaction history (not just the most recent 100 entries) by walking the v2 API's `prev` chain in addition to `next`. After this, `SUM(amount)` per user returns the true current share, matching what Torn's website shows.
|
||||
|
||||
**Architecture:** Modify `fetchAndStoreLogPage` so the pagination loop follows BOTH `_metadata.links.next` (forward in time, used when `from` is given) and `_metadata.links.prev` (backward in time, used on the first/oldest page). The two are mutually exclusive in practice — `prev` exists on the most-recent page only, `next` exists on the oldest page only — but following both is defensive and safe.
|
||||
|
||||
**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- PHP 8.1+ syntax only.
|
||||
- Idempotency: `INSERT … ON CONFLICT(id) DO NOTHING` (or `DO UPDATE` for re-fetch — see below).
|
||||
- API authentication: `Authorization: ApiKey <key>` header.
|
||||
- All other helpers stay byte-identical unless the task explicitly says otherwise.
|
||||
- Commits per task. Do not push.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Update `fetchAndStoreLogPage` to walk `prev`
|
||||
|
||||
**Files:**
|
||||
- Modify: `includes/utilities.php` (`fetchAndStoreLogPage` only)
|
||||
|
||||
**Interfaces:**
|
||||
- `fetchAndStoreLogPage($pdo, $user, $url)`: unchanged signature. The pagination loop additionally follows `_metadata.links.prev` to fetch older pages.
|
||||
|
||||
- [ ] **Step 1: Update the pagination loop**
|
||||
|
||||
In `includes/utilities.php`, find the `fetchAndStoreLogPage` function. Replace the entire function body with:
|
||||
|
||||
```php
|
||||
function fetchAndStoreLogPage($pdo, $user, $url) {
|
||||
if (!array_key_exists($user, USER_KEYS)) {
|
||||
throw new ApiKeyMissingException("User does not have an API key configured.");
|
||||
}
|
||||
|
||||
$insertStmt = $pdo->prepare(
|
||||
'INSERT INTO vault (id, user, timestamp, description, amount, running_balance) '
|
||||
. 'VALUES (:id, :user, :timestamp, :description, :amount, :running_balance) '
|
||||
. 'ON CONFLICT(id) DO UPDATE SET running_balance = EXCLUDED.running_balance, amount = EXCLUDED.amount'
|
||||
);
|
||||
|
||||
do {
|
||||
$responseData = executeApiCall($url, USER_KEYS[$user]);
|
||||
validateApiResponse($responseData);
|
||||
|
||||
processLogEntries($responseData['log'], $user, $insertStmt);
|
||||
|
||||
// Follow the older-pages chain so the very first call (which gets the
|
||||
// most recent page) also retrieves every older entry. After the prev
|
||||
// chain is exhausted, fall back to the newer-pages chain in case the
|
||||
// caller passed a `from=` URL and the result has a next link.
|
||||
$url = $responseData['_metadata']['links']['prev']
|
||||
?? $responseData['_metadata']['links']['next']
|
||||
?? null;
|
||||
} while ($url !== null);
|
||||
}
|
||||
```
|
||||
|
||||
The change from the previous version: the pagination cursor now prefers `prev` (older pages) over `next` (newer pages). Once the prev chain is exhausted, it falls back to next.
|
||||
|
||||
Note: the INSERT statement now uses `ON CONFLICT(id) DO UPDATE` (instead of `DO NOTHING`) for `running_balance` and `amount`. This allows a re-fetch to refresh those fields if the API ever returns different values for an existing entry id. This is necessary because the backfill may run more than once (e.g., after a re-fetch from a different starting URL).
|
||||
|
||||
- [ ] **Step 2: Verify the file parses**
|
||||
|
||||
Run: `php -l includes/utilities.php`
|
||||
Expected: `No syntax errors detected`.
|
||||
|
||||
- [ ] **Step 3: Run the golden-file test (no regression)**
|
||||
|
||||
Run: `php tests/process_log_entries_test.php`
|
||||
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add includes/utilities.php
|
||||
git commit -m "fetchAndStoreLogPage: walk prev chain to fetch full vault history"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: End-to-end smoke check
|
||||
|
||||
**Files:** No code changes.
|
||||
|
||||
- [ ] **Step 1: Run the test suite**
|
||||
|
||||
Run: `php tests/process_log_entries_test.php`
|
||||
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
|
||||
|
||||
- [ ] **Step 2: Verify no v1 references and no stray references remain**
|
||||
|
||||
```bash
|
||||
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
|
||||
grep -n "firstRun\|?key=" functions.php index.php includes/utilities.php includes/exceptions.php tests/
|
||||
```
|
||||
|
||||
Expected: no matches (same as before).
|
||||
|
||||
- [ ] **Step 3: Verify the live DB state has Zarathos=$0 and Symos=$152,663,117**
|
||||
|
||||
The DB was manually backfilled earlier in the session to contain the correct values. Re-verify:
|
||||
|
||||
```bash
|
||||
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
|
||||
php -r "
|
||||
\$db = new PDO('sqlite:data/vault.db');
|
||||
foreach (\$db->query('SELECT user, COUNT(*) AS n, SUM(amount) AS s FROM vault GROUP BY user') as \$r) {
|
||||
echo \$r['user'] . ': ' . \$r['n'] . ' entries, SUM(amount)=\$' . number_format(\$r['s']) . PHP_EOL;
|
||||
}
|
||||
"
|
||||
```
|
||||
|
||||
Expected: Zarathos: 374 entries (or similar), SUM(amount)=$0. Symos: 2461 entries (or similar), SUM(amount)=$152,663,117. (The DB has been manually populated; the new code preserves these values via `DO UPDATE`.)
|
||||
|
||||
- [ ] **Step 4: (Optional) Manual backfill via test instance**
|
||||
|
||||
If the test instance is running and the page is loaded, the new code should be a no-op (DB already populated). If the test instance has been restarted with an empty DB, the new code should populate it fully via the `prev` chain walk. Note: the user may need to clear the DB and reload the page to actually exercise the new code path end-to-end. Mark as not-required for this PR if the DB is already correct.
|
||||
|
||||
- [ ] **Step 5: Commit if any incidental fixes were needed**
|
||||
|
||||
If Step 1, 2, 3, or 4 produced any required fixes, commit them. Otherwise, no commit.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
**Spec coverage:**
|
||||
|
||||
| Requirement | Task |
|
||||
|---|---|
|
||||
| Follow `prev` chain in pagination | Task 1 |
|
||||
| Idempotent re-fetch (re-running updates rather than no-ops) | Task 1 (`ON CONFLICT DO UPDATE`) |
|
||||
| End-to-end smoke check | Task 2 |
|
||||
|
||||
**Placeholder scan:** No "TODO", "TBD", "implement later".
|
||||
|
||||
**Type consistency:** `fetchAndStoreLogPage(PDO $pdo, string $user, string $url): void` signature unchanged.
|
||||
|
||||
**Risks addressed:**
|
||||
- The `prev` walk adds up to N HTTP calls per user per backfill (N = ceil(entries / 100)). For Symos that's ~25 calls. Acceptable since backfill only runs once on first sync.
|
||||
- `ON CONFLICT DO UPDATE` means re-running the backfill will overwrite existing rows. This is correct because the API data is the source of truth — re-fetches may have updated values (e.g., if Torn fixes a stale entry).
|
||||
Reference in New Issue
Block a user