Files
Torn-Vault-Tracker/docs/superpowers/plans/2026-08-03-vault-balance-from-api-design.md
Keith Solomon 4339958b5e
Deploy website via FTP / Deploy (push) Skipped
Normalize line endings in vault balance plan doc
The plan was committed with LF endings; the editor normalized to CRLF on
save. Restore LF for consistency with the rest of the repo's plan docs.
2026-08-10 20:40:55 -05:00

624 lines
22 KiB
Markdown

# Vault Balance From Per-Entry API Value — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Fix the vault balance display so it shows the live vault total (per the Torn v2 API's `data.balance` field) instead of `SUM(amount)` of stored entries. The v2 API only returns the most recent ~100 entries per user, so summing deltas cannot yield the true current balance — only the per-entry running balance can.
**Architecture:** Add a `running_balance INTEGER` column to the `vault` table. Capture it from `$entry['data']['balance']` in `processLogEntries`. Replace balance SQL from `SUM(amount)` to `MAX(running_balance)` per user. Add a one-shot live-read backfill that broadcasts the latest entry's `data.balance` to all NULL rows of a user. Add a "Balance After" column to the rendered table.
**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies.
## Global Constraints
- PHP 8.1+ syntax only.
- Database schema gains one column (`running_balance INTEGER`). Existing columns and their semantics are unchanged.
- `INSERT … ON CONFLICT(id) DO NOTHING` is the only INSERT form used for log entries.
- API authentication: `Authorization: ApiKey <key>` header.
- Pagination: follow `_metadata.links.next` URL until null.
- Idempotency: every change must be safe to re-run (especially the schema migration and the backfill).
- Commits per task. Do not push.
- All existing helpers outside the rewritten ones stay byte-identical unless the task explicitly says otherwise.
---
### Task 1: Schema — add `running_balance` column (idempotent on existing DBs)
**Files:**
- Modify: `functions.php` (the `getDatabaseConnection` function only)
**Interfaces:**
- Consumes: PDO from `PDO(DB_DSN, …)`. Reads `PRAGMA table_info(vault)` to detect an existing column.
- Produces: a `vault` table whose schema includes `running_balance INTEGER`. For existing DBs, the column is added via `ALTER TABLE`.
- [ ] **Step 1: Update the CREATE TABLE statement**
In `functions.php`, replace the `CREATE TABLE IF NOT EXISTS vault` (currently at lines 32-38) with the version that includes `running_balance`:
```php
$createTableSQL = "CREATE TABLE IF NOT EXISTS vault (
id TEXT PRIMARY KEY,
user TEXT NOT NULL,
timestamp INTEGER NOT NULL,
description TEXT NOT NULL,
amount REAL NOT NULL,
running_balance INTEGER
);";
```
- [ ] **Step 2: Add the idempotent ALTER TABLE for existing DBs**
Replace the `if ($pdo === null)` block's `$pdo->exec($createTableSQL);` call with the following code (keep the rest of the block intact):
```php
$pdo->exec($createTableSQL);
// Add the running_balance column for databases created before the
// column was added. Idempotent: skip if it already exists.
$columns = $pdo->query("PRAGMA table_info(vault)")->fetchAll(PDO::FETCH_COLUMN, 1);
if (!in_array('running_balance', $columns, true)) {
$pdo->exec('ALTER TABLE vault ADD COLUMN running_balance INTEGER');
}
```
- [ ] **Step 3: Verify the file parses**
Run: `php -l functions.php`
Expected: `No syntax errors detected in functions.php`.
- [ ] **Step 4: Verify the migration is idempotent on the live DB**
Run once:
```bash
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
php -r "
\$db = new PDO('sqlite:data/vault.db');
foreach(\$db->query('PRAGMA table_info(vault)') as \$c) { echo \$c['name'] . ' ' . \$c['type'] . PHP_EOL; }
"
```
Expected: `running_balance INTEGER` appears in the column list.
Run a second time (Page-load will do this; or re-run the script above twice in a row). Expected: no error (the `ALTER TABLE` fires only when the column is missing).
- [ ] **Step 5: Commit**
```bash
git add functions.php
git commit -m "Schema: add running_balance column to vault table"
```
---
### Task 2: Rewrite `fetchAndStoreLogPage` and `processLogEntries` to capture `data.balance` (with failing test)
**Files:**
- Modify: `includes/utilities.php` (`fetchAndStoreLogPage` and `processLogEntries`)
- Modify: `tests/process_log_entries_test.php` (extend to assert `running_balance` round-trips)
**Interfaces:**
- `processLogEntries($logEntries, $user, $insertStmt)`: reads `$entry['data']['balance']` (int) and binds it as `:running_balance`. If `data.balance` is missing, binds NULL.
- `fetchAndStoreLogPage($pdo, $user, $url)`: INSERT statement now includes `running_balance`.
- [ ] **Step 1: Extend the failing test**
In `tests/process_log_entries_test.php`, after the existing "row count after re-run" assertion, add these new assertions for the `running_balance` column. The fixture has 100 entries; the most recent entry's `data.balance` is `69235831` (newest, withdraw) and the oldest entry's `data.balance` is `509622821` (oldest, deposit) — both per the fixture verified during task planning.
Append after the existing "row count after re-run" block:
```php
// round-trip the running_balance column from $entry['data']['balance']
$firstRunning = (int)$first['running_balance'];
$lastRunning = (int)$last['running_balance'];
assertSame(69235831, $firstRunning, 'first.running_balance');
assertSame(509622821, $lastRunning, 'last.running_balance');
```
- [ ] **Step 2: Run the test and verify it fails**
Run: `php tests/process_log_entries_test.php`
Expected: failure with `first.running_balance: expected 69235831, got ` (likely 0 or NULL because the column doesn't get inserted yet).
- [ ] **Step 3: Update `processLogEntries`**
In `includes/utilities.php`, replace the existing `processLogEntries` function (currently at lines 122-157) with:
```php
function processLogEntries($logEntries, $user, $insertStmt) {
foreach ($logEntries as $entry) {
$id = $entry['id'] ?? null;
if (!$id) {
consoleLog('Skipping entry with no id: ' . print_r($entry, true));
continue;
}
$timestamp = $entry['timestamp'] ?? null;
$description = $entry['details']['title'] ?? null;
$hasDeposit = isset($entry['data']['deposited']);
$hasWithdraw = isset($entry['data']['withdrawn']);
if ($hasDeposit) {
$amount = (int)$entry['data']['deposited'];
} elseif ($hasWithdraw) {
$amount = -((int)$entry['data']['withdrawn']);
} else {
consoleLog('Skipping entry ' . $id . ' with no deposited/withdrawn: ' . print_r($entry, true));
continue;
}
if ($timestamp === null || $description === null) {
consoleLog('Skipping entry ' . $id . ' missing timestamp or details.title: ' . print_r($entry, true));
continue;
}
$runningBalance = isset($entry['data']['balance']) ? (int)$entry['data']['balance'] : null;
$insertStmt->bindValue(':id', $id);
$insertStmt->bindValue(':user', $user);
$insertStmt->bindValue(':timestamp', $timestamp);
$insertStmt->bindValue(':description', $description);
$insertStmt->bindValue(':amount', $amount);
$insertStmt->bindValue(':running_balance', $runningBalance);
$insertStmt->execute();
}
}
```
- [ ] **Step 4: Update `fetchAndStoreLogPage`'s INSERT statement**
In `includes/utilities.php`, replace the `$insertStmt` block inside `fetchAndStoreLogPage` (currently at lines 252-256) with:
```php
$insertStmt = $pdo->prepare(
'INSERT INTO vault (id, user, timestamp, description, amount, running_balance) '
. 'VALUES (:id, :user, :timestamp, :description, :amount, :running_balance) '
. 'ON CONFLICT(id) DO NOTHING'
);
```
- [ ] **Step 5: Run the test and verify it passes**
Run: `php tests/process_log_entries_test.php`
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
- [ ] **Step 6: Verify the test file still parses**
Run: `php -l tests/process_log_entries_test.php`
Expected: `No syntax errors detected`.
- [ ] **Step 7: Commit**
```bash
git add includes/utilities.php tests/process_log_entries_test.php
git commit -m "Capture running_balance from v2 API data.balance"
```
---
### Task 3: Update `fetchVaultRecords` to include `running_balance`
**Files:**
- Modify: `includes/utilities.php` (`fetchVaultRecords` only)
**Interfaces:**
- Consumes: existing signature unchanged.
- Produces: rows that include the `running_balance` column (existing callers that ignore unknown keys continue to work).
- [ ] **Step 1: Replace `fetchVaultRecords` with explicit column list**
In `includes/utilities.php`, replace the body of `fetchVaultRecords` (currently at lines 168-189) with:
```php
function fetchVaultRecords($user = null) {
$pdo = getDatabaseConnection();
if ($user) {
$query = "SELECT id, user, timestamp, description, amount, running_balance FROM vault WHERE user = :user ORDER BY timestamp DESC";
$params = [':user' => $user];
} else {
$query = "SELECT id, user, timestamp, description, amount, running_balance FROM vault ORDER BY timestamp DESC";
$params = [];
}
try {
$stmt = $pdo->prepare($query);
$stmt->execute($params);
return $stmt->fetchAll(PDO::FETCH_ASSOC);
} catch (PDOException $e) {
echo "Error fetching records: " . $e->getMessage();
return [];
}
}
```
- [ ] **Step 2: Verify the file parses**
Run: `php -l includes/utilities.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/utilities.php
git commit -m "fetchVaultRecords: explicit column list including running_balance"
```
---
### Task 4: Switch `vaultLoop` to MAX(running_balance)
**Files:**
- Modify: `includes/utilities.php` (`vaultLoop` lives there; `generateBalance` and `getSpace` already wrap it via `functions.php` — they don't need changes)
**Interfaces:**
- `vaultLoop($name=null)`: returns the per-user vault balance (the most recent entry's `running_balance`) for the given user, or the sum of those across all users when `$name` is null. Goes directly to PDO instead of fetching records.
- `generateBalance($name=null)`: continues to wrap `vaultLoop`, returns the formatted value.
- `getSpace($name=null)`: continues to compute `limit - balance` against the same value.
- [ ] **Step 1: Replace `vaultLoop` with a SQL-based implementation**
In `includes/utilities.php`, replace `vaultLoop` (currently at lines 202-208) with:
```php
function vaultLoop ($name=null) {
$pdo = getDatabaseConnection();
if ($name === null) {
$stmt = $pdo->query(
'SELECT COALESCE(SUM(max_balance), 0) FROM '
. '(SELECT MAX(running_balance) AS max_balance FROM vault GROUP BY user)'
);
return (int)$stmt->fetchColumn();
}
$stmt = $pdo->prepare('SELECT MAX(running_balance) FROM vault WHERE user = :user');
$stmt->bindValue(':user', $name);
$stmt->execute();
return (int)$stmt->fetchColumn();
}
```
- [ ] **Step 2: Verify the file parses**
Run: `php -l includes/utilities.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/utilities.php
git commit -m "vaultLoop: use MAX(running_balance) per user instead of SUM(amount)"
```
---
### Task 5: Add `backfillRunningBalances` helper
**Files:**
- Modify: `includes/utilities.php` (append new function after `consoleLog`)
**Interfaces:**
- `backfillRunningBalances(PDO $pdo, string $user): void`
- Counts rows where `running_balance IS NULL` for the user. If zero, returns. Otherwise hits the API with `selections=log&log=5850,5851&limit=1&sort=DESC` and uses the first entry's `data.balance` to UPDATE all NULL rows for that user.
- [ ] **Step 1: Append the new function**
In `includes/utilities.php`, append this function at the end of the file (after `consoleLog`):
```php
/**
* Backfill the running_balance column for any user rows that have NULL.
*
* The v2 API's `data.balance` field is the vault balance after each entry.
* If the column is missing for a row, treating it as NULL means
* MAX(running_balance) ignores it. This routine fetches the latest entry's
* balance from the API and broadcasts it to all NULL rows of the user,
* so the displayed balance becomes accurate after the first sync following
* the schema upgrade.
*
* @param PDO $pdo Database connection.
* @param string $user The user whose NULL rows should be backfilled.
*
* @return void
*/
function backfillRunningBalances($pdo, $user) {
if (!array_key_exists($user, USER_KEYS)) {
throw new ApiKeyMissingException("User does not have an API key configured.");
}
$countStmt = $pdo->prepare('SELECT COUNT(*) FROM vault WHERE user = :user AND running_balance IS NULL');
$countStmt->bindValue(':user', $user);
$countStmt->execute();
if ((int)$countStmt->fetchColumn() === 0) {
return;
}
$url = 'https://api.torn.com/v2/user?selections=log&log=5850,5851&limit=1&sort=DESC';
$responseData = executeApiCall($url, USER_KEYS[$user]);
validateApiResponse($responseData);
if (empty($responseData['log'])) {
return;
}
$latest = $responseData['log'][0];
if (!isset($latest['data']['balance'])) {
return;
}
$balance = (int)$latest['data']['balance'];
$updateStmt = $pdo->prepare('UPDATE vault SET running_balance = :balance WHERE user = :user AND running_balance IS NULL');
$updateStmt->bindValue(':balance', $balance);
$updateStmt->bindValue(':user', $user);
$updateStmt->execute();
}
```
- [ ] **Step 2: Verify the file parses**
Run: `php -l includes/utilities.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/utilities.php
git commit -m "Add backfillRunningBalances helper for NULL running_balance rows"
```
---
### Task 6: Wire `backfillRunningBalances` into the entry points
**Files:**
- Modify: `functions.php` (`backfillUserLogs` and `syncUserLogs`)
**Interfaces:**
- `backfillUserLogs($user)`: after the pagination loop, call `backfillRunningBalances($pdo, $user)`.
- `syncUserLogs($user)`: after the pagination loop, call `backfillRunningBalances($pdo, $user)`.
- [ ] **Step 1: Update `backfillUserLogs`**
In `functions.php`, replace the body of `backfillUserLogs` (currently at lines 66-74) with:
```php
function backfillUserLogs($user) {
$url = 'https://api.torn.com/v2/user?selections=log&log=5850,5851';
$pdo = getDatabaseConnection();
do {
$next = fetchAndStoreLogPage($pdo, $user, $url);
$url = $next;
} while ($next !== null);
backfillRunningBalances($pdo, $user);
}
```
- [ ] **Step 2: Update `syncUserLogs`**
In `functions.php`, replace the body of `syncUserLogs` (currently at lines 87-106) with:
```php
function syncUserLogs($user) {
if (dbNew()) {
backfillUserLogs($user);
return;
}
$pdo = getDatabaseConnection();
$stmt = $pdo->prepare('SELECT MAX(timestamp) AS max_ts FROM vault WHERE user = :user');
$stmt->bindValue(':user', $user);
$stmt->execute();
$lastTs = (int)$stmt->fetch(PDO::FETCH_ASSOC)['max_ts'];
$url = "https://api.torn.com/v2/user?selections=log&log=5850,5851&from=" . ($lastTs + 1);
do {
$next = fetchAndStoreLogPage($pdo, $user, $url);
$url = $next;
} while ($next !== null);
backfillRunningBalances($pdo, $user);
}
```
- [ ] **Step 3: Verify the file parses**
Run: `php -l functions.php`
Expected: `No syntax errors detected`.
- [ ] **Step 4: Commit**
```bash
git add functions.php
git commit -m "Invoke backfillRunningBalances after sync/backfill"
```
---
### Task 7: Add "Balance After" column to `buildTable`
**Files:**
- Modify: `functions.php` (`buildTable` only)
**Interfaces:**
- `buildTable()`: emits a 5th `<td>` with `number_format($entry['running_balance'], 0)` per row. The `running_balance` field may be NULL on freshly-backfilled rows; render as `—` (em-dash) so the table doesn't show "0" by default.
- [ ] **Step 1: Update `buildTable`**
In `functions.php`, replace the `buildTable` function (currently at lines 155-181) with:
```php
function buildTable () {
$records = fetchVaultRecords();
foreach ($records as $entry) {
$user = $entry['user'];
$timestamp = $entry['timestamp'];
$description = $entry['description'];
$amount = $entry['amount'];
$sign = getSign($amount);
if ($description == 'Vault withdraw') {
$class = 'debit';
$amount = substr($amount, 1); // Remove the negative sign
} else {
$class = 'credit';
}
$amount = number_format($amount, 0); // Format the amount as an integer
$runningBalance = isset($entry['running_balance']) && $entry['running_balance'] !== null
? '$' . number_format((int)$entry['running_balance'], 0)
: '—';
echo '<tr class="'.$class.'">';
echo '<td>'.$user.'</td>';
echo '<td>'.date("F j, Y / H:i", $timestamp).'</td>';
echo '<td>'.$description.'</td>';
echo '<td>'.$sign.$amount.'</td>';
echo '<td>'.$runningBalance.'</td>';
echo '</tr>';
}
}
```
- [ ] **Step 2: Verify the file parses**
Run: `php -l functions.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add functions.php
git commit -m "buildTable: add Balance After column showing per-entry running_balance"
```
---
### Task 8: Add the column header in `index.php`
**Files:**
- Modify: `index.php` (one `<th>` element)
**Interfaces:**
- Just the HTML table header.
- [ ] **Step 1: Add the `<th>Balance After</th>` element**
In `index.php`, after the existing `<th><h4>Amount</h4></th>` (currently at line 69), add:
```php
<th><h4>Balance After</h4></th>
```
The result should read:
```php
<th><h4>User</h4></th>
<th><h4>Date / Time (TCT)</h4></th>
<th><h4>Operation</h4></th>
<th><h4>Amount</h4></th>
<th><h4>Balance After</h4></th>
```
- [ ] **Step 2: Verify the file parses**
Run: `php -l index.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add index.php
git commit -m "index.php: add Balance After column header"
```
---
### Task 9: End-to-end smoke check
**Files:** No code changes - just verification.
- [ ] **Step 1: Run the test suite**
Run: `php tests/process_log_entries_test.php`
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
- [ ] **Step 2: Verify the schema migration survived on the live DB**
Run:
```bash
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
php -r "
\$db = new PDO('sqlite:data/vault.db');
foreach(\$db->query('PRAGMA table_info(vault)') as \$c) { echo \$c['name'] . ' ' . \$c['type'] . PHP_EOL; }
echo 'Total rows: ' . \$db->query('SELECT COUNT(*) FROM vault')->fetchColumn() . PHP_EOL;
echo 'Rows with running_balance after backfill: ' . \$db->query('SELECT COUNT(*) FROM vault WHERE running_balance IS NOT NULL')->fetchColumn() . PHP_EOL;
echo 'Rows with NULL running_balance: ' . \$db->query('SELECT COUNT(*) FROM vault WHERE running_balance IS NULL')->fetchColumn() . PHP_EOL;
"
```
Expected: column list shows `running_balance INTEGER`. After Task 6's backfill has run on the live DB (manually once via the test instance), the NULL count should be 0.
- [ ] **Step 3: Probe the test instance**
Open the test instance in the browser (or `curl http://127.0.0.1:8001/`). The right-side "Balances" panel should show a positive total (`$615,423,760` = $69M Zarathos + $546M Symos), not a negative sum. The transactions table should show a 5th "Balance After" column with `$X` values.
- [ ] **Step 4: Verify no v1 references remain**
Run:
```bash
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
grep -n "firstRun\|?key=" functions.php index.php includes/utilities.php includes/exceptions.php
```
Expected: no matches.
- [ ] **Step 5: Commit if any incidental fixes were needed**
If Step 3 or 4 produced any required fixes, commit them. Otherwise, no commit.
---
## Self-Review Notes
**Spec coverage:**
| Requirement | Task |
|---|---|
| Add `running_balance` column to vault table | Task 1 |
| Idempotent schema migration for existing DBs | Task 1 (PRAGMA check + ALTER TABLE) |
| Capture `data.balance` from v2 API in `processLogEntries` | Task 2 |
| Update INSERT statement in `fetchAndStoreLogPage` | Task 2 |
| Update `fetchVaultRecords` to include new column | Task 3 |
| Replace `SUM(amount)` with `MAX(running_balance)` per user | Task 4 |
| Backfill NULL rows from latest API balance | Task 5 |
| Wire backfill into entry points | Task 6 |
| Add "Balance After" column to `buildTable` | Task 7 |
| Add header `<th>` in `index.php` | Task 8 |
| Test asserts `running_balance` round-trips | Task 2 |
| End-to-end smoke check | Task 9 |
**Placeholder scan:** No "TODO", "TBD", "implement later", or vague instructions.
**Type consistency:** The `backfillRunningBalances(PDO $pdo, string $user): void` signature in Task 5 matches the call sites in Task 6 (`backfillRunningBalances($pdo, $user)` in both `backfillUserLogs` and `syncUserLogs`). The `vaultLoop` SQL in Task 4 uses `MAX(running_balance)` for the per-user case and `SUM(max_balance)` over a per-user MAX subquery for the all-users case — both return int, matching the existing `(int)` cast in `generateBalance`/`getSpace`. The `fetchAndStoreLogPage` INSERT in Task 2 includes `:running_balance` matching the new bind in `processLogEntries`.
**Risks addressed:**
- The schema migration is idempotent (PRAGMA check) and safe for existing DBs.
- The backfill is a no-op when no NULL rows exist for a user.
- The `running_balance` column is nullable, so entries missing `data.balance` from the API don't break the insert.
- All existing SQL (purported to depend on the table schema) keeps working because the new column is `INTEGER` with null default.