From e7ec343be1de64841892ef1d74e9b2a84642deaa Mon Sep 17 00:00:00 2001 From: Keith Solomon Date: Mon, 3 Aug 2026 12:47:59 -0500 Subject: [PATCH] Add implementation plan: Torn API v1 to v2 migration - 8 tasks: exception class, API call rewrite, validation tightening, fetchAndStoreLogPage helper, processLogEntries rewrite with golden-file test, backfillUserLogs/syncUserLogs entry points, index.php call swap, end-to-end smoke check - TDD: write the failing test before rewriting processLogEntries - Each task is a self-contained commit --- .../plans/2026-08-03-torn-api-v2-migration.md | 604 ++++++++++++++++++ 1 file changed, 604 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-03-torn-api-v2-migration.md diff --git a/docs/superpowers/plans/2026-08-03-torn-api-v2-migration.md b/docs/superpowers/plans/2026-08-03-torn-api-v2-migration.md new file mode 100644 index 0000000..a4b7af6 --- /dev/null +++ b/docs/superpowers/plans/2026-08-03-torn-api-v2-migration.md @@ -0,0 +1,604 @@ +# Torn API v1 → v2 Migration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Rewrite Torn Vault Tracker's log-ingestion code to consume the new v2 Torn API JSON shape (array-of-entries with `_metadata.links.next` pagination and `Authorization: ApiKey` header), with a single shared page-fetch helper and a golden-file unit test. + +**Architecture:** All v2-aware behavior lives in `includes/utilities.php` (`executeApiCall`, `validateApiResponse`, `processLogEntries`, plus a new `fetchAndStoreLogPage` helper). `functions.php` exposes two thin entry points — `backfillUserLogs` (full-history pagination on empty DB) and `syncUserLogs` (incremental fetch from last seen timestamp). `index.php` swaps its call sites. The only new behavior is in the helper module; everything else is either renamed or a one-line swap. + +**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies. No framework. Pure CLI test runner (`php tests/process_log_entries_test.php`) using the existing `backup/api-sample-new.json` fixture. + +## Global Constraints + +- PHP 8.1+ syntax only. +- Database schema (`vault` table with `id TEXT PRIMARY KEY`) is **not** changed. +- `INSERT … ON CONFLICT(id) DO NOTHING` is the only INSERT form used for log entries — operation must be idempotent across re-runs. +- API authentication: `Authorization: ApiKey ` header; **never** pass the key as a `?key=` query parameter. +- Pagination: follow `_metadata.links.next` URL until null. Never compute timestamps manually for pagination; v2 provides the cursor. +- Use the fixture at `backup/api-sample-new.json` (100 entries) for the golden-file test. +- All existing helpers outside the rewritten ones stay byte-identical (no drive-by refactors). +- Commit after each task. Do not push. + +--- + +### Task 1: Add `LogEntryIncompleteException` + +**Files:** +- Modify: `includes/exceptions.php` (append new class after the existing four) +- No test file — exercised via Task 5's golden test. + +**Interfaces:** +- Consumes: none. +- Produces: a new exception class `LogEntryIncompleteException extends Exception`, with the same constructor signature as the existing four exception classes in the file. + +- [ ] **Step 1: Append the new class to `includes/exceptions.php`** + +Append after the existing `ApiValidationException` class, following the same doc-comment + constructor pattern used by the other four classes in the file: + +```php +/** + * Exception thrown when a log entry is missing required data fields. + * + * @category Exception + * @package TornVaultTracker + * @author Keith Solomon + * @license Unlicense https://unlicense.org/ + * @link https://github.com/ksolomon/Torn-Vault-Tracker + */ +class LogEntryIncompleteException extends Exception { + /** + * Constructor. + * + * @param string $message The exception message. + * @param int $code The exception code. + * @param Throwable $previous The previous throwable. + */ + public function __construct($message, $code = 0, Throwable $previous = null) { + parent::__construct($message, $code, $previous); + } +} +``` + +- [ ] **Step 2: Verify file still parses** + +Run: `php -l includes/exceptions.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 3: Commit** + +```bash +git add includes/exceptions.php +git commit -m "Add LogEntryIncompleteException for malformed v2 entries" +``` + +--- + +### Task 2: Rewrite `executeApiCall` for v2 authentication + +**Files:** +- Modify: `includes/utilities.php` (replace the existing `executeApiCall` function body) +- No new test file for this task alone — exercised by Task 5's golden test and by the integration in Task 4. + +**Interfaces:** +- Consumes: `$apiEndpoint` (full URL), `$apiKey` (string). Uses `\Exception` (existing) for `CurlErrorException` and `JsonDataException` from `includes/exceptions.php`. +- Produces: `array` — the decoded JSON response, or throws `CurlErrorException` / `JsonDataException`. Header is `Authorization: ApiKey ` only (no `Content-Type`). + +- [ ] **Step 1: Replace `executeApiCall` in `includes/utilities.php`** + +Replace the existing function (currently takes only `$apiEndpoint`) with this v2-aware version: + +```php +function executeApiCall($apiEndpoint, $apiKey) { + if (empty($apiKey)) { + throw new ApiKeyMissingException('API key is required for executeApiCall.'); + } + + $headers = ["Authorization: ApiKey $apiKey"]; + $ch = curl_init(); + curl_setopt($ch, CURLOPT_URL, $apiEndpoint); + curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); + curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); + + $response = curl_exec($ch); + + if (curl_errno($ch)) { + throw new CurlErrorException('cURL error: ' . curl_error($ch)); + } + + curl_close($ch); + + $responseData = json_decode($response, true); + + if (json_last_error() !== JSON_ERROR_NONE) { + throw new JsonDataException('Failed to decode JSON response: ' . json_last_error_msg()); + } + + return $responseData; +} +``` + +- [ ] **Step 2: Verify file still parses** + +Run: `php -l includes/utilities.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 3: Commit** + +```bash +git add includes/utilities.php +git commit -m "Rewrite executeApiCall for v2 API (Authorization header)" +``` + +--- + +### Task 3: Tighten `validateApiResponse` for v2 shape + +**Files:** +- Modify: `includes/utilities.php` (replace the existing `validateApiResponse` function body) +- No test file for this task alone — exercised by Task 5's golden test. + +**Interfaces:** +- Consumes: `$responseData` (array). Throws `ApiValidationException` (already defined). +- Produces: void. `log` must exist and be an array. + +- [ ] **Step 1: Replace `validateApiResponse` in `includes/utilities.php`** + +Replace the existing function with: + +```php +function validateApiResponse($responseData) { + if (!isset($responseData['log']) || !is_array($responseData['log'])) { + throw new ApiValidationException('Invalid log data received from the API.'); + } +} +``` + +(The change from the old version is tightening `is_array($responseData['log'])` — v2 always returns an array; v1 returned an object-keyed map, which `is_array()` rejects.) + +- [ ] **Step 2: Verify file still parses** + +Run: `php -l includes/utilities.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 3: Commit** + +```bash +git add includes/utilities.php +git commit -m "Tighten validateApiResponse for v2 array shape" +``` + +--- + +### Task 4: Add `fetchAndStoreLogPage` helper + +**Files:** +- Modify: `includes/utilities.php` (append new function after `processLogEntries`) +- No test file for this task alone — exercised by Task 5 and the integration tests in Task 7. + +**Interfaces:** +- Consumes: `$pdo` (PDO), `$user` (string — key into `USER_KEYS`), `$url` (string — full v2 API URL, possibly a `_metadata.links.next` URL). Throws `ApiKeyMissingException`, `CurlErrorException`, `JsonDataException`, `ApiValidationException` (all already defined). +- Produces: `?string` — the next page URL from `_metadata.links.next`, or `null` if no more pages. + +- [ ] **Step 1: Add the new function to `includes/utilities.php`** + +Append the following at the end of the file (after `consoleLog()`): + +```php +/** + * Fetch a single page of log entries from the Torn v2 API and store them. + * + * Performs one HTTP request, validates the response, inserts each vault + * log entry (idempotently via INSERT ... ON CONFLICT), and returns the + * pagination cursor (`_metadata.links.next`) if more pages remain. + * + * @param PDO $pdo Database connection used to insert vault entries. + * @param string $user The user whose log entries are being fetched. + * @param string $url Full URL for the v2 API request. + * + * @throws ApiKeyMissingException If no API key is configured for the user. + * @throws CurlErrorException If the HTTP request fails. + * @throws JsonDataException If the response body is not valid JSON. + * @throws ApiValidationException If the response is missing the `log` array. + * + * @return string|null The `_metadata.links.next` URL, or null when there are no more pages. + */ +function fetchAndStoreLogPage($pdo, $user, $url) { + if (!array_key_exists($user, USER_KEYS)) { + throw new ApiKeyMissingException("User does not have an API key configured."); + } + + $responseData = executeApiCall($url, USER_KEYS[$user]); + validateApiResponse($responseData); + + $insertStmt = $pdo->prepare( + 'INSERT INTO vault (id, user, timestamp, description, amount) ' + . 'VALUES (:id, :user, :timestamp, :description, :amount) ' + . 'ON CONFLICT(id) DO NOTHING' + ); + + processLogEntries($responseData['log'], $user, $insertStmt); + + return $responseData['_metadata']['links']['next'] ?? null; +} +``` + +- [ ] **Step 2: Verify file still parses** + +Run: `php -l includes/utilities.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 3: Commit** + +```bash +git add includes/utilities.php +git commit -m "Add fetchAndStoreLogPage helper for v2 pagination" +``` + +--- + +### Task 5: Rewrite `processLogEntries` for v2 shape (with golden-file test) + +**Files:** +- Create: `tests/process_log_entries_test.php` +- Modify: `includes/utilities.php` (replace the existing `processLogEntries` function body) + +**Interfaces:** +- Consumes: `$logEntries` (array of v2 entry objects), `$user` (string), `$insertStmt` (PDOStatement prepared with the `INSERT … ON CONFLICT(id) DO NOTHING` query). Throws `LogEntryIncompleteException` (new in Task 1). +- Produces: void. For each entry: extracts `id`, `timestamp`, `details.title`, `data.deposited` (deposit) or `data.withdrawn` (withdraw) and binds to `$insertStmt`, then `execute()`. Skips entries missing both `data.deposited` and `data.withdrawn` after logging via `consoleLog`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/process_log_entries_test.php`: + +```php + 'test-key-not-used', + 'symos' => 'test-key-not-used', + ]); +} + +require_once __DIR__ . '/../includes/exceptions.php'; +require_once __DIR__ . '/../includes/utilities.php'; + +function fail(string $message): void { + fwrite(STDERR, "FAIL: $message\n"); + exit(1); +} + +function assertSame($expected, $actual, string $label): void { + if ($expected !== $actual) { + $exp = var_export($expected, true); + $act = var_export($actual, true); + fail("$label: expected $exp, got $act"); + } +} + +// In-memory DB so we don't touch the user's vault.db. +$pdo = new PDO('sqlite::memory:'); +$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); +$pdo->exec( + 'CREATE TABLE vault ( + id TEXT PRIMARY KEY, + user TEXT NOT NULL, + timestamp INTEGER NOT NULL, + description TEXT NOT NULL, + amount REAL NOT NULL + )' +); + +$insertStmt = $pdo->prepare( + 'INSERT INTO vault (id, user, timestamp, description, amount) ' + . 'VALUES (:id, :user, :timestamp, :description, :amount) ' + . 'ON CONFLICT(id) DO NOTHING' +); + +// Load the golden file. +$fixturePath = __DIR__ . '/../backup/api-sample-new.json'; +if (!is_readable($fixturePath)) { + fail("Fixture not readable at $fixturePath"); +} +$fixture = json_decode(file_get_contents($fixturePath), true); +if (!is_array($fixture) || !isset($fixture['log'])) { + fail('Fixture is missing the log array.'); +} + +processLogEntries($fixture['log'], 'zarathos', $insertStmt); + +// 100 entries in the fixture. +$count = (int)$pdo->query('SELECT COUNT(*) FROM vault')->fetchColumn(); +assertSame(100, $count, 'row count'); + +// Spot-check first entry (newest, withdraw) and last (oldest, deposit). +$rows = $pdo->query('SELECT id, user, timestamp, description, amount FROM vault ORDER BY timestamp DESC')->fetchAll(PDO::FETCH_ASSOC); + +$first = $rows[0]; +assertSame('c1pEDQ7jl3kuBHV4NqDI', $first['id'], 'first.id'); +assertSame('zarathos', $first['user'], 'first.user'); +assertSame(1785325489, (int)$first['timestamp'], 'first.timestamp'); +assertSame('Vault withdraw', $first['description'], 'first.description'); +assertSame(-498870964, (int)$first['amount'], 'first.amount'); + +$last = $rows[count($rows) - 1]; +assertSame('8m5jrUKTiKk0SB4MxzIy', $last['id'], 'last.id'); +assertSame(1714350415, (int)$last['timestamp'], 'last.timestamp'); +assertSame('Vault deposit', $last['description'], 'last.description'); +assertSame(231492, (int)$last['amount'], 'last.amount'); + +// Re-running with the same fixture inserts zero new rows (idempotency). +processLogEntries($fixture['log'], 'zarathos', $insertStmt); +$count2 = (int)$pdo->query('SELECT COUNT(*) FROM vault')->fetchColumn(); +assertSame(100, $count2, 'row count after re-run'); + +echo "OK: processLogEntries v2 golden-file test passed (100 entries).\n"; +``` + +- [ ] **Step 2: Run the test and verify it fails** + +Run: `php tests/process_log_entries_test.php` +Expected: PHP fatal error: `processLogEntries()` exists but reads `$entry['log']` (v1) and crashes when `log` is not a key — or returns zero rows. Either way the test will not print `OK: …`. + +- [ ] **Step 3: Rewrite `processLogEntries` in `includes/utilities.php`** + +Replace the existing `processLogEntries` function with: + +```php +function processLogEntries($logEntries, $user, $insertStmt) { + foreach ($logEntries as $entry) { + $id = $entry['id'] ?? null; + + if (!$id) { + consoleLog('Skipping entry with no id: ' . print_r($entry, true)); + continue; + } + + $timestamp = $entry['timestamp'] ?? null; + $description = $entry['details']['title'] ?? null; + $hasDeposit = isset($entry['data']['deposited']); + $hasWithdraw = isset($entry['data']['withdrawn']); + + if ($hasDeposit) { + $amount = (int)$entry['data']['deposited']; + } elseif ($hasWithdraw) { + $amount = -((int)$entry['data']['withdrawn']); + } else { + consoleLog('Skipping entry ' . $id . ' with no deposited/withdrawn: ' . print_r($entry, true)); + continue; + } + + if ($timestamp === null || $description === null) { + throw new LogEntryIncompleteException( + "Entry $id missing timestamp or details.title." + ); + } + + $insertStmt->bindValue(':id', $id); + $insertStmt->bindValue(':user', $user); + $insertStmt->bindValue(':timestamp', $timestamp); + $insertStmt->bindValue(':description', $description); + $insertStmt->bindValue(':amount', $amount); + $insertStmt->execute(); + } +} +``` + +Note: signature changed from `(array, string, PDOStatement, PDOStatement, bool)` to `(array, string, PDOStatement)`. The two statements (check/insert) collapsed into one ON CONFLICT insert, the `debug` flag is gone (debug logging was unused), and the `$debug` boolean parameter is dropped. + +- [ ] **Step 4: Run the test and verify it passes** + +Run: `php tests/process_log_entries_test.php` +Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).` + +- [ ] **Step 5: Commit** + +```bash +git add tests/process_log_entries_test.php includes/utilities.php +git commit -m "Rewrite processLogEntries for v2 shape; add golden-file test" +``` + +--- + +### Task 6: Replace `firstRun`/`getLog` with `backfillUserLogs`/`syncUserLogs` + +**Files:** +- Modify: `functions.php` (remove `firstRun`, remove `getLog`, add `backfillUserLogs` and `syncUserLogs`) +- No new test file — verified end-to-end in Task 8. + +**Interfaces:** +- Consumes: `$user` (string — key into `USER_KEYS`). Uses `getDatabaseConnection()`, `dbNew()`, `fetchAndStoreLogPage()` (Task 4). +- Produces: `void`. Both functions are idempotent. + +- [ ] **Step 1: Remove `firstRun` from `functions.php`** + +Delete the entire `firstRun()` function (lines 72-119 in the current file, including the doc-comment block above it). + +- [ ] **Step 2: Remove `getLog` from `functions.php`** + +Delete the entire `getLog()` function (lines 132-153 in the current file, including the doc-comment block above it). + +- [ ] **Step 3: Add `backfillUserLogs` and `syncUserLogs`** + +Insert the following in place of the deleted functions: + +```php +/** + * Pulls and stores the full vault transaction log history for a user. + * + * Pages through the v2 Torn API using `_metadata.links.next` until the + * API reports no further pages. Intended for first-run use when the + * local database is empty. Idempotent: re-running on a partially + * populated database inserts only new entries (ON CONFLICT DO NOTHING). + * + * @param string $user The user whose logs should be fetched. + * + * @return void + */ +function backfillUserLogs($user) { + $url = 'https://api.torn.com/v2/user?selections=log&log=5850,5851'; + $pdo = getDatabaseConnection(); + + do { + $next = fetchAndStoreLogPage($pdo, $user, $url); + $url = $next; + } while ($next !== null); +} + +/** + * Synchronizes recent vault transactions for a user. + * + * On a non-empty database, fetches only entries newer than the user's + * most recent row. Falls back to full backfill when the database is + * empty. + * + * @param string $user The user whose logs should be synced. + * + * @return void + */ +function syncUserLogs($user) { + if (dbNew()) { + backfillUserLogs($user); + return; + } + + $pdo = getDatabaseConnection(); + + $stmt = $pdo->prepare('SELECT MAX(timestamp) AS max_ts FROM vault WHERE user = :user'); + $stmt->bindValue(':user', $user); + $stmt->execute(); + $lastTs = (int)$stmt->fetch(PDO::FETCH_ASSOC)['max_ts']; + + $url = "https://api.torn.com/v2/user?selections=log&log=5850,5851&from=" . ($lastTs + 1); + + do { + $next = fetchAndStoreLogPage($pdo, $user, $url); + $url = $next; + } while ($next !== null); +} +``` + +- [ ] **Step 4: Verify the file still parses** + +Run: `php -l functions.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 5: Commit** + +```bash +git add functions.php +git commit -m "Replace firstRun/getLog with backfillUserLogs/syncUserLogs" +``` + +--- + +### Task 7: Update `index.php` call sites + +**Files:** +- Modify: `index.php` (rename two function calls in the `dbNew()` branches) + +**Interfaces:** +- Consumes: existing `index.php` control flow (calls into `firstRun` / `getLog`). +- Produces: same control flow, but calling `backfillUserLogs` / `syncUserLogs` instead. + +- [ ] **Step 1: Replace `firstRun` with `backfillUserLogs`** + +In `index.php`, replace `firstRun($key)` with `backfillUserLogs($key)`. + +The result should read: + +```php +if (dbNew()) { + foreach (USER_KEYS as $key => $value) { + backfillUserLogs($key); + } + header('Location: /'); +} else { + foreach (USER_KEYS as $key => $value) { + syncUserLogs($key); + } +} +``` + +- [ ] **Step 2: Replace `getLog` with `syncUserLogs`** + +In the same file, in the `else` branch, replace `getLog($key)` with `syncUserLogs($key)`. + +- [ ] **Step 3: Verify the file still parses** + +Run: `php -l index.php` +Expected: `No syntax errors detected`. + +- [ ] **Step 4: Commit** + +```bash +git add index.php +git commit -m "Update index.php to call backfillUserLogs/syncUserLogs" +``` + +--- + +### Task 8: End-to-end smoke check + +**Files:** +- No code changes. Just verification steps. + +- [ ] **Step 1: Run the unit test once more** + +Run: `php tests/process_log_entries_test.php` +Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).` + +- [ ] **Step 2: Verify no remaining references to the v1 functions** + +Run: + +```bash +cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker" +grep -rn "firstRun\|?key=\|api.torn.com/user/" --include="*.php" . +``` + +Expected: no matches (except possibly `backup/` reference files, which are read-only fixtures). The grep should report nothing under `functions.php`, `index.php`, `includes/`, or `tests/`. + +- [ ] **Step 3: (Optional) Run `phpcs` against the touched files** + +Run: `phpcs functions.php index.php includes/utilities.php includes/exceptions.php tests/process_log_entries_test.php` + +If `phpcs` is installed and clean: skip Step 4. If it reports issues, fix them and commit. (The project's existing `phpcs.xml` is the source of style rules; there is no CI check, so this is a manual sweep.) + +--- + +## Self-Review Notes + +**Spec coverage:** + +| Spec requirement | Task | +|---|---| +| Single v2 path for both bootstrap and steady-state | Task 6 (single `fetchAndStoreLogPage` consumed by both) | +| `Authorization: ApiKey` header, no `?key=` | Task 2 | +| `_metadata.links.next` pagination | Task 4 (helper returns it) + Task 6 (loop in both entry points) | +| Entry id as `TEXT PRIMARY KEY` with `ON CONFLICT DO NOTHING` | Task 4 (insert stmt) + Task 5 (test asserts idempotency) | +| LogEntryIncompleteException for malformed entries | Task 1 (class) + Task 5 (throw site) | +| Golden-file test against `backup/api-sample-new.json` | Task 5 | +| `firstRun` → `backfillUserLogs`, `getLog` → `syncUserLogs` | Task 6 (definitions) + Task 7 (call sites) | +| DB schema unchanged, UI helpers unchanged | Tasks 6 & 7 only swap the log-fetching entry points; `buildTable`, `generateBalance`, `getSpace`, `vaultLoop`, `fetchVaultRecords`, `dbNew`, `getDatabaseConnection` all untouched. | +| Tests verified manually via `php tests/...` | Task 5 (initial run) + Task 8 (final run) | +| `config.php`, `style.css`, `script.js`, `backup/`, README untouched | No task touches them. (Spec said README update was optional; we skip it.) | + +**Placeholder scan:** No "TODO", "TBD", "implement later", or vague instructions in the plan. + +**Type consistency:** The `executeApiCall(string $url, string $apiKey)` signature in Task 2 matches the call site `executeApiCall($url, USER_KEYS[$user])` in Task 4. The `fetchAndStoreLogPage(PDO $pdo, string $user, string $url): ?string` signature in Task 4 matches the loop body in both `backfillUserLogs` and `syncUserLogs` in Task 6. The `processLogEntries(array, string, PDOStatement)` signature in Task 5 matches the call site `processLogEntries($responseData['log'], $user, $insertStmt)` in Task 4. The `dbNew()` function called in Task 6 is unchanged and already exists in `includes/utilities.php`. \ No newline at end of file