Files
Torn-Vault-Tracker/docs/superpowers/plans/2026-08-03-torn-api-v2-migration.md
Keith Solomon e7ec343be1 Add implementation plan: Torn API v1 to v2 migration
- 8 tasks: exception class, API call rewrite, validation tightening,
  fetchAndStoreLogPage helper, processLogEntries rewrite with golden-file
  test, backfillUserLogs/syncUserLogs entry points, index.php call swap,
  end-to-end smoke check
- TDD: write the failing test before rewriting processLogEntries
- Each task is a self-contained commit
2026-08-03 12:47:59 -05:00

604 lines
22 KiB
Markdown

# Torn API v1 → v2 Migration Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Rewrite Torn Vault Tracker's log-ingestion code to consume the new v2 Torn API JSON shape (array-of-entries with `_metadata.links.next` pagination and `Authorization: ApiKey` header), with a single shared page-fetch helper and a golden-file unit test.
**Architecture:** All v2-aware behavior lives in `includes/utilities.php` (`executeApiCall`, `validateApiResponse`, `processLogEntries`, plus a new `fetchAndStoreLogPage` helper). `functions.php` exposes two thin entry points — `backfillUserLogs` (full-history pagination on empty DB) and `syncUserLogs` (incremental fetch from last seen timestamp). `index.php` swaps its call sites. The only new behavior is in the helper module; everything else is either renamed or a one-line swap.
**Tech Stack:** PHP 8.1+, SQLite3 via PDO, cURL. No new dependencies. No framework. Pure CLI test runner (`php tests/process_log_entries_test.php`) using the existing `backup/api-sample-new.json` fixture.
## Global Constraints
- PHP 8.1+ syntax only.
- Database schema (`vault` table with `id TEXT PRIMARY KEY`) is **not** changed.
- `INSERT … ON CONFLICT(id) DO NOTHING` is the only INSERT form used for log entries — operation must be idempotent across re-runs.
- API authentication: `Authorization: ApiKey <key>` header; **never** pass the key as a `?key=` query parameter.
- Pagination: follow `_metadata.links.next` URL until null. Never compute timestamps manually for pagination; v2 provides the cursor.
- Use the fixture at `backup/api-sample-new.json` (100 entries) for the golden-file test.
- All existing helpers outside the rewritten ones stay byte-identical (no drive-by refactors).
- Commit after each task. Do not push.
---
### Task 1: Add `LogEntryIncompleteException`
**Files:**
- Modify: `includes/exceptions.php` (append new class after the existing four)
- No test file — exercised via Task 5's golden test.
**Interfaces:**
- Consumes: none.
- Produces: a new exception class `LogEntryIncompleteException extends Exception`, with the same constructor signature as the existing four exception classes in the file.
- [ ] **Step 1: Append the new class to `includes/exceptions.php`**
Append after the existing `ApiValidationException` class, following the same doc-comment + constructor pattern used by the other four classes in the file:
```php
/**
* Exception thrown when a log entry is missing required data fields.
*
* @category Exception
* @package TornVaultTracker
* @author Keith Solomon <ksolomon@gmail.com>
* @license Unlicense https://unlicense.org/
* @link https://github.com/ksolomon/Torn-Vault-Tracker
*/
class LogEntryIncompleteException extends Exception {
/**
* Constructor.
*
* @param string $message The exception message.
* @param int $code The exception code.
* @param Throwable $previous The previous throwable.
*/
public function __construct($message, $code = 0, Throwable $previous = null) {
parent::__construct($message, $code, $previous);
}
}
```
- [ ] **Step 2: Verify file still parses**
Run: `php -l includes/exceptions.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/exceptions.php
git commit -m "Add LogEntryIncompleteException for malformed v2 entries"
```
---
### Task 2: Rewrite `executeApiCall` for v2 authentication
**Files:**
- Modify: `includes/utilities.php` (replace the existing `executeApiCall` function body)
- No new test file for this task alone — exercised by Task 5's golden test and by the integration in Task 4.
**Interfaces:**
- Consumes: `$apiEndpoint` (full URL), `$apiKey` (string). Uses `\Exception` (existing) for `CurlErrorException` and `JsonDataException` from `includes/exceptions.php`.
- Produces: `array` — the decoded JSON response, or throws `CurlErrorException` / `JsonDataException`. Header is `Authorization: ApiKey <apiKey>` only (no `Content-Type`).
- [ ] **Step 1: Replace `executeApiCall` in `includes/utilities.php`**
Replace the existing function (currently takes only `$apiEndpoint`) with this v2-aware version:
```php
function executeApiCall($apiEndpoint, $apiKey) {
if (empty($apiKey)) {
throw new ApiKeyMissingException('API key is required for executeApiCall.');
}
$headers = ["Authorization: ApiKey $apiKey"];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $apiEndpoint);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$response = curl_exec($ch);
if (curl_errno($ch)) {
throw new CurlErrorException('cURL error: ' . curl_error($ch));
}
curl_close($ch);
$responseData = json_decode($response, true);
if (json_last_error() !== JSON_ERROR_NONE) {
throw new JsonDataException('Failed to decode JSON response: ' . json_last_error_msg());
}
return $responseData;
}
```
- [ ] **Step 2: Verify file still parses**
Run: `php -l includes/utilities.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/utilities.php
git commit -m "Rewrite executeApiCall for v2 API (Authorization header)"
```
---
### Task 3: Tighten `validateApiResponse` for v2 shape
**Files:**
- Modify: `includes/utilities.php` (replace the existing `validateApiResponse` function body)
- No test file for this task alone — exercised by Task 5's golden test.
**Interfaces:**
- Consumes: `$responseData` (array). Throws `ApiValidationException` (already defined).
- Produces: void. `log` must exist and be an array.
- [ ] **Step 1: Replace `validateApiResponse` in `includes/utilities.php`**
Replace the existing function with:
```php
function validateApiResponse($responseData) {
if (!isset($responseData['log']) || !is_array($responseData['log'])) {
throw new ApiValidationException('Invalid log data received from the API.');
}
}
```
(The change from the old version is tightening `is_array($responseData['log'])` — v2 always returns an array; v1 returned an object-keyed map, which `is_array()` rejects.)
- [ ] **Step 2: Verify file still parses**
Run: `php -l includes/utilities.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/utilities.php
git commit -m "Tighten validateApiResponse for v2 array shape"
```
---
### Task 4: Add `fetchAndStoreLogPage` helper
**Files:**
- Modify: `includes/utilities.php` (append new function after `processLogEntries`)
- No test file for this task alone — exercised by Task 5 and the integration tests in Task 7.
**Interfaces:**
- Consumes: `$pdo` (PDO), `$user` (string — key into `USER_KEYS`), `$url` (string — full v2 API URL, possibly a `_metadata.links.next` URL). Throws `ApiKeyMissingException`, `CurlErrorException`, `JsonDataException`, `ApiValidationException` (all already defined).
- Produces: `?string` — the next page URL from `_metadata.links.next`, or `null` if no more pages.
- [ ] **Step 1: Add the new function to `includes/utilities.php`**
Append the following at the end of the file (after `consoleLog()`):
```php
/**
* Fetch a single page of log entries from the Torn v2 API and store them.
*
* Performs one HTTP request, validates the response, inserts each vault
* log entry (idempotently via INSERT ... ON CONFLICT), and returns the
* pagination cursor (`_metadata.links.next`) if more pages remain.
*
* @param PDO $pdo Database connection used to insert vault entries.
* @param string $user The user whose log entries are being fetched.
* @param string $url Full URL for the v2 API request.
*
* @throws ApiKeyMissingException If no API key is configured for the user.
* @throws CurlErrorException If the HTTP request fails.
* @throws JsonDataException If the response body is not valid JSON.
* @throws ApiValidationException If the response is missing the `log` array.
*
* @return string|null The `_metadata.links.next` URL, or null when there are no more pages.
*/
function fetchAndStoreLogPage($pdo, $user, $url) {
if (!array_key_exists($user, USER_KEYS)) {
throw new ApiKeyMissingException("User does not have an API key configured.");
}
$responseData = executeApiCall($url, USER_KEYS[$user]);
validateApiResponse($responseData);
$insertStmt = $pdo->prepare(
'INSERT INTO vault (id, user, timestamp, description, amount) '
. 'VALUES (:id, :user, :timestamp, :description, :amount) '
. 'ON CONFLICT(id) DO NOTHING'
);
processLogEntries($responseData['log'], $user, $insertStmt);
return $responseData['_metadata']['links']['next'] ?? null;
}
```
- [ ] **Step 2: Verify file still parses**
Run: `php -l includes/utilities.php`
Expected: `No syntax errors detected`.
- [ ] **Step 3: Commit**
```bash
git add includes/utilities.php
git commit -m "Add fetchAndStoreLogPage helper for v2 pagination"
```
---
### Task 5: Rewrite `processLogEntries` for v2 shape (with golden-file test)
**Files:**
- Create: `tests/process_log_entries_test.php`
- Modify: `includes/utilities.php` (replace the existing `processLogEntries` function body)
**Interfaces:**
- Consumes: `$logEntries` (array of v2 entry objects), `$user` (string), `$insertStmt` (PDOStatement prepared with the `INSERT … ON CONFLICT(id) DO NOTHING` query). Throws `LogEntryIncompleteException` (new in Task 1).
- Produces: void. For each entry: extracts `id`, `timestamp`, `details.title`, `data.deposited` (deposit) or `data.withdrawn` (withdraw) and binds to `$insertStmt`, then `execute()`. Skips entries missing both `data.deposited` and `data.withdrawn` after logging via `consoleLog`.
- [ ] **Step 1: Write the failing test**
Create `tests/process_log_entries_test.php`:
```php
<?php
/**
* Golden-file test for processLogEntries() v2 behavior.
*
* Loads backup/api-sample-new.json, runs each entry through the v2
* processor against an in-memory SQLite database, and asserts the
* resulting rows match the expected (id, user, timestamp, description,
* amount) tuples derived from the fixture.
*
* Run: php tests/process_log_entries_test.php
* Exit code 0 = pass.
*/
declare(strict_types=1);
// Make USER_KEYS available without including config.php (it would force a
// real DB connection).
if (!defined('USER_KEYS')) {
define('USER_KEYS', [
'zarathos' => 'test-key-not-used',
'symos' => 'test-key-not-used',
]);
}
require_once __DIR__ . '/../includes/exceptions.php';
require_once __DIR__ . '/../includes/utilities.php';
function fail(string $message): void {
fwrite(STDERR, "FAIL: $message\n");
exit(1);
}
function assertSame($expected, $actual, string $label): void {
if ($expected !== $actual) {
$exp = var_export($expected, true);
$act = var_export($actual, true);
fail("$label: expected $exp, got $act");
}
}
// In-memory DB so we don't touch the user's vault.db.
$pdo = new PDO('sqlite::memory:');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec(
'CREATE TABLE vault (
id TEXT PRIMARY KEY,
user TEXT NOT NULL,
timestamp INTEGER NOT NULL,
description TEXT NOT NULL,
amount REAL NOT NULL
)'
);
$insertStmt = $pdo->prepare(
'INSERT INTO vault (id, user, timestamp, description, amount) '
. 'VALUES (:id, :user, :timestamp, :description, :amount) '
. 'ON CONFLICT(id) DO NOTHING'
);
// Load the golden file.
$fixturePath = __DIR__ . '/../backup/api-sample-new.json';
if (!is_readable($fixturePath)) {
fail("Fixture not readable at $fixturePath");
}
$fixture = json_decode(file_get_contents($fixturePath), true);
if (!is_array($fixture) || !isset($fixture['log'])) {
fail('Fixture is missing the log array.');
}
processLogEntries($fixture['log'], 'zarathos', $insertStmt);
// 100 entries in the fixture.
$count = (int)$pdo->query('SELECT COUNT(*) FROM vault')->fetchColumn();
assertSame(100, $count, 'row count');
// Spot-check first entry (newest, withdraw) and last (oldest, deposit).
$rows = $pdo->query('SELECT id, user, timestamp, description, amount FROM vault ORDER BY timestamp DESC')->fetchAll(PDO::FETCH_ASSOC);
$first = $rows[0];
assertSame('c1pEDQ7jl3kuBHV4NqDI', $first['id'], 'first.id');
assertSame('zarathos', $first['user'], 'first.user');
assertSame(1785325489, (int)$first['timestamp'], 'first.timestamp');
assertSame('Vault withdraw', $first['description'], 'first.description');
assertSame(-498870964, (int)$first['amount'], 'first.amount');
$last = $rows[count($rows) - 1];
assertSame('8m5jrUKTiKk0SB4MxzIy', $last['id'], 'last.id');
assertSame(1714350415, (int)$last['timestamp'], 'last.timestamp');
assertSame('Vault deposit', $last['description'], 'last.description');
assertSame(231492, (int)$last['amount'], 'last.amount');
// Re-running with the same fixture inserts zero new rows (idempotency).
processLogEntries($fixture['log'], 'zarathos', $insertStmt);
$count2 = (int)$pdo->query('SELECT COUNT(*) FROM vault')->fetchColumn();
assertSame(100, $count2, 'row count after re-run');
echo "OK: processLogEntries v2 golden-file test passed (100 entries).\n";
```
- [ ] **Step 2: Run the test and verify it fails**
Run: `php tests/process_log_entries_test.php`
Expected: PHP fatal error: `processLogEntries()` exists but reads `$entry['log']` (v1) and crashes when `log` is not a key — or returns zero rows. Either way the test will not print `OK: …`.
- [ ] **Step 3: Rewrite `processLogEntries` in `includes/utilities.php`**
Replace the existing `processLogEntries` function with:
```php
function processLogEntries($logEntries, $user, $insertStmt) {
foreach ($logEntries as $entry) {
$id = $entry['id'] ?? null;
if (!$id) {
consoleLog('Skipping entry with no id: ' . print_r($entry, true));
continue;
}
$timestamp = $entry['timestamp'] ?? null;
$description = $entry['details']['title'] ?? null;
$hasDeposit = isset($entry['data']['deposited']);
$hasWithdraw = isset($entry['data']['withdrawn']);
if ($hasDeposit) {
$amount = (int)$entry['data']['deposited'];
} elseif ($hasWithdraw) {
$amount = -((int)$entry['data']['withdrawn']);
} else {
consoleLog('Skipping entry ' . $id . ' with no deposited/withdrawn: ' . print_r($entry, true));
continue;
}
if ($timestamp === null || $description === null) {
throw new LogEntryIncompleteException(
"Entry $id missing timestamp or details.title."
);
}
$insertStmt->bindValue(':id', $id);
$insertStmt->bindValue(':user', $user);
$insertStmt->bindValue(':timestamp', $timestamp);
$insertStmt->bindValue(':description', $description);
$insertStmt->bindValue(':amount', $amount);
$insertStmt->execute();
}
}
```
Note: signature changed from `(array, string, PDOStatement, PDOStatement, bool)` to `(array, string, PDOStatement)`. The two statements (check/insert) collapsed into one ON CONFLICT insert, the `debug` flag is gone (debug logging was unused), and the `$debug` boolean parameter is dropped.
- [ ] **Step 4: Run the test and verify it passes**
Run: `php tests/process_log_entries_test.php`
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
- [ ] **Step 5: Commit**
```bash
git add tests/process_log_entries_test.php includes/utilities.php
git commit -m "Rewrite processLogEntries for v2 shape; add golden-file test"
```
---
### Task 6: Replace `firstRun`/`getLog` with `backfillUserLogs`/`syncUserLogs`
**Files:**
- Modify: `functions.php` (remove `firstRun`, remove `getLog`, add `backfillUserLogs` and `syncUserLogs`)
- No new test file — verified end-to-end in Task 8.
**Interfaces:**
- Consumes: `$user` (string — key into `USER_KEYS`). Uses `getDatabaseConnection()`, `dbNew()`, `fetchAndStoreLogPage()` (Task 4).
- Produces: `void`. Both functions are idempotent.
- [ ] **Step 1: Remove `firstRun` from `functions.php`**
Delete the entire `firstRun()` function (lines 72-119 in the current file, including the doc-comment block above it).
- [ ] **Step 2: Remove `getLog` from `functions.php`**
Delete the entire `getLog()` function (lines 132-153 in the current file, including the doc-comment block above it).
- [ ] **Step 3: Add `backfillUserLogs` and `syncUserLogs`**
Insert the following in place of the deleted functions:
```php
/**
* Pulls and stores the full vault transaction log history for a user.
*
* Pages through the v2 Torn API using `_metadata.links.next` until the
* API reports no further pages. Intended for first-run use when the
* local database is empty. Idempotent: re-running on a partially
* populated database inserts only new entries (ON CONFLICT DO NOTHING).
*
* @param string $user The user whose logs should be fetched.
*
* @return void
*/
function backfillUserLogs($user) {
$url = 'https://api.torn.com/v2/user?selections=log&log=5850,5851';
$pdo = getDatabaseConnection();
do {
$next = fetchAndStoreLogPage($pdo, $user, $url);
$url = $next;
} while ($next !== null);
}
/**
* Synchronizes recent vault transactions for a user.
*
* On a non-empty database, fetches only entries newer than the user's
* most recent row. Falls back to full backfill when the database is
* empty.
*
* @param string $user The user whose logs should be synced.
*
* @return void
*/
function syncUserLogs($user) {
if (dbNew()) {
backfillUserLogs($user);
return;
}
$pdo = getDatabaseConnection();
$stmt = $pdo->prepare('SELECT MAX(timestamp) AS max_ts FROM vault WHERE user = :user');
$stmt->bindValue(':user', $user);
$stmt->execute();
$lastTs = (int)$stmt->fetch(PDO::FETCH_ASSOC)['max_ts'];
$url = "https://api.torn.com/v2/user?selections=log&log=5850,5851&from=" . ($lastTs + 1);
do {
$next = fetchAndStoreLogPage($pdo, $user, $url);
$url = $next;
} while ($next !== null);
}
```
- [ ] **Step 4: Verify the file still parses**
Run: `php -l functions.php`
Expected: `No syntax errors detected`.
- [ ] **Step 5: Commit**
```bash
git add functions.php
git commit -m "Replace firstRun/getLog with backfillUserLogs/syncUserLogs"
```
---
### Task 7: Update `index.php` call sites
**Files:**
- Modify: `index.php` (rename two function calls in the `dbNew()` branches)
**Interfaces:**
- Consumes: existing `index.php` control flow (calls into `firstRun` / `getLog`).
- Produces: same control flow, but calling `backfillUserLogs` / `syncUserLogs` instead.
- [ ] **Step 1: Replace `firstRun` with `backfillUserLogs`**
In `index.php`, replace `firstRun($key)` with `backfillUserLogs($key)`.
The result should read:
```php
if (dbNew()) {
foreach (USER_KEYS as $key => $value) {
backfillUserLogs($key);
}
header('Location: /');
} else {
foreach (USER_KEYS as $key => $value) {
syncUserLogs($key);
}
}
```
- [ ] **Step 2: Replace `getLog` with `syncUserLogs`**
In the same file, in the `else` branch, replace `getLog($key)` with `syncUserLogs($key)`.
- [ ] **Step 3: Verify the file still parses**
Run: `php -l index.php`
Expected: `No syntax errors detected`.
- [ ] **Step 4: Commit**
```bash
git add index.php
git commit -m "Update index.php to call backfillUserLogs/syncUserLogs"
```
---
### Task 8: End-to-end smoke check
**Files:**
- No code changes. Just verification steps.
- [ ] **Step 1: Run the unit test once more**
Run: `php tests/process_log_entries_test.php`
Expected: `OK: processLogEntries v2 golden-file test passed (100 entries).`
- [ ] **Step 2: Verify no remaining references to the v1 functions**
Run:
```bash
cd "C:\Users\ksolo\Projects\Games\Torn\Torn Vault Tracker"
grep -rn "firstRun\|?key=\|api.torn.com/user/" --include="*.php" .
```
Expected: no matches (except possibly `backup/` reference files, which are read-only fixtures). The grep should report nothing under `functions.php`, `index.php`, `includes/`, or `tests/`.
- [ ] **Step 3: (Optional) Run `phpcs` against the touched files**
Run: `phpcs functions.php index.php includes/utilities.php includes/exceptions.php tests/process_log_entries_test.php`
If `phpcs` is installed and clean: skip Step 4. If it reports issues, fix them and commit. (The project's existing `phpcs.xml` is the source of style rules; there is no CI check, so this is a manual sweep.)
---
## Self-Review Notes
**Spec coverage:**
| Spec requirement | Task |
|---|---|
| Single v2 path for both bootstrap and steady-state | Task 6 (single `fetchAndStoreLogPage` consumed by both) |
| `Authorization: ApiKey` header, no `?key=` | Task 2 |
| `_metadata.links.next` pagination | Task 4 (helper returns it) + Task 6 (loop in both entry points) |
| Entry id as `TEXT PRIMARY KEY` with `ON CONFLICT DO NOTHING` | Task 4 (insert stmt) + Task 5 (test asserts idempotency) |
| LogEntryIncompleteException for malformed entries | Task 1 (class) + Task 5 (throw site) |
| Golden-file test against `backup/api-sample-new.json` | Task 5 |
| `firstRun``backfillUserLogs`, `getLog``syncUserLogs` | Task 6 (definitions) + Task 7 (call sites) |
| DB schema unchanged, UI helpers unchanged | Tasks 6 & 7 only swap the log-fetching entry points; `buildTable`, `generateBalance`, `getSpace`, `vaultLoop`, `fetchVaultRecords`, `dbNew`, `getDatabaseConnection` all untouched. |
| Tests verified manually via `php tests/...` | Task 5 (initial run) + Task 8 (final run) |
| `config.php`, `style.css`, `script.js`, `backup/`, README untouched | No task touches them. (Spec said README update was optional; we skip it.) |
**Placeholder scan:** No "TODO", "TBD", "implement later", or vague instructions in the plan.
**Type consistency:** The `executeApiCall(string $url, string $apiKey)` signature in Task 2 matches the call site `executeApiCall($url, USER_KEYS[$user])` in Task 4. The `fetchAndStoreLogPage(PDO $pdo, string $user, string $url): ?string` signature in Task 4 matches the loop body in both `backfillUserLogs` and `syncUserLogs` in Task 6. The `processLogEntries(array, string, PDOStatement)` signature in Task 5 matches the call site `processLogEntries($responseData['log'], $user, $insertStmt)` in Task 4. The `dbNew()` function called in Task 6 is unchanged and already exists in `includes/utilities.php`.