Keith Solomon
c2b138d687
- Compares old (api-sample.json) and new (api-sample-new.json) shapes - Specifies Authorization: ApiKey header instead of ?key= query param - Specifies following _metadata.links.next for pagination - Specifies entry.id (string) as PRIMARY KEY with INSERT ... ON CONFLICT DO NOTHING - Adds backfillUserLogs (replaces firstRun) and syncUserLogs (renamed from getLog) - Adds shared fetchAndStoreLogPage helper - Adds golden-file unit test (no live API)
197 lines
12 KiB
Markdown
197 lines
12 KiB
Markdown
# Torn API v1 → v2 Migration Design
|
|
|
|
**Date:** 2026-08-03
|
|
**Project:** Torn Vault Tracker
|
|
**Status:** Approved — pending implementation
|
|
|
|
## Problem
|
|
|
|
The Torn City API has moved from v1 (`api.torn.com/user/`) to v2 (`api.torn.com/v2/user`). The two protocols return different JSON shapes and use different authentication:
|
|
|
|
| Concern | v1 (old) | v2 (new) |
|
|
|---|---|---|
|
|
| Endpoint | `https://api.torn.com/user/?selections=log&…&key=APIKEY` | `https://api.torn.com/v2/user?selections=log&…` + `Authorization: ApiKey KEY` |
|
|
| `log` shape | Object keyed by entry id | Flat array of entry objects |
|
|
| Entry id | `$entry['log']` (numeric event type — duplicated across entries) | `$entry['id']` (string, unique per entry) |
|
|
| `title` location | `$entry['title']` (root) | `$entry['details']['title']` |
|
|
| `timestamp` location | `$entry['timestamp']` (root) | `$entry['timestamp']` (root, unchanged) |
|
|
| Pagination | `&to=$ts`, `&from=$ts` | `_metadata.links.next` URL with `from`/`to`/`sort`/`limit` params |
|
|
|
|
The project was already partially migrated (`firstRun()` in `functions.php:72` calls the v2 endpoint) but still reads the v1 JSON shape and v1 authentication, so `firstRun()` does not work. `getLog()` still uses the v1 endpoint entirely. Both code paths need to be brought onto v2 — a single coherent rewrite — and the helpers in `includes/utilities.php` (which both rely on) need to match.
|
|
|
|
Sample files live at `backup/api-sample.json` (v1, 2 entries) and `backup/api-sample-new.json` (v2, 100 entries with `_metadata`).
|
|
|
|
## Goals
|
|
|
|
- A single v2 path that handles both bootstrap (full history) and steady-state (recent sync).
|
|
- Idempotent re-runs: re-fetching the same data does not produce duplicate rows.
|
|
- Existing users with a populated `vault.db` come through the migration without manual intervention — only an empty DB triggers the full bootstrap.
|
|
- New code is covered by a unit-style golden-file test that doesn't touch the network.
|
|
|
|
## Non-Goals
|
|
|
|
- Adding new API endpoints beyond `user/selections=log&log=5850,5851`.
|
|
- Changing the local DB schema (table shape is already v2-compatible thanks to `id TEXT PRIMARY KEY`).
|
|
- UI changes — `index.php`, `style.css`, `script.js` stay identical.
|
|
- README updates — content already describes current behavior accurately.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
index.php
|
|
├── config.php
|
|
├── functions.php (high-level: backfillUserLogs, syncUserLogs, DB, UI helpers)
|
|
└── includes/
|
|
├── utilities.php (low-level: API + DB helpers; all v2-aware)
|
|
└── exceptions.php (unchanged)
|
|
```
|
|
|
|
The two-phase flow on each request:
|
|
|
|
1. **Bootstrap** (empty DB) → `backfillUserLogs($user)` paginates full history.
|
|
2. **Steady state** (DB populated) → `syncUserLogs($user)` fetches only newer entries.
|
|
|
|
Both call one shared `fetchAndStoreLogPage()` helper that performs a single page pull, stores rows idempotently, and returns either the next `_metadata.links.next` URL or `null`.
|
|
|
|
## Components and Responsibilities
|
|
|
|
### `includes/exceptions.php` — unchanged
|
|
|
|
Existing `ApiKeyMissingException`, `CurlErrorException`, `JsonDataException`, `ApiValidationException` cover all the failure modes we anticipate. Add one optional exception:
|
|
|
|
- `LogEntryIncompleteException` — raised when a fetched entry is missing `data.deposited` or `data.withdrawn` (defensive — shouldn't occur given we filter to `log=5850,5851`, but we shouldn't silently insert garbage if the API ever changes).
|
|
|
|
### `includes/utilities.php` — rewritten
|
|
|
|
| Function | Change |
|
|
|---|---|
|
|
| `executeApiCall(string $url, string $apiKey): array` | **Rewrite.** Drop `Content-Type: application/json`; add `Authorization: ApiKey <apiKey>`. Accepts a complete URL (so callers can pass through `_metadata.links.next`). If `$apiKey` is empty, throws `ApiKeyMissingException`. The `apiKey` is supplied by the caller (`fetchAndStoreLogPage` looks it up from `USER_KEYS[$user]`); this keeps `executeApiCall` agnostic about which user is being fetched. |
|
|
| `validateApiResponse(array)` | Tighten: `log` must exist and be an **array** (not an object-keyed map). |
|
|
| `processLogEntries(array $logEntries, string $user, …)` | **Rewrite for v2 shape.** Read `$entry['id']` for the PK, `$entry['timestamp']`, `$entry['details']['title']`, `$entry['data']['deposited']` (deposit) or `$entry['data']['withdrawn']` (withdraw). Use `INSERT … ON CONFLICT(id) DO NOTHING` so the operation is idempotent. Skip (and log) any entry whose `data` block lacks both fields rather than inserting bad data. |
|
|
| `fetchAndStoreLogPage(PDO $pdo, string $user, string $url): ?string` | **New.** Composes `executeApiCall` + `validateApiResponse` + `processLogEntries` for a single page. Looks the API key up via `USER_KEYS[$user]`. Returns `$data['_metadata']['links']['next'] ?? null`. |
|
|
| `dbNew()`, `ensureUserHasApiKey()`, `fetchVaultRecords()`, `vaultLoop()`, `getSign()`, `consoleLog()` | Unchanged. |
|
|
|
|
### `functions.php` — rewritten entry points
|
|
|
|
| Function | Change |
|
|
|---|---|
|
|
| `getDatabaseConnection()` | Unchanged. |
|
|
| `backfillUserLogs(string $user): void` | **New** (replaces `firstRun`). Starts at `INITIAL_URL = "https://api.torn.com/v2/user?selections=log&log=5850,5851"`, paginates with `_metadata.links.next` until exhausted. |
|
|
| `syncUserLogs(string $user): void` | **Renamed** from `getLog`. Uses the v2 endpoint with `&from=<last_seen_ts+1>` query param (only if the user has rows in the DB — otherwise delegates to `backfillUserLogs`). Falls back to following `_metadata.links.next` if more than one page comes back. |
|
|
| `generateBalance()`, `getSpace()`, `buildTable()` | Unchanged — they read from the DB, schema is unchanged. |
|
|
|
|
### `index.php` — one-line swap
|
|
|
|
Replace `firstRun` → `backfillUserLogs` and `getLog` → `syncUserLogs`. Logic for "is DB empty?" already lives in `dbNew()`, so the control flow is identical.
|
|
|
|
### `config.php` — unchanged
|
|
|
|
`USER_KEYS` map keyed by user already provides both API keys; the migration uses them both.
|
|
|
|
## Data Flow
|
|
|
|
### Bootstrap (empty DB)
|
|
|
|
```
|
|
backfillUserLogs(user) {
|
|
url = "https://api.torn.com/v2/user?selections=log&log=5850,5851"
|
|
do {
|
|
next = fetchAndStoreLogPage($pdo, $user, $url)
|
|
url = next
|
|
} while (next !== null)
|
|
}
|
|
|
|
fetchAndStoreLogPage(pdo, user, url): ?string {
|
|
headers = ["Authorization: ApiKey " . USER_KEYS[user]]
|
|
data = executeApiCall(url, USER_KEYS[user]) // throws on curl/JSON failure
|
|
validateApiResponse(data) // throws on shape error
|
|
|
|
processLogEntries(data['log'], user, …) // INSERT ... ON CONFLICT DO NOTHING
|
|
return data['_metadata']['links']['next'] ?? null
|
|
}
|
|
```
|
|
|
|
### Steady state (DB populated)
|
|
|
|
```
|
|
syncUserLogs(user) {
|
|
if (dbNew()) return backfillUserLogs(user) // safety net
|
|
|
|
last = $pdo->query("SELECT MAX(timestamp) FROM vault WHERE user = ?")
|
|
from = (int)last + 1
|
|
url = "https://api.torn.com/v2/user?selections=log&log=5850,5851&from=$from"
|
|
|
|
do {
|
|
next = fetchAndStoreLogPage($pdo, $user, $url)
|
|
url = next
|
|
} while (next !== null)
|
|
}
|
|
```
|
|
|
|
## Data Model
|
|
|
|
The `vault` table (defined in `functions.php:32-38`) already matches our needs:
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS vault (
|
|
id TEXT PRIMARY KEY, -- v2 entry id, e.g. "j8EzaeOu2lpWPcloqGYJ"
|
|
user TEXT NOT NULL,
|
|
timestamp INTEGER NOT NULL,
|
|
description TEXT NOT NULL,
|
|
amount REAL NOT NULL
|
|
);
|
|
```
|
|
|
|
Migration concern: existing rows were inserted with `$key` from the v1 object-keyed `log`, which means the current PK is the v1 alphanumeric object key (e.g. `"CTczoijHAKhnfAeYmagC"`). The v2 PK is the v2 unique id (e.g. `"j8EzaeOu2lpWPcloqGYJ"`). Format-wise both are strings, so the schema is compatible — **but** the values do not match and v2 inserts will collide on the PK only if the same id repeats, which it won't across the two schemes. Wiping `data/vault.db` on the first page load after deploy gives the cleanest result (recommended in install note). For users with an existing DB, the bootstrap path will simply keep fetching — v2 ids that already exist (by coincidence — vanishingly unlikely between v1 object keys and v2 entry ids) will be skipped via `ON CONFLICT(id) DO NOTHING`.
|
|
|
|
## Error Handling
|
|
|
|
| Failure | Behavior |
|
|
|---|---|
|
|
| Missing API key for a user | `ensureUserHasApiKey()` throws `ApiKeyMissingException` before any fetch. |
|
|
| `curl_exec` returns false | `executeApiCall` throws `CurlErrorException`. The bootstrap loop stops; index page renders whatever was stored so far. |
|
|
| `json_decode` fails | `executeApiCall` throws `JsonDataException`. Same as above. |
|
|
| Response missing `log` or wrong shape | `validateApiResponse` throws `ApiValidationException`. Loop stops. |
|
|
| Entry lacks `data.deposited` or `data.withdrawn` | `processLogEntries` raises `LogEntryIncompleteException` for that entry, skips it, continues with the rest. (Logged via the existing `consoleLog` and `debug.log` patterns.) |
|
|
| DB error | Handled by PDO exception mode; propagates. |
|
|
|
|
All exceptions are caught at the top of `index.php`'s bootstrap path so a single bad page doesn't take down the whole UI. The user sees whatever data is in the DB plus a non-fatal warning.
|
|
|
|
## Testing Strategy
|
|
|
|
A pure-PHP "golden file" test that does **not** hit the network:
|
|
|
|
1. Load `backup/api-sample-new.json`.
|
|
2. Mock `executeApiCall` to return the fixture (`[api-sample-new.json content]`).
|
|
3. Mock `processLogEntries`'s PDO statements (count `execute()` calls and capture params).
|
|
4. Assert: every entry produces exactly one `INSERT` with the expected `id`, `user`, `timestamp`, `description`, `amount`; that the loop terminates when `links.next` is null; and that a fixture with `links.next` set to a sentinel URL causes exactly one extra fetch.
|
|
|
|
This lives next to `phpcs.xml` (no test framework added; we just add `tests/process_log_entries_test.php` and document how to run it via `php tests/process_log_entries_test.php`). The test is run manually before each deploy.
|
|
|
|
We do **not** add live API smoke tests — they require real keys and quota. The existing `dbNew()` path on `index.php` is the only "live" verification, and it runs only on the user's first page-load.
|
|
|
|
## Files Touched
|
|
|
|
| File | Action |
|
|
|---|---|
|
|
| `functions.php` | Rewrite the two log-fetch entry points; keep DB + UI helpers. |
|
|
| `includes/utilities.php` | Rewrite `executeApiCall`, `validateApiResponse`, `processLogEntries`; add `fetchAndStoreLogPage`. |
|
|
| `includes/exceptions.php` | Add `LogEntryIncompleteException`. |
|
|
| `index.php` | Rename calls: `firstRun`→`backfillUserLogs`, `getLog`→`syncUserLogs`. |
|
|
| `tests/process_log_entries_test.php` | **New.** Golden-file unit test for v2 parsing. |
|
|
| `README.md` | Optional: note that `data/vault.db` will be re-populated on first page-load post-deploy. |
|
|
| `config.php`, `style.css`, `script.js`, `backup/` | **Untouched.** |
|
|
|
|
## Open Questions
|
|
|
|
None at design time. All clarifications resolved during brainstorming.
|
|
|
|
## Acceptance Criteria
|
|
|
|
1. `getLog`/`syncUserLogs` and `firstRun`/`backfillUserLogs` both call `https://api.torn.com/v2/user` with `Authorization: ApiKey …` header.
|
|
2. `processLogEntries` reads `id`, `timestamp`, `details.title`, and `data.deposited`/`data.withdrawn` from each entry; ignores any other shape.
|
|
3. Pagination follows `_metadata.links.next` until null.
|
|
4. Re-running on the same data produces no duplicate rows.
|
|
5. `tests/process_log_entries_test.php` passes against `backup/api-sample-new.json`.
|
|
6. `index.php`, `style.css`, `script.js`, `includes/exceptions.php` (existing classes), `config.php` behave identically to before for end users — except that the data they fetch is now the v2 shape.
|