MeshVault/docs/superpowers/plans/2026-10-11-m3-import-indexing-engine.md

M3: Import and Indexing Engine Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build scanning, validation, hashing, verified-copy primitives plus background worker, progress, and failure capture for managed imports.

Architecture: Three phases: (1) scan/copy primitives as reusable services in src/Indexer/; (2) repository layer for M4 #28 persistence; (3) background worker consuming import_jobs. Filesystem behind interfaces for testability.

Tech Stack: PHP 8.3+, PostgreSQL via Phinx, PSR-3 logging, PSR-4 MeshVault\, strict types, PSR-12. No new runtime deps.

Spec: notes/backlog-m3-m4.json issues 20-27 plus 28, and notes/Roadmap.md.

Global Constraints

• PHP 8.3+, strict_types=1, PSR-12, four spaces, LF, UTF-8 no BOM.
• New classes final readonly where possible; constructor injection.
• PDO repositories, no ORM.
• Streams only; never load whole files into memory.
• realpath() checks; reject symlinks, .., dest outside library root.
• SHA-256 streaming, 128 KiB default buffer.
• Collision suffix _NNN before extension.
• Free-space safety margin 5% default.
• Library root from validated AppConfig::libraryPath.
• Error codes: SCAN_EMPTY, SCAN_READ_ERROR, HASH_MISMATCH, DISK_FULL, COLLISION, SYMLINK_ESCAPE, PATH_TRAVERSAL, STL_INVALID, COPY_FAILED, DB_ERROR.
• Atomic worker claim on import_jobs.
• Unit plus integration tests; 2+ failure-path tests per feature.

Review Focus

1. Path safety vs traversal, symlinks, absolute escapes.
2. Streaming hash equality with hash_file(); mid-read changes.
3. Atomic publish temp-then-rename; no partial completes.
4. Worker idempotency on retry.
5. Disk-full pre-check plus mid-copy cleanup.

Task 1.1 actual scope/status

• Implemented only filesystem abstraction and path safety: FilesystemInterface, LocalFilesystem, PathSafety, PathSafetyException, and unit tests for both services. Native-style I/O failures return false; resource returns are PHPDoc and caller-owned. Source checks allow existing external sources; destinations stay within the library and may have missing suffixes. Relative paths, traversal, ancestor/dangling symlinks, root-prefix siblings, and ambiguous Windows names are checked with structured reasons.
• Validation on local PHP 8.3.35: initial RED 22 missing-class assertion failures; Windows alias regression also observed RED before its fix. Final focused suite: 23 tests, 102 assertions, 5 symlink skips (host cannot create them). composer test: foundation checks pass; 110 tests, 604 assertions, 7 skips total. New PHP files pass PSR-12; full composer lint still reports three unrelated existing errors in db/seeds/DefaultDataSeed.php and tests/Integration/MigrationTest.php (not changed).
• Drive-relative review fix: ensureSourceSafe() now rejects any uppercase/lowercase drive prefix without a following slash as INVALID_PATH before link inspection; constructor roots inherit the same check. Existing relative/destination behavior is unchanged. Regression RED: all 5 new cases failed with expected missing-rejection failures, including realpath-confirmed existing source/root paths with upper/lowercase drives. GREEN on Windows/PHP 8.3.35: focused tests/Unit/Indexer suite passed with 28 tests, 531 assertions, 5 symlink skips. composer test passed foundation checks and 115 tests, 1033 assertions, 7 skips. Scoped PHPCS using phpcs.xml.dist passed for src/Indexer/Filesystem/PathSafety.php and tests/Unit/Indexer/PathSafetyTest.php. Coverage also checks all 52 drive-letter variants with bare, file, and dot-relative suffixes.
• Limitations: checks are point-in-time, not race-proof; callers must control directory mutation around I/O. Symlink behavior needs execution on a host permitting links; no Linux runtime validation performed. No later M3 tasks, commits, or delegation.

Task 1.2 actual scope/status

• Implemented recursive FileEnumerator, FileEnumeratorInterface, and separate PSR-4 ScanResult, ScannedFile, ScanError readonly DTOs. Manifest entries expose canonical absolute sourcePath, safe forward-slash relativePath, size, and mimeType; totals include only successful entries. Default allowlist is stl/jpg/jpeg/png/gif/pdf/txt/md, matched case-insensitively; an explicit empty allowlist selects nothing. Empty directories return an empty result without errors.
• Safety ruling: the global rejection of all symlinks overrides any older follow-in-root-link wording. File, directory, dangling and ancestor links are reported as SYMLINK_ESCAPE, even if the link has a disallowed extension, and are never traversed. PathSafety checks run before canonicalization; the review fix below tightens POSIX source spelling. Traversal retains PATH_TRAVERSAL. Invalid/missing/non-directory roots and per-entry readability, resolution, stat, MIME and directory-read failures produce actionable structured SCAN_READ_ERROR errors; independent siblings continue.
• Minimal filesystem API addition: readDirectory(): array|false returns entry names without dot entries, backed by suppressed native scandir() failures. Existing metadata calls and injected filesystem doubles cover deterministic I/O failures; no new runtime dependencies or container changes. Iterative traversal avoids call stack recursion; root depth is zero and the configurable maximum is 128 by default. Exceeding it reports an error for that subtree without losing siblings.
• TDD/validation on Windows, PHP 8.3.35 (commands run from repository root):
  • RED: php vendor/phpunit/phpunit/phpunit tests/Unit/Indexer/FileEnumeratorTest.php produced 15 expected missing-FileEnumerator assertion failures before implementation.
  • GREEN, same command: 15 tests, 143 assertions, 4 symlink privilege skips.
  • Added direct adapter regression coverage for empty/nested listings, missing paths and non-directory reads; enumeration tests cover mixed-case nested defaults, disallowed extensions, custom/empty allowlists, invalid roots, six injected per-entry failure modes, root failures, sibling continuation and depth limits.
  • php vendor/squizlabs/php_codesniffer/bin/phpcbf --standard=phpcs.xml.dist src/Indexer tests/Unit/Indexer/FileEnumeratorTest.php tests/Unit/Indexer/LocalFilesystemTest.php fixed generated CRLF and multiline formatting; no behavioral changes.
  • php vendor/phpunit/phpunit/phpunit tests/Unit/Indexer: 44 tests, 679 assertions, 9 skips; success.
  • composer test: foundation checks pass; 131 tests, 1181 assertions, 11 skips; success (existing HTTP failure-path tests emit expected error log records).
  • php vendor/squizlabs/php_codesniffer/bin/phpcs --standard=phpcs.xml.dist src/Indexer tests/Unit/Indexer/FileEnumeratorTest.php tests/Unit/Indexer/LocalFilesystemTest.php: passes with no findings. Full-repository lint's previously documented unrelated failures were not addressed.
• Review fixes (task 1.2):
  • Policy ruling: POSIX source paths containing any backslash are rejected as INVALID_PATH before normalization, including constructor library roots. Windows backslash separators remain supported; relative/destination normalization is unchanged. FileEnumerator now uses the canonical path returned by safety checks for injected filesystem operations, retaining the adapter realpath call and its failure handling.
  • Safety failure messages retain reason tokens and scan codes, but add corrective guidance for missing, empty/NUL, traversal and symlink paths. Tests now require directory-selection guidance rather than merely the phrase "source root".
  • Added five tests: source/constructor platform policy, POSIX backslash-named ancestor symlink with a separate slash-spelled decoy tree, traversal root/entry codes, zero-byte metadata, and negative-size/empty-MIME sibling continuation. Existing failure doubles continue to pass, including injected realpath failures.
  • RED: Windows focused suite failed on missing-root guidance (49 tests, 688 assertions, 1 failure, 10 skips). A local WSL/PHP 8.5.4 standalone probe using real directories and a backslash-named ancestor symlink failed source and constructor rejection and demonstrated one accepted target file before the fix. After the fix all three probe checks passed, with no accepted files. Probe fixtures were cleaned up; the ignored .phpunit.cache/posix-review-probe.php remains because shell cleanup was blocked.
  • GREEN on Windows/PHP 8.3.35: php vendor/phpunit/phpunit/phpunit tests/Unit/Indexer: 49 tests, 722 assertions, 10 skips (nine host symlink-permission skips and one POSIX-only regression). composer test: foundation checks pass; 136 tests, 1224 assertions, 12 skips (the ten above plus two existing integration skips); expected HTTP error logs only. php vendor/squizlabs/php_codesniffer/bin/phpcs --standard=phpcs.xml.dist src/Indexer tests/Unit/Indexer: exit 0, no findings.
  • Linux PHPUnit was attempted via WSL but cannot run because its PHP lacks dom, mbstring and xmlwriter. No extensions were installed; the standalone native POSIX RED/GREEN probe is not a substitute for a full Linux suite.
• Limitations: symlink PHPUnit cases require a link-capable host; Windows skips are detailed above. Path checks are point-in-time, not race-proof, so callers must control source-tree mutation. No file payload is loaded by the enumerator or opened through readStream; native fileinfo may inspect content for MIME detection. Directory name lists and the returned manifest are materialized in memory (not file contents). Scan order is not a public guarantee. No free-space, hashing, copying, persistence or worker work; no commits/merges/pushes. Task 1.1 source safety is tightened only by the POSIX backslash policy above.

Tasks 1.3-3.4 and delivery layer: actual scope/status

• 1.3 SpaceChecker (src/Indexer/SpaceChecker.php) — 5% default margin, injectable free-space probe, DISK_FULL / SPACE_CHECK_FAILED reasons.
• 1.4 StreamingHasher (src/Indexer/StreamingHasher.php) — SHA-256 in 128 KiB chunks, validates the source through PathSafety before and after the read.
• 1.5 StlValidator (src/Indexer/StlValidator.php) — binary and ASCII STL, facet-count and terminator checks, STL_INVALID / SCAN_READ_ERROR.
• 1.6/1.7 VerifiedCopier (src/Indexer/VerifiedCopier.php) — temp copy, size and hash verification, link-based publication, _NNN collision suffix, cleanup on failure.
• 1.8 ImportService (src/Indexer/ImportService.php) — scan, free-space preflight, then one atomic manifest+job write. Scan errors become terminal Failed items.
• Phase 2 repositories — CatalogRepositoryInterface, ImportJobRepositoryInterface, PdoRepository, PdoCatalogRepository, PdoImportJobRepository. No schema changes.
• Phase 3 worker (src/Indexer/ImportWorker.php) — single advisory flock over the library, resumeInterrupted() under that lock, atomic claim, plan-before-publish durability, per-item failure isolation, graceful-stop callback that leaves the job Pending.
• Delivery layer — src/Console/Console.php (argument parsing, exit codes), ConsoleServices, ContainerConsoleServices, ImportWorkerFactory, LazyConsoleServices, bin/meshvault, container wiring in config/container.php, and import / worker / worker-once subcommands in dev.sh and dev.ps1.

Rulings

• Ruling: no /import HTTP route in M3 — the backlog's HTTP surface has no authentication yet, and any route accepting a source path would expose arbitrary server filesystem reads to unauthenticated callers. Deferred to M6 behind auth. Cost if wrong: the UI trigger lands one milestone later; the CLI covers the workflow meanwhile.
• Ruling: CLI exit codes are 0 success / 1 usage / 2 runtime — matches the existing wrappers' use of exit 2 for refusal. Cost if wrong: scripted callers need remapping.
• Ruling: services are constructed lazily in bin/meshvault — help and usage errors must not require a database or library directory. Cost if wrong: a misconfiguration surfaces one call later, still as exit 2.
• Ruling: grouping is by first directory component below the source root, never inferred from file contents. Cost if wrong: flat source trees produce one model per source directory.

Validation (Windows, PHP 8.3.35)

• composer test: foundation checks pass; 285 tests, 1796 assertions, 18 skips, no failures.
• composer lint: exit 0, no findings repository-wide. The two long-standing errors in db/seeds/DefaultDataSeed.php and tests/Integration/MigrationTest.php were trailing-newline violations and are now fixed; no logic in either file changed.
• vendor\bin\phpunit tests/Unit/Console: 22 tests, 71 assertions, 0 skips.
• vendor\bin\phpunit --filter TaskRunner tests/Integration: 1 test, 71 assertions — asserts the wrappers forward import, worker and worker-once to bin/meshvault with arguments intact.
• Live CLI exit-code probe: help=0, bogus=1, no args=1, import (no path)=1, worker --sleep=x=1, worker-once extra=1, import C:\nope (unconfigured env)=2.

Limitations

• PostgreSQL was not exercised. Integration tests run against SQLite built from the Phinx migrations; FOR UPDATE SKIP LOCKED is driver-detected but unverified on a live server. tests/Integration/MigrationTest.php still skips without DB credentials.
• Signals are POSIX-only. bin/meshvault installs SIGTERM/SIGINT handlers when pcntl is available; on Windows the worker stops only at --max-iterations.
• Graceful stop is verified between items, not mid-copy.
• Symlink unit tests skip on hosts that cannot create links (11 skips here).
• Nothing is committed, merged or pushed; all work is uncommitted in the working tree.