MeshVault/docs/superpowers/specs/2026-10-14-catalog-explicit-grouping-contract.md

Bounded #28 Catalog Contract — Explicit Grouping & Stable Identity

Status: Design phase — not implemented. Extends Tasks 1-4 contracts without breaking them. Baseline: main at 9dd8c07 (Tasks 1-4 integrated, #28 catalog core closed).

1. Problem Statement

The current LegacyGuessedImportGroupingAdapter implements guessed grouping:

• First directory component under source root → model name
• Files in source root → model named after source directory
• Stateless, fresh lookup on every call via ensureModel()
• No user confirmation, no persistence of grouping decisions

Gaps from reconciliation table:

• #18: Confirmed explicit grouping (user confirmation) missing
• #20: Explicit user-confirmed grouping (not guessed); stable model identity across imports; grouping persistence
• #24: Crash between publish and suffix save → duplicate suffixed copies; orphaned published files on catalog failure; replay idempotency

2. Design Principles

1. Retain PHP/PDO/Phinx and existing repositories — no schema rewrites, new migrations only
2. Extend, don't replace — CatalogServiceInterface/CatalogRepositoryInterface unchanged
3. Explicit grouping is opt-in — legacy guessed adapter remains for backward compatibility
4. Bounded scope — no UI, no scheduler, no async workers; pure service/contract layer
5. Lock order preserved — root → folder → model → file (same as Task 4)
6. Savepoint semantics — caller transaction owns commit/rollback; catalog operations use savepoints

3. New Contracts (Interfaces)

3.1 Explicit Grouping Adapter Interface

// src/Indexer/Contracts/ExplicitImportGroupingAdapterInterface.php
namespace MeshVault\Indexer\Contracts;

use MeshVault\Indexer\ScannedFile;

interface ExplicitImportGroupingAdapterInterface
{
    public function modelFor(string $sourcePath, ScannedFile $file, array $context = []): int;
    public function persistDecisions(int $jobId, array $decisions): void;
    public function recoverDecisions(int $jobId): array;
}

3.2 Explicit Grouping Adapter Implementation

// src/Indexer/ExplicitImportGroupingAdapter.php
namespace MeshVault\Indexer;

use MeshVault\Indexer\Contracts\ExplicitImportGroupingAdapterInterface;
use MeshVault\Indexer\ScannedFile;
use MeshVault\Repository\CatalogRepositoryInterface;
use MeshVault\Repository\ImportJobRepositoryInterface;
use DomainException;
use InvalidArgumentException;

final class ExplicitImportGroupingAdapter implements ExplicitImportGroupingAdapterInterface
{
    public function __construct(
        private readonly CatalogRepositoryInterface $catalog,
        private readonly ImportJobRepositoryInterface $jobs,
    ) {}

    public function modelFor(string $sourcePath, ScannedFile $file, array $context = []): int
    {
        if (isset($context['model_id'])) {
            $modelId = (int) $context['model_id'];
            $aggregate = $this->catalog->aggregate($modelId);
            if ($aggregate === null) {
                throw new DomainException('Explicit model not found.');
            }
            if (isset($context['model_name']) && $aggregate['model']['name'] !== $context['model_name']) {
                throw new DomainException('Model title mismatch.');
            }
            if (isset($context['folder_path'])) {
                $expectedFolder = '/' . ltrim($context['folder_path'], '/');
                if ($aggregate['folder']['path'] !== $expectedFolder) {
                    throw new DomainException('Model folder mismatch.');
                }
            }
            return $modelId;
        }

        if (isset($context['folder_path']) && isset($context['model_name'])) {
            $folderPath = '/' . ltrim($context['folder_path'], '/');
            $modelName = $context['model_name'];
            if ($context['create_new'] ?? false) {
                return $this->catalog->atomic(fn () => $this->catalog->createModel($folderPath, $modelName));
            }
            return $this->catalog->atomic(fn () => $this->catalog->ensureModel($folderPath, $modelName));
        }

        $parent = trim(dirname(str_replace('\\', '/', $file->relativePath)), '/');
        $group = $parent === '' || $parent === '.'
            ? basename(rtrim(str_replace('\\', '/', $sourcePath), '/'))
            : explode('/', $parent)[0];
        return $this->catalog->atomic(fn () => $this->catalog->ensureModel('/' . $group, $group));
    }

    public function persistDecisions(int $jobId, array $decisions): void
    {
        $this->jobs->atomic(function () use ($jobId, $decisions) {
            foreach ($decisions as $decision) {
                $this->jobs->recordGroupingDecision(
                    $jobId,
                    $decision['source_path'],
                    $decision['model_id'],
                    $decision['context'] ?? []
                );
            }
        });
    }

    public function recoverDecisions(int $jobId): array
    {
        return $this->jobs->getGroupingDecisions($jobId);
    }
}

3.3 Import Job Repository Extensions

// src/Repository/ImportJobRepositoryInterface.php (additions)
namespace MeshVault\Repository;

interface ImportJobRepositoryInterface
{
    // ... existing methods ...
    public function recordGroupingDecision(int $jobId, string $sourcePath, int $modelId, array $context): void;
    public function getGroupingDecisions(int $jobId): array;
}

3.4 Catalog Service Extensions (for stable identity)

// src/Catalog/CatalogServiceInterface.php (additions)
namespace MeshVault\Catalog;

interface CatalogServiceInterface
{
    // ... existing methods ...
    public function findModel(string $folderPath, string $modelName): ?array;
    public function listModelsInFolder(string $folderPath): array;
    public function getOrCreateModel(string $folderPath, string $modelName): array;
}

4. Database Migrations

4.1 Migration: 20261014120001_import_grouping_decisions.php

<?php
declare(strict_types=1);

use Phinx\Migration\AbstractMigration;

final class ImportGroupingDecisions extends AbstractMigration
{
    public function up(): void
    {
        $this->table('import_jobs_decisions', ['id' => false, 'primary_key' => ['job_id', 'source_path']])
            ->addColumn('job_id', 'integer', ['signed' => false])
            ->addColumn('source_path', 'text')
            ->addColumn('model_id', 'integer', ['signed' => false])
            ->addColumn('context', 'jsonb', ['default' => '{}'])
            ->addColumn('created_at', 'timestamp', ['default' => 'CURRENT_TIMESTAMP'])
            ->addForeignKey('job_id', 'import_jobs', 'id', ['delete' => 'CASCADE', 'update' => 'NO_ACTION'])
            ->addIndex(['model_id'], ['name' => 'idx_import_decisions_model'])
            ->create();
    }

    public function down(): void
    {
        $this->table('import_jobs_decisions')->drop()->save();
    }
}

4.2 Migration: 20261014120002_model_identity_stability.php

<?php
declare(strict_types=1);

use Phinx\Migration\AbstractMigration;

final class ModelIdentityStability extends AbstractMigration
{
    public function up(): void
    {
        // No schema changes — this migration documents the stability contract
        // Models are now immutable in identity (folder_id + name unique)
        // Files reference models via nullable FK (from Task 4)
        // Explicit grouping decisions recorded in import_jobs_decisions
    }

    public function down(): void
    {
        // No-op: identity stability is a contract, not a schema change
    }
}

5. Import Service Integration

// src/Indexer/ImportService.php (modified)
namespace MeshVault\Indexer;

use MeshVault\Indexer\Contracts\ExplicitImportGroupingAdapterInterface;
use MeshVault\Indexer\Contracts\FileEnumeratorInterface;
use MeshVault\Repository\ImportJobRepositoryInterface;
use Throwable;

final readonly class ImportService
{
    public function __construct(
        private FileEnumeratorInterface $enumerator,
        private ImportJobRepositoryInterface $jobs,
        private ExplicitImportGroupingAdapterInterface $groupingAdapter, // Changed from LegacyGuessed
        private SpaceChecker $space,
        private string $libraryRoot,
    ) {}

    /** @param array<string, mixed> $groupingContext Optional explicit grouping context per file */
    public function import(string $sourcePath, array $extensions = [], array $groupingContext = []): int
    {
        $scan = $this->enumerator->enumerate($sourcePath, $extensions);
        if ($scan->files === []) {
            $detail = $scan->errors === [] ? 'No importable files were found.' : $scan->errors[0]->message;
            throw new ImportException('SCAN_EMPTY', $detail);
        }
        $this->space->check($this->libraryRoot, $scan->totalBytes);

        try {
            return $this->jobs->atomic(
                fn (): int => $this->createJobWithGrouping($sourcePath, $scan, $groupingContext)
            );
        } catch (ImportException $error) {
            throw $error;
        } catch (Throwable $error) {
            throw new ImportException('DB_ERROR', 'Could not persist the import job.', $error);
        }
    }

    /** @return list<array<string, mixed>> */
    private function createJobWithGrouping(string $sourcePath, ScanResult $scan, array $groupingContext): int
    {
        $items = [];
        $decisions = [];

        foreach ($scan->files as $file) {
            $fileContext = $groupingContext[$file->relativePath] ?? $groupingContext['*'] ?? [];
            $modelId = $this->groupingAdapter->modelFor($sourcePath, $file, $fileContext);

            $items[] = [
                'source_path' => $file->sourcePath,
                'size' => $file->size,
                'model_id' => $modelId,
            ];

            $decisions[] = [
                'source_path' => $file->sourcePath,
                'model_id' => $modelId,
                'context' => $fileContext,
            ];
        }

        foreach ($scan->errors as $error) {
            $items[] = [
                'source_path' => $error->sourcePath,
                'size' => 0,
                'model_id' => null,
                'status' => 'Failed',
                'failure_reason' => $error->code . ': ' . $error->message,
            ];
        }

        $jobId = $this->jobs->create($sourcePath, $items);

        if ($decisions !== []) {
            $this->groupingAdapter->persistDecisions($jobId, $decisions);
        }

        return $jobId;
    }
}

6. Worker Recovery & Idempotency (Addresses #24)

6.1 Current Gap

• Worker persists planned name before copy
• Saves collision suffix after publication
• Crash between → duplicate suffixed copies on replay
• Catalog failure after publish → orphaned file on disk

6.2 Solution: Atomic Publish-Then-Register

// src/Indexer/ImportWorker.php (modified publish method)
private function publish(array $plan, string $destination): string
{
    $directory = dirname($destination);
    if (!is_dir($directory) && !@mkdir($directory, 0775, true) && !is_dir($directory)) {
        throw new ImportException('COPY_FAILED', 'Cannot create the managed destination directory.');
    }

    $destination = $this->pathSafety->ensureDestSafe($destination);
    clearstatcache(true, $destination);

    if (is_file($destination)) {
        if (hash_file('sha256', $destination) === $plan['hash']) {
            return $destination;
        }
    } else {
        $this->spaceChecker->check($directory, $plan['size']);
    }

    // CRITICAL CHANGE: Copy to TEMP, then atomic rename + catalog register in ONE transaction
    $tempPath = $destination . '.tmp.' . bin2hex(random_bytes(8));
    try {
        $this->copier->copy($plan['source'], $tempPath, $plan['size'], $plan['hash']);

        $managedPath = $this->relative($tempPath);
        $this->catalog->atomic(function () use ($tempPath, $destination, $plan, &$managedPath) {
            if (!@rename($tempPath, $destination)) {
                throw new ImportException('COPY_FAILED', 'Cannot finalize the published file.');
            }

            $managedPath = $this->relative($destination);

            $fileId = $this->catalog->registerFile(
                $plan['model_id'],
                $managedPath,
                $plan['hash'],
                $plan['size'],
                $plan['mime'] ?? null,
                'Present'
            );

            $this->jobs->updateItemManagedPath($plan['item_id'], $managedPath, $fileId);
        });

        return $destination;
    } catch (Throwable $error) {
        if (is_file($tempPath)) {
            @unlink($tempPath);
        }
        throw $error;
    }
}

6.3 New Job Repository Method

// src/Repository/ImportJobRepositoryInterface.php (addition)
public function updateItemManagedPath(int $itemId, string $managedPath, int $fileId): void;

7. Lock Order & Transaction Boundaries

Operation	Locks Acquired (Order)	Savepoint
createModel	Root → Folder → Model	Yes
ensureModel	Root → Folder → Model (SELECT FOR SHARE)	Yes
registerFile	Root → Folder → File	Yes
changeFileOwner	Root → File (model FK key-share)	Yes
updateModelMetadata	Model (FOR NO KEY UPDATE)	Yes
changeFileStatus	File (FOR UPDATE)	Yes
Explicit grouping decide	Root → Folder → Model (same as ensureModel)	Yes
Worker publish+register	Root → Folder → File (same as registerFile)	Yes

Invariant: Never hold job/item locks while entering catalog mutations. The worker's file lock (.meshvault-import.lock) is outside database transactions.

8. Test Plan (RED → GREEN)

8.1 New Test Cases

Test	Scenario	Expected
testExplicitGroupingWithModelId	Provide existing model_id in context	Reuses model, validates folder/title match
testExplicitGroupingNewModel	Provide folder_path + model_name + create_new	Creates new model at exact path
testExplicitGroupingFindOrCreate	Provide folder_path + model_name (no create_new)	Finds existing or creates
testExplicitGroupingAmbiguityRejected	Two models match folder/title (legacy data)	DomainException
testGroupingDecisionsPersisted	Import with explicit context	Decisions recoverable after job creation
testGroupingDecisionsRecovered	Resume interrupted job	Reuses recorded model_ids
testWorkerPublishAtomic	Crash simulation between copy/register	No orphan temp, no duplicate suffix
testWorkerReplayIdempotent	Re-run worker on completed item	Adopts existing file, no re-copy
testWorkerCatalogFailureRollback	Catalog error after file copy	Temp cleaned, no orphan on disk
testStableModelIdentity	Multiple imports of same source tree	Same model IDs reused (not recreated)

8.2 Mutation Tests (Negative Controls)

Mutation	Expected Failure
Remove FOR SHARE in ensureModel	Ambiguous title not detected
Remove temp-file atomic rename	Orphan temp files on crash
Skip decision persistence	Recovered job uses guessed grouping

9. Backward Compatibility

• LegacyGuessedImportGroupingAdapter unchanged — remains default in container
• New ExplicitImportGroupingAdapter is opt-in via container configuration
• No schema changes to existing tables (new table only)
• Existing ImportService::import() signature extended with optional $groupingContext

10. Container Wiring (config/container.php)

// Legacy (default)
$container->set(ImportService::class, fn ($c) => new ImportService(
    $c->get(FileEnumeratorInterface::class),
    $c->get(ImportJobRepositoryInterface::class),
    $c->get(LegacyGuessedImportGroupingAdapter::class),  // Existing
    $c->get(SpaceChecker::class),
    $c->get(string::class, 'library_root'),
));

// Explicit (new, opt-in)
$container->set(ExplicitImportGroupingAdapter::class, fn ($c) => new ExplicitImportGroupingAdapter(
    $c->get(CatalogRepositoryInterface::class),
    $c->get(ImportJobRepositoryInterface::class),
));

$container->set(ImportService::class, fn ($c) => new ImportService(
    $c->get(FileEnumeratorInterface::class),
    $c->get(ImportJobRepositoryInterface::class),
    $c->get(ExplicitImportGroupingAdapter::class),  // New
    $c->get(SpaceChecker::class),
    $c->get(string::class, 'library_root'),
));

11. Acceptance Criteria (Per #28 Bullets)

Bullet	Evidence
1. Transactional multipart aggregate	Existing CatalogService::atomic + aggregate() — Tasks 1-4
2. Shared service contracts consumed by import/metadata/status	ImportService uses CatalogServiceInterface; ImportWorker uses same; metadata/status via CatalogService — Tasks 1-4
3. Explicit reassignment/rollback/FK evidence	reassignFiles, detachFiles, changeFileOwner with FK KEY SHARE — Task 4
NEW: Explicit confirmed grouping	ExplicitImportGroupingAdapter with persisted decisions — This design
NEW: Stable model identity	findModel, getOrCreateModel, decision recovery — This design
NEW: Crash-safe publish	Atomic temp-rename + catalog register in savepoint — This design

12. Out of Scope (Future Milestones)

• UI for grouping confirmation (#14)
• Scanner incremental/resumable (#15)
• Missing-file replace/remove UI (#25)
• Progress/ETA/pause/resume contracts (#26)
• Retry/backoff/dead-letter contracts (#27)
• Full-text search/facets (#29)
• Duplicate detection/review queue (#30)

13. Implementation Sequence

1. Migration 1: import_jobs_decisions table
2. Interface: ExplicitImportGroupingAdapterInterface
3. Implementation: ExplicitImportGroupingAdapter
4. Repository extension: recordGroupingDecision, getGroupingDecisions, updateItemManagedPath
5. Catalog extension: findModel, listModelsInFolder, getOrCreateModel
6. ImportService integration: Optional $groupingContext parameter
7. Worker atomic publish: Temp copy → atomic rename + register in savepoint
8. Tests: RED → GREEN for all cases in §8
9. Mutation verification: Two negative controls
10. Documentation: Update ledger and project-status

14. Risk Assessment

Risk	Mitigation
Lock order violation	All new paths follow root→folder→model→file; code review + mutation tests
Savepoint leakage	All catalog ops use existing atomic() wrapper; no raw transactions
Decision table growth	One row per file per job; bounded by job size; cleaned with job
Temp file cleanup on crash	finally block in worker; temp files in library root (same FS)
Backward compatibility break	Legacy adapter unchanged; explicit adapter opt-in only
Migration failure on existing data	Preflight checks (Task 1 pattern); reject if unassigned/ambiguous

---

Next Steps: Implement RED tests first, then GREEN implementation per sequence in §13. No commits to main until fresh whole-branch review.