488 lines
19 KiB
Markdown
488 lines
19 KiB
Markdown
# Bounded #28 Catalog Contract — Explicit Grouping & Stable Identity
|
|
|
|
**Status:** Design phase — not implemented. Extends Tasks 1-4 contracts without breaking them.
|
|
**Baseline:** `main` at `9dd8c07` (Tasks 1-4 integrated, #28 catalog core closed).
|
|
|
|
## 1. Problem Statement
|
|
|
|
The current `LegacyGuessedImportGroupingAdapter` implements **guessed** grouping:
|
|
- First directory component under source root → model name
|
|
- Files in source root → model named after source directory
|
|
- Stateless, fresh lookup on every call via `ensureModel()`
|
|
- **No user confirmation**, no persistence of grouping decisions
|
|
|
|
Gaps from reconciliation table:
|
|
- **#18**: Confirmed explicit grouping (user confirmation) missing
|
|
- **#20**: Explicit user-confirmed grouping (not guessed); stable model identity across imports; grouping persistence
|
|
- **#24**: Crash between publish and suffix save → duplicate suffixed copies; orphaned published files on catalog failure; replay idempotency
|
|
|
|
## 2. Design Principles
|
|
|
|
1. **Retain PHP/PDO/Phinx and existing repositories** — no schema rewrites, new migrations only
|
|
2. **Extend, don't replace** — `CatalogServiceInterface`/`CatalogRepositoryInterface` unchanged
|
|
3. **Explicit grouping is opt-in** — legacy guessed adapter remains for backward compatibility
|
|
4. **Bounded scope** — no UI, no scheduler, no async workers; pure service/contract layer
|
|
5. **Lock order preserved** — root → folder → model → file (same as Task 4)
|
|
6. **Savepoint semantics** — caller transaction owns commit/rollback; catalog operations use savepoints
|
|
|
|
## 3. New Contracts (Interfaces)
|
|
|
|
### 3.1 Explicit Grouping Adapter Interface
|
|
|
|
```php
|
|
// src/Indexer/Contracts/ExplicitImportGroupingAdapterInterface.php
|
|
namespace MeshVault\Indexer\Contracts;
|
|
|
|
use MeshVault\Indexer\ScannedFile;
|
|
|
|
interface ExplicitImportGroupingAdapterInterface
|
|
{
|
|
public function modelFor(string $sourcePath, ScannedFile $file, array $context = []): int;
|
|
public function persistDecisions(int $jobId, array $decisions): void;
|
|
public function recoverDecisions(int $jobId): array;
|
|
}
|
|
```
|
|
|
|
### 3.2 Explicit Grouping Adapter Implementation
|
|
|
|
```php
|
|
// src/Indexer/ExplicitImportGroupingAdapter.php
|
|
namespace MeshVault\Indexer;
|
|
|
|
use MeshVault\Indexer\Contracts\ExplicitImportGroupingAdapterInterface;
|
|
use MeshVault\Indexer\ScannedFile;
|
|
use MeshVault\Repository\CatalogRepositoryInterface;
|
|
use MeshVault\Repository\ImportJobRepositoryInterface;
|
|
use DomainException;
|
|
use InvalidArgumentException;
|
|
|
|
final class ExplicitImportGroupingAdapter implements ExplicitImportGroupingAdapterInterface
|
|
{
|
|
public function __construct(
|
|
private readonly CatalogRepositoryInterface $catalog,
|
|
private readonly ImportJobRepositoryInterface $jobs,
|
|
) {}
|
|
|
|
public function modelFor(string $sourcePath, ScannedFile $file, array $context = []): int
|
|
{
|
|
if (isset($context['model_id'])) {
|
|
$modelId = (int) $context['model_id'];
|
|
$aggregate = $this->catalog->aggregate($modelId);
|
|
if ($aggregate === null) {
|
|
throw new DomainException('Explicit model not found.');
|
|
}
|
|
if (isset($context['model_name']) && $aggregate['model']['name'] !== $context['model_name']) {
|
|
throw new DomainException('Model title mismatch.');
|
|
}
|
|
if (isset($context['folder_path'])) {
|
|
$expectedFolder = '/' . ltrim($context['folder_path'], '/');
|
|
if ($aggregate['folder']['path'] !== $expectedFolder) {
|
|
throw new DomainException('Model folder mismatch.');
|
|
}
|
|
}
|
|
return $modelId;
|
|
}
|
|
|
|
if (isset($context['folder_path']) && isset($context['model_name'])) {
|
|
$folderPath = '/' . ltrim($context['folder_path'], '/');
|
|
$modelName = $context['model_name'];
|
|
if ($context['create_new'] ?? false) {
|
|
return $this->catalog->atomic(fn () => $this->catalog->createModel($folderPath, $modelName));
|
|
}
|
|
return $this->catalog->atomic(fn () => $this->catalog->ensureModel($folderPath, $modelName));
|
|
}
|
|
|
|
$parent = trim(dirname(str_replace('\\', '/', $file->relativePath)), '/');
|
|
$group = $parent === '' || $parent === '.'
|
|
? basename(rtrim(str_replace('\\', '/', $sourcePath), '/'))
|
|
: explode('/', $parent)[0];
|
|
return $this->catalog->atomic(fn () => $this->catalog->ensureModel('/' . $group, $group));
|
|
}
|
|
|
|
public function persistDecisions(int $jobId, array $decisions): void
|
|
{
|
|
$this->jobs->atomic(function () use ($jobId, $decisions) {
|
|
foreach ($decisions as $decision) {
|
|
$this->jobs->recordGroupingDecision(
|
|
$jobId,
|
|
$decision['source_path'],
|
|
$decision['model_id'],
|
|
$decision['context'] ?? []
|
|
);
|
|
}
|
|
});
|
|
}
|
|
|
|
public function recoverDecisions(int $jobId): array
|
|
{
|
|
return $this->jobs->getGroupingDecisions($jobId);
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3.3 Import Job Repository Extensions
|
|
|
|
```php
|
|
// src/Repository/ImportJobRepositoryInterface.php (additions)
|
|
namespace MeshVault\Repository;
|
|
|
|
interface ImportJobRepositoryInterface
|
|
{
|
|
// ... existing methods ...
|
|
public function recordGroupingDecision(int $jobId, string $sourcePath, int $modelId, array $context): void;
|
|
public function getGroupingDecisions(int $jobId): array;
|
|
}
|
|
```
|
|
|
|
### 3.4 Catalog Service Extensions (for stable identity)
|
|
|
|
```php
|
|
// src/Catalog/CatalogServiceInterface.php (additions)
|
|
namespace MeshVault\Catalog;
|
|
|
|
interface CatalogServiceInterface
|
|
{
|
|
// ... existing methods ...
|
|
public function findModel(string $folderPath, string $modelName): ?array;
|
|
public function listModelsInFolder(string $folderPath): array;
|
|
public function getOrCreateModel(string $folderPath, string $modelName): array;
|
|
}
|
|
```
|
|
|
|
## 4. Database Migrations
|
|
|
|
### 4.1 Migration: `20261014120001_import_grouping_decisions.php`
|
|
|
|
```php
|
|
<?php
|
|
declare(strict_types=1);
|
|
|
|
use Phinx\Migration\AbstractMigration;
|
|
|
|
final class ImportGroupingDecisions extends AbstractMigration
|
|
{
|
|
public function up(): void
|
|
{
|
|
$this->table('import_jobs_decisions', ['id' => false, 'primary_key' => ['job_id', 'source_path']])
|
|
->addColumn('job_id', 'integer', ['signed' => false])
|
|
->addColumn('source_path', 'text')
|
|
->addColumn('model_id', 'integer', ['signed' => false])
|
|
->addColumn('context', 'jsonb', ['default' => '{}'])
|
|
->addColumn('created_at', 'timestamp', ['default' => 'CURRENT_TIMESTAMP'])
|
|
->addForeignKey('job_id', 'import_jobs', 'id', ['delete' => 'CASCADE', 'update' => 'NO_ACTION'])
|
|
->addIndex(['model_id'], ['name' => 'idx_import_decisions_model'])
|
|
->create();
|
|
}
|
|
|
|
public function down(): void
|
|
{
|
|
$this->table('import_jobs_decisions')->drop()->save();
|
|
}
|
|
}
|
|
```
|
|
|
|
### 4.2 Migration: `20261014120002_model_identity_stability.php`
|
|
|
|
```php
|
|
<?php
|
|
declare(strict_types=1);
|
|
|
|
use Phinx\Migration\AbstractMigration;
|
|
|
|
final class ModelIdentityStability extends AbstractMigration
|
|
{
|
|
public function up(): void
|
|
{
|
|
// No schema changes — this migration documents the stability contract
|
|
// Models are now immutable in identity (folder_id + name unique)
|
|
// Files reference models via nullable FK (from Task 4)
|
|
// Explicit grouping decisions recorded in import_jobs_decisions
|
|
}
|
|
|
|
public function down(): void
|
|
{
|
|
// No-op: identity stability is a contract, not a schema change
|
|
}
|
|
}
|
|
```
|
|
|
|
## 5. Import Service Integration
|
|
|
|
```php
|
|
// src/Indexer/ImportService.php (modified)
|
|
namespace MeshVault\Indexer;
|
|
|
|
use MeshVault\Indexer\Contracts\ExplicitImportGroupingAdapterInterface;
|
|
use MeshVault\Indexer\Contracts\FileEnumeratorInterface;
|
|
use MeshVault\Repository\ImportJobRepositoryInterface;
|
|
use Throwable;
|
|
|
|
final readonly class ImportService
|
|
{
|
|
public function __construct(
|
|
private FileEnumeratorInterface $enumerator,
|
|
private ImportJobRepositoryInterface $jobs,
|
|
private ExplicitImportGroupingAdapterInterface $groupingAdapter, // Changed from LegacyGuessed
|
|
private SpaceChecker $space,
|
|
private string $libraryRoot,
|
|
) {}
|
|
|
|
/** @param array<string, mixed> $groupingContext Optional explicit grouping context per file */
|
|
public function import(string $sourcePath, array $extensions = [], array $groupingContext = []): int
|
|
{
|
|
$scan = $this->enumerator->enumerate($sourcePath, $extensions);
|
|
if ($scan->files === []) {
|
|
$detail = $scan->errors === [] ? 'No importable files were found.' : $scan->errors[0]->message;
|
|
throw new ImportException('SCAN_EMPTY', $detail);
|
|
}
|
|
$this->space->check($this->libraryRoot, $scan->totalBytes);
|
|
|
|
try {
|
|
return $this->jobs->atomic(
|
|
fn (): int => $this->createJobWithGrouping($sourcePath, $scan, $groupingContext)
|
|
);
|
|
} catch (ImportException $error) {
|
|
throw $error;
|
|
} catch (Throwable $error) {
|
|
throw new ImportException('DB_ERROR', 'Could not persist the import job.', $error);
|
|
}
|
|
}
|
|
|
|
/** @return list<array<string, mixed>> */
|
|
private function createJobWithGrouping(string $sourcePath, ScanResult $scan, array $groupingContext): int
|
|
{
|
|
$items = [];
|
|
$decisions = [];
|
|
|
|
foreach ($scan->files as $file) {
|
|
$fileContext = $groupingContext[$file->relativePath] ?? $groupingContext['*'] ?? [];
|
|
$modelId = $this->groupingAdapter->modelFor($sourcePath, $file, $fileContext);
|
|
|
|
$items[] = [
|
|
'source_path' => $file->sourcePath,
|
|
'size' => $file->size,
|
|
'model_id' => $modelId,
|
|
];
|
|
|
|
$decisions[] = [
|
|
'source_path' => $file->sourcePath,
|
|
'model_id' => $modelId,
|
|
'context' => $fileContext,
|
|
];
|
|
}
|
|
|
|
foreach ($scan->errors as $error) {
|
|
$items[] = [
|
|
'source_path' => $error->sourcePath,
|
|
'size' => 0,
|
|
'model_id' => null,
|
|
'status' => 'Failed',
|
|
'failure_reason' => $error->code . ': ' . $error->message,
|
|
];
|
|
}
|
|
|
|
$jobId = $this->jobs->create($sourcePath, $items);
|
|
|
|
if ($decisions !== []) {
|
|
$this->groupingAdapter->persistDecisions($jobId, $decisions);
|
|
}
|
|
|
|
return $jobId;
|
|
}
|
|
}
|
|
```
|
|
|
|
## 6. Worker Recovery & Idempotency (Addresses #24)
|
|
|
|
### 6.1 Current Gap
|
|
- Worker persists planned name **before** copy
|
|
- Saves collision suffix **after** publication
|
|
- Crash between → duplicate suffixed copies on replay
|
|
- Catalog failure after publish → orphaned file on disk
|
|
|
|
### 6.2 Solution: Atomic Publish-Then-Register
|
|
|
|
```php
|
|
// src/Indexer/ImportWorker.php (modified publish method)
|
|
private function publish(array $plan, string $destination): string
|
|
{
|
|
$directory = dirname($destination);
|
|
if (!is_dir($directory) && !@mkdir($directory, 0775, true) && !is_dir($directory)) {
|
|
throw new ImportException('COPY_FAILED', 'Cannot create the managed destination directory.');
|
|
}
|
|
|
|
$destination = $this->pathSafety->ensureDestSafe($destination);
|
|
clearstatcache(true, $destination);
|
|
|
|
if (is_file($destination)) {
|
|
if (hash_file('sha256', $destination) === $plan['hash']) {
|
|
return $destination;
|
|
}
|
|
} else {
|
|
$this->spaceChecker->check($directory, $plan['size']);
|
|
}
|
|
|
|
// CRITICAL CHANGE: Copy to TEMP, then atomic rename + catalog register in ONE transaction
|
|
$tempPath = $destination . '.tmp.' . bin2hex(random_bytes(8));
|
|
try {
|
|
$this->copier->copy($plan['source'], $tempPath, $plan['size'], $plan['hash']);
|
|
|
|
$managedPath = $this->relative($tempPath);
|
|
$this->catalog->atomic(function () use ($tempPath, $destination, $plan, &$managedPath) {
|
|
if (!@rename($tempPath, $destination)) {
|
|
throw new ImportException('COPY_FAILED', 'Cannot finalize the published file.');
|
|
}
|
|
|
|
$managedPath = $this->relative($destination);
|
|
|
|
$fileId = $this->catalog->registerFile(
|
|
$plan['model_id'],
|
|
$managedPath,
|
|
$plan['hash'],
|
|
$plan['size'],
|
|
$plan['mime'] ?? null,
|
|
'Present'
|
|
);
|
|
|
|
$this->jobs->updateItemManagedPath($plan['item_id'], $managedPath, $fileId);
|
|
});
|
|
|
|
return $destination;
|
|
} catch (Throwable $error) {
|
|
if (is_file($tempPath)) {
|
|
@unlink($tempPath);
|
|
}
|
|
throw $error;
|
|
}
|
|
}
|
|
```
|
|
|
|
### 6.3 New Job Repository Method
|
|
|
|
```php
|
|
// src/Repository/ImportJobRepositoryInterface.php (addition)
|
|
public function updateItemManagedPath(int $itemId, string $managedPath, int $fileId): void;
|
|
```
|
|
|
|
## 7. Lock Order & Transaction Boundaries
|
|
|
|
| Operation | Locks Acquired (Order) | Savepoint |
|
|
|-----------|------------------------|-----------|
|
|
| `createModel` | Root → Folder → Model | Yes |
|
|
| `ensureModel` | Root → Folder → Model (SELECT FOR SHARE) | Yes |
|
|
| `registerFile` | Root → Folder → File | Yes |
|
|
| `changeFileOwner` | Root → File (model FK key-share) | Yes |
|
|
| `updateModelMetadata` | Model (FOR NO KEY UPDATE) | Yes |
|
|
| `changeFileStatus` | File (FOR UPDATE) | Yes |
|
|
| **Explicit grouping decide** | Root → Folder → Model (same as ensureModel) | Yes |
|
|
| **Worker publish+register** | Root → Folder → File (same as registerFile) | Yes |
|
|
|
|
**Invariant:** Never hold job/item locks while entering catalog mutations. The worker's file lock (`.meshvault-import.lock`) is **outside** database transactions.
|
|
|
|
## 8. Test Plan (RED → GREEN)
|
|
|
|
### 8.1 New Test Cases
|
|
|
|
| Test | Scenario | Expected |
|
|
|------|----------|----------|
|
|
| `testExplicitGroupingWithModelId` | Provide existing model_id in context | Reuses model, validates folder/title match |
|
|
| `testExplicitGroupingNewModel` | Provide folder_path + model_name + create_new | Creates new model at exact path |
|
|
| `testExplicitGroupingFindOrCreate` | Provide folder_path + model_name (no create_new) | Finds existing or creates |
|
|
| `testExplicitGroupingAmbiguityRejected` | Two models match folder/title (legacy data) | DomainException |
|
|
| `testGroupingDecisionsPersisted` | Import with explicit context | Decisions recoverable after job creation |
|
|
| `testGroupingDecisionsRecovered` | Resume interrupted job | Reuses recorded model_ids |
|
|
| `testWorkerPublishAtomic` | Crash simulation between copy/register | No orphan temp, no duplicate suffix |
|
|
| `testWorkerReplayIdempotent` | Re-run worker on completed item | Adopts existing file, no re-copy |
|
|
| `testWorkerCatalogFailureRollback` | Catalog error after file copy | Temp cleaned, no orphan on disk |
|
|
| `testStableModelIdentity` | Multiple imports of same source tree | Same model IDs reused (not recreated) |
|
|
|
|
### 8.2 Mutation Tests (Negative Controls)
|
|
|
|
| Mutation | Expected Failure |
|
|
|----------|------------------|
|
|
| Remove FOR SHARE in `ensureModel` | Ambiguous title not detected |
|
|
| Remove temp-file atomic rename | Orphan temp files on crash |
|
|
| Skip decision persistence | Recovered job uses guessed grouping |
|
|
|
|
## 9. Backward Compatibility
|
|
|
|
- `LegacyGuessedImportGroupingAdapter` **unchanged** — remains default in container
|
|
- New `ExplicitImportGroupingAdapter` is opt-in via container configuration
|
|
- No schema changes to existing tables (new table only)
|
|
- Existing `ImportService::import()` signature extended with optional `$groupingContext`
|
|
|
|
## 10. Container Wiring (config/container.php)
|
|
|
|
```php
|
|
// Legacy (default)
|
|
$container->set(ImportService::class, fn ($c) => new ImportService(
|
|
$c->get(FileEnumeratorInterface::class),
|
|
$c->get(ImportJobRepositoryInterface::class),
|
|
$c->get(LegacyGuessedImportGroupingAdapter::class), // Existing
|
|
$c->get(SpaceChecker::class),
|
|
$c->get(string::class, 'library_root'),
|
|
));
|
|
|
|
// Explicit (new, opt-in)
|
|
$container->set(ExplicitImportGroupingAdapter::class, fn ($c) => new ExplicitImportGroupingAdapter(
|
|
$c->get(CatalogRepositoryInterface::class),
|
|
$c->get(ImportJobRepositoryInterface::class),
|
|
));
|
|
|
|
$container->set(ImportService::class, fn ($c) => new ImportService(
|
|
$c->get(FileEnumeratorInterface::class),
|
|
$c->get(ImportJobRepositoryInterface::class),
|
|
$c->get(ExplicitImportGroupingAdapter::class), // New
|
|
$c->get(SpaceChecker::class),
|
|
$c->get(string::class, 'library_root'),
|
|
));
|
|
```
|
|
|
|
## 11. Acceptance Criteria (Per #28 Bullets)
|
|
|
|
| Bullet | Evidence |
|
|
|--------|----------|
|
|
| **1. Transactional multipart aggregate** | Existing `CatalogService::atomic` + `aggregate()` — Tasks 1-4 |
|
|
| **2. Shared service contracts consumed by import/metadata/status** | `ImportService` uses `CatalogServiceInterface`; `ImportWorker` uses same; metadata/status via `CatalogService` — Tasks 1-4 |
|
|
| **3. Explicit reassignment/rollback/FK evidence** | `reassignFiles`, `detachFiles`, `changeFileOwner` with FK KEY SHARE — Task 4 |
|
|
| **NEW: Explicit confirmed grouping** | `ExplicitImportGroupingAdapter` with persisted decisions — This design |
|
|
| **NEW: Stable model identity** | `findModel`, `getOrCreateModel`, decision recovery — This design |
|
|
| **NEW: Crash-safe publish** | Atomic temp-rename + catalog register in savepoint — This design |
|
|
|
|
## 12. Out of Scope (Future Milestones)
|
|
|
|
- UI for grouping confirmation (#14)
|
|
- Scanner incremental/resumable (#15)
|
|
- Missing-file replace/remove UI (#25)
|
|
- Progress/ETA/pause/resume contracts (#26)
|
|
- Retry/backoff/dead-letter contracts (#27)
|
|
- Full-text search/facets (#29)
|
|
- Duplicate detection/review queue (#30)
|
|
|
|
## 13. Implementation Sequence
|
|
|
|
1. **Migration 1**: `import_jobs_decisions` table
|
|
2. **Interface**: `ExplicitImportGroupingAdapterInterface`
|
|
3. **Implementation**: `ExplicitImportGroupingAdapter`
|
|
4. **Repository extension**: `recordGroupingDecision`, `getGroupingDecisions`, `updateItemManagedPath`
|
|
5. **Catalog extension**: `findModel`, `listModelsInFolder`, `getOrCreateModel`
|
|
6. **ImportService integration**: Optional `$groupingContext` parameter
|
|
7. **Worker atomic publish**: Temp copy → atomic rename + register in savepoint
|
|
8. **Tests**: RED → GREEN for all cases in §8
|
|
9. **Mutation verification**: Two negative controls
|
|
10. **Documentation**: Update ledger and project-status
|
|
|
|
## 14. Risk Assessment
|
|
|
|
| Risk | Mitigation |
|
|
|------|------------|
|
|
| Lock order violation | All new paths follow root→folder→model→file; code review + mutation tests |
|
|
| Savepoint leakage | All catalog ops use existing `atomic()` wrapper; no raw transactions |
|
|
| Decision table growth | One row per file per job; bounded by job size; cleaned with job |
|
|
| Temp file cleanup on crash | `finally` block in worker; temp files in library root (same FS) |
|
|
| Backward compatibility break | Legacy adapter unchanged; explicit adapter opt-in only |
|
|
| Migration failure on existing data | Preflight checks (Task 1 pattern); reject if unassigned/ambiguous |
|
|
|
|
---
|
|
|
|
**Next Steps:** Implement RED tests first, then GREEN implementation per sequence in §13. No commits to `main` until fresh whole-branch review. |