keith/MeshVault
1
0 Fork 0
Code Issues 44 Pull Requests Actions Packages Projects Releases Wiki Activity

Background import worker and job queue #25

New Issue
Open
opened 2026-10-10 22:21:49 +00:00 by keith · 1 comment
keith commented 2026-10-10 22:21:49 +00:00
Owner
Copy link
Copy source

Context

Implement this MVP work package under M3 - Import & Indexing Engine, following the managed-storage decision in #1.

Acceptance criteria

  • Use PostgreSQL-backed jobs and a separate PHP worker; imports outlive HTTP requests.
  • Claim jobs atomically, serialize filesystem mutations, and detect stale jobs with recoverable states.
  • Restart without duplicate copies or catalog records; test graceful shutdown and idempotent retry.
  • Create models, folders and associated records from confirmed grouping manifest; integrate the catalog repository service when available.

Blocked by: #24, #11, #28, #28

Sequence note: implement #28 before the worker despite its M4 grouping; importer and web routes share repository services.

Sequence note: implement #28 before the worker despite its M4 grouping; importer and web routes share repository services.

## Context Implement this MVP work package under M3 - Import & Indexing Engine, following the managed-storage decision in #1. ## Acceptance criteria - [ ] Use PostgreSQL-backed jobs and a separate PHP worker; imports outlive HTTP requests. - [ ] Claim jobs atomically, serialize filesystem mutations, and detect stale jobs with recoverable states. - [ ] Restart without duplicate copies or catalog records; test graceful shutdown and idempotent retry. - [ ] Create models, folders and associated records from confirmed grouping manifest; integrate the catalog repository service when available. Blocked by: #24, #11, #28, #28 Sequence note: implement #28 before the worker despite its M4 grouping; importer and web routes share repository services. Sequence note: implement #28 before the worker despite its M4 grouping; importer and web routes share repository services.
keith added this to the M3 - Import & Indexing Engine milestone 2026-10-10 22:21:49 +00:00
keith added the area:indexertype:featurepriority:medium labels 2026-10-10 22:21:49 +00:00
keith commented 2026-10-12 18:55:35 +00:00
Author
Owner
Copy link
Copy source

Acceptance reconciliation at c1ef2ee: keeping open. Queue, worker and locking exist. Confirmed manifest and collision/DB-failure replay remain incomplete. Existing success-path tests do not establish full crash recovery. This is a source/test review, not a newly executed crash or restore experiment. Fresh Windows suite: 291 tests, 1801 assertions, 22 skips; lint passes. PostgreSQL fixture success does not waive these feature criteria.

Acceptance reconciliation at c1ef2ee: keeping open. Queue, worker and locking exist. Confirmed manifest and collision/DB-failure replay remain incomplete. Existing success-path tests do not establish full crash recovery. This is a source/test review, not a newly executed crash or restore experiment. Fresh Windows suite: 291 tests, 1801 assertions, 22 skips; lint passes. PostgreSQL fixture success does not waive these feature criteria.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
area:backend
PHP application logic
 area:db
PostgreSQL schema and migrations
 area:frontend
PHP views, Tailwind, browser JS
 area:indexer
Scanning, hashing, import pipeline
 area:infra
Docker, CI, deployment
 priority:high
Blocking or critical path
 priority:low
Nice to have
 priority:medium
Normal priority
 type:bug
Defect
 type:chore
Tooling, scaffolding, maintenance
 type:decision
Architecture decision record
 type:docs
Documentation
 type:feature
New user-facing capability
No labels area:indexer priority:medium type:feature
Milestone
No items
No Milestone M3 - Import & Indexing Engine
Projects
Clear projects
No projects
Assignees
Clear assignees
keith (Keith Solomon)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: keith/MeshVault#25