Tools for conservative, file-level recovery from a damaged ext4 filesystem on an LVM logical volume when one physical volume (PV) is missing, failed, or has been replaced with a blank disk.

These scripts were written after a failed PV in a linear LVM volume. The replacement PV could be recreated with the original UUID and geometry, which made the LV structurally activatable again, but the filesystem blocks that had lived on the failed disk were now effectively zero-filled. That means ordinary reads can appear to succeed even when the original data is gone.

The important rule is therefore:

Do not treat a successful debugfs dump as proof that a file is intact. First prove that every physical extent for that file is outside the known-dead filesystem block range.

The scripts do exactly that and only recover files classified SAFE.

Files

scan.sh       Scan an arbitrary source directory and classify files.
recover.sh    Recover only files classified SAFE by the matching scan.
salvage.py    Shared recovery engine.

Basic usage

Scan any directory inside the damaged filesystem:

./scan.sh '/Warez/Books'
./scan.sh '/Warez/3D Printing'
./scan.sh '/Media/Some Folder'

Then recover the SAFE files from the same path:

./recover.sh '/Warez/Books'

Always quote paths containing spaces.

The source tree is mirrored beneath /mnt/recovery:

/Warez/Books         -> /mnt/recovery/Warez/Books
/Warez/3D Printing   -> /mnt/recovery/Warez/3D Printing
/Media/Some Folder   -> /mnt/recovery/Media/Some Folder

Each scan creates uniquely named manifest and summary files under /mnt/recovery. The recovery run creates a TSV log and summary there as well.

Before using the scripts on another recovery

salvage.py currently contains recovery-specific constants near the top:

LV = "/dev/mapper/plex--new-plex--new"
DEST_MOUNT = Path("/mnt/recovery")
DEAD_START = 3418636288
DEAD_END = 4395389951

Do not blindly reuse the dead block range on another filesystem. Determine it again from the damaged LV layout using the process below.

The current scripts assume one contiguous dead filesystem-block range. If a failed PV supplied multiple non-contiguous LV segments, or more than one PV failed, the overlap logic must be changed to support multiple dead ranges.

Determining the known-dead block range

There are two coordinate systems involved:

dmsetup reports logical-volume offsets and lengths in 512-byte sectors.
ext4 extents reported by debugfs use filesystem blocks (4096 bytes on the filesystem this project was written for).

The dead range is the LV sector range backed by the failed/blank PV, converted into ext4 filesystem-block numbers.

1. Keep the damaged LV read-only

Activate the VG/LV as needed, but do not mount it read-write and do not run a repairing e2fsck before salvage is complete.

When working with a missing PV, partial activation may be required:

vgchange -ay --activationmode partial <vg-name>

Then force the LV read-only at the block-device layer:

blockdev --setro /dev/mapper/<vg--name>-<lv--name>
blockdev --getro /dev/mapper/<vg--name>-<lv--name>

blockdev --getro must return:

The scripts refuse to run unless the configured LV is read-only.

2. Determine the ext4 filesystem block size

Normal debugfs may refuse to open a badly damaged filesystem because of bitmap/checksum failures. Catastrophic mode (-c) is intentionally used here because it opens the filesystem read-only and avoids depending on allocation bitmaps.

DEBUGFS_PAGER=cat debugfs -c \
  -R 'stats' \
  /dev/mapper/<vg--name>-<lv--name> | grep 'Block size'

For this recovery the result was:

Block size: 4096

With a 4096-byte filesystem block, each filesystem block contains exactly 8 dmsetup sectors:

4096 / 512 = 8

3. Identify the LV segment that belongs to the failed/blank PV

Best case: the PV is still missing

With the LV activated in partial mode, inspect the device-mapper table:

dmsetup table /dev/mapper/<vg--name>-<lv--name>

During this incident the relevant table was:

0           15628050432 linear 8:16 2048
15628050432 11721039872 linear 8:32 2048
27349090304 7814029312  linear 252:0 0
35163119616 11721039872 linear 8:48 2048
46884159488 3907010560  linear 8:32 11721041920

The suspicious segment was:

27349090304 7814029312 linear 252:0 0

Here:

27349090304 = LV start sector
7814029312  = segment length in sectors
252:0       = device-mapper device backing that segment

Inspect the referenced mapper device:

cat /sys/dev/block/252:0/dm/name

Then inspect its table:

dmsetup table "$(cat /sys/dev/block/252:0/dm/name)"

For this failure it returned:

0 7814029312 error

That error target proved that the entire LV segment was the missing PV region.

If the failed PV has already been recreated on a blank replacement disk

Once the replacement PV is recreated and the historical VG metadata is restored, the LV may no longer contain an error target. The blank replacement disk will appear as an ordinary linear target.

In that case:

identify the replacement PV (pvs, lsblk, and the restored VG metadata are useful);
identify its major:minor number with lsblk -o NAME,MAJ:MIN;
inspect dmsetup table for the LV;
find every LV segment backed by that replacement PV;
treat those LV sector ranges as dead, because the replacement contains no original filesystem data.

Historical metadata is extremely useful here:

vgcfgrestore -l <vg-name>
lvs --segments -o lv_name,vg_name,seg_start,seg_size,devices
pvs -o pv_name,pv_uuid,pv_size,pv_free

If exact sector arithmetic is needed, prefer the exact dmsetup table values over rounded human-readable LVM sizes.

4. Convert the dead LV sector range to ext4 block numbers

dmsetup units are always 512-byte sectors.

Use:

START_BLOCK = START_SECTOR * 512 / FILESYSTEM_BLOCK_SIZE
BLOCK_COUNT = LENGTH_SECTORS * 512 / FILESYSTEM_BLOCK_SIZE
END_BLOCK   = START_BLOCK + BLOCK_COUNT - 1

For this recovery:

START_SECTOR         = 27349090304
LENGTH_SECTORS       = 7814029312
FILESYSTEM_BLOCK_SIZE = 4096

Therefore:

START_BLOCK = 27349090304 * 512 / 4096
            = 3418636288

BLOCK_COUNT = 7814029312 * 512 / 4096
            = 976753664

END_BLOCK   = 3418636288 + 976753664 - 1
            = 4395389951

The known-dead ext4 block range for this incident is therefore:

3418636288 - 4395389951

Equivalent Bash arithmetic:

block_size=4096
start_sector=27349090304
length_sectors=7814029312

start_block=$(( start_sector * 512 / block_size ))
block_count=$(( length_sectors * 512 / block_size ))
end_block=$(( start_block + block_count - 1 ))

printf 'dead range: %s-%s\n' "$start_block" "$end_block"

Before using the generic scripts on a different failure, update DEAD_START and DEAD_END in salvage.py with the calculated values.

What the classifications mean

SAFE
LOST
ETB_LOST
ETB_SURVIVES
UNKNOWN
OTHER

`SAFE`

All physical data extents that debugfs can resolve lie outside the known-dead filesystem-block range.

Only SAFE files are automatically recovered.

A SAFE classification means the file does not depend on the known failed-PV region. It is still sensible to validate particularly important files afterward (hash against another known copy, open/test archives, use application-specific validation, etc.).

`LOST`

At least one resolved data extent overlaps the dead filesystem-block range.

The file may be partially present, but it cannot be trusted as intact and is not automatically recovered.

`ETB_LOST`

The inode survives, but the external ext4 extent-tree block (ETB) needed to locate the file's data lives inside the dead block range.

Example seen during this incident:

EXTENTS:
(ETB0):4338778112

4338778112 is inside 3418636288-4395389951, so the mapping from file offsets to physical data blocks is gone.

This does not prove that all of the file's actual data was on the failed disk. It means ext4 has lost the map needed to find that data. Further recovery would require carving/forensic work rather than ordinary filesystem traversal.

`ETB_SURVIVES`

An external extent-tree block exists outside the known-dead range, but usable data extents were not resolved by the scanner. Investigate manually before deciding the file is lost.

`UNKNOWN`

The scanner could not obtain enough trustworthy extent information to classify the file.

Do not recover it automatically.

`OTHER`

The directory entry is not a regular file (for example, a symlink or special file). The generic recovery script currently handles regular files only.

Directory metadata can be lost too

A top-level directory entry can survive while the inode it points to is gone.

During this incident /Backups still appeared in the root directory, but:

Inode: 496369665   Type: bad type    Mode: 0000
Size: 0
Links: 0
Blockcount: 0

That means the directory's inode metadata was in the dead region. Without the directory inode, debugfs has no tree to traverse, even though some child inodes or file data may still physically survive elsewhere.

The generic scanner will stop immediately in this situation. Recovering such a tree requires forensic carving or targeted searches for known filenames/signatures and is outside the normal workflow.

Recovery destination

/mnt/recovery must be a separate mounted filesystem. Confirm it before starting:

findmnt /mnt/recovery
df -hT /mnt/recovery

Do not accidentally recover back onto the damaged LV.

The recovery drive used during this incident was exFAT. That is fine for file contents, but exFAT does not preserve normal Unix ownership/mode metadata. Do not rely on debugfs dump -p to preserve metadata on such a destination; it can report an ownership error after successfully writing the file.

The generic recovery script intentionally dumps file contents without -p.

Why the scan matters even if `debugfs dump` appears to work

The failed PV in this incident was replaced with a same-sized blank disk so the original LVM geometry could be reconstructed.

That creates an important trap: reads from the formerly missing region no longer necessarily return an I/O error. They can return zero-filled blocks from the replacement disk.

Therefore this is not sufficient:

debugfs -c -R 'dump <inode> /some/output/file' <LV>

A dump can have the expected byte count while containing zeroes/corruption where the original file used blocks on the dead PV.

The scanner prevents this by checking physical extents first and only permitting automatic recovery when every resolved extent avoids the dead block range.

Manual inspection commands

Inspect a directory:

DEBUGFS_PAGER=cat debugfs -c \
  -R 'ls -l "/path/to/directory"' \
  /dev/mapper/<vg--name>-<lv--name>

Inspect a file by path:

DEBUGFS_PAGER=cat debugfs -c \
  -R 'stat "/path/to/file"' \
  /dev/mapper/<vg--name>-<lv--name>

Inspect by inode (safer when filenames contain awkward characters):

DEBUGFS_PAGER=cat debugfs -c \
  -R 'stat <123456789>' \
  /dev/mapper/<vg--name>-<lv--name>

Ask debugfs for the blocks it can resolve:

DEBUGFS_PAGER=cat debugfs -c \
  -R 'blocks <123456789>' \
  /dev/mapper/<vg--name>-<lv--name>

Dump a manually verified-safe file by inode:

DEBUGFS_PAGER=cat debugfs -c \
  -R 'dump <123456789> /mnt/recovery/recovered-file' \
  /dev/mapper/<vg--name>-<lv--name>

Verify the output byte count against the inode's Size: value:

stat -c '%s' /mnt/recovery/recovered-file

For important files, also use a format-specific validator or compare a cryptographic hash against another known-good copy when available.

Script safety behavior

The generic scripts intentionally make several conservative choices:

the damaged LV must be kernel read-only;
/mnt/recovery must be a separate mounted filesystem;
filesystem root / is refused as a scan target;
.. path components are refused;
only regular files classified SAFE are recovered;
each file is dumped by inode to a temporary filename first;
the dumped byte size must exactly match the inode's Size: value;
only after the size check is the temporary file moved into the reconstructed tree;
existing destination files with the expected size are skipped, making recovery restartable;
at least 2 GiB free is preserved on /mnt/recovery.

Do not remove these checks just to make a troublesome file copy. Investigate the troublesome file separately instead.

Incident-specific reference values

These values apply to the original plex-new recovery only and are retained here as a worked example for future troubleshooting.

VG: plex-new
LV: plex-new
Filesystem: ext4
Filesystem block size: 4096 bytes

Failed/recreated PV UUID:
QKoHgS-dCwc-tqVD-7o6R-ZoQr-6ab7-kdjYSK

Failed PV geometry:
dev_size = 7814033408 sectors
pe_start = 2048 sectors
pe_count = 953861
extent size = 8192 sectors (4 MiB)

Dead LV segment:
start  = 27349090304 sectors
length = 7814029312 sectors

Dead ext4 filesystem blocks:
3418636288 - 4395389951

Historical five-segment LV layout after metadata restoration:

start 0        -> /dev/sdb(0)
start ~7.28T   -> /dev/sdc(0)
start ~12.74T  -> failed/replacement /dev/sda1(0)
start ~16.37T  -> /dev/sdd(0)
start ~21.83T  -> remainder of /dev/sdc

The exact dmsetup values, not these rounded TiB values, should be used for future arithmetic.

Practical recovery order

When space or time is limited, recover in value order rather than disk order:

irreplaceable personal/project data;
rare firmware, installers, keys, configs, archives, and documentation;
collections that are difficult to reconstruct;
replaceable media last.

Scan first, review the summary, then recover.

Once everything worth saving has been copied and independently verified, rebuild the filesystem/array cleanly rather than attempting to return this damaged ext4 filesystem to normal service.

README.md

LVM/ext4 Salvage Tools