HALDE

HALDE recovers files from Btrfs filesystems that can no longer be mounted. For every single file it tells you how strongly the surviving metadata supports the recovery. It never writes to a source.

=== HALDE Recovery summary ===
Entries discovered       : 6
Regular files discovered : 5
Known regular-file size  : 5.94 MiB
Recoverable files        : 4
Recoverable size         : 4.99 MiB
Checksum-verified files  : 4
Recovered successfully   : 4
Recovery failures/skips  : 0

CHECKSUM_MISMATCH                   1
CHECKSUM_VERIFIED_100               4
DIR                                 1

The CHECKSUM_MISMATCH line is the point of the tool. One file no longer matched its Btrfs checksum. HALDE reported it and did not write it. The other four files were verified and recovered.

01What HALDE is

HALDE is a forensic recovery tool for Btrfs. It does not repair filesystems and does not replace btrfs-progs.

Use HALDE if all of the following conditions apply:

• The Btrfs filesystem cannot be mounted.
• btrfs-progs and btrfs restore do not recover the files.
• At least one device of the filesystem still contains readable metadata and data.

HALDE was written after a three-device Btrfs filesystem lost one device and its superblock. The standard tools recovered nothing. HALDE answers one question: which files can still be proven intact, and where are they?

Design rule: HALDE reports a gap instead of guessing. A wrong file that looks correct is harder to detect than a file that is marked as not recoverable.

Two stages

halde_scan.py reads the sources and scans them for Btrfs B-tree blocks. It checks every block against its Btrfs checksum. It rebuilds the current trees and writes the results to the metadata directory.

halde_recover.py reads the metadata directory. It assigns a status to every entry in the selected directories. With --verify-data, it checks the data against the Btrfs checksums. With --recover, it writes the recoverable files to the target directory.

The scan reads the damaged media once. After the scan, all further work uses the metadata directory and the data sections of the sources.

Terms

02What it can and cannot do

03Requirements

HALDE consists of two Python files. Nothing has to be compiled or installed. The required packages depend on the filesystem you recover.

Always required

Required under a condition

If your filesystem meets the condition and the package is missing, HALDE cannot recover the affected files. HALDE reports this condition.

zlib extents need no extra package. LZO is not implemented. The scanner needs no third-party package, except xxhash under the condition above.

Find out which packages you need

1. Start an inventory run with --show problem.
2. If the run stops with an xxhash error, install xxhash. Repeat step 1.
3. Search the output for NEEDS_ZSTD.
4. If NEEDS_ZSTD appears, install zstandard. Repeat step 1.

Btrfs geometry    : nodesize=16384 sectorsize=4096 checksum=crc32c
ZSTD support      : yes

./halde_recover.py --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img --dir '#256' --show problem \
  | grep NEEDS_ZSTD

Install the packages

# Debian, Ubuntu, Linux Mint
sudo apt install python3-zstandard python3-xxhash

# Fedora, RHEL
sudo dnf install python3-zstandard python3-xxhash

# Arch
sudo pacman -S python-zstandard python-xxhash

# openSUSE
sudo zypper install python3-zstandard python3-xxhash

# Without system packages
python3 -m venv /tmp/halde-venv
/tmp/halde-venv/bin/pip install zstandard xxhash
/tmp/halde-venv/bin/python3 ./halde_recover.py --meta-dir ...

# Check
python3 -c 'import zstandard, xxhash; print("both available")'

HALDE bundles neither package. zstandard is licensed under BSD-3-Clause, xxhash under BSD-2-Clause. HALDE also works with older zstandard builds that lack the allow_extra_data argument.

Only for the test suite

btrfs-progs 6.14 or newer. The test suite builds its own Btrfs image files with mkfs.btrfs. Older versions skip the subvolume and compression cases. The release gate treats a skipped case as a failure. HALDE itself does not need btrfs-progs.

Disk space

• The TSV files need about 250 bytes per directory entry. A test filesystem with 2,525 files produced a metadata directory of 592 KB.
• metadata.sqlite3 needs about 200 bytes per B-tree block found. It includes blocks of older generations. On an aged filesystem it can be several times larger than the current tree.

--no-metadata-tsv omits the largest export. The recoverer does not need it. The target directory needs space for the recovered data. The target directory must be on a different filesystem than the filesystem you recover.

04Before you start

sudo ddrescue -d -r3 /dev/sdb /srv/rescue/sdb.img /srv/rescue/sdb.map
sudo ddrescue -d -r3 /dev/sdc /srv/rescue/sdc.img /srv/rescue/sdc.map
sudo ddrescue -d -r3 /dev/sdd /srv/rescue/sdd.img /srv/rescue/sdd.map

Both stages accept image files wherever they accept devices. Keep the map files. They record which regions ddrescue could not read. HALDE cannot reconstruct this information.

Identify the disks

Names such as /dev/sdb can change between boots. Record the serial number of each disk.

ls -l /dev/disk/by-id/
lsblk -o NAME,SIZE,SERIAL,MODEL

Output protection

• HALDE opens every source with O_RDONLY.
• HALDE refuses an output location on the same block device as a source. This check includes device-mapper and MD layers.
• HALDE refuses an output directory that contains a source image file.
• An output directory on the same filesystem as a source image file is permitted. HALDE prints a warning about free space.

05How it works

Btrfs stores its metadata in copy-on-write B-trees. Every tree block has a header. The header contains the filesystem UUID, the logical address of the block, a generation number, the owning tree and a checksum of the block. HALDE uses this header to identify and verify blocks.

1. Superblocks. Btrfs stores superblock copies at 64 KiB, 64 MiB and 256 GiB on each device. HALDE reads all copies. It discards copies with an invalid checksum. It takes geometry and identity from the newest valid copy.
2. Physical scan. HALDE reads every source sequentially. At each sector boundary it checks for the filesystem UUID in the header. It checks each candidate against its Btrfs checksum. HALDE uses no block that fails this check.
3. Index. HALDE records every accepted block in SQLite: source, physical offset, logical address, generation, owning tree and level. The index includes blocks of older generations.
4. Current trees. HALDE starts from the superblock roots and the backup root slots. It follows parent pointers. Every step requires an exact generation match. If a block is missing, HALDE records the gap. It does not substitute an older block.
5. Export. HALDE writes the chunk mapping, directory index, inode metadata, device extents, subvolume list and checksums as TSV files.
6. Classification and recovery. The recoverer maps every data extent through the CHUNK_TREE to a physical offset on a supplied source. It checks the allocation accounting of the inode. Only then does it assign a trust class.

Conditions for RECOVERABLE_100

A file receives RECOVERABLE_100 if all of the following conditions apply:

• Its directory entry, inode and all EXTENT_DATA records come from metadata with a valid checksum.
• The extents account for the allocated size recorded in the inode.
• Every data extent maps through a CHUNK_ITEM with profile DATA=single.
• Every device required by this mapping was supplied.

With --verify-data, HALDE also compares the data on disk with the reconstructed CSUM_TREE. If all checksums match, the file receives CHECKSUM_VERIFIED_100.

06Statuses and trust classes

Every report row has exactly one status. The trust classes are the statuses that mark a file as recoverable. On this page, colour marks the status category only.

07Quick start

1. Scan all surviving sources in one run.

sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --device dev2=/srv/rescue/sdc.img \
  --device dev3=/srv/rescue/sdd.img \
  --output /srv/rescue/meta

2. Read scan-report.txt. Check: the report lists the trees and the missing blocks.
3. List the subvolumes with --list-subvols.
4. Start an inventory run.

less /srv/rescue/meta/scan-report.txt
./halde_recover.py --meta-dir /srv/rescue/meta --list-subvols

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 2=/srv/rescue/sdc.img \
  --device 3=/srv/rescue/sdd.img \
  --dir 'Documents' --verify-data --show all

5. Check: no NOTE: devid N ... not supplied line names a device you still have.
6. If such a line names an available device, add it with --device. Repeat step 4.
7. Recover the files to a different filesystem.

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 2=/srv/rescue/sdc.img \
  --device 3=/srv/rescue/sdd.img \
  --dir 'Documents' --verify-data \
  --recover --target /srv/restored

8. Check: the summary shows Recovery failures/skips : 0.
9. If the value is not 0, read the column recovery_error in the report.

08halde_scan.py

Stage 1. The scanner reads the sources and writes the metadata directory. It opens every source read-only.

halde_scan.py [--version] [--license] [--device [NAME=]PATH]...
              [--output DIR] [--scan-block-mib N] [--checkpoint-gib N]
              [--eio-budget N] [--subvol SPEC]... [--max-subvol-walks N]
              [--resume] [--rescan] [--no-metadata-tsv] [--self-test]

Options

--device [NAME=]PATH

A surviving device, mapper node or image file. Repeat the option for each device. NAME labels the source in the exported files. Without NAME, HALDE derives a name from the path. Pass all surviving devices of the filesystem in one run.

--output DIR

The metadata directory. Default: ./halde-scan. The directory must not be on a source and must not contain a source image file.

--scan-block-mib N

Sequential read size in MiB. Default: 64. Larger values read faster. Smaller values reduce the data affected by one unreadable region.

--checkpoint-gib N

Progress is saved every N GiB. Default: 16. Larger values reduce seeks if output and source share rotating disks. After a crash, the scan rereads at most N GiB.

--eio-budget N

Maximum failed split reads per scan block. Default: 64. After that, the scanner zero-fills the rest of the unreadable region. A zero-filled region cannot pass a Btrfs checksum and therefore never becomes metadata. For failing media, use ddrescue instead of raising this value.

--subvol SPEC

Walk a subvolume and export its inventory. SPEC is a subvolume id, a subvolume path such as @home, or all. Repeatable. subvolumes.tsv lists all subvolumes also without this option.

--max-subvol-walks N

Maximum number of subvolumes walked in one run. Default: 32.

--resume

Continue an interrupted scan from its checkpoints. For a completed scan, --resume skips the physical scan. Use it to export further subvolumes.

--rescan

Discard the saved scan state of the supplied sources and start again.

--no-metadata-tsv

Omit metadata.tsv. This file lists all blocks of all generations. The recoverer does not need it.

--license

Print the licence text and exit. --licence is an alias.

--version

Print the version and exit.

--self-test

Run the built-in tests and exit. On success, the output is one line.

Output files

Scan report

Current root   tree: root=30490624 gen=9 blocks=1 missing=0 source=backup:dev1:mirror1:slot2
Current chunk  tree: root=22036480 gen=9 blocks=1 missing=0 source=backup:dev1:mirror1:slot2
Current fs     tree: root=30408704 gen=6 blocks=1 missing=0 source=root-tree:30490624:gen9:keyoff0
Current dev    tree: root=30425088 gen=6 blocks=1 missing=0 source=root-tree:30490624:gen9:keyoff0
Current csum   tree: root=30441472 gen=6 blocks=1 missing=0 source=root-tree:30490624:gen9:keyoff0

Subvolumes and snapshots
------------------------
 subvolid  walked   entries  path
      256  yes            1  @home  (read-only)
      257  yes            2  @
      258  yes            1  @/var

Check the value missing first. A value above 0 means that a parent block points to a block that was not found. The inventory below that point is incomplete. HALDE marks affected files with a degraded-scope trust class.

09halde_recover.py

Stage 2. The recoverer reads the metadata directory, assigns statuses and writes files on request. It opens every source read-only. It writes only to the target directory and to the report.

halde_recover.py --meta-dir DIR [--device DEVID=PATH]... [--dir SPEC]...
                 [--subvol SPEC] [--list-subvols] [--root-inode N]
                 [--recover --target DIR] [--overwrite] [--report PATH]
                 [--verify-data] [--strict] [--allow-devextent-mapping]
                 [--no-restore-times] [--restore-ownership]
                 [--restore-special-bits] [--show all|recoverable|problem]
                 [--version] [--license] [--self-test]

Selection

--meta-dir DIR

The metadata directory written by halde_scan.py.

--device DEVID=PATH

Maps a Btrfs devid to its source. Repeat the option for each surviving device. scan-report.txt and scan-info.json list the devids. If the source has a valid superblock, HALDE checks the mapping and refuses a mismatch.

--dir SPEC

The directory to inventory or recover. Repeatable. SPEC is a path from the tree root (Documents/2024), a unique directory name (Documents) or an inode number (#256 for the tree root).

--root-inode N

The inode where path resolution starts. Default: 256 for the default tree, the root directory of the subvolume with --subvol.

--list-subvols

List the subvolumes found by the scanner and their export state, then exit.

--subvol SPEC

Work in a subvolume instead of the default tree. SPEC is a numeric id or a subvolume path. The subvolume must be exported with halde_scan.py --subvol. HALDE accepts a name or path only if it is unique.

Writing

--recover

Write the recoverable files. Without this option, HALDE writes only the report.

--target DIR

The target directory. Required with --recover. It must be on a different filesystem than the source and must not contain a source image file.

--overwrite

Replace existing files in the target directory. Default: existing files are kept.

--report PATH

Path of the TSV report. Default: <target>/halde-recovery-report.tsv, without --target ./halde-recovery-report.tsv.

Evidence

--verify-data

Read the data of every recoverable file and compare every sector with the reconstructed CSUM_TREE. Files with a mismatch receive CHECKSUM_MISMATCH and are not written. Use this option for damaged media.

--strict

Write only RECOVERABLE_100 and CHECKSUM_VERIFIED_100 files. The report still lists degraded-scope and DEV_EXTENT-heuristic files.

--allow-devextent-mapping

Permit a mapping derived from DEV_EXTENT records if the CHUNK_TREE could not be reconstructed. HALDE accepts this mapping only if it is unique. Affected files receive RECOVERABLE_DEVEXTENT_HEURISTIC.

File metadata

--no-restore-times

Do not restore modification and access times. By default, HALDE restores both with nanosecond precision.

--restore-ownership

Also restore uid and gid. Requires root. If HALDE does not run as root, it prints a warning before it starts.

--restore-special-bits

Also restore setuid, setgid and sticky bits. Requires --restore-ownership. Without this option, HALDE clears these bits.

Output

--show all | recoverable | problem

Selects the per-file lines printed after the summary. Default: recoverable. problem prints all files that are not written.

--license

Print the licence text and exit. --licence is an alias.

--version

Print the version and exit.

--self-test

Run the built-in tests and exit. On success, the output is one line.

10The TSV report

Every run writes a tab-separated report. The report has one row per directory entry. The report is the complete record. The terminal summary condenses it.

Evaluate the report

The helper col finds a column by name. The commands therefore also work if a later version adds columns.

# Helper: print the column number for a column name
col() { awk -F'\t' -v n="$1" 'NR==1{for(i=1;i<=NF;i++) if($i==n){print i;exit}}' "$2"; }

R=halde-recovery-report.tsv

# Count files per status
awk -F'\t' -v c=$(col status $R) 'NR>1{print $c}' $R | sort | uniq -c | sort -rn

# List all files that were not written, with the reason
awk -F'\t' -v s=$(col status $R) -v r=$(col recovered_path $R) \
           -v w=$(col reason $R) -v p=$(col path $R) \
  'NR>1 && $r=="" {print $s"\t"$p"\t"$w}' $R

# List files whose data could not be fully verified
awk -F'\t' -v d=$(col data_checksum_status $R) -v m=$(col checksums_missing $R) \
           -v p=$(col path $R) \
  'NR>1 && ($d=="PARTIAL" || $d=="UNAVAILABLE") {print $p, $d, $m}' $R

# Sum the recovered bytes
awk -F'\t' -v r=$(col recovered_path $R) -v z=$(col size $R) \
  'NR>1 && $r!="" {n+=$z} END {printf "%.2f GiB\n", n/1073741824}' $R

# List the devids that are still required
awk -F'\t' -v m=$(col missing_devids $R) 'NR>1 && $m!="" {print $m}' $R | sort -u

11Examples

All examples use image files in /srv/rescue and the metadata directory /srv/rescue/meta.

Start an inventory run

An inventory run writes only the report. Start every session with an inventory run.

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --dir '#256' --show all

Metadata directory: /srv/rescue/meta
HALDE recover     : 0.4.10
Selected tree     : default FS_TREE, root inode 256
ZSTD support      : yes
Btrfs geometry    : nodesize=16384 sectorsize=4096 checksum=crc32c
Surviving devid 1: /srv/rescue/sdb.img
Devices in filesystem: 1 expected, 1 supplied
Selected directory: 'inode-256' -> inode 256
Discovered current entries under selection: 8
Current tree leaves readable: 1/1 (tree 5)

Recover one directory

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --dir 'Documents' --verify-data \
  --recover --target /srv/restored

HALDE writes the files to /srv/restored/Documents/. Repeat --dir to select several directories.

Recover the whole tree

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 2=/srv/rescue/sdc.img \
  --dir '#256' --verify-data \
  --recover --target /srv/restored

Recover with a missing device

Supply all devices you still have. HALDE names every missing device before the per-file results.

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 3=/srv/rescue/sdd.img \
  --dir '#256' --verify-data --show problem

Surviving devid 1: /srv/rescue/sdb.img
Surviving devid 3: /srv/rescue/sdd.img
Devices in filesystem: 3 expected, 2 supplied
NOTE: devid 2 (scanned as /srv/rescue/sdc.img) was recorded by the scanner but
not supplied. Files whose data lives on it will be reported LOST_PARTIAL. If that
device or an image of it still exists, pass --device 2=PATH and re-run.

LOST_PARTIAL                  184.21 MiB inode=41203    Videos/holiday.mkv
LOST_PARTIAL                    2.10 GiB inode=41288    Videos/archive.tar

HALDE does not write these files. The column missing_devids names devid 2. If an image of this device turns up later, add it with --device and repeat the run. A new scan is not required.

Recover from a subvolume

Ubuntu and openSUSE store user data in the subvolumes @ and @home. Check for subvolumes first.

./halde_recover.py --meta-dir /srv/rescue/meta --list-subvols

 subvolid  exported   entries  path
      256  no               0  @
      257  no               0  @home
      258  no               0  @/var

Subvolumes marked 'no' were detected but not exported. Re-run
halde_scan.py with --resume --subvol ID to export one of them.

# --resume skips the physical scan of a completed scan.
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta \
  --resume --subvol all

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --subvol @home --dir '#256' \
  --verify-data --recover --target /srv/restored/home

The inventory marks a subvolume mount point with [S]. HALDE does not follow it as a directory. If you select a mount point with --dir, HALDE names the correct option:

ERROR: '@' is a subvolume mount point (subvolume 257), not a directory of this
tree. Its contents live in a separate tree. Recover it with --subvol 257
instead, and see --list-subvols.

Recover from a compressed filesystem

Condition: the filesystem contains zstd extents. Install zstandard first. Without the module, affected files receive NEEDS_ZSTD.

python3 -c 'import zstandard' || sudo apt install python3-zstandard

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --verify-data --recover --target /srv/restored

Files with LZO extents receive UNSUPPORTED_COMPRESSION and are not written.

Scan failing media

Copy the disk with ddrescue first. Then scan the image file with smaller reads and a lower retry limit.

sudo ddrescue -d -r3 /dev/sdb /srv/rescue/sdb.img /srv/rescue/sdb.map

sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta \
  --scan-block-mib 8 \
  --eio-budget 16

Resume or restart a scan

# Continue an interrupted scan
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta --resume

# Discard the saved state and scan again
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta --rescan

Recover without a CHUNK_TREE

If the scanner could not reconstruct the CHUNK_TREE, the recoverer stops:

ERROR: authoritative chunk-items.tsv mapping is unavailable; rerun the scanner
or use --allow-devextent-mapping for an explicit reduced-confidence fallback

Permit the DEV_EXTENT fallback. Combine it with --verify-data.

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --allow-devextent-mapping \
  --verify-data --recover --target /srv/restored

WARN: DEV_EXTENT mapping fallback enabled. Results using it have reduced
confidence unless independently checksum-verified.

RECOVERABLE_DEVEXTENT_HEURISTIC     4.77 MiB inode=573817   large.bin
RECOVERABLE_100                     500.00 B inode=573815   small_inline.bin

Recover only files without caveats

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --verify-data --strict \
  --recover --target /srv/restored-strict

Write strict and non-strict results to separate target directories. The directory then states which files carry caveats.

Restore ownership and timestamps

# Timestamps are restored by default.
# Ownership and special bits require root and explicit options.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --recover --target /srv/restored \
  --restore-ownership --restore-special-bits

# Keep the time of recovery instead
sudo ./halde_recover.py ... --no-restore-times

File names that are not valid UTF-8

No option is required. HALDE carries names byte-exactly in base64 and writes them unchanged. The terminal shows them escaped.

CHECKSUM_VERIFIED_100       1.21 KiB inode=573820   sub/broken_\udcff\udcfename.bin

Compare with a backup

This comparison does not depend on HALDE. Perform it if a backup exists.

cd /srv/restored
find . -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/recovered.sha256

cd /srv/backup
sha256sum -c /tmp/recovered.sha256 2>&1 | grep -v ': OK$'

Check: the second command prints nothing. Every printed line names a file that differs or is missing in the backup.

Complete session

# 1. Copy all disks to image files.
for d in sdb sdc sdd; do
  sudo ddrescue -d -r3 /dev/$d /srv/rescue/$d.img /srv/rescue/$d.map
done

# 2. Scan all image files in one run.
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --device dev2=/srv/rescue/sdc.img \
  --device dev3=/srv/rescue/sdd.img \
  --output /srv/rescue/meta

# 3. Read the scan report and list the subvolumes.
less /srv/rescue/meta/scan-report.txt
./halde_recover.py --meta-dir /srv/rescue/meta --list-subvols

# 4. Start an inventory run and list the problems.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 3=/srv/rescue/sdd.img \
  --dir '#256' --verify-data --show problem \
  --report /srv/rescue/inventory.tsv

# 5. Recover.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 3=/srv/rescue/sdd.img \
  --dir '#256' --verify-data \
  --recover --target /srv/restored

# 6. Compare with the backup.
sha256sum -c /tmp/backup.sha256

12Diagnostics

The following messages require a decision.

Metadata scope degraded

WARN: metadata scope degraded (current metadata copy/copies unreadable during
recovery); per-file structural proof is retained as RECOVERABLE_DEGRADED_SCOPE
where possible

Meaning: FS_TREE metadata could not be read. Files with complete own records stay recoverable. The inventory can be incomplete. Action: none required. Do not treat a missing file as proof that it did not exist.

Redundant copy used

WARN: metadata checksum mismatch dev1:38830080 logical=30441472; trying another copy

Meaning: one copy of a metadata block failed its checksum. HALDE used the DUP or RAID1 copy. Action: none required.

Checksum evidence not used

NOTE: 5 disk-backed file(s) are reported recoverable on structural evidence
alone. csum-items.tsv holds 1 checksum range(s) that were not consulted, so
corrupted data could have gone unnoticed. Re-run with --verify-data to check the
bytes against the Btrfs CSUM_TREE.

Meaning: the run did not use --verify-data. Action: repeat the run with --verify-data.

Verification incomplete

NOTE: --verify-data was requested, but 2 disk-backed recoverable file(s) were
not fully checksum-verified (PARTIAL=2). They remain recoverable on structural
evidence and may still be written. Inspect data_checksum_status/checksums_missing
in the TSV report and compare important files against an independent backup.

Meaning: the CSUM_TREE does not cover all sectors of these files. This is not a checksum mismatch. Action: read checksums_missing in the report. Compare the files with a backup.

Device not supplied

NOTE: devid 2 (scanned as /srv/rescue/sdc.img) was recorded by the scanner but
not supplied. Files whose data lives on it will be reported LOST_PARTIAL.

Meaning: the scan recorded this device, but the run did not include it. Action: if the device or an image exists, add it with --device and repeat the run.

Wrong device mapping

ERROR: source identity mismatch for --device 2=/srv/rescue/sdc.img; checksum-valid
superblock(s) report [('c51c4a56-...', 1)], expected fsid=c51c4a56-... devid=2

Meaning: the source mapped to devid 2 reports devid 1. Action: correct the --device mapping.

No readable leaf

ERROR: not a single current tree leaf could be read. Check that the --device
DEVID=/path mappings match scan-info.json and that the sources are readable.

Typical causes: a wrong devid mapping, a truncated image file, or a metadata directory of a different filesystem. Action: check these three causes in this order.

XXHASH64 without the module

ERROR: this filesystem uses Btrfs XXHASH64 checksums; install Python module 'xxhash'

Meaning: the filesystem uses XXHASH64. HALDE stops before scanning. Action: install xxhash and repeat the command.

13Verifying HALDE

Self-tests

./halde_scan.py --self-test
./halde_recover.py --self-test

Check: each command prints one line ending in self-test OK. The self-tests cover the structure parsers, the CRC32C test vector, profile rejection, compression edge cases, subvolume resolution, output protection, status assignment and the embedded licence.

End-to-end test suite

The test suite builds Btrfs image files with mkfs.btrfs. It fills them with test files, damages them in defined ways and runs both stages. It compares every recovered file with its original by SHA-256. It does not access real disks.

# Development run. Older btrfs-progs skip some cases.
./halde_e2e_test.sh /tmp/halde-e2e

# Release gate. Requires btrfs-progs 6.14+ and zstandard. SKIP counts as failure.
./halde_e2e_test.sh --release-gate /tmp/halde-gate

  PASS  halde_scan.py --license matches LICENCE
  PASS  halde_recover.py --license matches LICENCE
  PASS  all 7 files byte-identical with restored timestamps
  PASS  setuid/setgid/sticky cleared without --restore-special-bits
  PASS  fallback to the redundant copy was taken
  PASS  one bad leaf cost only 21 files, not all 2500
  PASS  fallback recovery is still byte-identical
  PASS  recoverer refused a target that contains the source image
  PASS  recovered a known file from each of 3 subvolumes
  PASS  --dir on a subvolume mount point is refused with a usable hint
  PASS  zlib extents recovered byte-identically
  PASS  zstd extents recovered byte-identically
  PASS  --verify-data detects corrupted data behind intact metadata
  PASS  the mismatching file is refused, not silently written
  PASS  an unsupplied device is named up front, not only per file
  PASS  LZO extents are refused rather than guessed at

=== result ===
release gate: genuine multi-subvolume case ran
All executed cases passed.

btrfs-progs 6.14, python-zstandard 0.25.0. Several checks were tested for their ability to fail. For each of them, the defect it guards against was reintroduced. The suite then reported FAIL and exited with a non-zero status.

Real damaged filesystem

HALDE was run against the damaged three-device filesystem it was written for. The run used --verify-data. The recovered files were compared byte for byte with an independent backup. Files with data on the missing device received LOST_PARTIAL and were not written. Status: passed.

Development

HALDE was developed in an adversarial review loop between two AI models, GPT-5.6 "Ada" and Claude Opus, with a human operator. Each model reviewed and attacked the other's work. CHANGES.md records the defects found and fixed. Examples: a wrong structure offset that disabled the backup-root fallback, a mapping that would have returned wrong data on RAID0, invented paths at subvolume mount points, and compressed files that were reported as verified and then not written.

14Limits

Status of all items: not implemented, unless stated otherwise.

• DATA profiles. HALDE reconstructs data only for the profile single. It detects RAID0, RAID1, RAID10, RAID5, RAID6, RAID1C3 and RAID1C4 and refuses them.
• Generations. HALDE follows only the current generation. The scanner records older blocks but does not use them for recovery. Files deleted before the failure are not found.
• Inventory gaps. HALDE cannot name directory entries from lost FS_TREE blocks. HALDE reports that a gap exists. It cannot state what the gap contained.
• Compression. zlib and zstd: implemented. LZO: not implemented.
• Symlinks, xattrs, ACLs. The report lists them. HALDE does not restore them.
• Directory metadata. Recovered directories receive default permissions and the current time. Only files receive their original timestamps.
• Memory. The tree walk keeps one record per block in memory. Memory use grows with the size of the tree. Filesystems with more than a few hundred thousand blocks were not tested.
• Hardlinks. HALDE recovers every directory entry separately. Hardlinks become independent copies.
• Sparse files. HALDE preserves holes recorded by Btrfs. A hole that Btrfs stored as zeros is recovered as zeros.

15Download

1. Unpack the archive.
2. Check the file checksums. Check: every line ends in OK.
3. Run both self-tests. Check: each prints self-test OK.

unzip halde-0.4.10.zip
cd halde-0.4.10
sha256sum -c SHA256SUMS
./halde_scan.py --self-test
./halde_recover.py --self-test

HALDE implements the Btrfs on-disk structures from public Btrfs documentation and from public kernel and btrfs-progs headers. HALDE contains no source code from these projects and bundles no third-party Python package.

16Licence

HALDE is licensed under the Playsheep Source-Available Licence 1.0. This licence is not an OSI-approved open-source licence. Both programs print the text with --license.

Summary

This summary has no legal force. The licence text below is binding.

• You may use HALDE for any purpose free of charge. This includes use in organisations and paid work for third parties, for example data recovery for a customer.
• Files recovered with HALDE are not subject to the licence.
• You may copy, modify and redistribute HALDE free of charge. Keep the licence and all notices. Mark modified files.
• Without written permission you must not sell HALDE or include it in a paid product.
• Without written permission you must not offer HALDE as a hosted or managed service. This applies to free services as well.
• HALDE is provided without warranty.

Licence text

Playsheep Source-Available Licence 1.0
Copyright (c) 2026 J. Philipp de Graaff, www.playsheep.de

1. Definitions
   "Software" means the software and its documentation supplied under this
   licence, in source or binary form. The accompanying copyright notice or
   project information identifies the covered software.
   "Derivative" means any work based on the Software or containing a
   substantial part of it.
   "You" means the person or organisation exercising the permissions below.
   Third-party components identified as separately licensed remain governed
   by their own licences. This licence does not restrict the rights granted
   by those licences.

2. Permission to use
   You may use the Software for any purpose, without charge. This expressly
   includes use inside a company, an authority or any other organisation,
   and use as a tool while performing paid work for a third party.

   Documents and other input data do not become subject to this licence
   merely because they are processed with the Software. You may use, copy,
   modify, redistribute and sell documents produced with the Software,
   including formatting styles incorporated by its export function, without
   the restrictions imposed by this licence. This output exception does not
   grant rights in third-party content or permit distribution of the
   application itself as an exported document.

3. Permission to copy, modify and redistribute
   You may copy the Software, modify it and redistribute the original or a
   Derivative, free of charge, provided that:
   a) this licence and the copyright and authorship notices are retained in
      full and remain visible in the source;
   b) modified files are marked as modified, stating who changed them and when;
   c) a Derivative is not presented as the original software, and the software's
      name is not used in a way that suggests the copyright holder endorses it.

4. Restriction on sale and commercial distribution
   Without prior written permission from the copyright holder, you may not:
   a) sell the Software or a Derivative, or licence it for a fee;
   b) distribute the Software or a Derivative as, or as part of, a product or
      service that is offered for payment, where the Software provides a
      substantial part of that product's or service's function or value;
   c) offer the Software or a Derivative as a hosted or managed service.
   Section 4(c) applies whether or not the service is offered for payment.
   Section 2 is not limited by this section. Using the Software as a tool to
   do paid work remains permitted.
   If you would like to do something this section does not allow, please ask.
   Separate commercial licences are available.

5. No trademark or patent licence
   This licence grants no trademark rights in the software's name, the name
   Playsheep, or any logo, and no rights under any patent.

6. No warranty
   The Software is provided "as is", without warranty of any kind, express
   or implied, including but not limited to warranties of merchantability,
   fitness for a particular purpose and non-infringement. The Software may
   contain errors and may produce incomplete or incorrect results.
   Mandatory statutory rights remain unaffected.

7. Limitation of liability
   To the fullest extent permitted by applicable law, the copyright holder
   shall not be liable for any loss of data, loss of profit, business
   interruption or any other direct, indirect, incidental, special or
   consequential damages arising out of the use of or inability to use the
   Software. Liability that cannot be excluded under applicable mandatory
   law remains unaffected. This includes, in particular, liability for intent,
   gross negligence, and culpable injury to life, body or health.

8. Termination
   Your rights under this licence end automatically if you breach it. They
   are reinstated if you cure the breach within 30 days of becoming aware of it.

9. Severability and language
   If any provision is held unenforceable, the remaining provisions stay in
   force. The English text is the binding version. Any translation is provided
   for convenience only.

Contact for commercial licensing and matters not covered by this licence:
license@playsheep.de or via the website https://playsheep.de/