HALDE

HALDE recovers files from Btrfs filesystems that can no longer be mounted, and tells you for every single file how strongly its recovery is supported by the metadata that survived. It never writes to a source device.

=== HALDE Recovery summary ===
Entries discovered       : 6
Regular files discovered : 5
Known regular-file size  : 5.94 MiB
Recoverable files        : 4
Recoverable size         : 4.99 MiB
Checksum-verified files  : 4
Recovered successfully   : 4
Recovery failures/skips  : 0

CHECKSUM_MISMATCH                   1
CHECKSUM_VERIFIED_100               4
DIR                                 1

01What HALDE is

HALDE is a forensic recovery tool for Btrfs. It is not a repair tool and it is not a replacement for btrfs-progs. It exists for one specific situation: the filesystem will not mount, btrfs restore gave you nothing useful, but the surviving devices or images still hold usable metadata and file data.

It was written after a three-device Btrfs span lost one disk. The superblock was gone, mounting failed at every attempt, and the standard tools had nothing left to offer. HALDE was built to answer a narrower question than "can you repair this": which files can I still prove are intact, and where are they?

The design rule throughout is that an honest gap beats a plausible guess. Where the evidence runs out, HALDE says so and stops. A file that looks recovered but is silently wrong is worse than a file that is openly marked as unrecoverable, because you will only find out about the first one years later.

Two programs

halde_scan.py reads the surviving devices, scans them physically for Btrfs B-tree blocks, verifies every block against its own checksum, reconstructs the current trees by following exact-generation parent pointers, and writes everything it found into a separate metadata directory as SQLite and TSV.

halde_recover.py reads that metadata directory, builds an inventory of the directories you ask for, classifies every file, optionally checks the data against the reconstructed CSUM_TREE, and writes out the files that pass.

The split matters in practice. Scanning a failing disk is slow and you only want to do it once. Once the metadata directory exists you can inventory, re-inventory and recover as often as you like without touching the damaged media again.

02What it can and cannot do

03Requirements

HALDE is two Python files. There is nothing to compile, nothing to install and no package to fetch before it will run. What you do need depends on the filesystem you are recovering, so this section is more detailed than it looks like it should be.

Always required

Required only in specific cases

These two are not conveniences. If your filesystem needs one and you do not have it, the affected files simply cannot be recovered. HALDE says so clearly instead of producing partial output.

zlib is decompressed with the Python standard library, so compress=zlib filesystems need nothing extra. LZO is not implemented at all and no module will change that. The scanner itself needs no third-party package except xxhash in the case above; only the recoverer touches zstandard.

How to find out what you need

The scanner prints the checksum algorithm, and the recoverer prints whether zstd support is available, in the first few lines of every run:

Btrfs geometry    : nodesize=16384 sectorsize=4096 checksum=crc32c
ZSTD support      : yes

If ZSTD support says no, that is not automatically a problem; it only matters if the filesystem actually used zstd. Run the inventory first, without --recover, and look for the status:

./halde_recover.py --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img --dir '#256' --show problem \
  | grep NEEDS_ZSTD

Any output means install the module and run again. Nothing was written in the meantime, so nothing is lost.

Installing them

# Debian, Ubuntu, Linux Mint
sudo apt install python3-zstandard python3-xxhash

# Fedora, RHEL
sudo dnf install python3-zstandard python3-xxhash

# Arch
sudo pacman -S python-zstandard python-xxhash

# openSUSE
sudo zypper install python3-zstandard python3-xxhash

# Anywhere, without touching the system packages
python3 -m venv /tmp/halde-venv
/tmp/halde-venv/bin/pip install zstandard xxhash
/tmp/halde-venv/bin/python3 ./halde_recover.py --meta-dir ...

# Confirm
python3 -c 'import zstandard, xxhash; print("both available")'

Only for running the test suite

btrfs-progs 6.14 or newer, for mkfs.btrfs. The suite builds its own Btrfs images to test against. Older versions work but skip the subvolume and compression cases, which is fine for a quick check and not sufficient for a release gate. You do not need btrfs-progs to use HALDE.

Space

Two figures, both measured rather than estimated.

• The exported TSV files scale with the number of directory entries, at roughly 250 bytes each. A test filesystem with 2,525 files produced a 592 KB metadata directory.
• metadata.sqlite3 holds one row per B-tree block found anywhere on the disks, at about 200 bytes per block. That includes historical blocks from older generations, so on an aged filesystem it can be several times the size of the current tree. This is the entry that can grow unexpectedly.

--no-metadata-tsv suppresses the largest single export and costs you nothing the recoverer needs. Budget generously anyway, and put the metadata directory somewhere with room. The recovery target obviously needs space for the recovered data, and must be on a different filesystem from the one you are recovering.

04Before you start

If the disk is physically failing, image it first. HALDE reads a lot and reads it repeatedly across runs. A dying disk that is read repeatedly usually gets worse, and every retry costs you sectors you may still need.

sudo ddrescue -d -r3 /dev/sdb /srv/rescue/sdb.img /srv/rescue/sdb.map
sudo ddrescue -d -r3 /dev/sdc /srv/rescue/sdc.img /srv/rescue/sdc.map
sudo ddrescue -d -r3 /dev/sdd /srv/rescue/sdd.img /srv/rescue/sdd.map

Both HALDE stages accept plain files everywhere they accept devices, so the rest of this page works unchanged on images. Note the map files: if ddrescue had to give up on some regions, the map tells you which ones, and that is information HALDE cannot reconstruct for you.

Identify devices properly

/dev/sdX names move between boots. Use stable identifiers when you write down which device was which.

ls -l /dev/disk/by-id/
lsblk -o NAME,SIZE,SERIAL,MODEL

Never write to the source

HALDE opens every source with O_RDONLY and refuses an output location that overlaps a raw block-device source, including through device-mapper and MD ancestry. It also refuses an output directory that contains one of your source images, since a recovered file could otherwise overwrite it. Putting the recovered tree next to your images on the same scratch filesystem is fine and only produces a warning about free space.

05How it works

Btrfs stores its metadata in copy-on-write B-trees. Every tree block carries a header with the filesystem UUID, its own logical address, a generation number, the tree it belongs to, and a checksum over the whole block. That header is the lever HALDE uses.

1. Read the superblocks. Btrfs keeps mirrors at 64 KiB, 64 MiB and 256 GiB on each device. HALDE reads all of them, discards any that fail their checksum, and takes the geometry and identity from the newest trustworthy one.
2. Scan physically. Every device is read sequentially. At each sector boundary HALDE looks for the filesystem UUID at the header offset, then validates the candidate against its Btrfs checksum. A block that does not pass its checksum is never used for anything.
3. Index everything found. Each accepted block is recorded in SQLite with its device, physical offset, logical address, generation, owning tree and level. This is the raw evidence, and it includes historical blocks as well as current ones.
4. Rebuild the current trees. Starting from the superblock roots and the backup root slots, HALDE walks each tree by following parent pointers and demanding an exact generation match at every step. No silent substitution of an older block for a missing newer one.
5. Export. The chunk mapping, directory index, inode metadata, device extents, subvolume list and reconstructed checksums are written as TSV.
6. Classify and recover. The second stage maps every extent through the authoritative CHUNK_TREE to a physical offset on a device you supplied, checks that the inode's allocation accounting adds up, and only then calls a file recoverable.

What "proven" means here

A file is RECOVERABLE_100 when all of the following hold: its directory entry, inode and every EXTENT_DATA record were read from checksum-valid metadata; the sum of its extents matches the byte count recorded in the inode; every data extent maps through a real CHUNK_ITEM with a DATA=single profile; and every device that mapping needs was supplied. Add --verify-data and the on-disk bytes are additionally compared against the reconstructed CSUM_TREE, which promotes the file to CHECKSUM_VERIFIED_100.

06Trust classes

Every file in the report carries exactly one status. The colour on this page is the classification and nothing else.

07Quick start

Three commands. Scan, look, recover.

sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --device dev2=/srv/rescue/sdc.img \
  --device dev3=/srv/rescue/sdd.img \
  --output /srv/rescue/meta

less /srv/rescue/meta/scan-report.txt
less /srv/rescue/meta/filesystem-tree.txt
./halde_recover.py --meta-dir /srv/rescue/meta --list-subvols

# Dry run. Writes nothing except the TSV report.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 2=/srv/rescue/sdc.img \
  --device 3=/srv/rescue/sdd.img \
  --dir 'Documents' --verify-data --show all

# Same command plus --recover --target, once you are happy.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 2=/srv/rescue/sdc.img \
  --device 3=/srv/rescue/sdd.img \
  --dir 'Documents' --verify-data \
  --recover --target /srv/restored

Without --recover the second stage never writes anything but the report, so running it first to look around costs nothing.

08halde_scan.py

Stage one. Reads the surviving devices and writes a metadata directory. It never opens a source for writing.

halde_scan.py [--device [NAME=]PATH]... [--output DIR]
              [--scan-block-mib N] [--checkpoint-gib N] [--eio-budget N]
              [--subvol SPEC]... [--max-subvol-walks N]
              [--resume] [--rescan] [--no-metadata-tsv] [--self-test]

Options

--device [NAME=]PATH

A surviving device, mapper node or image file. Repeat once per device. The optional name is how this source is labelled in the exported artefacts; if you leave it out, a name is derived from the path. Pass every surviving device of the filesystem in a single run.

--output DIR

Where the metadata directory goes. Default ./halde-scan. Must not be on a source device, and must not contain one of your source images.

--scan-block-mib N

Sequential read size in MiB, default 64. Larger is faster on healthy media. Smaller is gentler on failing media and limits how much is lost to a single bad region.

--checkpoint-gib N

How often progress is persisted, default every 16 GiB. Larger values mean far fewer seeks when the output directory and the source share rotating disks. A hard crash costs you at most a re-read of this amount.

--eio-budget N

How many failed split reads are tolerated per scan block before the rest of the bad region is zero-filled and the scan moves on, default 64. Zero-filled regions can never be mistaken for metadata, because every candidate block still has to pass its Btrfs checksum. Raising this on a dying disk is usually the wrong move; use ddrescue instead.

--subvol SPEC

Also walk a subvolume and export its own inventory. SPEC is a numeric subvolume id, a subvolume path such as @home, or the word all. Repeatable. Subvolumes are always listed in subvolumes.tsv even without this option; walking them is opt-in because snapshots share most of their metadata.

--max-subvol-walks N

Refuse to walk more than N subvolume roots in one run, default 32. A guard against spending hours on a snapshot farm.

--resume

Continue an interrupted scan from its checkpoints. A scan that already finished resumes instantly, which makes this the cheap way to re-export with different --subvol settings.

--rescan

Discard the saved scan state for the supplied devices and start over.

--no-metadata-tsv

Skip the metadata.tsv export, which lists every block from every generation and can be very large. Everything the recoverer needs is elsewhere.

--self-test

Run the built-in structure and checksum tests and exit.

What it writes

Reading the scan report

Current root   tree: root=30490624 gen=9 blocks=1 missing=0 source=backup:dev1:mirror1:slot2
Current chunk  tree: root=22036480 gen=9 blocks=1 missing=0 source=backup:dev1:mirror1:slot2
Current fs     tree: root=30408704 gen=6 blocks=1 missing=0 source=root-tree:30490624:gen9:keyoff0
Current dev    tree: root=30425088 gen=6 blocks=1 missing=0 source=root-tree:30490624:gen9:keyoff0
Current csum   tree: root=30441472 gen=6 blocks=1 missing=0 source=root-tree:30490624:gen9:keyoff0

Subvolumes and snapshots
------------------------
 subvolid  walked   entries  path
      256  yes            1  @home  (read-only)
      257  yes            2  @
      258  yes            1  @/var

The number that matters most is missing. A non-zero count means the walk found a parent pointer to a block that could not be recovered, so the inventory below it has holes. HALDE carries that fact forward into every file it reports from that tree.

09halde_recover.py

Stage two. Reads a metadata directory, classifies files, and optionally writes them. Sources stay read-only; the only things written are the recovery target and the report.

halde_recover.py --meta-dir DIR [--device DEVID=PATH]... [--dir SPEC]...
                 [--subvol SPEC] [--list-subvols] [--root-inode N]
                 [--recover --target DIR] [--overwrite] [--report PATH]
                 [--verify-data] [--strict] [--allow-devextent-mapping]
                 [--no-restore-times] [--restore-ownership]
                 [--restore-special-bits]
                 [--show all|recoverable|problem] [--self-test]

Selecting what to work on

--meta-dir DIR

The directory produced by halde_scan.py.

--device DEVID=PATH

Maps a Btrfs devid to the read-only source that holds it. Repeat once per surviving device. The devid is the filesystem's own number for that device, not a position; scan-report.txt and scan-info.json tell you which is which. HALDE verifies the mapping against the device's superblock where one survives, and refuses a mismatch.

--dir SPEC

Which directory to inventory. Repeatable. Accepts a path from the tree root (Documents/2024), a bare directory name if it is unambiguous (Documents), or an inode number (#256 for the tree root itself).

--root-inode N

The inode that path resolution starts from. Defaults to 256 for the default tree, and to the subvolume's own root directory when --subvol is used. You rarely need this.

--list-subvols

Print the subvolumes the scanner found, whether each was exported, and exit.

--subvol SPEC

Work inside a subvolume instead of the default tree. A numeric id or a subvolume path. The subvolume must have been exported with halde_scan.py --subvol. Numeric ids are authoritative; a name or path is accepted only when it is unambiguous.

Writing files out

--recover

Actually write the files. Without it, nothing is written except the report.

--target DIR

Destination root. Required with --recover. Must be on a different filesystem from the source, and must not contain your source images.

--overwrite

Replace destination files that already exist. Off by default, so a repeated run does not silently clobber an earlier recovery.

--report PATH

Where the TSV report goes. Defaults to <target>/halde-recovery-report.tsv, or the current directory when not recovering.

Evidence and trust

--verify-data

Read the recoverable files' extents back off the disk and compare every sector against the reconstructed CSUM_TREE. Files that fail are reported as CHECKSUM_MISMATCH and never written. Slower, and on damaged media it is the difference between a structural claim and a proven one. Use it.

--strict

Recover only RECOVERABLE_100 and CHECKSUM_VERIFIED_100. Degraded-scope and DEV_EXTENT-heuristic files are still reported but not written.

--allow-devextent-mapping

Permit a reduced-confidence logical-to-physical fallback derived from unique DEV_EXTENT records, for the case where the CHUNK_TREE could not be reconstructed at all. Files mapped this way are labelled RECOVERABLE_DEVEXTENT_HEURISTIC and never silently presented as authoritative.

File metadata

--no-restore-times

Do not restore modification and access times. They are restored by default, at nanosecond precision, because a rescued tree where everything is dated today is much harder to work with.

--restore-ownership

Also restore uid and gid. Needs root, and only makes sense when restoring onto the same system. HALDE warns up front if you ask for this without root.

--restore-special-bits

Also restore setuid, setgid and sticky bits. Accepted only together with --restore-ownership. Without it, special bits are always cleared, so a recovery run as root cannot turn a foreign 04755 file into a root-owned setuid binary.

Output control

--show all | recoverable | problem

Which per-file lines to print after the summary. Default recoverable. problem prints everything that will not be recovered, which is usually the more interesting list.

--self-test

Run the built-in logic tests and exit.

10The TSV report

Every run writes a tab-separated report with one row per directory entry. This is the authoritative record; the terminal summary is a condensation of it.

Useful one-liners

# A small helper: look each column up by name, so these keep working
# even if a future version adds a column.
col() { awk -F'\t' -v n="$1" 'NR==1{for(i=1;i<=NF;i++) if($i==n){print i;exit}}' "$2"; }

R=halde-recovery-report.tsv

# Count files per status
awk -F'\t' -v c=$(col status $R) 'NR>1{print $c}' $R | sort | uniq -c | sort -rn

# Everything that was not written, with the reason
awk -F'\t' -v s=$(col status $R) -v r=$(col recovered_path $R) \
           -v w=$(col reason $R) -v p=$(col path $R) \
  'NR>1 && $r=="" {print $s"\t"$p"\t"$w}' $R

# Files whose data could not be fully checksum-verified
awk -F'\t' -v d=$(col data_checksum_status $R) -v m=$(col checksums_missing $R) \
           -v p=$(col path $R) \
  'NR>1 && ($d=="PARTIAL" || $d=="UNAVAILABLE") {print $p, $d, $m}' $R

# Total recovered bytes
awk -F'\t' -v r=$(col recovered_path $R) -v z=$(col size $R) \
  'NR>1 && $r!="" {n+=$z} END {printf "%.2f GiB\n", n/1073741824}' $R

# Which devices are still needed
awk -F'\t' -v m=$(col missing_devids $R) 'NR>1 && $m!="" {print $m}' $R | sort -u

11Examples

Look before you touch anything

The inventory run writes nothing. Start here every time.

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --dir '#256' --show all

Metadata directory: /srv/rescue/meta
HALDE recover     : 0.4.9b
Selected tree     : default FS_TREE, root inode 256
ZSTD support      : yes
Btrfs geometry    : nodesize=16384 sectorsize=4096 checksum=crc32c
Surviving devid 1: /srv/rescue/sdb.img
Devices in filesystem: 1 expected, 1 supplied
Selected directory: 'inode-256' -> inode 256
Discovered current entries under selection: 8
Current tree leaves readable: 1/1 (tree 5)

Recover one directory

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --dir 'Documents' \
  --verify-data \
  --recover --target /srv/restored

Recovered files land under /srv/restored/Documents/. Repeat --dir to select several directories in one run.

Recover the whole tree

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 2=/srv/rescue/sdc.img \
  --dir '#256' \
  --verify-data \
  --recover --target /srv/restored

A multi-device filesystem with one disk gone

Pass the devices you still have. HALDE tells you which one is absent before it shows you a single file, so a forgotten disk cannot be mistaken for a dead one.

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 3=/srv/rescue/sdd.img \
  --dir '#256' --verify-data --show problem

Surviving devid 1: /srv/rescue/sdb.img
Surviving devid 3: /srv/rescue/sdd.img
Devices in filesystem: 3 expected, 2 supplied
NOTE: devid 2 (scanned as /srv/rescue/sdc.img) was recorded by the scanner but
not supplied. Files whose data lives on it will be reported LOST_PARTIAL. If that
device or an image of it still exists, pass --device 2=PATH and re-run.

LOST_PARTIAL                  184.21 MiB inode=41203    Videos/holiday.mkv
LOST_PARTIAL                    2.10 GiB inode=41288    Videos/archive.tar

Those files are not written, and the report names devid 2 in the missing_devids column. If you later find an image of that disk, add it and re-run; nothing needs to be scanned again.

Subvolumes

On a standard Ubuntu or openSUSE layout your data is not in the default tree at all, it is in @ and @home. Check first.

./halde_recover.py --meta-dir /srv/rescue/meta --list-subvols

 subvolid  exported   entries  path
      256  no               0  @
      257  no               0  @home
      258  no               0  @/var

Subvolumes marked 'no' were detected but not exported. Re-run
halde_scan.py with --resume --subvol ID to export one of them.

# --resume skips the physical scan, so this takes minutes, not hours.
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta \
  --resume --subvol all

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --subvol @home --dir '#256' \
  --verify-data --recover --target /srv/restored/home

A subvolume mount point inside a tree is marked [S] in the inventory and is never walked as an ordinary directory, because its directory entry points at a separate tree rather than at an inode. Asking for it directly gives you the right instruction instead of a wrong answer:

ERROR: '@' is a subvolume mount point (subvolume 257), not a directory of this
tree. Its contents live in a separate tree. Recover it with --subvol 257
instead, and see --list-subvols.

Compressed filesystems

zlib and zstd are handled transparently, including inline compressed extents. zstd needs the Python module; without it those files are reported NEEDS_ZSTD rather than silently skipped.

python3 -c 'import zstandard' || sudo apt install python3-zstandard

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --verify-data --recover --target /srv/restored

LZO is not implemented. Files using it are reported UNSUPPORTED_COMPRESSION and never written, because a half-decoded file is worse than an honest refusal.

Failing media

sudo ddrescue -d -r3 /dev/sdb /srv/rescue/sdb.img /srv/rescue/sdb.map

sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta \
  --scan-block-mib 8 \
  --eio-budget 16

Unreadable regions are zero-filled after the budget is spent and the scan continues. Nothing is lost by this: a zero-filled region can never pass a Btrfs checksum, so it can never be mistaken for metadata.

Resuming an interrupted scan

# Ctrl-C, power cut, whatever. Just add --resume.
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta --resume

# Start over from scratch for these devices
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --output /srv/rescue/meta --rescan

When the CHUNK_TREE is gone

If the chunk tree could not be reconstructed, HALDE refuses to guess and says so:

ERROR: authoritative chunk-items.tsv mapping is unavailable; rerun the scanner
or use --allow-devextent-mapping for an explicit reduced-confidence fallback

The fallback derives the mapping from DEV_EXTENT records instead. It is only accepted when the result is unambiguous, and everything recovered through it is labelled:

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --allow-devextent-mapping \
  --verify-data --recover --target /srv/restored

WARN: DEV_EXTENT mapping fallback enabled. Results using it have reduced
confidence unless independently checksum-verified.

RECOVERABLE_DEVEXTENT_HEURISTIC     4.77 MiB inode=573817   large.bin
RECOVERABLE_100                     500.00 B inode=573815   small_inline.bin

Combine it with --verify-data. A heuristic mapping that then matches the Btrfs checksums is a far stronger result than one that does not.

Only what is beyond doubt

sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --verify-data --strict \
  --recover --target /srv/restored-strict

Two passes are often the right approach: a strict pass into one directory, then a normal pass into another, so you always know which pile is which.

Restoring ownership and timestamps

# Times are restored by default at nanosecond precision.
# Ownership and special bits are opt-in and need root.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta --device 1=/srv/rescue/sdb.img \
  --dir '#256' --recover --target /srv/restored \
  --restore-ownership --restore-special-bits

# Or leave every timestamp at recovery time
sudo ./halde_recover.py ... --no-restore-times

Filenames that are not valid UTF-8

No special handling needed. Byte-exact names are carried through the metadata in base64 and recreated as they were. They appear escaped on the terminal so your shell survives, and unescaped on disk.

CHECKSUM_VERIFIED_100       1.21 KiB inode=573820   sub/broken_\udcff\udcfename.bin

Checking a recovery against a backup

Always do this if you have anything to compare against. It is the only check that does not depend on HALDE being right.

cd /srv/restored
find . -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/recovered.sha256

cd /srv/backup
sha256sum -c /tmp/recovered.sha256 2>&1 | grep -v ': OK$'

# or file by file
cmp --silent /srv/restored/Documents/report.odt \
             /srv/backup/Documents/report.odt && echo IDENTICAL

A full session, start to finish

# 1. Image all three disks.
for d in sdb sdc sdd; do
  sudo ddrescue -d -r3 /dev/$d /srv/rescue/$d.img /srv/rescue/$d.map
done

# 2. Scan them together, once.
sudo ./halde_scan.py \
  --device dev1=/srv/rescue/sdb.img \
  --device dev2=/srv/rescue/sdc.img \
  --device dev3=/srv/rescue/sdd.img \
  --output /srv/rescue/meta

# 3. Read the report and the inventory.
less /srv/rescue/meta/scan-report.txt
grep -c '^' /srv/rescue/meta/current-files.tsv
./halde_recover.py --meta-dir /srv/rescue/meta --list-subvols

# 4. Dry run over everything, look at the problems.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 3=/srv/rescue/sdd.img \
  --dir '#256' --verify-data --show problem \
  --report /srv/rescue/dryrun.tsv

# 5. Recover.
sudo ./halde_recover.py \
  --meta-dir /srv/rescue/meta \
  --device 1=/srv/rescue/sdb.img \
  --device 3=/srv/rescue/sdd.img \
  --dir '#256' --verify-data \
  --recover --target /srv/restored

# 6. Verify against whatever backup you have.
sha256sum -c /tmp/backup.sha256

12Diagnostics you may see

HALDE says out loud when it knows something you should know. These are the messages worth acting on.

Metadata scope degraded

WARN: metadata scope degraded (current metadata copy/copies unreadable during
recovery); per-file structural proof is retained as RECOVERABLE_DEGRADED_SCOPE
where possible

Some FS_TREE metadata could not be read. Files whose own records are complete stay recoverable, but the inventory around them has holes. Files you remember having may simply not be listed.

A redundant copy was used

WARN: metadata checksum mismatch dev1:38830080 logical=30441472; trying another copy

One physical copy of a metadata block failed its checksum and HALDE fell back to the DUP or RAID1 mirror. No action needed. It is working as intended.

Checksum evidence was not used

NOTE: 5 disk-backed file(s) are reported recoverable on structural evidence
alone. csum-items.tsv holds 1 checksum range(s) that were not consulted, so
corrupted data could have gone unnoticed. Re-run with --verify-data to check the
bytes against the Btrfs CSUM_TREE.

You ran without --verify-data on a filesystem where verification was possible. Run it again with the flag.

Verification could not be completed

NOTE: --verify-data was requested, but 2 disk-backed recoverable file(s) were
not fully checksum-verified (PARTIAL=2). They remain recoverable on structural
evidence and may still be written. Inspect data_checksum_status/checksums_missing
in the TSV report and compare important files against an independent backup.

The reconstructed CSUM_TREE does not cover every sector of those files. That is not corruption, it means the evidence is incomplete. Check checksums_missing in the report and compare against a backup where you can.

A device was recorded but not supplied

NOTE: devid 2 (scanned as /srv/rescue/sdc.img) was recorded by the scanner but
not supplied. Files whose data lives on it will be reported LOST_PARTIAL.

Either that disk is genuinely gone, or you forgot to pass it. Check before concluding the data is lost.

The device mapping is wrong

ERROR: source identity mismatch for --device 2=/srv/rescue/sdc.img; checksum-valid
superblock(s) report [('c51c4a56-...', 1)], expected fsid=c51c4a56-... devid=2

The image you mapped to devid 2 actually reports devid 1. Swap the numbers. HALDE checks this before doing any work rather than producing plausible nonsense.

Nothing readable at all

ERROR: not a single current tree leaf could be read. Check that the --device
DEVID=/path mappings match scan-info.json and that the sources are readable.

Usually a wrong devid mapping, a truncated image, or a metadata directory from a different filesystem.

XXHASH64 without the module

ERROR: this filesystem uses Btrfs XXHASH64 checksums; install Python module 'xxhash'

This fires within seconds, before the physical scan, so you do not lose hours to it.

13Verifying HALDE itself

A recovery tool asks you to trust it with the one copy of your data you have left. It should be able to show its work.

Built-in tests

./halde_scan.py --self-test
./halde_recover.py --self-test

These check the on-disk structure parsers against known values, the CRC32C implementation against its standard test vector, chunk profile rejection, compression edge cases, subvolume resolution, the output guards and the trust classification logic.

The end-to-end suite

The suite builds real Btrfs filesystems with mkfs.btrfs, fills them with awkward files, damages them in controlled ways, runs both stages and compares every recovered file against its original with SHA-256. It touches no real disk. Everything happens on image files in a scratch directory.

# Development run. May skip cases your btrfs-progs cannot construct.
./halde_e2e_test.sh /tmp/halde-e2e

# Release gate. Requires btrfs-progs 6.14+ and python zstandard.
# Skips are forbidden; a skipped case is a failed gate.
./halde_e2e_test.sh --release-gate /tmp/halde-gate

  PASS  scanner completed on a clean filesystem
  PASS  all 7 files byte-identical with restored timestamps
  PASS  setuid/setgid/sticky cleared without --restore-special-bits
  PASS  fallback to the redundant copy was taken
  PASS  one bad leaf cost only 21 files, not all 2500
  PASS  fallback recovery is still byte-identical
  PASS  recoverer refused a target that contains the source image
  PASS  three subvolume roots enumerated
  PASS  recovered a known file from each of 3 subvolumes
  PASS  --dir on a subvolume mount point is refused with a usable hint
  PASS  zlib extents recovered byte-identically
  PASS  zstd extents recovered byte-identically
  PASS  --verify-data detects corrupted data behind intact metadata
  PASS  the mismatching file is refused, not silently written
  PASS  LZO extents are refused rather than guessed at
  PASS  an unsupplied device is named up front, not only per file

=== result ===
release gate: genuine multi-subvolume case ran
All executed cases passed.

Thirty-five checks, no failures, no skips. Several of them have been verified to be capable of failing, by deliberately reintroducing the defect each one guards against and confirming the suite reports FAIL and exits non-zero. That habit came from finding the opposite: a case that had silently reported SKIP on every run for four releases, so the suite looked green while never touching the feature it was written for. A test that cannot fail is worth as much as a test that was skipped, and this one is checked for both.

The real thing

Beyond the synthetic suite, HALDE was run against the original damaged three-device filesystem it was written for, with --verify-data, and the recovered files were compared byte for byte against an independent backup. Files whose data lived on the missing device were correctly classified as not recoverable rather than being written out as partial garbage.

How it was built

HALDE was written in an adversarial loop between two AI models, GPT-5.6 "Ada" and Claude Opus, with a human operator in the middle. Each reviewed the other's work and tried to break it. Defects found this way, and fixed, include a wrong structure offset that silently disabled the backup-root fallback, a mapping that would have produced correct-looking but wrong data on RAID0 layouts, fabricated pathnames at subvolume mount points, compressed files that were declared verified and then refused, and several places where the tool knew something and did not say it. The commit history of that argument is in CHANGES.md.

14Limits and honest caveats

• Only single DATA is reconstructed. RAID0, RAID1, RAID10, RAID5, RAID6, RAID1C3 and RAID1C4 data are detected and refused. HALDE does not attempt stripe reconstruction, because a wrong stripe calculation produces plausible-looking garbage.
• Only the current generation. Older tree generations are recorded by the scanner but never followed. A file deleted before the failure will not appear.
• The inventory can be short. Directory entries in lost FS_TREE blocks cannot be named. HALDE tells you when this is the case, but it cannot tell you what is missing.
• LZO is not implemented. zlib and zstd are.
• Symlinks, xattrs and ACLs are not restored. They are listed in the report so you know they existed.
• Directory metadata is not restored. Recovered directories get default permissions and the current time, only files carry their original timestamps.
• Memory scales with the tree. The walk holds discovered blocks in memory. On a filesystem with tens of millions of files, expect the scanner to want several GB.
• Hardlinks become copies. Each directory entry is recovered independently.
• Sparse files stay sparse where the filesystem recorded holes, but a hole that Btrfs materialised as zeros comes back as zeros.
• This is not an OSI open-source licence. See below.

15Download

unzip halde-0.4.9b.zip
cd halde-0.4.9b

# Every shipped file is covered by SHA256SUMS
sha256sum -c SHA256SUMS

chmod +x halde_scan.py halde_recover.py halde_e2e_test.sh
./halde_scan.py --self-test
./halde_recover.py --self-test

The Btrfs on-disk structure layouts were implemented from public Btrfs documentation and public kernel and btrfs-progs headers. No source code was copied from either project, and no third-party Python package is bundled.

16Licence

HALDE is source-available, not OSI open source. In plain terms: use it for anything including paid data-recovery work, copy it, change it, keep the attribution, and talk to the author before selling it. The binding text is below.

HALDE Source-Available Licence 1.0
Copyright (c) 2026 J. Philipp de Graaff, www.playsheep.de

1. Definitions
   "HALDE" means this software and its documentation, in source or binary form.
   "Derivative" means any work based on HALDE or containing a substantial part
   of it.
   "You" means the person or organisation exercising the permissions below.

2. Permission to use
   You may use HALDE for any purpose, without charge. This expressly includes
   use inside a company, an authority or any other organisation, and it
   expressly includes using HALDE as a tool while performing paid work for a
   third party, such as data recovery for a customer.

3. Permission to copy, modify and redistribute
   You may copy HALDE, modify it and redistribute the original or a Derivative,
   free of charge, provided that:
   a) this licence and the copyright and authorship notices are retained in
      full and remain visible in the source;
   b) modified files are marked as modified, stating who changed them and when;
   c) a Derivative is not presented as the original HALDE, and the name HALDE
      is not used in a way that suggests the copyright holder endorses it.

4. Restriction on sale and commercial distribution
   Without prior written permission from the copyright holder, you may not:
   a) sell HALDE or a Derivative, or licence it for a fee;
   b) distribute HALDE or a Derivative as, or as part of, a product or service
      that is offered for payment, where HALDE provides a substantial part of
      that product's or service's function or value;
   c) offer third parties paid access to HALDE or a Derivative itself as hosted,
      software-as-a-service, or managed software.

   Section 2 is not limited by this section. For the avoidance of doubt, a
   person or organisation may charge for data recovery or other professional
   services performed using HALDE, provided the customer is paying for the
   professional service or its result and not for access to HALDE itself.

   If you would like to do something this section does not allow, please ask.
   Separate commercial licences may be available by written agreement.

5. Name and marks
   This licence does not grant any right to use HALDE or any HALDE logo as a
   trademark, except for truthful identification of the original software and
   for the attribution required by this licence.

6. No warranty
   HALDE is provided "as is", without warranty of any kind, express or implied,
   including but not limited to warranties of merchantability, fitness for a
   particular purpose and non-infringement. Data recovery is inherently
   uncertain. HALDE may fail to recover data, may present incomplete results,
   and may report a file as recoverable when it is not.

7. Limitation of liability
   To the fullest extent permitted by applicable law, the copyright holder
   shall not be liable for loss of data, loss of profit, business interruption
   or other direct, indirect, incidental, special or consequential damages
   arising from the use of or inability to use HALDE.

   Nothing in this licence excludes or limits liability to the extent that such
   exclusion or limitation is prohibited by mandatory applicable law,
   including, where applicable, liability for intent, gross negligence, or
   injury to life, body or health.

8. Termination
   Your rights under this licence end automatically if you breach it. They are
   reinstated if you cure the breach within 30 days of becoming aware of it.

9. Severability and language
   If any provision is held unenforceable, the remaining provisions stay in
   force. The English text is the binding version. Any translation is provided
   for convenience only.

Contact for commercial licensing and anything this licence does not cover:
halde@playsheep.de

Kurzfassung auf Deutsch

Rechtlich nicht bindend, verbindlich ist der englische Text oben.

  Benutzen    Frei, auch in Firmen und Behörden, auch bei bezahlter Arbeit für
              Kunden, zum Beispiel in der Datenrettung.
  Kopieren    Frei, solange Lizenz und Urhebervermerk erhalten bleiben.
  Verändern   Erlaubt. Geänderte Dateien bitte als geändert kennzeichnen.
  Verkaufen   Nur nach Absprache. Wer HALDE selbst verkauft, es in ein
              kostenpflichtiges Produkt einbaut oder bezahlten Zugang zu HALDE
              als Software anbietet, braucht eine schriftliche Vereinbarung.
  Dienste     Bezahlte Datenrettung oder andere professionelle Arbeit mit HALDE
              als Werkzeug ist ausdrücklich erlaubt.
  Garantie    Keine. Datenrettung ist unsicher, Benutzung auf eigenes Risiko.
  Fragen      halde@playsheep.de