duperemove \- Find duplicate extents and print them to stdout

\fBduperemove\fR \fI[options]\fR \fIfiles...\fI

.PP

\fBduperemove\fR is a simple tool for finding duplicated extents and

submitting them for deduplication. When given a list of files it will

hash their contents on a block by block basis and compare those hashes

to each other, finding and categorizing extents that match each

other. When given the \fB-d\fR option, \fBduperemove\fR will submit those

extents for deduplication using the btrfs-extent-same ioctl.

Duperemove has two major modes of operation one of which is a subset

of the other.

When run without \fB-d\fR (the default) duperemove will print out one or

more tables of matching extents it has determined would be ideal

candidates for deduplication. As a result, readonly mode is useful for

seeing what duperemove might do when run with \fB-d\fR. The output could

also be used by some other software to submit the extents for

deduplication at a later time.

It is important to note that this mode will not print out \fBall\fR

instances of matching extents, just those it would consider for

deduplication.

Generally, duperemove does not concern itself with the underlying

representation of the extents it processes. Some of them could be

compressed, undergoing I/O, or even have already been deduplicated. In

dedupe mode, the kernel handles those details and therefore we try not

to replicate that work.

This functions similarly to readonly mode with the exception that the

duplicated extents found in our "read, hash, and compare" step will

actually be submitted for deduplication. An estimate of the total data

deduplicated will be printed after the operation is complete. This

estimate is calculated by comparing the total amount of shared bytes

in each file before and after the dedupe.

\fIfiles\fR can refer to a list of regular files and directories or be

a hyphen (-) to read them from standard input.

If a directory is specified, all regular files within it will also be

scanned.

.TP

Enable recursive dir traversal.

.TP

De-dupe the results - only works on \fIbtrfs\fR.

.TP

Opens files readonly when deduping. Primarily for use by privileged

users on readonly snapshots.

.TP

Use the specified block size. The default is \fB128K\fR.

.TP

Print numbers in human-readable format.

.TP

Be verbose.

.TP

Use N threads for I/O. This is used by the file hashing and dedupe

stages. Default is automatically detected based on number of host

cpus.

.TP

Read hashes from a hashfile. A file list is not required with this

option. Dedupe can be done if duperemove is run from the same base

directory as is stored in the hash file (basically duperemove has to

be able to find the files).

.TP

Write hashes to a hashfile. These can be read in at a later date and

deduped from.

.TP

Defaults to no. While checksumming a file, duperemove can optionally

lookup file extent state to see whether a given file block is already

shared. This information can later be used to optimize the search for

duplicate extents. There are some caveats to this, so please read

below.

On btrfs, extents which have been snapshotted are reported as shared,

as more than one inode points to them. A deduped extent also gets

reported as shared for the same reasons. Internally duperemove can not

yet make the distinction between the two. If \fB--lookup-extents\fR is

turned on, duperemove will consider a shared extent to have already

been deduped. On a snapshotted file system this might cause all or

most of the extents to be skipped for dedupe.

If you are not making snapshots on the fs you are deduping, this

option will allow duperemove to make better decisions on which extents

to dedupe.

A future version of duperemove will remove this restriction, allowing

us to default this option to on.

.TP

Prints help text.

.TP

Deprecated, see \fB--io-threads\fR above.

Please see the \fBFAQ.md\fR file which should have been included with your duperemove package.

Deduplication is currently only supported by the \fIbtrfs\fR filesystem.

.BR filesystems(5)

.BR btrfs(8)