Find and remove duplicate files on your system with duff

Those who like me like to collect certain information (as HD wallpapers, etc) we have the problem that over time we begin to have duplicate information in our system, the typical example may be a repeated Linux wallpaper (each file with a different name). The real problem may not be space but rather obsessive-compulsive orderlies like me; we simply cannot be calm and at peace knowing that in our precious collection there is something wrong, duplicated, wrong 😀

Luckily in Linux there is always a tool to solve our problems, for this case of duplicate files we have the software: duff

You can find it in your repository, on distros like Debian or based on it a simple: sudo apt-get install duff it will be enough to install it, in ArchLinux I imagine it to be sudo pacman -S duff

Once we install it, using it is really very simple, for example I will look for all the duplicate files in my folder Pictures (/ home / kzkggaara / Pictures /):

duff -r  /home/kzkggaara/Pictures

Parameter -r it is so that it searches recursively and then it only remains to define in which folder it should search, in this example / home / kzkggaara / Pictures

This will show you something like:

As you can see, it tells us that there are 2 duplicate files in Pictures / Paisajes / Landscapes + Green / and also two duplicate files in Pictures / Mangas / Naruto / Sharingans /

This is only a small part of the whole screenshot… I just noticed that I have a lot of duplicate files 0_oU

Now the question is imposed: How to delete duplicate files?

With the following command, it will search for all duplicate files and delete one of them, that is, they are two identical files ... one of them will be deleted:

duff -e0 -r /home/kzkggaara/Pictures/ | xargs -0 rm

This command I repeat, it will eliminate duplicate files that are in the directory that you indicate, always leaving only 1 file of all the others that may be identical.

I am still testing command combinations that may be useful with duff… xargs, as well as in the man duff reading how to customize the output of it, I recommend you read the help of duff

Anyway, there is not much more to add at the moment, I hope you have found this command useful 😀

regards


The content of the article adheres to our principles of editorial ethics. To report an error click here!.

27 comments, leave yours

Leave a Comment

Your email address will not be published. Required fields are marked with *

*

*

  1. Responsible for the data: Miguel Ángel Gatón
  2. Purpose of the data: Control SPAM, comment management.
  3. Legitimation: Your consent
  4. Communication of the data: The data will not be communicated to third parties except by legal obligation.
  5. Data storage: Database hosted by Occentus Networks (EU)
  6. Rights: At any time you can limit, recover and delete your information.

  1.   st0rmt4il said

    Thanks for the tip brother, could you in the future make some posts on how to use "xargs"?

    Thank you!

    1.    KZKG ^ Gaara said

      I'm not an expert with xargs yet, to explain it simply (very simply), xargs allows you to execute commands by passing the output or result of what was executed before as a parameter (that is, what is before the pipe or |)

    2.    dhunter said

      One detail, every time you use xargs try to send it data without \ other than passing the -0 and then receiving in xargs with -0 this is very important for file names that contain space like "my file is this.mp3".

      An example with find.

      find -iname "to-delete" ./ -print0 | xargs -0 rm

  2.   cat said

    thanks for the article, sometimes there are some .desktop (especially wine) repeated

    1.    KZKG ^ Gaara said

      A pleasure 🙂

      1.    cat said

        I do take advantage of clarifying that in Arch and derivatives it is installed with yaourt, not with pacman

        1.    st0rmt4il said

          Thanks for the clarification: D!

  3.   Joaquin said

    Very good!
    Added to the collection of special tools 🙂

    1.    KZKG ^ Gaara said

      Thanks for the comment 😀

  4.   Carper said

    Excellent contribution, thank you very much XD.
    regards

    1.    KZKG ^ Gaara said

      Thanks to you for commenting 🙂

  5.   F3niX said

    It's good. I will try to remove duplicate mp3s from my 100gb uu collection

    1.    KZKG ^ Gaara said

      Thanks for commenting, good luck 🙂

  6.   Blaire pascal said

    Good input, as always. Thank you.

    1.    KZKG ^ Gaara said

      hahaha thanks, I always try to put interesting things 😉

  7.   VulkHead said

    Very good contribution .. Thank you!

    1.    KZKG ^ Gaara said

      Thank you ^ - ^

  8.   jems said

    I will start to erase all my duplicate files from my disk that has been filled, I hope to empty about 10Gb,

  9.   damncookie said

    Ohh thank you very much this will help me a lot in my work 😉

  10.   albert I said

    Thank you very much for presenting little programs that solve problems like this one, so common among internet users.

    1.    KZKG ^ Gaara said

      Thanks to you for reading and commenting.

      regards

  11.   kuk said

    this had me crazy> _ <thanks !!!

  12.   Joan said

    Thank you very much I had a mess with duplicate photos that I did not know what to do !!!!
    Thank you

  13.   Alexander said

    Sorry, I have tried duff to delete duplicate files with the syntax you indicate but the terminal output tells me that the rm operand is missing
    I write something like this in terminal:
    duff -e0 -r / media / alejandro / backing / libraries / ebooksepub / | xargs -0 rm
    and I get this: rm: an operand is missing
    Try 'rm –help' for more information

    Please could you tell me the operand when, as in my case, it is about deleting thousands of duplicate files in my epub library?

    1.    KZKG ^ Gaara said

      Try putting rm -R at the end

      1.    Alexander said

        Thanks for your attention, but unfortunately your proposal did not work, I show you the output:
        alejandro @ alejandro-ubuntu-mate-1504: ~ $ duff -e0 -r / media / alejandro / backup / libraries / ebooksepub / | xargs -0 rm -R
        rm: an operand is missing
        Try 'rm –help' for more information.

  14.   Hector said

    Good Tip.
    the problem with the rm operand is that it doesn't have to delete. If the command is executed without the pipe, it is verified that there are no duplicates.
    regards