Detect and Remove Duplicate Files

If you are like me you are probably downloading lots of files from the internet every day. Those files are using a large part of my hard disk and every now and then I have to start deleting some of the files to recover some free space for future files that I want to download to my PC.

I also have a rather large amount of images on my PC, could well be over 30000 and I always had the impression that there have to be duplicates all over the place. Never had the time to manually check them and delete the duplicates, would take forever.

Therefore I decided to take a look at tools that compare and delete duplicate files automatically. I found two that are ideal for that purpose, one is more of a general duplicate finder and one has been specifically designed for images.

CloneSpy

Clonespy is able to locate duplicate files even if they have a different file name or size. Its freeware of course, simply unzip to a location and run the file. The program gives you some options on how to handle the duplicate files, move them, delete them or export a list of the files without removing any.

CloneSpy's interface may look complicated on first glance but you will grow accustomed to it quickly. Basically, what you do is add one or more folders to Pool 1. You can then start the comparison of all files in the folders with a click on "start scanning".

The program supports several methods of detecting whether files are duplicates. The most thorough is a checksum-based check, but you can also check for same name or approximate size instead.

In addition to making those changes, you may also check a list of files against a previously created checksum file, or perform automatic operations on duplicate files. It is for instance possible to automatically delete the newer or smaller file among other things.

You may also want to open the options at least once, preferably before you start scanning for duplicate files. There you find many preferences that determine the outcome of scans, for instance how file attributes like read-only should be handled, file types you want excluded from the scan, or define a minimum size of files you want the program to scan.

Unique Filer

The second tool which is unfortunately nagware is called Unique Filer. I have not found a freeware tool that can perform the same operations as Unique Filer for you.

It allows you to find image dupes even if the images have been resized, rescaled, resampled and color corrected. It also finds normal dupes of course, but that's not what makes it so special.

If you have many images that you downloaded over the years I recommend Unique Filer to you, you can use all its functions without registering, but see a nagscreen all 10 files that you opt to delete. And those are a lot, from my 30000 files more than 5000 have been dupes.

Please note that the program has not been updated since 2001. While it may still work on newer systems, I have not tested that personally and can't vouch for that therefore.

Update: Unique Filer is unfortunately not available anymore. Try an alternative like Double Killer instead.

Advertisement