AllDup is a powerful duplicate file finder for Windows
Depending on how you use your computer and download and transfer files, you may end up with a large collection of files on it.
If you prefer to download in bulk for instance, then you may end up with duplicate files. The same is true if you have a large collection of files, for instance music or pictures, then it may be difficult to keep an overview of what you have and what you still want which may result in duplicate downloads as well.
Going through a collection of thousands of pictures or music files manually is not really an option as it would take a long time to do so.
That's where programs like AllDup come into play. It is a portable program for Windows that scans alle files in directories that you specify for duplicate contents.
The usefulness of a duplicate file finder lives and falls with the search criteria that it offers. A program that can only search for duplicates based on file names for instance will skip duplicates that have different names.
AllDup ships with eight different search options that you can combine. It is possible to search by name and extension, but also by size, content (byte to byte), attributes, file modification and creation date, or hard link.
If you select file contents, you get additional customization options that allow you to ignore id3 and exif data.
Once you have made your selection there, you select one or multiple folders that you want scanned by the program. It is possible to go all in and select all root folders of all drives and partitions connected to the system but that is usually not a good idea.
First, it takes way to long to scan all those files for duplicate and second, you will get hits in Windows folders which are better left alone. The program scans all subfolders automatically.
Before you hit start, you may want to go through the search options, file and folder filter menus as they hold interesting options as well.
Search Options for instance allows you to enable the scanning the contents of zip and rar files, and to exclude files that are larger or smaller than a defined size.
File and folder filters on the other hand allow you to exclude files and folders from the scan, or turn things around and only include select files or folders in the scan.
The program scans only the following file types by default: bmp, gif, jpeg, jpg, mp3 and png. While that speeds up the scan, it ignores other file types you may be interested in such as flac, doc or avi.
The results window looks like an icon editor on first glance as it displays several icon toolbars at the top.
Move the mouse cursor over an icon to get a tooltip which hints at what it does. Probably the most useful ones are the file selectors in the last toolbar as it enables you to automatically select files. A click may select all files but the first one for instance or all files but the one with the shortest name.
Other filters of interest allow you to remove files of select folders from the listing or expand all groups automatically.
Once you have select one or multiple files you right-click on the selection to display a context menu with options to delete the selection.
The context menu displays other options, for instance to add a file to the ignore list, to open it on the local system or to select all files of the same path as well.
Search results can be saved and exported to a txt or csv file. If you save search results you can load the results at a later point in time.
It will take some time before you get used to all the features that the program offers. While it is possible to ignore most of them and get great results, it is necessary to go through some of the menus such as file filters before you run scans as you may end up with no or only partial results otherwise.
Two areas could use improvement: first, the preview option is handy as it shows a preview of the file directly in the interface when enabled. While that is the case, only one preview is shown at a time. It would be useful if you could display two images side by side to determine if they are indeed identical or which has the better quality.
Second, an option to match partial file names as well.
With all that said, AllDup is an excellent powerful duplicate file finder for Windows.
> The usefulness of a duplicate file finder lives and falls with the search criteria that it offers.
I’ve been using dupfinders for decades and I’ve found that almost all support sufficient search criteria to do a decent job of finding the duplicates. I can’t remember when I’ve used one that doesn’t at least do a reasonable level of checksum hashing. If they commonly used lame criteria then it would be an issue but they don’t.
I have disparate collections that I have split off and recombined. For example, I might have some filesystem corruption and lose some MP3 files, and in the meantime I find a dump of an SD card that I hurriedly evacuated, and it contains a portion of the same collection. A dupfinder will allow me to combine them properly, albeit saving same songs where metadata has been changed in one file but not the other.
In my experience, their usefulness hinges entirely on the post-search selection criteria instead. Usually I want to prioritize a certain folder to keep, possibly a few, and allow it to delete (or move) file from all other folders. It is rare to find one that operates like that.
Rick, so are you giving this software a thumb up or down?
Adam, I have not tested this specific program and was neither giving it a thumbs up nor a thumbs down. My comment was about stated criteria for making that decision.
It is my experience that the stated criteria, comparing by hashes, is completely bog standard and has been for decades; you might as well judge a car by whether or not it comes with wheels. I proposed instead judging a dupfinder by how powerful and flexible it is in helping you select which files to eliminate and which to keep. That is an important area where they rarely are good enough and always vary.
An alternative to AllDup is DoubleKiller, which is available here:
http://www.bigbangenterprises.de/en/doublekiller/
It is an excellent suggestion. I have reviewed the program back in 2007: https://www.ghacks.net/2007/01/28/remove-duplicate-files/
I have a client who before I got to him, over three migrations and three different IT people ended up with five to seven (yup, five to seven) copies of everything – and his everything was some 150,000 – 200,000 UNIQUE files (he runs a publishing service among other things) – saved msg’s, draft upon draft upon drafts of journals entries and so on. Items were in “my documents”, and c:/temp and c/batch*/ and on the desktop (various folders copied form my documents) and so on. That means I had to wade through 750,000 to 1 million files. It probably wasn’t quite that bad, as not everything was duplicated, and a lot maybe only once. Utter chaos. In order to search and find things, his entire drive was content indexed – this 5+ times duplication was being backed up, his HDD was 90% full, (and I will not mention the 60+GB in 5 PST files – some not even loaded and he couldn’t even search). Things got so bad he started saving items on is desktop, then folders of goodies and then his desktop got so bad the last IT guy gave him “fences”. This was his documents only – his pictures (a ton of personal ones was ok’ish, and his videos number about 5 and his music was on a separate dedicated music PC)
Long story short – Everything was moved into ONE location – My Documents – some folders were renamed if there was a clash. Nothing was deleted. Then I ran batches using a portable CloneSpy ( http://www.clonespy.com/ – to make portable choose portable during the install ). In two to three long afternoons I had it squeaky clean – First overall pass, only exact duplicates (name and size) on a longer path were removed. Second overall pass items that were the same name but a different size, the oldest was removed (first of all a hundred or so were spot checked). Some of these passes/batches would find 5 files and I’d just click the one i wanted to keep – it made it quite fast, plus you could automate a lot of it.
Then it was a matter of reorganizing his sub folders in My Documents into some semblance of order/sanity and merging a few folders. Now his machine didn’t have to back up as much and indeed a ton of indexing was turned off. We actually got him a new PC with a dedicated boot drive, secondary drive for all data, did away with windows content indexing (he uses copernic) and .. his PSTs 18 months later is down to a grand total of 25Gb and falling (and all live and searchable) as he daily removes any crap out of it.
TL;DR: CloneSpy :) That AllDup looks like a complete and utter piece of sh*t if I might say so.
That’s a lot of files and work :) I agree that AllDup looks kinda messy but it is a great program.
The current version of DoubleKiller has several limitations over the pro version of the software that Alldup does not.
Xeno, I did not use DoubleKiller for a long time. What are those limitations?
Please excuse the delayed reply, sometimes it can be difficult for me to reply as promptly as I would like to.
From a cheap-stakes perspective…
The freeware version of DoubleKiller that is currently available as it is today has a lot of restrictions compared to the payware version.
When you compare the freeware version of DoubleKiller to that of AllDup (which is offered completely as freeware), the functionality that is offered within the latter program comes across as being the more attractive software package over the former.
[ http://www.bigbangenterprises.de/en/doublekillerpro/comp.htm ]
[ http://www.alldup.de/alldup_help/alldup.htm ]
Another alternative to AllDup is Ashisoft Free Duplicate File Finder, here ashisoft.com
As I suspected, this program is useless for finding duplicate photos and the only way to check it’s accuracy is against another duplicate file finding program which I do have. In steps, “Duplicate Photo Finder” by Firmtool, which in Windows Vista and earlier could be launched from the context menu, but in Windows 7-64b I have to enter the folder using the standard Windows File Explorer interface.
I pointed ALLDup at a folder I know has duplicates, selected every conceivable filter that applies to images, and even selected the long slow options and it reported zero duplicates in 1176 files. I pointed Duplicate Photo Finder (DPF) at the same folder and of my 1176 files, it found 501 images and lists them by percentage of how much similar they were and it automatically selects the biggest and best version and places it into the original column while it leaves the poor quality, cropped, watermarked, B-n-W version in the duplicate column and lists them according to percentage they are similar by and the sized taken up by the files it deems duplicates.
The percentage can be set for a minimum value to not include in the results, or simply give the program no restrictions. The duplicates or similars it finds can even be two photos in a series where two practically identical shots but the difference is a tiny part moved, or turned direction and DPF lists how close they are similar to each other and from that we can select individual files or grouped files to be deleted. The program is a shareware program and all the free version does is find files, it won’t give delete options and it only finds files that are 100% similar.
Often I find two identical photos to the naked eye but the program says they are not 100% and it’s because one is raw uncompressed and the 2nd is a high compressed version thus lower quality so it put the high compressed version in the duplicate column. Other times and it is rare I find it finds two entirely different photos, with zero identical context in any way shape or form and it will report it to be 100%. So before I allow it to mass delete the files from the main and subfolders in one go, I usually go through them to check manually and it even allows us to exchange the two from side to side to see if we ourselves can catch the visual differences and deem it a true duplicate or something to remove from the list entirely.
http://duplicatefinder.firmtools.com/
So thanks for the information in ALLDup, … I’m sure it can do wonders for someone but for my purposes DPF meets and exceeds my needs.
p.s. Just to be fair I tried ALLDup on several image folders and each time it reported zero duplicates whereas DPF found tonnes… I guess it is time to clean out my file server…
The program does not offer options to compare two images visually. If you have the same image as a jpg and a raw file, then it won’t detect them as duplicates.
I do agree that it cannot be used for that purpose and that other tools are superior in this regard.
Thanks for the info!
LegoActionFigure, don’t confuse a duplicate image finder with a duplicate file finder. While images are a type of file, in this case the distinction is HUGE.
A duplicate file finder’s job is to compare files byte by byte and NEVER eliminate a file does not have an exact byte-by-byte duplicate. A data archivist/hoarder like myself is unwilling to discard a single unique byte, even if it is likely to be meaningless.
A duplicate image, audio, or video finder’s job is to forego those byte-by-byte omparisons and heuristically compare the contents of the file the way that a human would experience it, providing the ability to eliminate similar/same content where (as you noted) formats differ, or even where a single character of EXIF metadata has changed.
…to me they’re just files, so I see no difference other than context… an image file is after all just a file which contains a image if when viewed through the appropriate program will display said image. A text file is a document which is just a file with text in a format an appropriate program will display said text. So it is all the same to me. My mistake was to assume that was a universal concept.
I tried the program again and pointed it at various documents, (PDF, DOC, TXT, MS, MP3, M4A, AVI, MP4 etc…) and it still didn’t find any duplicates even if I purposely made duplicates and renamed them. So I guess I don’t understand what ALLDup is suppose to do. …
Martin’s sample image shows what seems obvious as cover art files (JPG), so this is what gave me a hint it may be used for images. It found nadda.. It’s either useless, or I keep good accounts of where everything is on my file server. The only thing i don’t keep track of closer, are image files because I bulk download them per page, or per site using a spider downloader. So for my needs, DPF works perfectly.
> it still didn’t find any duplicates even if I purposely made duplicates and renamed them
Interesting. How did you make the test duplicates that it failed to detect?
If, for example, you opened a file in MS Word and then chose “Save As” to make a copy, that’s not going to create a duplicate file. That’s merely going to create another file with most of the same document content. Word is likely to write some new metadata into the file (which you don’t see just looking at the document, but might see by viewing its properties) and maybe even change the file format slightly.
If you made a copy using a command prompt and a command like “copy file1.doc file2.doc” or right-dragging a file to a blank spot in the folder and choosing “Copy Here” from the menu, there may be some kind of malfunction going on, perhaps in AllDup.
Since we’re on the topic of duplicate content finders, by any chance can you recommend a duplicate video finder that will rate by similarity?
A duplicate is a duplicate. If you’re looking for a buplicate or a duplicoot or a dipsticklicaty then you’re looking for something else. When looking for a duplicate ALLDup is looking for 100%. If you thought you were getting a Corvette and you actually got a Chevette you may have gotten some percentage of a similarity but it’s not a duplicate, it’s not even a piss-poor substitute.
26 Best Free Duplicate File Finders
http://listoffreeware.com/list-of-best-free-duplicate-file-finder/
Wow really your only allowed one preview at a time, that would be like going to purchase sun glasses but one only eye has one lens. Martin I don’t care what you say, but honestly you’ve had to stop more than once and just say what were you thinking. Only one preview panel seriously lol, it ant going to stop me from trying it out though.
Thanks for Tutorial Martin
AllDup is excellent, the only problem is that it doesn’t support Unicode.
Thanks, I’d all but given up on finding an existing duplicate finder that looked at content, one of the times when google was not helpful with its results.
I’ve long used FASTCOPY for safely/quickly copying large volumes of files/folders. Just realized it appears to only compare date/size rather than content. Have many corrupt files that are not being overwritten by the non-corrupt file with same size/date. Can you recommend the a trusted software that will identify content variables; thereby, keeping both by adding a number to the the second file?
For duplicates, I use Duplicate Cleaner; albeit almost too precise in some ways, yet it is absolutely the most reliable! Would very much appreciate any input.
FYI: Prefer Windows Software but have an iMac if needed for best possible solution/app.
As Martin said:
.. AllDup is an excellent powerful duplicate file finder for Windows.
Well, it’s 2020 and AllDup is still working great, and still 100% free with no crap.
Also, perhaps this is new with it:
It has a 1, 2, 3 wizard that makes it very easy to use.
It has both portable and a proper Windows installer.
A lot of thought and skill has gone into AllDup. It reminds me of Advanced Renamer, which likewise is excellent, powerful, and easy to use.