I plan to use photos in my graph applications. But first I need to sort through and remove the dups of which there are many. So I set out to do this programmatically:

Generate a hash of each photo’s contents:
$ for file in *; do echo $(md5 "$file"); done > ed1

List the hash values and count the number of files.
$ cat ed1 | awk '{print $5}' | wc -l
2195


Now list the hash values and sort unique. How any unique files are there?
$ cat ed1 | awk '{print $5}' | sort -u | wc -l
2102


We have 93 dups.

Identify the unique files:

$ cat ed1 | sort -k5 -u | wc -l
2102


From here, basically diff the two groups and remove the duplicate files. I then loaded these files back into aperture. To my chagrin, not all dups were removed. Looking at the remaining dups I discovered that the time stamps were different by about 8 hours. This was enough to produce different hash values.

So, I’m basically doing this manually by combining all the photos into sets of Aperture libraries by year. Some years I need to break down by quarter or month. It’s tedious, but effective. Sometimes, manual is the best way!