I have a bunch of exact-duplicate pictures that I've acquired over the years. I'd like to create a list of all them so I can eventually delete some. My idea was simple: dump the hash and location of every image file under a path in MongoDB for later analysis. This is what I came up with:
import com.david.mongodocs.ImageEntry;
import com.mongodb.MongoClient;
import org.apache.commons.codec.digest.DigestUtils;
import org.mongodb.morphia.Datastore;
import org.mongodb.morphia.Morphia;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
public class MD5Deduplicator {
private static Datastore datastore;
public static void main(String[] args) throws Exception {
long startTime = System.nanoTime();
Morphia morphia = new Morphia();
morphia.mapPackage("com.david.mongodocs");
datastore = morphia.createDatastore(new MongoClient(), "md5Deduplicator");
datastore.ensureIndexes();
logDuplicates(Paths.get(args[0]));
System.out.println("Completed scan in " + (System.nanoTime() - startTime )+ " nanosecs");
}
private static void logDuplicates(Path path) throws IOException {
Files.walk(path).parallel()
.filter(Files::isReadable)
.filter(Files::isRegularFile)
.forEach(filePath -> {
try {
String contentType = Files.probeContentType(filePath);
if (contentType != null && contentType.startsWith("image")) {
FileInputStream fis = new FileInputStream(filePath.toFile());
String md5 = DigestUtils.md5Hex(fis);
fis.close();
ImageEntry imageEntry = new ImageEntry(filePath.toAbsolutePath().toString(), md5);
datastore.save(imageEntry);
}
} catch (Exception e) {
e.printStackTrace();
}
});
}
}
ImageEntry class:
package com.david.mongodocs;
import org.mongodb.morphia.annotations.Entity;
import org.mongodb.morphia.annotations.Id;
import org.mongodb.morphia.annotations.Indexed;
@Entity
public class ImageEntry {
@Id
public final String filePath;
@Indexed
public final String md5;
public ImageEntry(String filePath, String md5) {
this.filePath = filePath;
this.md5 = md5;
}
}
From the visualVM profiler it looks like the slowest part of this is the md5hex function (I was a little surprised, I was expecting the slowest part to be related to the FileInputStream
or save function). Is there perhaps an alternative hashing function or md5 implementation I should use?
I'm also a little worried that Files.walk().parallel()
might be putting a little bit too much faith in Java defaults and may not be the most optimized way to parallelise this.