1
\$\begingroup\$

This finds duplicate video files using size, duration and partial filename. I have a mass of 17000+ videos and it seems to do an OK job; the amount of duplicates has certainly reduced drastically. Even so there should be many more methods of finding duplicates using these three (or possibly more?) properties.

I don't entirely trust the code here. Some duplicates remain that could've been spotted with better filename comparison. Once it recycled both duplicates of a file! I was baffled. Never found the problem and I'm still using it. Also it's quite slow for a massive collection.

echo.>%temp%\vdat
for /f "delims=" %%v in ('dir d:\videos /b /s /a-d 2^>nul') do (
    for /f "tokens=2 delims=," %%i in ('ffprobe -v quiet -show_entries "format=duration" -of csv "%%v"') do echo %%i/%%v/%%~zv
)>>%temp%\vdat
for /f "tokens=1,2,3* delims=/" %%i in ('sort "%temp%\vdat"') do if exist "%%j" call :strict "%%j" "%%i" "%%k"
for /f "tokens=1,2,3* delims=/" %%i in ('sort "%temp%\vdat"') do if exist "%%j" call :loose "%%j" "%%i" "%%k"
exit /b

:strict
:: if same exact duration+size then remove. blind to filenames. which is nice.
if "%~2" equ "%lastdura%" if "%~3" equ "%lastsize%" recycle -f "%lastfull%" && echo recycled identical duplicate "%lastfile%"
set "lastfile=%%~nx1" & set "lastfull=%%~1" & set "lastdura=%%~2" & set "lastsize=%%~3"
exit /b

:loose
:: if same exact duration+size within the megabyte+same start of filename then remove.
if "%~2" equ "%lastdura%" echo %~3 | findstr /b /c:"%lastsize:~,-6%" >nul && echo "%~nx1" | findstr /i /c:"%lastbite%" >nul && recycle -f "%lastfull%" >nul && echo recycled near-identical "%lastfile%"
set "lastfile=%%~nx1" & set "lastfull=%%~1" & set "lastdura=%%~2" & set "lastsize=%%~3"
set "lastbite=%lastfile:~0,21%"
exit /b

Note that it uses recycle.exe instead of del so for testing you might wanna change that into something else if you don't have it. Also it is set to look for videos in d:\videos and ffprobe.exe is needed for fetching duration.

Even minor feedback very welcomed.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.