# Word list scrapper Merger and parse for duplicate entries trying too optimize and convert to python [closed]

#!/bin/bash
reset

echo "Creating output folder to Desktop/CombinedText and final passwordlist will show up as Final0Duplicates.txt"

##Functions##

##output file detection and creation##

function Dependency_Check
{
file2='~/Desktop/backupwordlist'
if [ ! -f "$file2" ] touch "$file2"
file2='~/Desktop/CombinedText'
if [ ! -f "$file3" ] touch "$file3"
}
## sort and merge .txt files and list files##

function Combined_Sort
{
##checks if file1 is not in directory##
file1='Final.list'
if [ ! -f "$file1" ] then echo "$file1 is not present"
echo -e "\e[41m\e[5m[+]-------------------creating $file1 and adding all .txt files to it-----------[+]\e[0m" cat *.txt > Final.txt reset ##text_Troll sleep 2s echo -e "\e[42m\e[5m[+]-------------------Working on Final List-------------------------------------[+]\e[0m" cat Final.txt > Final.list reset echo -e "\e[42m\e[5m[+]--------final list finished removing 1.list-------[+]\e[0m" echo -e "\e[41m\e[5mremoving Duplicates from Final list\e[0m" sleep 10s reset sort -u Final.list > Final0Duplicates.txt find . -name Final0Duplicates.txt -exec cp {} ~/Desktop/backupwordlist \; sleep 10s else echo "$file1 is present"
echo "combining list files to 2.txt"
cat *.list > 2.txt
sleep 30s
reset
sleep 20s
echo "Working on Final List"
cat *.txt > Final.list
echo "final list finished removing 1.list"
echo "removing Duplicates from Final list"
sort -u Final.list > Final0Duplicates.txt
find . -name Final0Duplicates.txt -exec cp {} ~/Desktop/backupwordlist \;
sleep 10s

fi
}

##Crawls all subdirectories and extracts txt documents to desktop folder##

function crawl
{

find . -name \*.txt -exec cp {} ~/Desktop/CombinedText \;
}

reset
crawl
reset
echo "crawling"
Dependency_Check
echo "Creating Dependency Folder on desktop "
Combined_Sort
echo "All Text Documents have been Combined and sorted."
sleep 6


Okay so a break down of this project started as a pseudo code project out of boredom. but I was looking to see how this can be preformed in python. There are several functions in this code as a break down.

Functions

Dependency_Check checks to see if the output folders are on the Desktop if not it creates it to be honest this is probably an unnecessary function

Combined_Sort This takes all of the text documents and merges them into one text document than removes duplicate entries and outputs a file Final0Duplicates.txt to ~/Desktop/backupwordlist

Crawler This function crawls all subdirectories of the working folder and grabs all files with .txt extension copies them to ~/Desktop/CombinedText

GOAL

My Goal with this program is to possibly create this as a powershell script to work with windows or a batch file using system variables instead of importing a ton of libraries with python. any help would be greatly appriciated, Later this will be released on github as a tool for pentesters to build a master dictionary attack file. Feel free to use this code it is fully fuctional as of 4/9/2019 on 64 bit kali linux latest build and patch.

• Hey, welcome to Code Review! Here we review working code and try to make it better. This is not the right place to ask for help in rewriting your code in another language. Have a look at our help center for more information on what is on-topic here. – Graipher Apr 9 '19 at 16:15
• Also, there seems to be a lot of pointless sleeping, running of needlessly expensive commands (cat Final.txt > Final.list instead of cp Final.txt Final.list) and weird comments (##text_Troll). Is this code meant as a joke? – Graipher Apr 9 '19 at 16:20
• so you're telling me you reimplemented cat ./**/*.txt | sort | uniq >> CombinedText badly? – Vogel612 Apr 10 '19 at 10:49
• Vogel612 Your a wizard and apparently I could have done this with that magic one liner of yours.....face palm with a pan – user11335084 Apr 10 '19 at 13:57
• @Vogel612: cat ./**/*.txt | sort -u >> CombinedText would be even faster. – Graipher Apr 11 '19 at 14:29