File Organiser in Python

Question

main.py

The "main.py" script is a Python program designed to organize files within a specified directory based on their file extensions. It enhances file organization and grouping for better management by moving files into subdirectories that correspond to specific categories, such as programming languages, file types, and data formats. This script provides a way to efficiently organize and categorize files in a given folder, making it easier to find and manage files of different types.

---

Notes on main.py

• The script uses constants defined in "constants.py" to categorize files into different groups.
• It provides a clear structure for adding new categories or modifying existing ones.
• The "constants.py" file allows customization of file extensions and categories.

---

Code

"""_File Organizer_
@date:
    August 28, 2023

@version:
    1.0

@license:
    This code is licensed under the MIT License.

@file:
    main.py

@description:
    This script is a Python program designed to organize files 
    within a specified directory into different categories 
    based on their file extensions. It aims to enhance file 
    organization and grouping for better management. The script 
    achieves this by moving files into subdirectories that correspond
    to specific categories such as programming languages, file types, and data formats.
    
@usage:
    Run this script with the necessary dependencies installed and provide the name of the folder
    you would like to organized. The script will organize the files in that folder.
    
    
@example:
    $ python main.py
"""

"""_imports_"""
import os
import shutil
import sys
from typing import List
from tqdm import tqdm

from constants import *

#---
# Imported for testing purposes only
#---
from test import run_test, delete_dir

"""_summary_
_desc_: returns the extension of a file name.

_param_: file_name (str): The name of the file.

_returns_: str: The extension of the file name.
"""
def get_extension(file_name: str) -> str:
    return "." + file_name.split(".")[-1].lower()            

def move_file(subcategory_dir: str, file: str) -> None:
    if not os.path.exists(os.path.join(TARGET_DIR_, subcategory_dir)):
        os.makedirs(os.path.join(TARGET_DIR_, subcategory_dir))
 
    source_path = os.path.join(TARGET_DIR_, file)
    destination_path = os.path.join(TARGET_DIR_, subcategory_dir, os.path.basename(file))
    shutil.move(source_path, destination_path)
    
"""_summary_
_desc_: organizes files into different categories based on their file extensions.

_param_: files (List[str]): A list of file paths.
    
_returns_: None: This function does not return anything.
"""          
def organize_files(files: List[str]) -> None:
    #for file in files:
    for file in tqdm(files, desc="\n[organizing files]", unit="file"):
        if file.lower() != MAKEFILE_:
            extension = get_extension(file)
        else:
            extension = "Makefile"
        
        for extensions, category in file_extensions.items():
            if extension in extensions:
                category_dir = category
                if extension in [".cpp", ".hpp", ".asm", "Makefile"]:
                    subdir_name = "Cpp"
                elif extension in [".c", ".h", ".pch", ".o", ".a"]:
                    subdir_name = "C"
                elif extension in [".java"]:
                    subdir_name = "Java"
                elif extension in [".py", ".pyw", ".ipynb"]:
                    subdir_name = "Python"
                elif extension in [".js", ".html", ".css", ".scss", ".sass", ".php", ".ts"]:
                    subdir_name = "Web"
                elif extension in [".rb"]:
                    subdir_name = "Ruby"
                elif extension in [".cs", ".vs"]:
                    subdir_name = "CSharp"
                elif extension in [".rs"]:
                    subdir_name = "Rust"
                elif extension in [".xlsx", ".xls"]:
                    subdir_name = "Spreadsheets"
                elif extension in [".pptx", ".ppt"]:
                    subdir_name = "Presentations"
                elif extension in [".csv", ".dat", ".db", ".dbf", ".log", ".mdb", ".sav", ".sql", ".json", ".xml"]:
                    subdir_name = "Data-Sets"
                elif extension in [".git"]:
                    subdir_name = "Git"
                elif extension in [".exe"]:
                    subdir_name = "Executables"
                else:
                    #check if no subdirs are present
                    subdir_name = OTHER_STR_
                    
                move_file(os.path.join(category_dir, subdir_name), file)
                break
        else:
            move_file(os.path.join(OTHER_STR_), file)

"""_summary_
    _desc_: exits the program.
    _returns_: None
    _param_: exit_code: int corresponding to the exit code
"""
def quit(exit_code: int) -> None:
    sys.exit(exit_code)

"""_summary_
    _desc_: establishes the current directory and returns a list of files in the directory.

    _returns_: List[str]: A list of file names in the current directory.

    _raises_: FileNotFoundError: If the target directory is not found.
"""
def establish_current_dir() -> List[str]:
    try:
        with os.scandir(TARGET_DIR_) as entries:
            files = [entry.name for entry in entries] # if entry.is_file()
            return files
        
    except FileNotFoundError:
        print(ERROR_MSG_)   
        quit(exit_code=1)
    
def main() -> None:
    run_test()
    files = establish_current_dir()
    organize_files(files)
    input("\nPress [Enter] to continue...")
    delete_dir()
    

if __name__ == "__main__":
  main()

constants.py

The "constants.py" file contains constants used throughout the program to categorize files into different groups. It defines file extensions and their corresponding categories, making it easy to maintain and expand the categorization system. You can customize the values in this file to fit your specific needs.

Code

"""_File Organizer_
@date:
    August 28, 2023

@version:
    1.0

@license:
    This code is licensed under the MIT License.

@file:
    constants.py

@description:
    This file contains the constants used throughout the program
    
@usage:
    Change the value of the TARGET_DIR_ variable to the folder you would like organized.
"""

'''_Could be changed to "Desktop" or any folder you would like organized_'''

TARGET_DIR_ = "__TEST__"

'''_List is unhashable, using tuples instead_'''

IMAGES_ = (".jpeg", ".png", ".jpg", ".gif", 
           ".bmp", ".svg", ".tiff", ".heic", ".webp")

TEXT_ = (".txt", ".rtf", ".md")

VIDEO_ = (".mp4", ".mkv", ".avi", ".mov", 
          ".wmv", ".flv", ".ogg", ".webm")

AUDIO_ = (".mp3", ".wav", ".m4a", ".flac")

APPLICATIONS_ = (".exe", ".lnk", ".msi", ".app")

CODE_ = (".cpp", ".hpp", ".asm", ".js", ".html", ".css", ".scss", 
        ".sass", ".php", ".ts", ".c", ".h", ".pch", ".o", ".a", 
        ".java", ".rb", ".rs", ".py", ".pyw", ".ipynb", ".cs", ".vs"
        ".xcodeproj")

INSTALL_ = (".dmg", ".pkg", ".deb", ".rpm")

COMPRESSED_ = (".zip", ".rar", ".tar", ".gz", ".7z")

DOCUMENTS_ = (".doc", ".pdf", ".docx", ".ppt", ".pptx", ".odt", ".ods",
              ".xls", ".xlsx", ".csv", ".dat", ".db", ".dbf", ".log", 
              ".mdb", ".sav", ".sql", ".json", ".xml")

OTHER_ = ("",)

'''_file categories used in organize_files()_'''
file_extensions = {
    IMAGES_: "Images",
    TEXT_: "Text",
    VIDEO_: "Video",
    AUDIO_: "Audio",
    APPLICATIONS_: "Applications",
    CODE_: "Code",
    INSTALL_: "Install",
    COMPRESSED_: "Compressed",
    DOCUMENTS_: "Documents",
    OTHER_: "Other"
}

'''_other constants defined for clarity_'''
MAKEFILE_ = "makefile"

OTHER_STR_ = "Other"

ERROR_MSG_ = f"[{TARGET_DIR_} directory not found]\n"

test.py

The "test.py" script provides a testing environment for the file organizer program. It includes functions to create random files with different extensions in a target directory. This allows you to test the file organizer's functionality without affecting your actual files.

---

Notes on test.py

• The script uses random file extensions to simulate a variety of file types for testing purposes.

• It ensures that the target directory is created before running tests.

• This testing script is separate from the main program and is useful for verifying the file organizer's functionality.

• Use the run_test() function to create a set of random files in the specified target directory.

• Use the delete_dir() function to remove the target directory and its contents after testing.

---

Code

"""_File Organizer_
@date:
    August 28, 2023

@version:
    1.0

@license:
    This code is licensed under the MIT License.

@file:
    contants.py

@description:
    This file contains the constants used throughout the program
    
@usage:
    Chnage the value of the TARGET_DIR_ variable to the folder you would like organized.
"""

"""_imports_"""
import os
import shutil
from typing import List
import random
import string

from constants import *

_MAX_ = 100

"""_summary_
_desc_: creates a new directory and moves a randomly generated file into it.

_param_: None

_returns_: None
"""
def make_dir() -> None:
    random_sequence = ''.join(random.choice(string.ascii_lowercase) for _ in range(4))
    
    all_extensions = (
        IMAGES_ + TEXT_ + VIDEO_ + AUDIO_ + APPLICATIONS_ + CODE_ +
        INSTALL_ + COMPRESSED_ + DOCUMENTS_ + OTHER_
    )
    random_extension = random.choice(all_extensions)
    
    file_name = random_sequence + random_extension
    
    with open(file_name, "w") as f:
        pass
    
    shutil.move(file_name, os.path.join(TARGET_DIR_, file_name))

"""_summary_
_desc_: create a directory if it doesn't exist.

_param_: None

_returns_: None
"""
def establish() -> None: 
    if not os.path.exists(TARGET_DIR_):
        os.makedirs(TARGET_DIR_)

"""_summary_
_desc_: deletes the directory specified by the constant TARGET_DIR_.

_param_: None

_returns_: None
"""
def delete_dir() -> None:
    shutil.rmtree(TARGET_DIR_)

"""_summary_
_desc_: run the test by establishing a connection and creating directories.

_param_: None

_returns_:
    None
"""
def run_test() -> None:
    establish()
    for i in range(_MAX_):
        make_dir()

Things on the TODO list:

• Command line Args -> e.g:

pyhton3 main.py "Downloads"

• Using .aksdirectory() from the Tkinter library
• Eventually, make a GUI

Note

I got the C++ disease in my freshman year and it's still sort of going on. I am sorry in advance if I insulted anyone with the use of unnecessary underscores, Doxygen style documentation, documenting my functions above the prototype and not inside the body, and defining a separate file for my constants (sort of like a .h file).

Answer 1

isort

Pep-8 asks that you organize imports into three sections. As author there's no need to devote a moment's thought to that; just "$ isort ." every now and again, and you're done.

from constants import *

The * star makes the meaning of this source code less obvious to the Gentle Reader; it slightly obscures it. Prefer from constants import TARGET_DIR_.

Also, wazzup with that trailing _ underscore? Typically we follow that convention for local variables like dir_ or map_, when we want to avoid shadowing a builtin. That's not what happening here. Plus, the identifier is part of your Public API, so spelling matters, more than for a local variable.

---

docstrings

"""_summary_
_desc_: returns the extension of a file name.

_param_: file_name (str): The name of the file.

_returns_: str: The extension of the file name.
"""

I didn't understand that module-level docstring at all; it's not helping me.

OIC, you meant to write a function docstring:

def get_extension(file_name: str) -> str:
    """_summary_ ...
    """
    return "." + file_name.split(".")[-1].lower()            

Nuke the _summary_ tag, it isn't helpful. Or cite the URL reference of the coding standard you're trying to conform to.

The _desc_ is obvious from the well-chosen function name -- elide it.

Similarly for _param_ and _returns_, but they additionally have redundant type annotations, which I fear can get out of sync with the signature's annotations. Recommend you specify type in just one place: the function signature. That way the mypy linter (type checker) can offer you automated assistance.

def get_extension(file_name: str) -> str:

This is a perfectly clear and lovely signature that you can keep as-is. But consider being even more clear by asking caller to pass in a Path.

The implementation is perfectly nice. I will note in passing that you chose to pass up both of these standard methods: splitext and suffix. Maintenance engineers will already be familiar with those, yet you're asking them to learn a new related API which has similar but subtly different behavior. This increases cognitive load and makes code defect injection during maintenance more likely.

---

sphinx

OIC, way at the end you offered a Note about conforming to Doxygen format. Hmmmm. Recommend you adopt sphinx, along with standard """docstrings""" between signature and function body.

And as far as choosing names goes, it is pretty important to use pep-8 conventions if you want to productively collaborate with folks in the python community. C++ names are very nice, in that environment, but they don't work well in other environments like Scheme, Forth, or others. Learning a new language is about much more than keyword syntax details.

---

document parameters

def move_file(subcategory_dir: str, file: str) -> None:

That is a perfectly nice signature, with well-named parameters. But it needs a """docstring""" that spells out we want a relative directory spec. Equivalently you might choose to assert or otherwise raise an error if that parameter starts with a slash.

    if not os.path.exists(os.path.join(TARGET_DIR_, subcategory_dir)): ...

Recall that makedirs and mkdir accept an exist_ok flag.

---

carefully choose what your Public API returns

_returns_: None: This function does not return anything.
"""          
def organize_files(files: List[str]) -> None:
    #for file in files:

Keep comments informative. After i += 1 we don't write "increments the integer index i by exactly one", and after a return of None we don't write "This function does not return anything." The code gives the details; the comments describe the motivation ("how" vs "why").

The commented for loop was fine during debugging. But now you're ready to tidy things up and merge down to main so other project members can interact with and maintain this code. That's when it is time to delete commented code, prior to requesting review.

        if file.lower() != MAKEFILE_:
            extension = get_extension(file)
        else:
            extension = "Makefile"

Ok, that's just crazy. Clearly there are "LICENSE" files and many others which lack a dot suffix. Special casing each of them as trouble arises won't scale.

Above I mentioned "subtly different" behavior for your helper plus a pair of standard helpers. Now we're seeing the chickens come home to roost. You implemented in a particular way down in the helper, and at this point you're telling the caller to "lump it!", that's just the way it is. The behavior the caller wants here is to receive "" empty string, rather than ".Makefile". Design each helper with the caller(s) in mind. Calling code should be simple, operating at a higher level, closer to your business goal than to the bits and bytes. Implementing the helper as you did was fine. But when you noticed this awkward edge case, that's when it would have made sense to go refactor the helper.

Messing around with upper M vs lower m in Makefile is not well-motivated here (no explanatory comment) and is likely to lead to bugs.

---

nesting

    for file in tqdm(files, desc="\n[organizing files]", unit="file"):
        ...        
        for extensions, category in file_extensions.items():
            if extension in extensions:
                ...
                if extension in [".cpp", ".hpp", ".asm", "Makefile"]:
                    subdir_name = "Cpp"
                elif extension in [".c", ".h", ".pch", ".o", ".a"]:
                    subdir_name = "C"
                elif ...

Wow, that's a lot of nesting, there.

I bet you could have identified one or two business concepts within that, suggesting a function name for an Extract Helper refactor.

The elif chain is not the end of the world. Consider replacing with match. And you should especially consider replacing with a dict.

Whatever you do, bury it down in some helper function. The move_file() / break / move_file() special casing is awkward.

It appears that Constants.py (unclear why that should be captilized) was originally intended to shoulder some responsibility for categorizing, but that was subsequently abandoned. DRY -- keep e.g. the CODE list in just one place, and don't invent synonymous names like WEB.

Downcasing "Makefile" seems odd. More generally your docstrings should be explaining what case-insensitive behavior caller should expect, if any.

---

In establish_current_dir() you

os.scandir(TARGET_DIR_)

which is fine. Apparently a single level of directory nesting suffices for your use case. If you ever need more flexibility, remember that os.walk() is always ready to recurse for you, and glob("**/*.txt") can recurse with a little less effort.

If you retain scandir, I recommend that you process sorted(os.scandir(...)) so the progress report happens in a consistent and human-meaningful way.

---

test.py

This is not a terrific module name to use for your code.

It regrettably is already defined as a standard part of python -- any SO contributor can successfully run $ python -c 'import test'

Your PYTHONPATH env var disambiguate which one gets used, either standard or your custom code. Depending on env var directory order is a recipe for confusion and higher maintenance costs. Choose another name, even if only to call it "test1.py".

---

use unit tests

Use the run_test() function to create a set of random files in the specified target directory.

Use the delete_dir() function to remove the target directory and its contents after testing.

Those are needlessly complicated instructions. Consider adding a $ make test target which implements them. That gives you an opportunity to add an rm -r line which makes the cleanup crystal clear.

And especially consider inheriting from unittest.TestCase in the standard way, so $ python -m unittest *.py, or $ pytest, will run the tests.

An automated test is "self evaluating", it "knows the right answer", often modeled with an assertEquals(a, b) statement. Your current test code looks for "correctness" in the sense of "we didn't crash!" There's an opportunity to probe the return value of smaller units like helpers, especially after you've broken out a couple of helper functions.

Answer 2

Just some quick notes about my first impressions. I haven't run the code.

Docstrings

I think you could reduce the amount of text in the docstrings. If you have already looked at some packages in Pypi or Github you may have noticed they are structured in a rather standard way.

For example there will be a separate LICENSE file. There will be also a README.md (or rst) file attached. The version can be written in a separate file, often it goes in __init__, but I often put it in a pyproject.toml file nowadays.

As for installation, usage etc, there can be separate files as well. You can use markdown or rst format to enhance those files with tables, links, hyperlinks, footnotes etc, so they will render better than plain text files when seen on web pages (eg Github).

Naming conventions

The use and abuse of underscore is somewhat unPythonic at places. This is a habit you'll have to break, because the underscore has semantic meaning in Python, and there are coding norms like PEP8, that programmers are expected to follow. A good IDE and some linter (eg Flake8) will help you normalize the code.

I am a bit puzzled at the way some functions/objects are named.

This:

_MAX_ = 100

should be more explicit. In your context it could be something like MAX_DIRS.

This function is poorly named:

def establish() -> None: 
    if not os.path.exists(TARGET_DIR_):
        os.makedirs(TARGET_DIR_)

The docstring says:

desc: create a directory if it doesn't exist.

Again, the name is too generic to be meaningful, and it does not accurately describe the purpose. Good code should be intuitive. If the functions and other object names are named sensibly, the code makes more sense to an outsider but it also helps the programmer to focus on logic, and not remembering what that darn function does again, especially when you haven't touched it for a long time.

For example you already have delete_dir, without looking at the code we can easily guess what it does.

On the other hand, function establish_current_dir does not have a telling name. If I had to guess what it does without looking at the code, then I would suppose it's the equivalent of the bash pwd command. Not quite.

Then you have a function named make_dir, which does too many things, but doesn't do the one thing we would expect. This is unfortunate. As I said already, good code should be intuitive but also predictable.

As the saying goes, each function should do just one thing but do it well. So it's perfectly appropriate to split the steps in several distinct functions, and call them in order. This is something you are already doing in your code, but you need to go further.

Configuration files

When you have so many hardcoded values it becomes obvious that a config file of some sort is preferable.

Instead of Constants.py it would make sense to use a YAML file, that you can read with the pyyaml library for example. It is suitable for hierarchical data like you've got here, and these files are easy to edit. Suggested reading: YAML: The Missing Battery in Python

I recommend to implement the same approach in your organize_files function. You are handling a long list of file types, which could grow vastly over time. YAML sounds appropriate to externalize that data. This will de-clutter your code and make it more manageable. At the very least you could have done like Constants.py. Also consider sorting the keys to avoid adding duplicates.

Don't reinvent the wheel

This function can be replaced by the pathlib module, thus:

def get_extension(file_name: str) -> str:
    return "." + file_name.split(".")[-1].lower()         

becomes:

from pathlib import Path  

Path(file_name).suffix

In general it's a good idea to use Google when you have to implement a function like this. Because there are many possible implementations but some are better than others. It's worth comparing the options, and - important - test the code in a variety of scenarios and edge cases.

Let's see how your code behaves with a file without extension: get_extension("test") returns: '.test'. Probably this is not what you want. Rather, you would expect an empty string right?

This is one thing you need to get right, because improper parsing of file parts could have adverse effects in your application.

Copy safety

I have a feeling this code could be dangerous in some circumstances. If you are moving a file to some location where a file with the same name already exists, will you overwrite the existing file without warning?