my program requires a hash to be generated for files and folders for data integrity between transfers and file system events (if hash is different, file is not ready). For data integrity, I wish to use SHA-256 and for checking if a file is ready for transfer, I am using a custom hash that uses small parts of the start, middle and end and the size of the file. Please let me know if you need implementations, but I'm hoping that the interfaces are so self-explanatory that you don't need it.
I am somewhat new to dependency injection, so I wanted to ask you guys for opinions on any improvements, code smells, etc.
With these requirements, I have the following interfaces created:
The file system interface
class IFileSystem(ABC):
@abstractmethod
def open(
self,
path: str,
mode: str = "r",
buffering: int = -1,
encoding: str | None = "utf-8",
errors: str | None = None,
newline: str = "",
) -> IO:
"""Opens a file at a given path on the file system
Args:
path: A path to a file on the filesystem.
mode: The mode of opening, identical to the default python `open` method. Defaults to "r"
buffering: Buffering policy (-1 to use default buffering, 0 to disable buffering, 1 to select line
buffering, of any positive integer to indicate a buffer size).. Defaults to -1.
encoding: Encoding for text files. Defaults to utf-8.
errors: What to do with unicode. Defaults to None.
newline: The newline format of the file. Defaults to "".
Returns:
IO: A working IO object for the file on the filesystem
"""
@abstractmethod
def list_dir(self, path: str) -> Iterable[FileSystemResource]:
"""Gets a list of all the resources in a directory on the file system. May be a file, directory
Args:
path: The path to the directory to list.
Returns:
List[FileSystemResourcer]: The file system resources in the directory
"""
@abstractmethod
def stat(self, path: str) -> FileSystemResource:
"""Retrieves the stat of a single resource at the given path
Args:
path: The path to the resource
Returns:
FileSystemResource: The info of the resource
"""
@abstractmethod
def copy(self, src: str, dest: str):
pass
@abstractmethod
def remove(self, path: str):
pass
@abstractmethod
def copy_dir(self, src: str, dest: str):
pass
@abstractmethod
def remove_dir(self, path: str):
pass
@abstractmethod
def walk(
self, base_path: str
) -> Iterable[tuple[str, list[FileSystemResource], list[FileSystemResource]]]:
"""Walks the filesystem from the given path
Args:
path: The path to the first location to walk
Yields:
Iteratable[tuple[str, list[str], list[str]]]: Identical to python walk functionality
"""
class FileSystemResourceType(Enum):
UNKNOWN = auto()
DIRECTORY = auto()
FILE = auto()
CHARACTER = auto()
BLOCK_SPECIAL_FILE = auto()
FIFO = auto()
SOCKET = auto()
SYMLINK = auto()
@dataclass
class FileSystemResource:
path: str
name: str
is_dir: bool
size: int
type: FileSystemResourceType
accessed: Optional[datetime]
created: Optional[datetime]
metadata_changed: Optional[datetime]
modified: Optional[datetime]
and the hashing interface
class IPathHasher(ABC):
@abstractmethod
def partial_hash_directory(self, path: str, fs: IFileSystem) -> str:
"""Hashes a directory starting from the given path. This hash will not guarantee content is complete, as it
should sacrifice completeness for performance. Should be a reliable but not absolute indicator that a file has
been completely copied
Args:
path: The path to the directory
Returns:
str: The hash
"""
@abstractmethod
def partial_hash(self, path: str, fs: IFileSystem) -> str:
"""Hashes a file object that is opened in binary mode. To open in binary mode, use `open(..., mode="rb")`.
This hash is not suitable for ensuring content is complete, as it only uses a small amount of a file to hash it
for performance reasons
Args:
file: The file object. Usually made with open or BytesIO()
Returns:
str: The hash of the file
"""
@abstractmethod
def complete_hash(self, path: str, fs: IFileSystem) -> str:
"""Hashes a file such that the entire content of the file is used to ensure that the entirety of 2 files are identical
Args:
file: The file to hash
Returns:
str: The hash
"""
@abstractmethod
def complete_hash_directory(self, path: str, fs: IFileSystem) -> str:
"""Hashes a directory using the complete hash algorithm. The hash should be completely unique to the directory.
Collisions may lead to lost data.
Args:
path: The path to the directory
Returns:
str: The hash.
"""
Some potential changes I see that could be good but I am unsure about;
- Use FileSystemResource objects instead of string paths everywhere except
list_dir(...),stat(...)and thedestinputs tocopyandcopy_dirin the IFileSystem (or essentially anything that generates a FileSystemResource) - Same as above but for the
IPathHasher - Separate the
IPathHasherinto anIPartialResourceHasherandICompleteResourceHasherand change the interface tohashandhash_dir, then inject as needed. I am not sure how this works with IOC containers. Its simple enough in pure DI, but I honestly can't see how this works with a container. I'm only 1/4 way through a DI book and havent reached containers yet :p
I'm really keen for your responses because I have put a lot of thought into this program and I believe I finally have a fully fleshed out question to ask you guys that will give me a lot of insight for my own learning. Thanks.
