Note: this code will be run using portable WinPython 3.7.6.0 (Python 3.7.6, package list here) on Win10x64. Installing new packages or upgrading the Python version are not an option for this project.
Disclosure: this code is heavily inspired by Janos' excellent answer here.
I've written a piece of crap that's going to grow a lot over time, so I thought I'd get the base reviewed instead of posting everything at once. After all, at this stage I can still easily scrap half of it without problems. Current version has already been a great help.
Situation
At work, we keep a lot of datasheets from the parts we use. The library of parts ever grows and the datasheets are updated by the manufacturer without notice. For some of our parts we're obliged to keep the documentation in order and to prevent accidentally missing relevant docs we keep the datasheets of everything even remotely electrical (including push-buttons, LEDs, etc.). Oh, and in 3 languages per part. Which means I'm talking about a couple thousand documents for some of our suppliers.
Directory structure looks like this:
|> current lib
|> supplier1
part1_de.pdf
part1_en.pdf
part1_nl.pdf
part2_de.pdf
part2_en.pdf
part2_nl.pdf
|> supplier2
Etc.
So if we have a 6ES7132-6BF01-0BA0 from Siemens, we have:
|> current lib
|> Siemens
6ES7132-6BF01-0BA0_de.pdf
6ES7132-6BF01-0BA0_en.pdf
6ES7132-6BF01-0BA0_nl.pdf
There are a couple of things that can go wrong with this:
- Missing files (only
_de
available instead of all 3). - Files not signed (
6ES71326BF010BA0_de.pdf
instead of6ES7132-6BF01-0BA0_de.pdf
). - Invalid language-separator signs (older files have the language separated by
-
instead of_
). - Old language (older Dutch files are named
_ne
instead of_nl
). - Incorrect language (we're not interested in
_it
). - Incorrect/missing extension (all files should be
.pdf
). - Outdated documents (the older it gets, the more likely the data has been adjusted).
This comes pretty exact, since the files are externally referenced on an unfortunately hard-coded list. Back-ups are taken care of, so the directory only has to contain recent files without worrying about the archive.
I'm still working on the first 2 points of that list (I can check against a CSV containing all the items) but that's for next version. Those are out-of-scope for this question but perhaps something to keep in mind. The rest is covered in this one by explicitly white-listing the last couple of characters.
Approach
Set a couple of globals, create a white-list, iterate over the files and classify each file as either 'Correct'
, 'Invalid'
or 'OutOfDate'
(the latter based on time modified (mtime, creation date is unreliable). Return both the results and the total count of checked files. Print in a readable format.
Currently both result
and count
are used for manual reference, later the result
will be passed to the next function. One supplier will be checked at a time. Current code considers files older than 90 days out-of-date. Actual list of suppliers is 20+ entries. I'm currently not automatically iterating over all suppliers because for some different standards apply. Those are exempt (come to think of it, perhaps I should put those on a blacklist and iterate over the list but that still seems risky). Current code works as expected for this revision.
Code
import os
import pprint
import time
MAX_AGE = 90
# 60 seconds in a minute, 60 minutes in an hour, 24 hours in a day
MINUTES = 60
HOURS = 60*MINUTES
DAYS = 24*HOURS
PATH_TO_FILES = "path/to/local/copy/of/lib/"
EXT = ".pdf"
WHITELIST = ['_nl' + EXT, '_en' + EXT, '_de' + EXT]
SUPPLIERS = ['Siemens', 'IFM']
def classify_files(basedir, limit):
"""
Classify files in *basedir* not modified within *limit*
:param basedir: directory to search
:param limit: timelimit for treshold
Return result (dict) and count (int).
"""
report = {
'Correct' : [],
'Invalid' : [],
'OutOfDate': []
}
mod_time_limit = time.time() - limit
count = 0
for filename in os.listdir(basedir):
path = os.path.join(basedir, filename)
count += 1
if not filename[-7:] in WHITELIST:
report['Invalid'].append(filename)
elif os.path.getmtime(path) < mod_time_limit:
report['OutOfDate'].append(filename)
else:
report['Correct'].append(filename)
return report, count
def main():
for supplier in SUPPLIERS:
print("\n{}:".format(supplier))
result, count = classify_files(
PATH_TO_FILES + supplier,
MAX_AGE * DAYS
)
pprint.pprint(result)
print("{} files checked".format(count))
if __name__ == '__main__':
main()
Issues
One of the most glaring issues is the hardcoded 7
here:
if not filename[-7:] in WHITELIST:
I'm not a fan of entire if/else
structure. Feels off. And I should probably use a different method of returning the result. How I handle times feels hacky, but it's a lot more straight-forward than converting and messing-around with datetime
. I should probably be using Correct
/Incorrect
or Valid
/Invalid
instead of the current Correct
/Invalid
combination, naming is hard. The count looks ugly (perhaps summing the length of the resulting lists would be better), and I'm not sure what validations and exception-handlers would make sense here.
I know the keys of report
are probably not according to PEP8 and the extra white-space definitely isn't, but it looks better this way.
Speed is currently not a concern. Naming, maintainability, readability and extendability are priority. Currently configuration by globals is fine, when it's clear what the final thing is going to look like they will be replaced by either config-files or arguments.
Complicated one-liners are sub-optimal. My Python might be good enough to understand it, but it would be preferable if colleagues less proficient can follow it as well.