6
\$\begingroup\$

Explanation: I am working on String comparison in which I want to cross-compare the list of the same input string. For the same string, I am continuing the loop but if a single string will have similarity greater than 90% than it should not be appended to the refined Input list. The code on which I am working is down there:

Source Code:

from DatabaseOperations import DatabaseOperations
import csv
from similarity.jarowinkler import JaroWinkler

def refineInputFileData(inputList):
    jarowinkler = JaroWinkler()
    refinedInputList = []
    refinedDict = {}
    for outer_keyword in inputList:
        if outer_keyword == "-":
            refinedInputList.append(outer_keyword)
            continue
        for inner_keyword in inputList:
            if outer_keyword == inner_keyword:
                continue

            similarity = jarowinkler.similarity(outer_keyword, inner_keyword)
            match_percentage = similarity * 100

            if (match_percentage >= 90):
                break
            else:
                refinedDict[outer_keyword] = ""
                refinedInputList.append(outer_keyword)
    return refinedInputList
if __name__ == '__main__':
    inputList = getInputDataList()
    print(inputList)
    refineList = refineInputFileData(inputList)

Data List:

inputList = ['2018 form 1040 schedule 1', '2018 schedule 1', '2018 form 1040 schedule 2', '2018 form 1040 schedule a', 'schedule 1 2018', '2018 form 1040 schedule 3', '2018 form 1040 schedule 1 instructions', '2018 form 1040 schedule 4', '2018 form 1040 schedule 5', '1040 schedule 1', '2018 schedule 2', '1040 line 11', '2018 1040 schedule 1', '2018 schedule 1 instructions', '2018 schedule 3', '2018 schedule 5', '2018 1040 schedule a', '1040 line 12', 'form 1040 line 11', 'schedule 1 tax form 2018', 'schedule 2 2018', 'schedule 5 2018', '1040 schedule a 2018', 'schedule 3 tax form 2018', 'schedule 1 line 22', 'form 1040 schedule 1', 'schedule 2 tax form 2018', 'schedule 3 2018', 'schedule 4 tax form 2018', '1040 form 2018 schedule 1', '1040 line 9', '1040 schedule 1 2018', '2018 form 1040 schedule 6', '2018 form 1040 schedules', '2018 1040 schedule 2', '1040 form 2018 schedule a', 'schedule 5 tax form 2018', 'new 1040 schedules', 'form 1040 schedule a 2018', 'schedule 1040 for 2018', 'schedule 1 instructions 2018', '2018 schedule 1040', 'new schedule a', 'irs schedule 1 2018', 'new form 1040 schedules', 'line 22 schedule 1', 'irs schedule 3', 'irs 2018 schedule 1', 'form 1040 schedules 2018', 'new tax schedules 1 6', '2018 schedule a form 1040', '2018 federal tax schedule 1', 'form 1040 schedule 3', '2018 tax form schedule 3', 'what is schedule 1 tax form', 'irs schedule e instructions', '2018 tax return schedule', '1040 schedules 2018', '1040 schedule e instructions', 'schedule a tax form 2018', 'form 1040 schedule e pdf', 'what is schedule 4 other taxes', 'schedule 3 of income tax act', '2018 schedule se', '2018 irs schedule 1', '2018 form 1040 schedule e', 'how to fill out 1040 form 2018', 'schedule e instructions 2018', '2018 schedule a tax form', 'what will the 2018 1040 look like', '2018 form 1040 draft', 'new irs schedules', 'irs form schedule e form 1040', 'irs schedule se instructions', 'tax schedule e 2018', 'what is schedule e form 1040', '2018 tax year schedule a', 'proposed new tax form', 'irs 2018 forms schedule 1', '2018 tax forms schedule a', 'irs schedule one', 'new irs schedule 1', '1040 schedule e 2018', 'tax season 2018 schedule', 'irs form schedule a 2018', 'new tax schedules for 2018', 'schedule se instructions 2018', 'what is schedule e in tax return', 'irs 1040 line 21', 'schedule se 2018 instructions', 'irs gov schedule se', '2018 schedule e instructions', 'irs 2018 schedule a', 'form 1040 schedule e line 16 instructions', 'schedules 1 6 irs', 'irs schedule 3 for 2018', 'schedule e tax return', '2018 draft 1040', '2017 form 1040 schedule e', '2018 instructions schedule c', 'Keyword', '2018 irs schedule 1 form', 'irs schedule e 2018', 'schedule 3 income tax act', 'form 1040 line 12b', 'schedule a form 2018', 'irs 1040 schedule 2', '2018 schedule 4 instructions', '2018 tax forms schedule 1', '2018 schedule a form', 'form 1040 schedule e instructions', 'irs gov schedule c instructions', 'schedule one tax form', 'irs form 1040 schedule e for 2018', 'final 1040 form 2018', 'irs schedule 5 for 2018', 'tax schedule a 2018']
\$\endgroup\$
2
\$\begingroup\$

Specific suggestions:

  1. main doesn't do anything with refineList.
  2. Rather than multiply by 100 a bunch of times you can simply check similarity >= 0.9
  3. The threshold should be configurable. You have many options for that, including at least a mandatory or optional parameter (using argparse) or a configuration file (using configparser).
  4. In the same vein the list of inputs could be taken from lines (or NUL-separated strings) in standard input.
  5. Why is there special casing for outer_keyword == "-"?
  6. refinedDict is superfluous - it's set but never read.
  7. Since match_percentage >= 90 is the last check in that loop you can get rid of the else clause:

    if similarity < 0.9:
        refined_input_list.append(outer_keyword)
    

General suggestions:

  1. black can automatically format your code to be more idiomatic.
  2. isort can group and sort your imports automatically.
  3. flake8 with a strict complexity limit will give you more hints to write idiomatic Python:

    [flake8]
    max-complexity = 4
    ignore = W503,E203
    
  4. I would then recommend adding type hints everywhere and validating them using a strict mypy configuration:

    [mypy]
    check_untyped_defs = true
    disallow_untyped_defs = true
    ignore_missing_imports = true
    no_implicit_optional = true
    warn_redundant_casts = true
    warn_return_any = true
    warn_unused_ignores = true
    
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.