2
\$\begingroup\$

I have a program which parse HTML content to get a script src attribute and script attribute. A function have directory (contain HTML files) as input and extract to other directory as output.

However, I think my code is not clean. The following is my code.

def extract_int_js(src, dst):

    for file in os.listdir(src):
        filepath = os.path.join(src, file)
        with open(filepath) as f:
            html = f.read()
        parser = HtmlParser(html)

        for text in parser.script_text:
            yield text


def extract_src_js(src, dst):
    for file in os.listdir(src):
        filepath = os.path.join(src, file)
        with open(filepath) as f:
            html = f.read()
        parser = HtmlParser(html)

        for src in parser.src:
            yield src

Do you have any solution to improve this code since there are duplicated line in 2 functions.

\$\endgroup\$
5
\$\begingroup\$

You can use getattr function to merge your two functions into a single one.

def extract_attribute(src, attr_name):
    for file in os.listdir(src):
        filepath = os.path.join(src, file)
    with open(filepath) as f:
        html = f.read()
    parser = HtmlParser(html)

    yield from getattr(parser, attr_name)

Note that in Python 3.3+ there is yield from statement so you can yield values directly from iterable without writing for loop

Now if you still want to have two separate functions instead of passing attribute_name argument each time you can use function partial from functools

from functools import partial

extract_src_js = partial(extract_attribute, attr_name='src')
extract_int_js = partial(extract_attribute, attr_name='script_text')

Please also note that I've removed dst parameter from your functions since it was not used, however if it's just a part of your function you wanted us to review, just return it back.

\$\endgroup\$
  • \$\begingroup\$ Thank you for your clean code :D. However, can I use decorator in this case ? \$\endgroup\$ – fluke-ng Apr 13 '17 at 13:09
  • \$\begingroup\$ @No-Encrypt yes you can, basically, all the common part could be done within a wrapper part of the decorator. In this particular case feel like decorator will add some inexplicit logics, which you usually want to avoid. But that is my own opinion, you can go for decorator if you like it more. \$\endgroup\$ – Alex Apr 13 '17 at 13:21
3
\$\begingroup\$

If you always need both attributes and not one individually, and both have the same length, you could just iterate over both at the same time:

def extract_int_js(src, dst):

    for file in os.listdir(src):
        filepath = os.path.join(src, file)
        with open(filepath) as f:
            html = f.read()
        parser = HtmlParser(html)

        for src, text in zip(parser.script_text, parser.src):
            yield src, text

You can use yield from here as well and factor out the reading of the files to another generator:

def read_files(src, dst):
    for file in os.listdir(src):
        filepath = os.path.join(src, file)
        with open(filepath) as f:
            yield f.read()


def extract_int_js(src, dst):
    for html in read_files(src, dst):
        parser = HtmlParser(html)
        yield from zip(parser.script_text, parser.src)

These can be used like this:

for src, text in extract_int_js(src, dst):
    print(src, text)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.