Skip to main content
added 87 characters in body
Source Link
Graipher
  • 41.1k
  • 7
  • 69
  • 133

An easy speed-up is not using a global object that you continually modify. This is especially important since you also save the output file over and over, which is completely unnecessary (unless you care about the current state if you abort the program).

from itertools import chain
from pathlib import Path
...

def opdata_collector(obj):
    if obj.name != 'START':
        return {}
    count = 0
    obj.recurs_drop(dropper)
    obj.recurs_change(rename_attribs)
    obj = rename_attribs(obj)
    final_dict = dict(obj.attribs)
    for child in obj.children:
        if str(child.name) in tags_to_leave:
            final_dict.update(dict(child.attribs))
        elif str(child.name) in value_data:
            final_dict.update({f"ValueData{key_name}{count}": value
                               for key_name, value in child.attribs.items()})
            count += 1
    return final_dict


if __name__ == '__main__':
    directory = Path('C:/Users/z647818/Desktop/Erdi/Files/test')
    dfdata = pd.concat((XMLParser.parse(directory / file_name).recurs_collect(opdata_collector)
                    for file_name in os.listdir(directory)
                    if file_name.endswith(".xml")),
                 df = ignore_index=Truepd.DataFrame(chain.from_iterable(data))
    df.dropna().to_csv("./output.csv")

On top of that the actual opdata_collector can probably also be improved, but this should give you a nice boost.

I also removed unneeded lines, used a dictionary comprehension in the inner loop, turned the if into an elif (you probably don't want a tag to be processed twice), spelled out count (no need to conserve bytes), used an f-string instead of string addition and packed the dataframe generation in one big generator expression.

An easy speed-up is not using a global object that you continually modify. This is especially important since you also save the output file over and over, which is completely unnecessary (unless you care about the current state if you abort the program).

def opdata_collector(obj):
    if obj.name != 'START':
        return {}
    count = 0
    obj.recurs_drop(dropper)
    obj.recurs_change(rename_attribs)
    obj = rename_attribs(obj)
    final_dict = dict(obj.attribs)
    for child in obj.children:
        if str(child.name) in tags_to_leave:
            final_dict.update(dict(child.attribs))
        elif str(child.name) in value_data:
            final_dict.update({f"ValueData{key_name}{count}": value
                               for key_name, value in child.attribs.items()})
            count += 1
    return final_dict


if __name__ == '__main__':
    directory = 'C:/Users/z647818/Desktop/Erdi/Files/test'
    df = pd.concat((XMLParser.parse(file_name).recurs_collect(opdata_collector)
                    for file_name in os.listdir(directory)
                    if file_name.endswith(".xml")),
                   ignore_index=True)
    df.to_csv("./output.csv")

On top of that the actual opdata_collector can probably also be improved, but this should give you a nice boost.

I also removed unneeded lines, used a dictionary comprehension in the inner loop, turned the if into an elif (you probably don't want a tag to be processed twice), spelled out count (no need to conserve bytes), used an f-string instead of string addition and packed the dataframe generation in one big generator expression.

An easy speed-up is not using a global object that you continually modify. This is especially important since you also save the output file over and over, which is completely unnecessary (unless you care about the current state if you abort the program).

from itertools import chain
from pathlib import Path
...

def opdata_collector(obj):
    if obj.name != 'START':
        return {}
    count = 0
    obj.recurs_drop(dropper)
    obj.recurs_change(rename_attribs)
    obj = rename_attribs(obj)
    final_dict = dict(obj.attribs)
    for child in obj.children:
        if str(child.name) in tags_to_leave:
            final_dict.update(dict(child.attribs))
        elif str(child.name) in value_data:
            final_dict.update({f"ValueData{key_name}{count}": value
                               for key_name, value in child.attribs.items()})
            count += 1
    return final_dict


if __name__ == '__main__':
    directory = Path('C:/Users/z647818/Desktop/Erdi/Files/test')
    data = (XMLParser.parse(directory / file_name).recurs_collect(opdata_collector)
            for file_name in os.listdir(directory)
            if file_name.endswith(".xml"))
    df = pd.DataFrame(chain.from_iterable(data))
    df.dropna().to_csv("./output.csv")

On top of that the actual opdata_collector can probably also be improved, but this should give you a nice boost.

I also removed unneeded lines, used a dictionary comprehension in the inner loop, turned the if into an elif (you probably don't want a tag to be processed twice), spelled out count (no need to conserve bytes), used an f-string instead of string addition and packed the dataframe generation in one big generator expression.

Source Link
Graipher
  • 41.1k
  • 7
  • 69
  • 133

An easy speed-up is not using a global object that you continually modify. This is especially important since you also save the output file over and over, which is completely unnecessary (unless you care about the current state if you abort the program).

def opdata_collector(obj):
    if obj.name != 'START':
        return {}
    count = 0
    obj.recurs_drop(dropper)
    obj.recurs_change(rename_attribs)
    obj = rename_attribs(obj)
    final_dict = dict(obj.attribs)
    for child in obj.children:
        if str(child.name) in tags_to_leave:
            final_dict.update(dict(child.attribs))
        elif str(child.name) in value_data:
            final_dict.update({f"ValueData{key_name}{count}": value
                               for key_name, value in child.attribs.items()})
            count += 1
    return final_dict


if __name__ == '__main__':
    directory = 'C:/Users/z647818/Desktop/Erdi/Files/test'
    df = pd.concat((XMLParser.parse(file_name).recurs_collect(opdata_collector)
                    for file_name in os.listdir(directory)
                    if file_name.endswith(".xml")),
                   ignore_index=True)
    df.to_csv("./output.csv")

On top of that the actual opdata_collector can probably also be improved, but this should give you a nice boost.

I also removed unneeded lines, used a dictionary comprehension in the inner loop, turned the if into an elif (you probably don't want a tag to be processed twice), spelled out count (no need to conserve bytes), used an f-string instead of string addition and packed the dataframe generation in one big generator expression.