An easy speed-up is not using a global object that you continually modify. This is especially important since you also save the output file over and over, which is completely unnecessary (unless you care about the current state if you abort the program).
from itertools import chain
from pathlib import Path
...
def opdata_collector(obj):
if obj.name != 'START':
return {}
count = 0
obj.recurs_drop(dropper)
obj.recurs_change(rename_attribs)
obj = rename_attribs(obj)
final_dict = dict(obj.attribs)
for child in obj.children:
if str(child.name) in tags_to_leave:
final_dict.update(dict(child.attribs))
elif str(child.name) in value_data:
final_dict.update({f"ValueData{key_name}{count}": value
for key_name, value in child.attribs.items()})
count += 1
return final_dict
if __name__ == '__main__':
directory = Path('C:/Users/z647818/Desktop/Erdi/Files/test')
data = (XMLParser.parse(directory / file_name).recurs_collect(opdata_collector)
for file_name in os.listdir(directory)
if file_name.endswith(".xml"))
df = pd.DataFrame(chain.from_iterable(data))
df.dropna().to_csv("./output.csv")
On top of that the actual opdata_collector
can probably also be improved, but this should give you a nice boost.
I also removed unneeded lines, used a dictionary comprehension in the inner loop, turned the if
into an elif
(you probably don't want a tag to be processed twice), spelled out count
(no need to conserve bytes), used an f-string
instead of string addition and packed the dataframe generation in one big generator expression.