1
\$\begingroup\$
import pandas as pd
import pandas_read_xml as pdx
from lxml import etree
url2='http:'
url='http:'
tree=etree.parse(url)
st=etree.tostring(tree.getroot())

# The list "all_leaves" includes every node and leaf in the tree

from io import StringIO, BytesIO
#st above is encoded as binary

all_leaves=[]
events = ("start", "end")
# "start" are the xml tags start
# "end" are the xml tags with "</"
parent='root' #root of the xml tree
context = etree.iterparse(BytesIO(st), events=events)
for n, actel in enumerate(context):
  if actel[0]=='end': #this marks the end of a node
    parent = all_leaves[n-1][1] #use previous parent
    all_leaves.append([actel[0],parent,actel[1].tag])
    #append the action, parent and tag
  else:
    all_leaves.append([actel[0],parent,actel[1].tag])
    if actel[0]=='start': #the current become parent since this
                          #algorithm is depth first
      parent = actel[1].tag

# "leaves" includes only the first row of each table
leaves=[]
for entry in all_leaves:
  if (entry not in leaves) and (entry[0]=='start'):
    leaves.append(entry)


# These are the names of the tables
nf_nodes=[]
for entry in leaves[2:]:#ignore the first two:
  nf_nodes.append(entry[1])
nf_nodes=list(set(nf_nodes))


# The dictionary "all_dic" has keys labeled as the node
# above each set of leaves (i.e. DataFrames) with value equal to the corresponding
# DataFrame or leav
all_dic={}
for l in nf_nodes:
    all_dic[l]=pdx.read_xml(url, ['DRFAssessments',l])

I have the following code below that reads and parses xml files. The following code takes about 10:30 seconds to run. I wanted to see if there were any suggestions or any edits that could be made to the code to increase the speed. The code is used to parse the xml file before it is uploaded to a MYSQL database. It also parses 20 tables if that is any useful

\$\endgroup\$
1
  • \$\begingroup\$ Welcome to Code Review! It would be helpful to know the context of the question, since a lot of posts are about parsing XML and increasing its speed. What specifically are you parsing and could you edit the title to match that? Furthermore, it would likely also help to have (a shortened) sample input file so that readers can actually try the code. Hope you get some good answers! \$\endgroup\$
    – ferada
    Mar 2 '21 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.