Questions tagged [lxml]
lxml is a full-featured, high performance Python library for processing XML and HTML.
22
questions
1
vote
1
answer
67
views
Counting Ads in webpage using XPath and EasyList in Python
I have the following the function the retrieves a given webpage and returns the number of adverts that are on the page using a shortened version of EasyList (17,000 rules). Using multiprocessing, this ...
- 167
4
votes
1
answer
2k
views
Python lxml transform xml to dictionary
I need to download a fairly large xml (up to 2 GB) then load it into memory or possibly an empty db on a stand alone machine on a semi-regular basis. I need to grab a large amount of very specific ...
- 229
2
votes
1
answer
120
views
How to divide this Python code into different functions?
I'm making a code that, starting from an xml file:
stores the index of child elements of a tag and the child elements as key, values in a dictionary;
deletes keys whose values contain a certain ...
- 169
1
vote
1
answer
138
views
Parsing xml files, ideas to design my code
I begun to work for a company which do not have enough time to practice code review. As a beginner programmer I would like to improve my skills on real working cases.
I have been asked to facilitate ...
- 103
3
votes
0
answers
66
views
OOP parsing for DB dump XML Python
I am parsing large XML (DB dump). So, the content looks like many different objects with their properties and dependencies (foreign keys). My way to do it - using lxml.etree (iterparse):
...
- 131
4
votes
1
answer
2k
views
Nested for-loops in xml parsing
I wrote program that reads and processes xml files submitted by external users. The part of the code in question iterates over xml node's children, appends a dictionary - a future row in a table, and ...
- 143
2
votes
1
answer
57
views
Flattening XML and selecting nodes based on input to convert to CSV
I have some XML that I want to flatten based on its XPath. The task is to acquire certain nodes that are passed as input. That said, I have to get to parent nodes and look for nodes that might be part ...
- 121
1
vote
1
answer
118
views
Python xml-parsing script
I have a Python code that deals with parsing xml-feeds of footwear online stores. I want to let the code look more professional for the sake of optimal reusability for other programmers.
It includes ...
- 55
3
votes
0
answers
47
views
Applying a transformation to all the text nodes inside a given node using lxml.html?
I would like to apply a transformation, say uppercase, to all the text nodes inside a given node. I came up with the following code which works fine for my sample ...
- 307
8
votes
1
answer
4k
views
Python CSV to XML converter
I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
- 83
2
votes
1
answer
2k
views
Parsing scraped data from html table
I've written a simple python web scraper that parses text from an html table and stores the scraped data in List of dictionaries. The code works and doesn't seem to have any glaring issues performance-...
- 459
2
votes
1
answer
370
views
Parsing XML documents with LXML
I'm new to LXML, and parsing XML documents in general. I have written a small section of code that appears to do what I need it too do, but it feels like I have overly complicated it. Is there any way ...
- 123
4
votes
1
answer
4k
views
Finding data on XML using Python's LXML
Using Python's LXML I must read an XML file and print from each "basic" and "expert" tag, the name and email text from it. I've done a script that works but I don't think is the best way of doing this....
- 141
4
votes
1
answer
270
views
Richelieu - product scraper (follow-up)
Intro
This is a follow-up question to my earlier post. Although @alecxe had very nice remarks in hist last answer, I decided to get all the products a bit different.
Instead of searching for a space (...
- 331
5
votes
1
answer
213
views
Richelieu - product scraper
I wanted to see how would I deal with a large amount of data being scraped and written into a CSV file, so I decided to get the info out of a random website.
First off, I found a way to search for ...
- 331
7
votes
2
answers
793
views
E-mail crawler for yellowpages
I've written a script in python to scrape e-mail addresses from different pizza shops located in los-angeles available in yellowpage traversing multiple pages. It is able to go one-layer deep and dig ...
- 2,471
2
votes
1
answer
290
views
URL List generator
The below code will read a downloaded HTML file. Uses BeautifulSoup and lxml to parse the document and extract all the URLs ...
5
votes
1
answer
132
views
Scraping required fields exhausting a full website
I've written some script in python using lxml library which is capable of dealing with a webpage with complicatedly layed-out next page links. My scraper is able to parse all the next page links ...
- 2,471
2
votes
1
answer
196
views
Making two classes work ambidextrously
I've written some script in python following OOP guidelines. I've created two classes in my script in such a way so that both of them are able to make requests and produce flawless results. It can ...
- 2,471
6
votes
1
answer
2k
views
LXML parsing for an HTML file
I wrote this code to parse an HTML file which is basically a report generated by a software. This file is 40+ MB in size. The code is 100% perfect. The only problem that I am facing is the parsing ...
- 63
3
votes
1
answer
464
views
Parsing a full site using python
I've created a crawler which is scraping name, phone number and web address of each profile from houzz website. Hope I did it the right way. Here is what I've written:
...
- 2,471
6
votes
2
answers
1k
views
Extracting a div from parsed HTML
It seems that lxml/etree are generally imported as from lxml import etree -- why is that? It keeps the code tidier, and while the potential namespace ambiguity ...
- 161