Timeline for Basic search engine
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 5, 2014 at 3:15 | comment | added | Blake Walsh | Oh yeah you're right it's been a long time since I've used BS reference. But the point remains that as the BS docs say Beautiful Soup will never be as fast as the parsers it sits on top of. However for me it's not about performance (though performance is nice), I just find lxml to be more pythonic. | |
| Oct 5, 2014 at 0:01 | comment | added | Ralph | Thanks, I found robotparser and put the updated version here: link I am confused about what you are saying about BS always using it's own parser, you can specify the lxml html parser by writing 'BeautifulSoup(markup, "lxml")' | |
| Oct 4, 2014 at 23:50 | comment | added | Blake Walsh |
@Ralph BS actually uses the best 'ElementTree' available, it always uses it's own regular-expression based parser. lxml parses and builds the element tree entirely at the speed of c which is why it's faster. With 1) You only use N threads (i.e. 4 or 8), and each thread handles multiple pages in series. You could use greenlets for a 'thread per page' approach. 2) Google is your friend, robotparser is part of the Python standard library.
|
|
| Oct 3, 2014 at 23:52 | comment | added | Ralph | I installed lxml, it parses significantly faster than the stock parser beautifulsoup uses with python 2.7, apparently beautifulsoup automatically uses the best available parser you have (without having to actually import lxml or even specifying parser when making a beautifulsoup command, just having it installed is enough). I have two questions for you though: 1.)if I use a new thread for every page I am processing, is there a limit to how many threads I use? 2.)I noticed your code, like mine, also doesn't mention robots.txt. Is there a library that interprets robots.txt? | |
| Oct 3, 2014 at 23:27 | history | answered | Blake Walsh | CC BY-SA 3.0 |