Timeline for Getting to Wikipedia's "Philosophy" article using Python
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 21, 2015 at 11:09 | answer | added | SuperBiasedMan | timeline score: 1 | |
| Dec 21, 2015 at 9:56 | comment | added | oliverpool | @Sumit : so even if you drastically improve the Python code, you can't get much more than a 10% improvement without a dump | |
| Dec 21, 2015 at 9:28 | comment | added | Sumit |
@oliverpool: I just did a timing test using line_profiler (which basically tells you about the time each line in your code took) and apparently in both __init__() and trace(), fetching the contents of the GET request is what took the most time (over 90% in both cases). I changed the headers to include Accept-Encoding: gzip (MediaWiki supports gzip) but the improvement wasn't much (39.5474 to 37.3156 seconds with "Koseč" as the starting point).
|
|
| Dec 20, 2015 at 22:07 | comment | added | oliverpool |
it takes a long time to fetch the result of a MediaWiki API query, parse it as JSON, extract the parsed HTML text, remove text within parentheses and some tags (e.g. <i>), then get the first link do you have some timing measurement to support this? Do you know which part is the slowest? Which part can you actually improve?
|
|
| Dec 20, 2015 at 20:35 | comment | added | BenC |
In trace(), the expression next_page[:len('/wiki/')] != '/wiki/' is equivalent to not next_page.startswith('/wiki/').
|
|
| Dec 20, 2015 at 17:07 | history | edited | 200_success |
edited tags
|
|
| Dec 20, 2015 at 16:58 | answer | added | Caridorc | timeline score: 5 | |
| Dec 20, 2015 at 15:03 | history | edited | Sumit | CC BY-SA 3.0 |
make example code consistent with pydoc
|
| Dec 20, 2015 at 14:55 | history | edited | Sumit | CC BY-SA 3.0 |
added original module-level pydoc that I had removed initially, in case anyone pointed that out in an answer + correction in example code
|
| Dec 20, 2015 at 12:38 | review | First posts | |||
| Dec 20, 2015 at 12:58 | |||||
| Dec 20, 2015 at 12:37 | history | asked | Sumit | CC BY-SA 3.0 |