Timeline for Yellow Pages scraper
Current License: CC BY-SA 3.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 30, 2015 at 12:05 | vote | accept | Halcyon Abraham Ramirez | ||
| Jul 30, 2015 at 7:51 | history | edited | lvc | CC BY-SA 3.0 |
Clarifications based on comments
|
| Jul 30, 2015 at 6:09 | comment | added | lvc |
You will get an empty return from that method if you call it a second time on the same instance, because the accumulation list doesn't persist across calls but the loop variable does (and you only reset it in __init__). I ultimately recommend having neither of them persist, which fixes this problem. Your points about the infinite loop and the fields being single valued change my recommendations a little, and I will update my answer.
|
|
| Jul 30, 2015 at 2:51 | comment | added | Halcyon Abraham Ramirez |
I did prepare for malformed data hence the line if not email
|
|
| Jul 30, 2015 at 2:50 | comment | added | Halcyon Abraham Ramirez |
I get what you mean. but I explicitly tell it to stop at page two for the sake of bailing out the loop early if I wanted it i'll put it at while True so it'll scrape everything. but I just wanted one page for the tests sake. the outer list is remember it other wise if it weren't remembered I would have had an empty output... while I agree that the associations may break if there is more than one value in those fields but going through yellow pages there will always just be one value one address one name one number for the selected xpath that it's in. except for the email
|
|
| Jul 30, 2015 at 0:08 | comment | added | lvc | If you want to support multi-valued fields, your data structure should be set up to explicitly support that they are multivalued. If you only want to support single valued fields, your code should behave appropriately by storing only the zeroth value that is scraped. It is always good practice to be prepared for malformed data, and either clean it up if you can do that sensibly, or error out early. As written, your code will break silently and subtly. Pick a rule and stick to it. Storing records instead of lists of fields helps with that. | |
| Jul 30, 2015 at 0:00 | comment | added | lvc |
The outer list isn't remembered. The second time that function is called, self.page is already 2, so the while loop is skipped, and it returns an empty list. For associating fields with the right records, storing them in the same order works if there's the same number of every field for every person - but the rest of your code suggests that that doesn't have to be true. If one person has three phone numbers (say, home, work and mobile) and two email addresses, those associations will break.
|
|
| Jul 29, 2015 at 19:55 | comment | added | Halcyon Abraham Ramirez |
list isn't remembered but extends an empty list outside of the loop before the next iteration.so the values are passed and therefore the values are remembered. Completely fail to correlate the names and emails and numbers to appropriate people? wut? at first glance that might be. but the zip joined back all those values together because they were extracted on the same page and extended into a list in the same order so zipping them fixed that. As for everything else you said besides those points. I completely agree with you. whitespace, variable naming, and class use. and thank you
|
|
| Jul 29, 2015 at 7:07 | history | answered | lvc | CC BY-SA 3.0 |