53

Is there a way to simplify this code? Yes. Don't scrape Wikipedia. Your first thought before "should I need to scrape this thing?" should be "Is there an API that can give me the data I want?" In this case, there super is. There are many informative links such as this StackOverflow question, but in the end reading the API documentation really is the right ...


44

Let me tell you about IMPORTHTML()... So here's all the code you need in Google Sheets: =IMPORTHTML("https://en.wikipedia.org/wiki/Transistor_count", "table", 2) The import seems to work fine: And it's possible to download the table as CSV: Processor,Transistor count,Date of introduction,Designer,MOS process,Area "MP944 (20-bit, *6-chip*)",,1970[14] (...


40

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a GET request to https://www.googleapis.com/youtube/v3/videos?key=…&part=snippet&chart=mostpopular, you will get the same information in a documented JSON format. Using the Python client, the code looks like: ...


28

That's a beautiful excuse for writing code, and the final product is quite nice as well. ★ First of all, congrats to your Java-ridden mind for not forcing classes into Python where they aren't needed. ★ You import but do not use OrderedDict and operator. Remove unused imports. ★ It is customary to write your code in a way that allows it to be used as ...


26

import*: Please avoid importing all modules at all costs. Import each module you use separately Regex ಠ_ಠ Regex is evil, the worst. seriously. stop it. right now. kill python.exe right now and change it. You use beautifulsoup, a dom parsing library, literally a few lines later, but you choose to use regex. You sir, are evil. I would suggest looking ...


22

You do not need HTML parsing at all. OEIS has a nice JSON output format. https://oeis.org/search?fmt=json&q=id:A000045 So the core functionality of your program can be written as something like import sys import urllib2 import json f = urllib2.urlopen("https://oeis.org/search?fmt=json&q=id:%s" % sys.argv[1]) doc = json.loads(f.read()) comment = ...


18

Here a few suggestions (in no particular order) for improvements on your code: Make variables have locality The line: items = 1 is at the top of the method, but is only used by the loop at the bottom of the method. It is likely better located just in front of the loop that increments it. Same applies to: tickerdict = {} You can break strings across ...


16

In addition to what @StephenRauch wrote in his excellent answer, let me add one more comment: Your class is completely useless. Not its functionality, which is fine, but it being a class is completely unnecessary. It does not inherit anything from a parent or has children which inherit from it, it has no attributes and therefore no state, and the one ...


15

Context manager You open a file at the beginning of the program and close it explicitly at the end. Python provides a nice way to allocate and release resources (such as files) easily: they are called Context managers. They give you the guarantee that the cleanup is performed at the end even in case of exception. In your case, you could write: with open('...


15

Make sure to follow naming conventions. You name two variables inappropriately: My_table = soup.find('table',{'class':'wikitable sortable'}) My_second_table = My_table.find_next_sibling('table') Those are just normal variables, not class names, so they should be lower-case: my_table = soup.find('table',{'class':'wikitable sortable'}) my_second_table = ...


12

Let's start with the obvious: this code doesn't run. You're missing ans = starter() so that further (el)if ans.lower() == ... doesn't miserably fail with a NameError. Likely, you define q() but never use it. And you also appears to have other useless stuff floating around: why use both textblob and indicoio to perform sentiment analysis? You also seem to ...


12

This seems like a cool project, and a nice first introduction to web scraping! I'll cover some general advice first, and then address your main concern: speed. General You use two spaces for indentation. PEP-8 recommends 4 spaces (most code editors have the option to automatically convert tabs to spaces, which I highly recommend). While on the subject, PEP-...


11

If you're after speed, I'd suggest scrapy. I was looking for an excuse to try it out and saw your question. When I ran your code on the first 10 pages of the NA leaderboard, it took a little over 4 seconds. Running the below takes about 0.3 seconds, presumably due to initiating all the HTTP requests in parallel: test.py: class LolSpider(scrapy.Spider): ...


11

A couple of suggestions: Use requests instead of urllib. This is mostly a matter of preference but I think you'll find it easier to work with. An example on how this would look is: import requests page = requests.get('http://rabota.ua/zapros/python/%D0%BA%D0%B8%D0%B5%D0%B2').content Fix the bug In some cases you'll get: Traceback (most recent call ...


11

First of all, usual things (I feel like I'm suggesting these things in most of the web-scraping related discussions): initialize session as requests.Session() and use session.get() instead of requests.get() - this would speed things up because the underlying TCP connection will be re-used for subsequent queries to the same domain use SoupStrainer to limit ...


10

Very nice code in general! Here are a few points, though: Think about using os.makedirs instead of os.mkdir. The former allows creating nested directories (so os.makedirs("~/long/new/path") will create all other folders in betweetn as well, if needed). You should never have a bare except. This prevents you to e.g. CTRL-C to abort the program, since it will ...


10

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing: you could get some speed and memory improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired ...


9

The second option is better, because the possible error is explicit. However, in lots of case in Python, you should follow EAFP and go for the try statement. However, we can do better. get(value, default) In BeautifulSoup, attributes behave like dictionaries. This means you can write img_tag.get('class', '') to get the class if it exists, or the empty ...


9

I'm just going to review these eight lines of code. You'll see that there's plenty here for one answer. try: from gtts import gTTS except: answer = input("Your system does not have Google's Text to Speech API installed. Do you want to install it?") if 'y' or 'Y' in answer: os.system('python -m pip install --upgrade gTTS') else: ...


9

At first glance, I'd say that this code isn't bad. I am particularly impressed by this line: regex = r"\b"+ re.escape(word) + r"\b" because you took the care to escape the word, and to ensure that it is starts and ends at word boundaries. (You might want to do case-insensitive searches, though? Also, you might want to treat all whitespace as equivalent, ...


9

I have no familiarity with any of the libraries used here, so I can't comment on their usage. What I will mention though is the giant chunk of profile.set_preference calls in the middle of the script. It would be much cleaner and less repetitive to save the string/bool pairs of options as a dictionary (or another "paired" structure), then just iterate over ...


8

A few things I can think of are: parse_text can be marked as a private method. def _parse_text(self):. I understand this won't actually be hidden, but its a convention followed in python to treat methods starting with an underscore as private. In MovieScript __init__ method maybe have the url as a kwarg (defaulting to http://www.springfieldspringfield.co.uk,...


8

I can't see any reason why you would need to pop elements off the arrays returned from .select() -- you could just do something like message_soup.select('.times')[0].string Either approach will throw an exception if the message doesn't contain a .times class, so you could add some exception handling: try: stars = message_soup.select('.times')[0]....


8

As it happens, there are positive answers to each of your questions: with Beautiful Soup you can remove the WebRupee span with replace_with() entirely.... webrupee_element = soup.find('span', {'class': 'WebRupee'}) webrupee_element.replace_with('') ... then, when you get the text value of the product_price_tag_element.text it will not have the symbol. ...


8

Nice script! A few points on top of what @Graipher already pointed out. Passing url parameters to requests This is a hacky way to pass query parameters: url = "http://vimcolors.com/?page=" # ... page_source = requests.get(url + str(page + 1)) The proper way to do this is to pass the parameters in a dictionary: url = "http://vimcolors.com" # ... ...


8

Notes about your code You can also simplify the way you look for the links to follow: for link in soup.select("a[href]"): downloadImages(link["href"], level - 1) Here, we are enforcing the a elements to have href values, not checking for elements to be found (since the loop body would just not be executed in this case). As far as managing the urls you'...


8

I would use requests.Session, which allows re-using the connection. This speeds up successive requests (as you are doing here). I would factor out the getting and souping into a function, to be more DRY. This uses the Python 3.x exclusive yield from to consume its argument as an iterator: def get(session, url, selector): res = session.get(url) soup =...


8

You should also split the code in functional parts, and not mix the IO (print(...)) with generating the result base_url First you generate the base_url def make_base_url(request_type): baseurl='https://finviz.com/screener.ashx?v=111' if request_type == 'sp500': firstpage=baseurl+'&f=idx_sp500&o=ticker' elif request_type == "...


7

Since your list SITES_INFO_HELPER remains constant throughout the entire program, that list should be a tuple instead Tuples are faster than lists and are write-protected, which doesn't make a difference to you because you are writing to them. Read more about when you should use a tuple and when you should use a list here. Good job using ArgumentParser in ...


7

Use modern versions of things The most obvious thing here is use Python 3. This will help massively with your Unicode problems, because Python 3 maintains a stricter separation between things that Python 2 conflated. In some cases, your errors will just be artifacts of Python 2's way of doing things, and will just go away. In others, you'll get errors that ...


Only top voted, non community-wiki answers of a minimum length are eligible