23

Cargo Fmt There's a very common tool accessible through cargo which can format all of the code in your project according to Rust's official style guide. Many major open source Rust libraries use this tool (and even enforce it through CI on pull requests), which you can access through cargo fmt. You can also customize its output using a .rustfmt config file. ...


13

I tried using __ once and found it to produce more problems than it's worth. Take: def __foo(): return 'foo' class Bar: def __init__(self): self.__baz = __foo() Bar() NameError: name '_Bar__foo' is not defined I'd suggest you follow PEP 8 and only use it for classes that will be subclassed and where you need to prevent name collisions. ...


9

Check for failure metals = requests.get('https://www.lme.com/').text should be response = requests.get('https://www.lme.com/') response.raise_for_status() metals = response.text String literals don't need parens date = datetime.datetime.now() time = (f'{date.day}/{date.month}/{date.year} | {date.hour}:{date.minute}') would only need parens if it spans ...


7

Bugs Lines 32 and 42: for titles, info in zip(self, titles, self.comicinfo): for dol_amount in comic_prices: both have unresolved variable references - titles and comic_prices. Bare class declaration class ComicScraper(): # Class ComicScraper for scraping comic books can be class ComicScraper: """ For scraping comic books ...


7

Disclaimer: Don't know Python nor PEP 8. In __download_robots(self) you could reverse the if condition and return early which saves one level of indentation. if self.__url is None: raise InvalidUrlException("\n\nInvalidUrlException: Invalid Url.\n") ..the remaining code this applies to get_allowed(self), get_disallowed(self) and get_all(self) ...


7

Optional I find it less useful to include None in an enum like Evaluation, and more useful to write Optional[Evaluation] where appropriate. It's useful to be able to choose whether you have a value that cannot be None at a certain point, or otherwise, based on context. In other words, this: sentiment: Literal[ Evaluation.POSITIVE, Evaluation....


7

Some style issues first: Package names should be all lowercase ASCII letters. No camelCase, PascalCase, snake_case or kebab-case. So tennisproject and scanner. Local variables should never be uppercase SNAKE_CASE, but camelCase. So atpUrlPrefix instead of ATP_URL_PREFIX and so on. You probably want those to be class constants anyways, which use uppercase ...


5

Python uses snake_case, not camelCase for variable and function names. csvFormat for example, should be csv_format. In two places you do this weird thing: with open(. . .) as file: . . . file.close with already closes file, so there's no need to manually close it. file.close doesn't do anything, and a good IDE would warn you of that. You need to add ()...


5

Clippy Clippy helps automatically scan for common mistakes in code. Many of these are from its automatic review. Aside from the suggestions in other answers, consider: Don't explicitly use return in fetch_url, and just leave it as the one word body. Don't use &String: use &str on lines 16 and 59. You should never (unless you're doing some weird ...


5

Seems like you have as many filing_dates as you have URLs, so you should have those together and handle them similarly. Your problem seems to come from the fact that you're losing the intel of which row comes from which URL, and so your only option becomes to set one date for the full dataframe. Here's an updated version saving the dates at the same time ...


5

More constants This: self.url: str = 'https://www.instagram.com/{name}/' appears to be a constant, so it can join the others in class scope. While you're doing that, you can also pull the URL from self.driver.get('https://www.instagram.com/accounts/login') into a constant; and also pull the base URL out. In other words: class InstagramPV: ...


5

Splitting lines Don't call splitlines() here: org_file = open('config/organisation.txt', 'r') return org_file.read().splitlines() The file object itself is an iterator over its lines. Also, use a context manager to ensure file closure: with open('config/organisation.txt', 'r') as f: return {line.rstrip() for line in f} This is a set comprehension. You ...


4

Wikipedia has an API available. You can make calls to e.g. "https://en.wikipedia.org/w/api.php?action=query&prop=links&titles={title}&pllimit=500" and get back a list of titles the Wikipedia page title links to, without all the internal links (but with non-existing pages and things like categories). Incidentally, there is even a python package ...


4

Duplicated statements in an if-block if find_name.startswith('https'): self._search_name = urllib.parse.urlparse(find_name).path.split('/')[1] return self._search_name else: return self._search_name should just be if find_name.startswith('https'): self._search_name = urllib.parse.urlparse(find_name).path.split('/...


4

Requests raising This pattern: if search.ok: ... else: search.raise_for_status() is redundant. Just call raise_for_status(), which will be a no-op if the response is OK. check_availability still has a confused return. If you're returning False, is there ever a point where it could return True? If not, why return a value at all? This boils down to ...


4

General There are so, so many examples of people scraping stock tracker sites. For beginners it's an understandable urge: you can see the data on the web, and you want to be able to translate those data using a script. The first thing you should reach for is an API, not a scraper. Scrapers are fragile, inefficient, and sometimes immoral - most website ...


4

Naming Variable names should be snake_case, and should represent what they are containing. I would also use req instead of r. The extra two characters aren't going to cause a heartache. Constants You have the same headers dict in four different places. I would instead define it once at the top of the file in UPPER_CASE, then just use that wherever you ...


4

Usage of requests Strongly consider replacing your use of bare urllib with requests. It's much more usable. Among other things, it should prevent you from having to worry about an SSL context. Type hints def is_downloadable(url): can be def is_downloadable(url: str) -> bool: And so on for your other functions. Boolean expressions content_type = ...


4

Line continuations Multi-line strings like this: intro = \ "DesmOEIS\n" \ "A tool for converting OEIS sequences to Desmos lists.\n" \ "\n" \ "Type id=<OEIS id> (without the brackets) to convert a sequence. \n" \ "Type help for a list of all valid commands. \n" \ "Type exit to close the ...


4

Break up logic into functions Having separate functions for each of the following steps will make the code easier to read. Get game names Scrape game information Display game information Guard your script's entry point I'd recommend moving the script execution flow under an if __name__ == "__main__": guard. Doing this allows you to import the ...


4

I choose to focus on the Single Responsibility Principle in this answer. To move closer to the Single Responsibility Principle, I think some of the functionality needs to be moved around. I have decomposed the class and removed all state that is shared on the class (rhetorical question: is the class responsible for state or functionality? If it is "both&...


3

Global constants chromedriver_path should be capitalized. Otherwise: I assume that you manually change it from None to some meaningful value for your local system. Try not to do this - instead, accept that path as an environmental variable, in a config file, or as a command-line parameter. Captain Obvious This: """ :param username: username :...


3

Your escaping for the search term is not sufficient. Other characters than spaces might be in the user entered string, which also need to be escaped. Luckily, requests can do it all for you: def rechercher_google(search_term, number_results, language_code): ''' input: Nom de l'entreprise à siretiser, le nombre de page et la langue output: La ...


3

You have quite a few methods that need to run for all words. Currently you always make sure that you output a list again, but you could use generator expressions to avoid these intermediate lists. from typing import Generator, Iterable REMOVE_PUNCTUATION = str.maketrans('', '', string.punctuation) def filter_words(words: Iterable[str], max_length: int) -&...


3

First of all, as per PEP8 I'd recommend you follow the best practices. Naming conventions allNagiosInfo -> this is most likely a constant, so: ALL_NAGIOS_INFO nagiosEntry, nagiosBaseURL, nagiosUsername, nagiosPassword, nagiosAuthType -> nagios_entry, nagios_base_url, nagios_username, nagios_password, nagios_auth_type. And so on with the naming, you got ...


3

Optimization and restructuring Function's responsibility The initial approach makes the read_page function depend on both requests and BeautifulSoup modules (though BeautifulSoup functionality/features is not actually used there). Then, a soup instance is passed to news_scraper(soup) function.To reduce dependencies let read_page function extract the remote ...


3

Just one small contribution from me: I think your utilization of BeautifulSoup is not optimal. For example this bit of code is wasteful as it does not warrant using the map function: self.titles = list( map(BeautifulSoup.get_text, self.soup.find_all('div', class_='comic-title'))) What does the map function do ? From the documentation (emphasis is mine):...


3

I think you can even get rid of the regular expressions. I prefer to use the BS4 functions. Instead of: categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))] This statement is equivalent using a CSS selector: categories = [link['href'] for link in soup.select("a[href*=\/category\/]")] That means: fetch all the a href ...


3

Combined imports from selenium.common.exceptions import ElementNotInteractableException from selenium.common.exceptions import NoSuchElementException from selenium.common.exceptions import ElementClickInterceptedException from selenium.common.exceptions import TimeoutException should be from selenium.common.exceptions import ( ...


3

Requirements Move these: # pip install requests # pip install beautifulsoup4 into a pip-compatible requirements.txt: requests beautifulsoup4 Type hints They will help. I don't know if id_ is an int or str here, but: def get_url(id_: int) -> str: Path parsing This: l = url.split("/") return l[4] if len(l) > 4 else "" is a little ...


Only top voted, non community-wiki answers of a minimum length are eligible