Remove \n newline from a list of string dumped from github used for a Italian stopwords vocabulary

Question

I have a set of Italian words in a txt file hosted on GitHub. I use it as stopwords vocabulary for further NLP tasks.

When I dump from GitHub it returns me, obviously, also \n

Better or more efficient way to replace the special character compared with my working code?

import urllib.request

stopwords_link = "https://raw.githubusercontent.com/stopwords-iso/stopwords-it/master/stopwords-it.txt"
file_name = urllib.request.urlopen(stopwords_link)
   
list_nowords = [x.decode("utf-8") for x in file_name]
splitted = [x.split('\n')[0] for x in list_nowords]

As usual I am a self-taught and this is the best place where I can learn, discover new ways and aspects on how to code efficiently with Python

Answer 1

Code is very readable, and that's a good start.

file_name is in fact the file_content; splitted (split) should be just stop_words; stopwords_list is a constant, so by PEP 8, should be uppercase STOPWORDS_LIST.

The problem I see here is that you are decoding each word, when in fact you should decode the whole file, just once.

This is just personal preference, but I tend to always use the requests library, because, in my experience, is always a step ahead of what I need. requests tries to understand the encoding, based on the response, so there should be no need to decode.

A bit of data validation is also needed. The file could have been moved, the network or Github could be down, so it always better to be prepared for the worse. In this case, if the network is down (Connection Timeout) or the file is not there (404) there isn't a big difference: your application should handle the exception and the execution should not proceed. While a connection timeout would normally raise an exception, a 404 (or any other HTTP error) would still be considered a valid HTTP response, that's why requests has a handy method when you just want 2** responses: raise_for_status.

Then, to remove the \n there is actually the method splitlines() which is exactly what you need.

import requests

STOPWORDS_LIST= "https://raw.githubusercontent.com/stopwords-iso/stopwords-it/master/stopwords-it.txt"

try:
    response = requests.get(STOPWORDS_LIST)
    response.raise_for_status()  
    file_content = response.text
    words = file_content.splitlines()
except Exception as e:
    print(f"There was an error: {str(e)}")

This piece of code should put inside a function, so it is clearer what it does and restrict its scope. I'm also adding a type to the function, that can be useful to avoid errors at compile time. Maybe some day you will have stopwords in other languages, so I'm just writing a main as example.

from typing import List

import requests


def download_stop_words(url: str) -> List[str]:
    try:
        response = requests.get(url)
        response.raise_for_status()
        return response.text.splitlines()
    except Exception as e:
        print(f"There was an error: {str(e)}")

if __name__ == '__main__': # this avoids the program being executed when the module is imported 
    STOP_WORDS = {
        "it": "https://raw.githubusercontent.com/stopwords-iso/stopwords-it/master/stopwords-it.txt"
    }
    for lang, url in STOP_WORDS.items():
        stop_words = download_stop_words(url)
        print(f"In '{lang}' there are {len(stop_words)} stop words")