Python program to identify cities, with spelling check

Question

For this Code, I am provided a text file that contain several cities. I am suppose to identify the cities mentioned and print their state and country.

Requirements: If a mentioned city is in two or more countries, I ask the user to mention which city they are talking about. In addition if there is a slight typo, I ask the user if they meant a certain city instead. For example if they type 'Dalls' instead of 'Dallas' I need to provide the user options such as 'Do you mean Dallas instead of Dalls'.

Problem: My code has been working pretty well, but I feel it can be significantly simplified. Please let me know what concepts I can use to further simplify my code.

File Explanations: I am using a text called 'world-cities.csv', 'TEXT.txt' and 'usa.txt'. 'world-cities.csv' is a file that contains a lot of cities in the world. 'TEXT.txt' is a file that contains the sentences that I will be analyzing for cities. 'usa.txt' contains common words in the english language. I used it to compare to 'TEXT.txt' to remove common words. I had a problem with words such as 'and' showing up as typos. So this was a bootleg method to get rid of them.

I am trying not to use imports like geotext, and instead focus more on finding the most logically efficient way to program. If I am wrong about this, let me know.

import pandas as pd
import re

#imported dataset
dataset = pd.read_csv('world-cities.csv')

#assigned certain parts of data set to variable
data = dataset.iloc[:,:-1]
city = dataset.iloc[:,0]
state = dataset.iloc[:,2]
country = dataset.iloc[:,1]


#opened and imported textfile
txtfile = open('TEXT.txt','r')
txtfile = txtfile.read()
words = open('usa.txt','r')
words = words.read()


#getting rid of punctation
altered = re.sub("[.,:]",'',txtfile)
templist = [] #holds the cities(state and country) info of the places 
mentioned in the paragraph
final = [] #final array
all_cities = [] #holds all the cities mentioned, used to check for 
repeating cities
repeat = {} #contains only city names
repeatinfo = [] #contain all infor about repeating cities
stupid = 0
close = 0
typo = []
typodict = {}
typecount = 0
finaltypo = []
altered2 = []
ban = []
temp = []



#finding out where the talked about cities are 
for x in altered.split():
    count = 0
    zcount = 0
    for y in city:
        if x == y:
            zcount +=1
            templist.append([city[count], state[count], country[count]])
            all_cities.append(city[count])
        count+=1
    if zcount > 1:
        repeat[x] = zcount

#finding two worded cities   
for count,y in enumerate(city):
    if y in altered:
        if y not in all_cities:
            if len(y.split()) >1:
                zcount +=1
                templist.append([city[count], state[count], 
country[count]])
                all_cities.append(city[count])
                temp.append(city[count])



#removing one city words identified from two worded cities (York vs New 
York)
#splitting words 
split = []
for x in temp:
    for y in x.split():
        split.append(y)

#removing from all_cities
for x in all_cities:
    for y in split:
        if y in all_cities:
            all_cities.remove(y)
            ban.append(y)


#removing from templist
for x in ban:
    for y in templist:
        if x == y[0]:
            templist.remove(y)

#removing form repeat
#print(repeat)
for x in list(repeat):
    if x in ban:
        repeat.pop(x)


#put in all assumed Typos
for x in altered.split():
    if x not in all_cities:
        x = x.lower()
        if x not in words:
            typo.append(x)



#narrow down options of typos
many = 0
for a in typo:
    for b in city:
        b = b.lower()
        if len(a) >= (len(b)-1) and len(a) <= (len(b)+1):
            if a[0] == b[0] or a[-1::] == b[-1::]:
                if a[0:3] == b[0:3] or a[-3::] == b[-3::]:
                    #print(f'{a} vs {b}')
                    many = 0
                    for x in a:
                        if x in b:
                            many+=1
                        if many >= (len(b)-1) and many <= (len(b)+1):
                            typodict[b] = a

#let user choose if it is a typo or not
print('TYPO Checking')
for a in typo:
    p =0
    q = 0
    while(p < len(typo) and q == 0):
        for x,y in typodict.items():
            go2 = True
            while(go2 and q==0):
                if y == a:
                    user2 = input(f" Did you mean to type '{x}' instead of 
'{y}'? Enter 'y' or 'n': ")
                    user2 = user2.lower()
                    if user2 == 'y':
                        go2 = False
                        finaltypo.append(x)
                        p+=1
                        q+=1
                    elif user2 == 'n':
                        go2 = False
                    else: 
                        print('You have entered a invalid value')
                else:
                    go2 = False



#adding typoed cities into list
for x in finaltypo:
    x = x.capitalize()
    count = 0
    zcount = 0
    for y in city:
        if x == y:
            zcount +=1
            templist.append([city[count], state[count], country[count]])
            all_cities.append(city[count])
        count+=1
    if zcount > 1:
        repeat[x] = zcount

#finding out what cities repeat and adding all their information to repeat 
info
for x in repeat:
    rcount = 0
    for y in city:
        if x == y: 
            repeatinfo.append([city[rcount], state[rcount], 
country[rcount]])
        rcount +=1

#determining which country they mean when they mentioned repeating cities
print('\n Which City?')
for x,y in repeat.items():
    i = 0
    e = 0
    while(i < y and e == 0):
        go = True
        for c in repeatinfo: 
            go = True
            while(go and e == 0):
                if x == c[0]:
                    user = input(f'Do you mean {x} in {c[1]},{c[2]} enter y 
or n: ') 
                    user = user.lower()
                    i +=1
                    if user == 'y':
                        final.append(f' {x} in {c[1]}, {c[2]}')
                        go = False
                        i +=1
                        e +=1
                    elif user == 'n':
                        go = False
                        i+=1
                    else:
                        print('You have entered a invalid input')
                else: 
                    go = False


#removing repeating cities from templist
for y in list(templist):
    if y[0] in list(repeat):
        templist.remove(y)

#adding remaining elements of templist to final list
for y in list(templist):
    final.append(f' {y[0]} in {y[1]}, {y[2]}')

#printing final output
print('\n You have entered the following cities:')               
for x in final:
    print(x)

Answer 1

Here are some general comments:

Close files

Close files when you are done with them. Or better yet use a with statement to automatically close the file at the end of the code block (the indented part)

with open('TEXT.txt','r') as txtfile:
    text = txtfile.read()

Variable names

Try to use meaningful variable names. That doesn't mean they have to be long names, depending on the context i,j,k or x,y,z can be meaningful names. 'temp', 'stupid' probably aren't meaningful.

And try to be consistent about a variables represent. In your code, sometimes x is a word in the text file, other times it is a two word city, any city, a typo, or a text line in the final output. So when someone (or you) reading your code sees the variable x, they must look around to see what x is this time.

Use functions

The program is one big run of code. That makes it hard to understand and reason about. To help structure and manage the program, try breaking it up into several functions that each do one thing. A rule of thumb is that a function should fit on your screen so you can see all of it at once.

Separate functions lets you test portions of your program to make sure they work as intended. That makes debugging easier. For example, does the code to check if a one word city (e.g. York) should be a two word city (e.g. New York) work correctly if both cities are in the text? Can you test it to make sure? What about cities with more than two words (e.g., Rio de Janeiro).

Functions can also help you keep related code together. In this program, the textfile is read in one place, it is processed to remove (some) punctuation in another place, and split into words in two other places.

Data structures

Choosing the right data structure can make a big difference in how complex your code is. For example, data, city, state, and country are parallel lists. So your code keeps track of an index into the city so it can access the other information. If you used a dict keyed by the city, the indexes can be eliminated. The value of a dict entry can be a list of tuples with the state, country, and data info. The list has multiple entries if more than one city has the same name:

cities = { 'Dallas':[('Texas', 'USA', 'dallas data')],
           'York'  :[('North Yorkshire', 'England', 'york england data'),
                     ('Maine', 'USA', 'york maine data')],
           'Mexico':[('Kansas', 'USA', 'mexico kansas data']
           ... etc.
         }

Obviously, this would be built from the world_cities.csv file (see below).

Standard library

The standard library has lots of useful functions. Some that may be useful here are collections, csv, and difflib:

collections.defaultdict is useful for building dictionaries on the fly.

You only use pandas to read in the csv file, but you can use csv from the standard library. csv.reader or csv.DictReader can be used to read a csv file:

import collections
import csv

cities = defaultdict(list)

with open('world_cities.csv', newline='') as csvfile:
    reader = csv.reader(csvfile)
        for row in reader:
            city = row[0]
            state = row[2]
            country = row[1]
            data = row[-1]
            cities[city].append(state, country, data)

And difflib.get_close_matches() can be used to search for words in a list that are close enough to a search term. There are parameters to control how close the match needs to be, and a maximum number of matches:

import difflib

# after you built the cities dict above, you would use
# city_names = list(cities.keys())
# but for illustration:
city_names = ['York', 'New York', 'Devon', 'Peoria', 'Dallas', ]

difflib.get_close_matches('York', city_names)        ==> ['York', 'New York']

difflib.get_close_matches('dalas', city_names)       ==> ['Dallas']