# Utilization of Steam APIs and web-scraping

Some background info here:

• This is a small fun project I made utilizing Steam APIs and web-scraping
• This is the first time I've ever used Python, so I'm not very familiar with the language
• The purpose of the code is determine whatever Steam user is a phisher, and I determined that using my observations/research, and this is the algorithm I developed.

The code is very inefficient as it takes 3-4 minutes to complete. I'm asking if the code can be improved so that it's more efficient and it runs faster, because I want to accomplish more with this script but with the original 3-4 minute run time that may make the code horribly slow and inefficient. (NOTE: I don't want to change the function of the code, just minor or major things that can make the code run faster).

import urllib
import itertools
import urllib2
import time
from datetime import datetime
from bs4 import BeautifulSoup
import requests
import json
import xml.etree.ElementTree as ET
from xml.dom.minidom import parseString

API_KEY = 'XXX'
steamIDs = []
myFriends = []

class steamUser:
def __init__(self, name, steamid, isPhisher):
self.name = name
self.steamid = steamid
self.isPhisher = isPhisher

def setPhisherStatus(self, phisher):
self.isPhisher = phisher

@app.route('/DeterminePhisher/<steamid>')
def getFriendList(steamid):
r = requests.get('http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key='+API_KEY+'&steamid='+steamid+'&relationship=all')
data = r.json()
for i in range(len(data['friendslist']['friends'])):
steamIDs.append(data['friendslist']['friends'][i]['steamid'])
return isPhisher(steamIDs)

def isPhisher(ids):
for l in chunksiter(ids, 50):
sids = ','.join(map(str, l))
r = requests.get('http://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key='+API_KEY+'&steamids=' + sids)
data = r.json();
for i in range(len(data['response']['players'])):
steamFriend = data['response']['players'][i]
n = steamUser(steamFriend['personaname'], steamFriend['steamid'], False)
if((('%s' % steamFriend['communityvisibilitystate']) == '1') and (('%s' % steamFriend['personastate']) == '0')):
#checks if user's  profile is private and is offline
#communityvisibilitystate=1 could also mean that the user has set his/her profile to friends only. These people are not phishers.
url =  'http://steamcommunity.com/profiles/{0}?xml=1'.format(steamFriend['steamid'])
dat = requests.get(url)
if 'profilestate' not in steamFriend:
n.setPhisherStatus(True);
if len(parseString(dat.text.encode('utf-8')).getElementsByTagName('privacyState')) > 0:
privacy = str(parseString(dat.text.encode('utf-8')).getElementsByTagName('privacyState')[0].firstChild.wholeText)
else:
privacy = ''
if (privacy == 'private'):
n.setPhisherStatus(True)
elif 'profilestate' not in steamFriend:
n.setPhisherStatus(True);
else:
#at this point the user may still be a phisher, and is trying to look like a normal steam user. If the user hasn't been reported and the account removed, we can then look at the public profile comments to determine if the user is a phisher or not.
for row in steamprofile('div', {'class': 'commentthread_comment  '}):
if ('phisher' in comment) or ('scammer' in comment):
n.setPhisherStatus(True)
myFriends.append(n);

regularusers = ''
phisherusers = ''
for j in myFriends:
if(j.isPhisher == True):
phisherusers = ((phisherusers) + ('%s is a phisher\n' % j.name) + ', ')
else:
regularusers = ((regularusers) + ('%s is a regular user' % j.name) + ', ')
return phisherusers + ' ' + regularusers

def chunksiter(l, chunks):
i,j,n = 0,0,0
rl = []
while n < len(l)/chunks:
rl.append(l[i:j+chunks])
i+=chunks
j+=j+chunks
n+=1
return iter(rl)

app.run(debug=True)

• Try to profile your code to see where the bottleneck is. If you find that almost all of the time is spent in requests.get, the bottleneck is the Steam server. Oct 28, 2014 at 6:48

It's a bit difficult to read this code, because it violates many Pythonic coding practices.

The base url http://api.steampowered.com/ISteamUser is duplicated in many places. It would be better to extract to a global variable. You might want to do likewise to all the API urls, for example:

API_BASEURL = 'http://api.steampowered.com/ISteamUser'
API_GET_FRIENDS = API_BASEURL + '/GetFriendList/v0001/?key={}&steamid={}&relationship=all'


Instead of iterating over a range of indexes like this:

for i in range(len(data['friendslist']['friends'])):
steamIDs.append(data['friendslist']['friends'][i]['steamid'])


It's better to iterate over elements:

for friend in data['friendslist']['friends']:
steamIDs.append(friend['steamid'])


Apply this everywhere possible.

This is really strange:

if((('%s' % steamFriend['communityvisibilitystate']) == '1') and (('%s' % steamFriend['personastate']) == '0')):


This is equivalent and better:

if steamFriend['communityvisibilitystate'] == '1' and steamFriend['personastate'] == '0':


Also, do you really need to compare against string values '1' and '0' instead of integers 1 and 0? If the type of these attributes is int, then you could further simplify and make the comparisons more natural as:

if steamFriend['communityvisibilitystate'] and not steamFriend['personastate']:


I haven't tested this with a steam account, but I recommend you try it.

Non-empty lists are truthy in Python. So instead of this:

if len(parseString(dat.text.encode('utf8')).getElementsByTagName('privacyState')) > 0:


This is equivalent and better:

if parseString(dat.text.encode('utf8')).getElementsByTagName('privacyState'):


You don't need to compare boolean values to True, you can use them directly. So instead of this:

if(j.isPhisher == True):


This is simpler and better:

if j.isPhisher:

• If steamFriend['communityvisibilitystate'] is an int, the "really strange" bit would convert it to str. In that case your version is not equivalent. But of course the best approach is to find out which JSON data type is expected here, an do a direct comparison. Oct 29, 2014 at 8:17
• @JanneKarila That's exactly what I had to do. It was an int and I wanted to change it into a string. Oct 30, 2014 at 22:45
• Also thank-you for those small points in my code. It really helped. Oct 30, 2014 at 22:55
• @VishwaIyer But why not simply compare steamFriend['communityvisibilitystate'] == 1? Oct 31, 2014 at 6:13
• @janos All of your suggestions worked, I tested them. I updated the code in the question. Is there anything else I can do to make the code more efficient? Nov 1, 2014 at 12:33

you should use the time module to make timing measurements for the non-deterministic calls to find out which calls are actually cause the bottleneck. like all the requests.get() calls.

(edit: Janne Karila already suggested the profile module, probably a more appropriate solution that I haven't used before (I guess I'm still a noob))