2
\$\begingroup\$

Some background info here:

  • This is a small fun project I made utilizing Steam APIs and web-scraping
  • This is the first time I've ever used Python, so I'm not very familiar with the language
  • I used Flask in conjunction with this code (google "Flask" if you don't know what I'm talking about)
  • The purpose of the code is determine whatever Steam user is a phisher, and I determined that using my observations/research, and this is the algorithm I developed.

The code is very inefficient as it takes 3-4 minutes to complete. I'm asking if the code can be improved so that it's more efficient and it runs faster, because I want to accomplish more with this script but with the original 3-4 minute run time that may make the code horribly slow and inefficient. (NOTE: I don't want to change the function of the code, just minor or major things that can make the code run faster).

import urllib
import itertools
import urllib2
import time
from datetime import datetime
from bs4 import BeautifulSoup
from flask import Flask
import requests
import json
import xml.etree.ElementTree as ET
from xml.dom.minidom import parseString

app = Flask(__name__)
API_KEY = 'XXX'
steamIDs = []
myFriends = []

class steamUser:
    def __init__(self, name, steamid, isPhisher):
        self.name = name
        self.steamid = steamid
        self.isPhisher = isPhisher

    def setPhisherStatus(self, phisher):
        self.isPhisher = phisher

@app.route('/DeterminePhisher/<steamid>')
def getFriendList(steamid):
    r = requests.get('http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key='+API_KEY+'&steamid='+steamid+'&relationship=all')
    data = r.json()
    for i in range(len(data['friendslist']['friends'])):
        steamIDs.append(data['friendslist']['friends'][i]['steamid'])
    return isPhisher(steamIDs)

def isPhisher(ids):
    for l in chunksiter(ids, 50):
        sids = ','.join(map(str, l)) 
        r = requests.get('http://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key='+API_KEY+'&steamids=' + sids)
        data = r.json();
        for i in range(len(data['response']['players'])):
            steamFriend = data['response']['players'][i]
            n = steamUser(steamFriend['personaname'], steamFriend['steamid'], False)
            if((('%s' % steamFriend['communityvisibilitystate']) == '1') and (('%s' % steamFriend['personastate']) == '0')): 
                #checks if user's  profile is private and is offline
                #communityvisibilitystate=1 could also mean that the user has set his/her profile to friends only. These people are not phishers.
                url =  'http://steamcommunity.com/profiles/{0}?xml=1'.format(steamFriend['steamid'])
                dat = requests.get(url)
                if 'profilestate' not in steamFriend:
                    n.setPhisherStatus(True);
                if len(parseString(dat.text.encode('utf-8')).getElementsByTagName('privacyState')) > 0:
                    privacy = str(parseString(dat.text.encode('utf-8')).getElementsByTagName('privacyState')[0].firstChild.wholeText)
                else:
                    privacy = ''
                if (privacy == 'private'):
                    n.setPhisherStatus(True)
            elif 'profilestate' not in steamFriend:
                n.setPhisherStatus(True);
            else:
                #at this point the user may still be a phisher, and is trying to look like a normal steam user. If the user hasn't been reported and the account removed, we can then look at the public profile comments to determine if the user is a phisher or not.
                steamprofile = BeautifulSoup(urllib.urlopen('http://steamcommunity.com/profiles/'+steamFriend['steamid']).read())
                for row in steamprofile('div', {'class': 'commentthread_comment  '}):
                    comment = row.find_all('div', 'commentthread_comment_text')[0].get_text().lower()
                    if ('phisher' in comment) or ('scammer' in comment):
                        n.setPhisherStatus(True)
            myFriends.append(n);

    regularusers = ''
    phisherusers = ''
    for j in myFriends:
        if(j.isPhisher == True):
            phisherusers = ((phisherusers) + ('%s is a phisher\n' % j.name) + ', ')
        else:
            regularusers = ((regularusers) + ('%s is a regular user' % j.name) + ', ')
    return phisherusers + ' ' + regularusers

def chunksiter(l, chunks):
    i,j,n = 0,0,0
    rl = []
    while n < len(l)/chunks:        
        rl.append(l[i:j+chunks])        
        i+=chunks
        j+=j+chunks        
        n+=1
    return iter(rl)

app.run(debug=True)
\$\endgroup\$
  • 1
    \$\begingroup\$ Try to profile your code to see where the bottleneck is. If you find that almost all of the time is spent in requests.get, the bottleneck is the Steam server. \$\endgroup\$ – Janne Karila Oct 28 '14 at 6:48
4
\$\begingroup\$

It's a bit difficult to read this code, because it violates many Pythonic coding practices.

The base url http://api.steampowered.com/ISteamUser is duplicated in many places. It would be better to extract to a global variable. You might want to do likewise to all the API urls, for example:

API_BASEURL = 'http://api.steampowered.com/ISteamUser'
API_GET_FRIENDS = API_BASEURL + '/GetFriendList/v0001/?key={}&steamid={}&relationship=all'

Instead of iterating over a range of indexes like this:

for i in range(len(data['friendslist']['friends'])):
    steamIDs.append(data['friendslist']['friends'][i]['steamid'])

It's better to iterate over elements:

for friend in data['friendslist']['friends']:
    steamIDs.append(friend['steamid'])

Apply this everywhere possible.


This is really strange:

if((('%s' % steamFriend['communityvisibilitystate']) == '1') and (('%s' % steamFriend['personastate']) == '0')): 

This is equivalent and better:

if steamFriend['communityvisibilitystate'] == '1' and steamFriend['personastate'] == '0':

Also, do you really need to compare against string values '1' and '0' instead of integers 1 and 0? If the type of these attributes is int, then you could further simplify and make the comparisons more natural as:

if steamFriend['communityvisibilitystate'] and not steamFriend['personastate']:

I haven't tested this with a steam account, but I recommend you try it.


Non-empty lists are truthy in Python. So instead of this:

if len(parseString(dat.text.encode('utf8')).getElementsByTagName('privacyState')) > 0:

This is equivalent and better:

if parseString(dat.text.encode('utf8')).getElementsByTagName('privacyState'):

You don't need to compare boolean values to True, you can use them directly. So instead of this:

if(j.isPhisher == True):

This is simpler and better:

if j.isPhisher:
\$\endgroup\$
  • \$\begingroup\$ If steamFriend['communityvisibilitystate'] is an int, the "really strange" bit would convert it to str. In that case your version is not equivalent. But of course the best approach is to find out which JSON data type is expected here, an do a direct comparison. \$\endgroup\$ – Janne Karila Oct 29 '14 at 8:17
  • \$\begingroup\$ @JanneKarila That's exactly what I had to do. It was an int and I wanted to change it into a string. \$\endgroup\$ – Vishwa Iyer Oct 30 '14 at 22:45
  • 1
    \$\begingroup\$ Also thank-you for those small points in my code. It really helped. \$\endgroup\$ – Vishwa Iyer Oct 30 '14 at 22:55
  • \$\begingroup\$ @VishwaIyer But why not simply compare steamFriend['communityvisibilitystate'] == 1? \$\endgroup\$ – Janne Karila Oct 31 '14 at 6:13
  • 1
    \$\begingroup\$ @janos All of your suggestions worked, I tested them. I updated the code in the question. Is there anything else I can do to make the code more efficient? \$\endgroup\$ – Vishwa Iyer Nov 1 '14 at 12:33
0
\$\begingroup\$

you should use the time module to make timing measurements for the non-deterministic calls to find out which calls are actually cause the bottleneck. like all the requests.get() calls.

(edit: Janne Karila already suggested the profile module, probably a more appropriate solution that I haven't used before (I guess I'm still a noob))

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.