Steam Community Market Strange Part Scraper

Question

A while back I wrote a simple little PHP script that searches the Steam community market for any TF2 strange weapons with strange parts on the first page of results for that weapon type. It works by retrieving the listings for each weapon using curl, runs a regex to retrieve the internal JS variable containing the item metadata, and parses the JSON and looks in the descriptions for any strange part strings:

#!/usr/bin/php

<?php

# list of weapon types to look for

$weapons = array(
  0 => "Scattergun",
  1 => "Pistol",
  2 => "Rocket%20Launcher",
  3 => "Direct%20Hit",
  4 => "Shotgun",
  5 => "Flame%20Thrower",
  6 => "Flare%20Gun",
  7 => "Reserve%20Shooter",
  8 => "Axtinguisher",
  9 => "Grenade%20Launcher",
  10 => "Loch-n-Load",
  11 => "Stickybomb%20Launcher",
  12 => "Eyelander",
  13 => "Minigun",
  14 => "Tomislav",
  15 => "Dalokohs%20Bar",
  16 => "Killing%20Gloves%20of%20Boxing",
  17 => "Pomson%206000",
  18 => "Widowmaker",
  19 => "Rescue%20Ranger",
  20 => "Frontier%20Justice",
  21 => "Wrangler",
  22 => "Sniper%20Rifle",
  23 => "Bazaar%20Bargain",
  24 => "SMG",
  25 => "Jarate",
  26 => "Bushwacka",
  27 => "Revolver",
  28 => "Ambassador",
  29 => "Knife",
);

for($wep_counter = 0; $wep_counter < count($weapons); $wep_counter++) {

  $url = "http://steamcommunity.com/market/listings/440/Strange%20". $weapons[$wep_counter];       # get listing for that weapon type

  $handle = curl_init($url);
  curl_setopt($handle, CURLOPT_RETURNTRANSFER, TRUE);

  $resultraw = curl_exec($handle);

  preg_match("/var g_rgAssets = (\{.*\});/", $resultraw, $resultsearch);                           # retrieve the JSON item metadata from the page

  $json_string = rtrim($resultsearch[1]);
  $json_data = json_decode($json_string,true);

  for($count = 0; $count < count($json_data["440"]["2"]); $count++) {                              # iterate through each item
    $item = array_values($json_data["440"]["2"])[$count];
    if(array_key_exists("descriptions",$item)) {                                                   # check for an item description; if it has a strange part it'll be listed in the description
      for($descriptions = 0; $descriptions < count($item["descriptions"]); $descriptions++) {
        $description = array_values($item["descriptions"])[$descriptions];
        if(array_key_exists("color", $description) && $description["color"] == "756b5e") {         # if any of the description strings have the color 756b5e (color of the strange part indicator)...
          print $url . ", ";                                                                       # output what item it is and the index on the page for easy retrieval 
          print $count + 1 . "\n";
        }  
      }  
    }  
  }

  print "Sleeping" . "\n";                                                                         # sleep for 15 seconds to avoid rate limiting
  sleep(15);
}

?>

I'd like to get some feedback on how I can make it cleaner/clearer/better. In particular, the big $weapons array at the beginning is super ugly; is there a way I could have that be more compact? Perhaps something with an explode?

Answer 1

Not exactly a PHP expert but your $weapons array:

$weapons = array(
  0 => "Scattergun",
  1 => "Pistol",
  2 => "Rocket%20Launcher",
  3 => "Direct%20Hit",
  .
  .
  .
  29 => "Knife",
);

You do not need an associative array for this. In fact, I would discourage this. Although there may be no good reason to do this, imagine you want to add a weapon in the middle, then you have to change a lot of numbers.

Similarly, I do not know how PHP internally represents associative arrays, but they may be slower to access than a standard array. C++ implementations often use Red-Black Trees to implement something similar to associative arrays and other languages may use other data structures.

Just use the standard array:

$weapons = array(
  "Scattergun",
  "Pistol",
  "Rocket%20Launcher",
  "Direct%20Hit",
   .
   .
   .
  "Knife",
);

Use a foreach loop:

for($wep_counter = 0; $wep_counter < count($weapons); $wep_counter++) {

  $url = "http://steamcommunity.com/market/listings/440/Strange%20". $weapons[$wep_counter];       # get listing for that weapon type

becomes:

foreach ($weapons as $weapon) { # I don't believe you are mutating a value so I don't believe the "&" prefix is necessary.

  $url = "http://steamcommunity.com/market/listings/440/Strange%20". $weapon; 

There could be other issues but I'm not too much of a PHP expert, but these issues pertain to a lot of languages. I would definitely be sure to keep these in mind when using other languages as well.

Answer 2

Your code is quite highly nested. Consider separating different parts of logic into separate functions. For example, you could create a function which fetches the initial data from the url.

As Dair mentioned, using foreach would help improve readability as well, you have this construct several times:

for($descriptions = 0; $descriptions < count($item["descriptions"]); $descriptions++) {
    $description = array_values($item["descriptions"])[$descriptions];

which would become

foreach($item["descriptions"] as $description)

or

foreach($item["descriptions"] as $count => $description)

if you need the index