# Steam Community Market Strange Part Scraper

A while back I wrote a simple little PHP script that searches the Steam community market for any TF2 strange weapons with strange parts on the first page of results for that weapon type. It works by retrieving the listings for each weapon using curl, runs a regex to retrieve the internal JS variable containing the item metadata, and parses the JSON and looks in the descriptions for any strange part strings:

#!/usr/bin/php

<?php

# list of weapon types to look for

$weapons = array( 0 => "Scattergun", 1 => "Pistol", 2 => "Rocket%20Launcher", 3 => "Direct%20Hit", 4 => "Shotgun", 5 => "Flame%20Thrower", 6 => "Flare%20Gun", 7 => "Reserve%20Shooter", 8 => "Axtinguisher", 9 => "Grenade%20Launcher", 10 => "Loch-n-Load", 11 => "Stickybomb%20Launcher", 12 => "Eyelander", 13 => "Minigun", 14 => "Tomislav", 15 => "Dalokohs%20Bar", 16 => "Killing%20Gloves%20of%20Boxing", 17 => "Pomson%206000", 18 => "Widowmaker", 19 => "Rescue%20Ranger", 20 => "Frontier%20Justice", 21 => "Wrangler", 22 => "Sniper%20Rifle", 23 => "Bazaar%20Bargain", 24 => "SMG", 25 => "Jarate", 26 => "Bushwacka", 27 => "Revolver", 28 => "Ambassador", 29 => "Knife", ); for($wep_counter = 0; $wep_counter < count($weapons); $wep_counter++) {$url = "http://steamcommunity.com/market/listings/440/Strange%20". $weapons[$wep_counter];       # get listing for that weapon type

$handle = curl_init($url);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, TRUE);$resultraw = curl_exec($handle); preg_match("/var g_rgAssets = (\{.*\});/",$resultraw, $resultsearch); # retrieve the JSON item metadata from the page$json_string = rtrim($resultsearch[1]);$json_data = json_decode($json_string,true); for($count = 0; $count < count($json_data["440"]["2"]); $count++) { # iterate through each item$item = array_values($json_data["440"]["2"])[$count];
if(array_key_exists("descriptions",$item)) { # check for an item description; if it has a strange part it'll be listed in the description for($descriptions = 0; $descriptions < count($item["descriptions"]); $descriptions++) {$description = array_values($item["descriptions"])[$descriptions];
if(array_key_exists("color", $description) &&$description["color"] == "756b5e") {         # if any of the description strings have the color 756b5e (color of the strange part indicator)...
print $url . ", "; # output what item it is and the index on the page for easy retrieval print$count + 1 . "\n";
}
}
}
}

print "Sleeping" . "\n";                                                                         # sleep for 15 seconds to avoid rate limiting
sleep(15);
}

?>


I'd like to get some feedback on how I can make it cleaner/clearer/better. In particular, the big $weapons array at the beginning is super ugly; is there a way I could have that be more compact? Perhaps something with an explode? • @Dair's answer pretty much covered everything, but I'd like to say that you can always use the Steam API as well, they usually provide all that kind of information and more – Quill May 1 '16 at 0:30 ## 2 Answers Not exactly a PHP expert but your $weapons array:

$weapons = array( 0 => "Scattergun", 1 => "Pistol", 2 => "Rocket%20Launcher", 3 => "Direct%20Hit", . . . 29 => "Knife", );  You do not need an associative array for this. In fact, I would discourage this. Although there may be no good reason to do this, imagine you want to add a weapon in the middle, then you have to change a lot of numbers. Similarly, I do not know how PHP internally represents associative arrays, but they may be slower to access than a standard array. C++ implementations often use Red-Black Trees to implement something similar to associative arrays and other languages may use other data structures. Just use the standard array: $weapons = array(
"Scattergun",
"Pistol",
"Rocket%20Launcher",
"Direct%20Hit",
.
.
.
"Knife",
);


Use a foreach loop:

for($wep_counter = 0;$wep_counter < count($weapons);$wep_counter++) {

$url = "http://steamcommunity.com/market/listings/440/Strange%20".$weapons[$wep_counter]; # get listing for that weapon type  becomes: foreach ($weapons as $weapon) { # I don't believe you are mutating a value so I don't believe the "&" prefix is necessary.$url = "http://steamcommunity.com/market/listings/440/Strange%20". $weapon;  There could be other issues but I'm not too much of a PHP expert, but these issues pertain to a lot of languages. I would definitely be sure to keep these in mind when using other languages as well. • Being not much of a PHP expert myself, I completely forgot you could declare a regular list like that. Oops. Anyways thanks. – a spaghetto Apr 30 '16 at 13:38 • Also, if you ever do need to use a for() loop using count($weapons). Make sure you put the count() value in a variable and than use that variable in the loop. count() is used every iteration if you use it within the for statement so it will slightly decrease your performance if you're looping through a big array – Crecket May 3 '16 at 10:30

Your code is quite highly nested. Consider separating different parts of logic into separate functions. For example, you could create a function which fetches the initial data from the url.

As Dair mentioned, using foreach would help improve readability as well, you have this construct several times:

for($descriptions = 0;$descriptions < count($item["descriptions"]);$descriptions++) {
$description = array_values($item["descriptions"])[$descriptions];  which would become foreach($item["descriptions"] as $description)  or foreach($item["descriptions"] as $count =>$description)


if you need the index