8
\$\begingroup\$

This code works fine, but I believe it has optimization problems. Please review this.

Also, please keep in mind that it stops after each iteration of the loop foreach($mascow_sub_area as $subway).

    <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<?php
    set_time_limit(0);
    ini_set('memory_limit', '-1');
    require_once("../../dom/dom.php");
    $connection = mysqli_connect("localhost", "root", "", "delivery_clud");
    mysqli_set_charset($connection, "utf8");
    function spider($url){
        $yql_base_url = "http://query.yahooapis.com/v1/public/yql";
        $url = "select * from html where url = '$url'";

        $yql_query_url = $yql_base_url . "?q=" . urlencode($url);
        $yql_query_url .= "&format=json"; 
        $session = curl_init($yql_query_url);
        curl_setopt($session, CURLOPT_RETURNTRANSFER,true);    
        $json = curl_exec($session);
        curl_close($session);
        return $json;
    }
    $mascow_sub_area = array(//'Aviamotornaya','Avtozavodskaya','Akademicheskaya','Aleksandrovskiy_Sad',
    'Alekseevskaya',
    'Alma-Atinskaya','Altufevo','Annino','Arbatskaya','Aeroport','Babushkinskaya','Bagrationovskaya','Barrikadnaya','Baumanskaya','Begovaya','Belorusskaya','Belyaevo','Bibirevo','Biblioteka_imeni_Lenina','Borisovo','Borovitskaya','Botanicheskiy_Sad','Bratislavskaya','Bulvar_Admirala_Ushakova','Bulvar_Dmitriya_Donskogo','Buninskaya_Alleya','Varshavskaya','VDNKh','Vladykino','Vodnyy_Stadion','Voykovskaya','Volgogradskiy_Prospekt','Volzhskaya','Volokolamskaya','Vorobevy_Gory','Vystavochnaya','Vykhino','Delovoy_Tsentr','Dinamo','Dmitrovskaya','Dobryninskaya','Domodedovskaya','Dostoevskaya','Dubrovka','Zhulebino','Zyablikovo','Izmaylovskaya','Kaluzhskaya','Kantemirovskaya','Kakhovskaya','Kashirskaya','Kievskaya','Kitay-gorod','Kozhukhovskaya','Kolomenskaya','Komsomolskaya','Konkovo','Krasnogvardeyskaya','Krasnopresnenskaya','Krasnoselskaya','Krasnye_Vorota','Krestyanskaya_Zastava','Kropotkinskaya','Krylatskoe','Kuznetskiy_Most','Kuzminki','Kuntsevskaya','Kurskaya','Kutuzovskaya','Leninskiy_prospekt','Lermontovskij_prospekt','Lubyanka','Lyublino','Marksistskaya','Marina_roshcha','Marino','Mayakovskaya','Medvedkovo','Mezhdunarodnaya','Mendeleevskaya','Mitino','Molodezhnaya','Myakinino','Nagatinskaya','Nagornaya','Nakhimovskiy_prospekt','Novogireevo','Novokosino','Novokuznetskaya','Novoslobodskaya','Novoyasenevskaya','Novye_Cheremushki','Oktyabrskaya','Oktyabrskoe_Pole','Orekhovo','Otradnoe','Okhotnyy_Ryad','Paveletskaya','Park_Kultury','Park_Pobedy','Partizanskaya','Pervomayskaya','Perovo','Petrovsko-Razumovskaya','Pechatniki','Pionerskaya','Planernaya','Ploshchad_Ilicha','Ploshchad_Revolyutsii','Polezhaevskaya','Polyanka','Prazhskaya','Preobrazhenskaya_Ploshchad','Proletarskaya','Prospekt_Vernadskogo','Prospekt_Mira','Profsoyuznaya','Pushkinskaya','Pyatnickoe_shosse','Rechnoy_Vokzal','Rizhskaya','Rimskaya','Ryazanskiy_Prospekt','Savelovskaya','Sviblovo','Sevastopolskaya','Semenovskaya','Serpukhovskaya','Slavyanskiy_Bulvar','Smolenskaya','Sokol','Sokolniki','Sportivnaya','Sretenskiy_bulvar','Strogino','Studencheskaya','Sukharevskaya','Skhodnenskaya','Taganskaya','Tverskaya','Teatralnaya','Tekstilshchiki','Teletsentr','Teplyy_Stan','Timiryazevskaya','Tretyakovskaya','Trubnaya','Tulskaya','Turgenevskaya','Tushinskaya','Ulitsa_1905_goda','Ulitsa_Akademika_Koroleva','Ulitsa_Akademika_Yangelya','Ulitsa_Gorchakova','Ulitsa_Milashenkova','Ulitsa_Podbelskogo','Ulitsa_Sergeya_Eyzenshteyna','Ulitsa_Skobelevskaya','Ulitsa_Starokachalovskaya','Universitet','Filevskiy_Park','Fili','Frunzenskaya','Tsaritsyno','Tsvetnoy_bulvar','Cherkizovskaya','Chertanovskaya','Chekhovskaya','Chistye_Prudy','Chkalovskaya','Shabolovskaya','Shipilovskaya','Shosse_Entuziastov','Shchelkovskaya','Shchukinskaya','Elektrozavodskaya','Yugo-Zapadnaya','Yuzhnaya','Yasenevo');
    $categories = array(1=>"1",2=>"2");//array(1=>array('pizza','sushi','shashliki','pirogi','burger'),2=>array('farm','dairy','delicatessen','confectionery','gastronomy'));
    foreach($categories as $key => $cati){
        {
            if($key == 2){
                $url = "http://www.delivery-club.ru/entities/groceries/farm/#group=%D0%93%D0%B0%D1%81%D1%82%D1%80%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%8F&group=%D0%A4%D0%B5%D1%80%D0%BC%D0%B5%D1%80%D1%81%D0%BA%D0%B8%D0%B5+%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D1%8B&group=%D0%9C%D0%BE%D0%BB%D0%BE%D1%87%D0%BD%D1%8B%D0%B5+%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D1%8B&group=%D0%94%D0%B5%D0%BB%D0%B8%D0%BA%D0%B0%D1%82%D0%B5%D1%81%D1%8B&group=%D0%9A%D0%BE%D0%BD%D0%B4%D0%B8%D1%82%D0%B5%D1%80%D1%81%D0%BA%D0%B8%D0%B5+%D0%B8%D0%B7%D0%B4%D0%B5%D0%BB%D0%B8%D1%8F&group=%D0%92%D0%BE%D0%B4%D0%B0%2C+%D0%A7%D0%B0%D0%B9%2C+%D0%9A%D0%BE%D1%84%D0%B5&show=all";
                get_data("Goods", str_get_html(file_get_contents($url)));
            }
            else{
            foreach($mascow_sub_area as $subway){
            //echo "Key: $key----Category: $sub.<br />";
                $url = "http://www.delivery-club.ru/ajax/entities/?mode=food&cat_id=$cati&mo_mode=null&district=null&okrug=null&cuisine=region&params=null&ajax_changer_subway=$subway";
                $html = str_get_html(file_get_contents($url));
                //echo $html;return;
                //$html = "";
                //echo "test<hr />";

                get_data($subway, $html);
                }
            }
        }

    }
    function get_data($subway, $html){

                $data = array();
                $data['subway'] = $subway;

                $val = ($html->find('.dum'));

                foreach($html->find('.dum') as $one){

                    foreach($one->find(".full_link") as $link){
                        $data['name'] = (trim(strip_tags("$link->innertext")));
                        $data['link'] = trim(strip_tags("http://www.delivery-club.ru$link->href"));
                    //        print_r($data);
                        //
                    }

                    foreach($one->find('.info') as $main){
                        foreach($main->find('.min_sum') as $min_sum){
                            $data['min_order'] = (trim(strip_tags("$min_sum->innertext")));
                        }//
                        foreach($main->find('.distanse') as $dis){
                            $d = explode(":", strip_tags($dis->innertext));
                            $data['distance'] = (trim($d[1]));
                        }
                        foreach($main->find('.specialization') as $main_cat){
                            $d = explode(":", strip_tags($main_cat->innertext));
                            $data['specialization'] = (trim($d[1]));
                        }
                        //$ul = $main->find('ul');
                        $price ;
                        foreach($main->find('.price') as $li){ $price[] = trim(strip_tags($li->innertext));}
                        $data['delivery_cost'] = end($price);$price = array();
                        foreach($main->find('.categories') as $cat){
                            $d = explode(":", strip_tags($cat->innertext));
                            $data['categories'] = (trim($d[1]));
                        }
            //            foreach($main->find('.price', 0) as $extra){
            //                echo "----Delivery Charges".$extra->innertext;
            //            }
                        foreach($main->find('.time') as $time){
                            $data['delivery_time'] =(trim(strip_tags($time->innertext)));
                        }
                        foreach($main->find('.plus') as $plus){
                            $data['plus'] = trim(strip_tags($plus->innertext));
                        }
                        foreach($main->find('.minus') as $minus){
                            $data['minus'] = trim(strip_tags($minus->innertext));
                        }
                        foreach($main->find('.hint') as $minus){
                            $data['description'] = (trim(strip_tags($minus->innertext)));
                        }
                        $u = ( parse_url($data['link']));
                        $u['fragment'] =  str_replace(end($u),"info/#".end($u), end($u));
                        $data['city'] = 'Moscow';
                        $next_scrap = $u['scheme']."://".$u['host'].$u['path'].$u['fragment'];
                        sleep(3);
                        $minified = str_get_html(file_get_contents($next_scrap));
                        if(!$minified || $minified == "") continue;
                        foreach($minified->find('.tabs') as $tabs){
                            preg_match('/[0-9]+/', $tabs->innertext, $found);
                            if(count($found)){
                                $data['reviews'] = $found[0];
                            }
                        }
                        foreach($minified->find(".thumb") as $plus){
                            $p_r = str_replace('%', "", $plus->innertext);
                            $data['positive_reviews'] = ceil($data['reviews']*($p_r/100));
                            $data['negative_reviews'] = $data['reviews']-$data['positive_reviews'];
                            break;
                        }
                        $i = 0;
                        $address = array();
                        foreach($minified->find('.default_list') as $ul){
                            foreach($ul->find('li') as $li){
                                $address[$i]['address'] = trim(strip_tags($li->innertext));
                                foreach($li->find('meta') as $meta){
                                    $address[$i][] = strip_tags($meta->content);
                                }
                           //     echo $li->innertext."<br />";
                           $i+=1;
                            }
                        }
                        //print_r($address);
                        $data['address'] = '';
                        foreach($address as $val){
                            $data['address'] .= "( address=>".(trim($val['address']));
                            $data['address'] .= " --- longitude=>".trim($val[1]);
                            $data['address'] .= " --- latitude=>".trim($val[0])." ) || ";
                        }
                    }
                    //file_put_contents("output.txt", 
                    //print_r($data);
                    insert($data);
                }
    }
    function insert($data)
{
    //open connection from the connection file
    //it'll reccieve table as a String and data in the form of array(key value pair) and adjust those key values to a query
    // $where is an array that will take array as first one and impose that as an 'WHERE' clause and will set its values.
    global $connection;
    $fields = array();
    $values = array();
    foreach ($data as $key => $value) {
        $fields[] = $key;
        $values[] = "'" . mysqli_real_escape_string($connection, $value) . "'";
    }
    if (count($fields) == count($values)) {
        $insert = implode(",", $fields);
        $val = implode(",", $values);
    }
    $sql = "INSERT INTO data($insert) VALUES($val)";

    $sql .= " ON DUPLICATE KEY UPDATE link = '".mysqli_real_escape_string($connection,$data['link'])."'";
    mysqli_query($connection, ($sql))or die(mysqli_error($connection)."<br />".print($sql));

}
\$\endgroup\$
  • \$\begingroup\$ You have up to five levels of nesting in foreach loop. Its Big O can't be good. stackoverflow.com/questions/21372927/big-o-and-nested-loops \$\endgroup\$ – CodeWorks Apr 30 '14 at 11:39
  • \$\begingroup\$ I only found about 2 lines of MySQLi at the very end, perhaps you should not have tagged that as MySQLi question... for all it's worth they look fine. \$\endgroup\$ – Phrancis May 22 '14 at 0:10
8
\$\begingroup\$

I want to focus principally on this section of your code:

$mascow_sub_area = array(//'Aviamotornaya','Avtozavodskaya','Akademicheskaya','Aleksandrovskiy_Sad',
'Alekseevskaya',
'Alma-Atinskaya','Altufevo','Annino','Arbatskaya','Aeroport','Babushkinskaya','Bagrationovskaya','Barrikadnaya','Baumanskaya','Begovaya','Belorusskaya','Belyaevo','Bibirevo','Biblioteka_imeni_Lenina','Borisovo','Borovitskaya','Botanicheskiy_Sad','Bratislavskaya','Bulvar_Admirala_Ushakova','Bulvar_Dmitriya_Donskogo','Buninskaya_Alleya','Varshavskaya','VDNKh','Vladykino','Vodnyy_Stadion','Voykovskaya','Volgogradskiy_Prospekt','Volzhskaya','Volokolamskaya','Vorobevy_Gory','Vystavochnaya','Vykhino','Delovoy_Tsentr','Dinamo','Dmitrovskaya','Dobryninskaya','Domodedovskaya','Dostoevskaya','Dubrovka','Zhulebino','Zyablikovo','Izmaylovskaya','Kaluzhskaya','Kantemirovskaya','Kakhovskaya','Kashirskaya','Kievskaya','Kitay-gorod','Kozhukhovskaya','Kolomenskaya','Komsomolskaya','Konkovo','Krasnogvardeyskaya','Krasnopresnenskaya','Krasnoselskaya','Krasnye_Vorota','Krestyanskaya_Zastava','Kropotkinskaya','Krylatskoe','Kuznetskiy_Most','Kuzminki','Kuntsevskaya','Kurskaya','Kutuzovskaya','Leninskiy_prospekt','Lermontovskij_prospekt','Lubyanka','Lyublino','Marksistskaya','Marina_roshcha','Marino','Mayakovskaya','Medvedkovo','Mezhdunarodnaya','Mendeleevskaya','Mitino','Molodezhnaya','Myakinino','Nagatinskaya','Nagornaya','Nakhimovskiy_prospekt','Novogireevo','Novokosino','Novokuznetskaya','Novoslobodskaya','Novoyasenevskaya','Novye_Cheremushki','Oktyabrskaya','Oktyabrskoe_Pole','Orekhovo','Otradnoe','Okhotnyy_Ryad','Paveletskaya','Park_Kultury','Park_Pobedy','Partizanskaya','Pervomayskaya','Perovo','Petrovsko-Razumovskaya','Pechatniki','Pionerskaya','Planernaya','Ploshchad_Ilicha','Ploshchad_Revolyutsii','Polezhaevskaya','Polyanka','Prazhskaya','Preobrazhenskaya_Ploshchad','Proletarskaya','Prospekt_Vernadskogo','Prospekt_Mira','Profsoyuznaya','Pushkinskaya','Pyatnickoe_shosse','Rechnoy_Vokzal','Rizhskaya','Rimskaya','Ryazanskiy_Prospekt','Savelovskaya','Sviblovo','Sevastopolskaya','Semenovskaya','Serpukhovskaya','Slavyanskiy_Bulvar','Smolenskaya','Sokol','Sokolniki','Sportivnaya','Sretenskiy_bulvar','Strogino','Studencheskaya','Sukharevskaya','Skhodnenskaya','Taganskaya','Tverskaya','Teatralnaya','Tekstilshchiki','Teletsentr','Teplyy_Stan','Timiryazevskaya','Tretyakovskaya','Trubnaya','Tulskaya','Turgenevskaya','Tushinskaya','Ulitsa_1905_goda','Ulitsa_Akademika_Koroleva','Ulitsa_Akademika_Yangelya','Ulitsa_Gorchakova','Ulitsa_Milashenkova','Ulitsa_Podbelskogo','Ulitsa_Sergeya_Eyzenshteyna','Ulitsa_Skobelevskaya','Ulitsa_Starokachalovskaya','Universitet','Filevskiy_Park','Fili','Frunzenskaya','Tsaritsyno','Tsvetnoy_bulvar','Cherkizovskaya','Chertanovskaya','Chekhovskaya','Chistye_Prudy','Chkalovskaya','Shabolovskaya','Shipilovskaya','Shosse_Entuziastov','Shchelkovskaya','Shchukinskaya','Elektrozavodskaya','Yugo-Zapadnaya','Yuzhnaya','Yasenevo');
$categories = array(1=>"1",2=>"2");//array(1=>array('pizza','sushi','shashliki','pirogi','burger'),2=>array('farm','dairy','delicatessen','confectionery','gastronomy'));
foreach($categories as $key => $cati){
    {
        if($key == 2){
            $url = "http://www.delivery-club.ru/entities/groceries/farm/#group=%D0%93%D0%B0%D1%81%D1%82%D1%80%D0%BE%D0%BD%D0%BE%D0%BC%D0%B8%D1%8F&group=%D0%A4%D0%B5%D1%80%D0%BC%D0%B5%D1%80%D1%81%D0%BA%D0%B8%D0%B5+%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D1%8B&group=%D0%9C%D0%BE%D0%BB%D0%BE%D1%87%D0%BD%D1%8B%D0%B5+%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D1%8B&group=%D0%94%D0%B5%D0%BB%D0%B8%D0%BA%D0%B0%D1%82%D0%B5%D1%81%D1%8B&group=%D0%9A%D0%BE%D0%BD%D0%B4%D0%B8%D1%82%D0%B5%D1%80%D1%81%D0%BA%D0%B8%D0%B5+%D0%B8%D0%B7%D0%B4%D0%B5%D0%BB%D0%B8%D1%8F&group=%D0%92%D0%BE%D0%B4%D0%B0%2C+%D0%A7%D0%B0%D0%B9%2C+%D0%9A%D0%BE%D1%84%D0%B5&show=all";
            get_data("Goods", str_get_html(file_get_contents($url)));
        }

Commented out code

There are some values and some code commented out. Why? If it's not needed, just remove it.

Values:

$mascow_sub_area = array(//'Aviamotornaya','Avtozavodskaya','Akademicheskaya','Aleksandrovskiy_Sad',

Code:

$categories = array(1=>"1",2=>"2");//array(1=>array('pizza','sushi','shashliki','pirogi','burger'),2=>array('farm','dairy','delicatessen','confectionery','gastronomy'));    $categories = array(1=>"1",2=>"2");//array(1=>array(

Hard-coded arbitrary values

You hard-coded dozens of arbitrary areas into the massive $mascow_sub_area array, then you make PHP iterate over each value in multiple arrays to check for conditions. I would suggest that you make use of MySQL more and store your values there. This would have certain advantages:

  1. Easy to add, update and remove values from a table without having to change the PHP script at all.

  2. Takes advantage of the speed of SQL query optimizer to fetch and compare data.

  3. Then just pass the result set back to PHP.

And that brings me to...

Wrong tool for the job.

Use your database!

What you are doing, I'm sure you realize, is iterating through arrays multiple levels deep, comparing data. I can't imagine that being very fast at all. As was commented:

You have up to five levels of nesting in foreach loop. Its Big O can't be good. – CodeWorks Apr 30 at 11:39

You can calculate with up to 5 levels of nesting you would get up to \$O(n^5)\$. I have not counted all those area names but it must be at least 30, probably a lot more... going through 3 levels of nesting \$O(30^3) = 27 000\$

Granted, this is oversimplified and probably not that bad, in concept it's a pretty scary Big O.

I try to keep this in mind:

hammer

To a RDBMS like MySQL, comparing multiple arrays worth of data using tables is trivial, because it does so as a set rather than one by one. I've had a similar problem, but in reverse using loops in SQL, and once I had the code reviewed someone suggested to do it by set instead, and it sped it up dramatically.

To me, I would see $mascow_sub_area, $categories, and maybe even your URL concatenation logic be set-based in SQL, though I'm not sure that specifically would be best tool for the job.

Example:

create table mascow_sub_area(
    area_id int not null identity,
    area_name varchar(255) not null
);
insert into mascow_sub_area(area_name) values
('Atinskaya'),('Altufevo'),('Annino') -- etc.
\$\endgroup\$
  • 1
    \$\begingroup\$ The database suggestion is good, but I would not say that the loop is a real O(n) as it's not looping through the same array. \$\endgroup\$ – Simon Forsberg Sep 9 '14 at 22:38
  • \$\begingroup\$ The performance analysis is sloppy. The runtime will likely be dominated by the fetching and inserting, both of which are probably unavoidable. The HTML parsing and descent into the DOM tree is fine — each inner loop will only look at the relevant portion of the document. Runtime should be nowhere near "\$O(n^5)\$" — whatever \$n\$ means (you didn't specify). It's probably more like \$O(N(S+L))\$, where \$S\$ is the number of subway stations, \$L\$ is the total number of store locations, and \$N\$ is the average number of DOM nodes per page. \$\endgroup\$ – 200_success Sep 9 '14 at 22:51
  • \$\begingroup\$ @200_success would you mind to edit that into the answer properly? I was thinking of \$n\$ as being the number of values in the $mascow_sub_area array, who knows how many that is so I made a conservative guess of 30. I just don't understand what you commented enough to do it justice... \$\endgroup\$ – Phrancis Sep 9 '14 at 23:14
  • \$\begingroup\$ Consider striking out the complexity analysis from your answer then. Probably the only thing remarkable to say about the code's complexity is that the code is structured in a way that makes it hard to see where the complexity lies. \$\endgroup\$ – 200_success Sep 9 '14 at 23:19
  • \$\begingroup\$ Fair enough. I did that. \$\endgroup\$ – Phrancis Sep 9 '14 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.