10
\$\begingroup\$

Every day Duga reports the number of unanswered questions and percentage of answered questions (after she calls the API endpoint /info) in The 2nd Monitor.

I created a script to read the messages by her in the past few months and plot the percentage of answered questions on a line chart. A demonstration of it can be seen in this playground example.

What would you change? Obviously there are other techniques for fetching and parsing the data... but is this approach acceptable?

<?php
$dd = new DomDocument();
$internalErrors = libxml_use_internal_errors(true);
$html = file_get_contents('https://chat.stackexchange.com/search?q=%22RELOAD!+There+are%22&user=125580&room=8595&pagesize=150');
$data = [];
$labels = [];
if ($html) {
    $dd->loadHtml($html);
    $xml = simplexml_import_dom($dd);

    $messages = $xml->xpath('//div[@class="content"]');
    $timestamps = $xml->xpath('//div[@class="timestamp"]');

    if (count($messages) == count($timestamps)) {
        foreach(array_reverse($timestamps, true) as $index => $timestamp) {
            preg_match('#(\d{2}.\d{4})#',(string)$messages[$index], $matches);
            if (count($matches) > 1) {
                $labels[] = (string)$timestamp;
                $data[] = $matches[1];
            }
        }
    }
?>
    <html>

    <head>
        <script src="//www.chartjs.org/dist/2.7.2/Chart.bundle.js"></script>
        <script src="//www.chartjs.org/samples/latest/utils.js"></script>
    </head>

    <body>
        <div style="width:75%;">
            <canvas id="canvas"></canvas>
        </div>
        <script type="text/javascript">
            Chart.defaults.global.legend.display = false;
            const config = {
                type: 'line',
                data: {
                    labels: <?=json_encode($labels);?>,
                    datasets: [{
                        label: 'Unanswered Percentage',
                        backgroundColor: window.chartColors.red,
                        borderColor: window.chartColors.red,
                        data: <?=json_encode($data);?>,
                        fill: false
                    }]
                },
                options: {
                    title: {
                        display: true,
                        text: 'Answered percentage in recent months'
                    },
                    scales: {
                        xAxes: [{
                            display: true,
                            scaleLabel: {
                                display: true,
                                labelString: 'Date'
                            }
                        }],
                        yAxes: [{
                            display: true,
                            scaleLabel: {
                                display: true,
                                labelString: 'Percentage'
                            }
                        }]
                    }
                }
            };
            document.addEventListener('DOMContentLoaded', _ => {
                var ctx = document.getElementById('canvas').getContext('2d');
                window.myLine = new Chart(ctx, config);
            });
        </script>
    </body>

    </html>
<?php
}
else {
    echo 'Unable to fetch messages';
}
?>

Sample output:

Chart.defaults.global.legend.display = false;
const config = {
  type: 'line',
  data: {
    labels: ["Mar 11 12:00 AM", "Mar 12 12:00 AM", "Mar 13 12:00 AM", "Mar 14 12:00 AM", "Mar 15 12:00 AM", "Mar 16 12:00 AM", "Mar 17 12:00 AM", "Mar 18 12:00 AM", "Mar 19 12:00 AM", "Mar 20 12:00 AM", "Mar 21 12:00 AM", "Mar 22 12:00 AM", "Mar 23 12:00 AM", "Mar 24 12:00 AM", "Mar 25 12:00 AM", "Mar 26 12:00 AM", "Mar 27 12:00 AM", "Mar 28 12:00 AM", "Mar 29 12:00 AM", "Mar 30 12:00 AM", "Mar 31 12:00 AM", "Apr 1 12:00 AM", "Apr 2 12:00 AM", "Apr 3 12:00 AM", "Apr 4 12:00 AM", "Apr 5 12:00 AM", "Apr 6 12:00 AM", "Apr 7 12:00 AM", "Apr 8 12:00 AM", "Apr 9 12:00 AM", "Apr 10 12:00 AM", "Apr 11 12:00 AM", "Apr 12 12:00 AM", "Apr 13 12:00 AM", "Apr 14 12:00 AM", "Apr 15 12:00 AM", "Apr 16 12:00 AM", "Apr 17 12:00 AM", "Apr 18 12:00 AM", "Apr 19 12:00 AM", "Apr 20 12:00 AM", "Apr 22 12:00 AM", "Apr 23 12:00 AM", "Apr 24 12:00 AM", "Apr 25 12:00 AM", "Apr 26 12:00 AM", "Apr 27 12:00 AM", "Apr 28 12:00 AM", "Apr 29 12:00 AM", "Apr 30 12:00 AM", "May 1 12:00 AM", "May 2 12:00 AM", "May 3 12:00 AM", "May 4 12:00 AM", "May 5 12:00 AM", "May 6 12:00 AM", "May 7 12:00 AM", "May 8 12:00 AM", "May 9 12:00 AM", "May 10 12:00 AM", "May 11 12:00 AM", "May 12 12:00 AM", "May 13 12:00 AM", "May 14 12:00 AM", "May 15 12:00 AM", "May 16 12:00 AM", "May 17 12:00 AM", "May 18 12:00 AM", "May 19 12:00 AM", "May 20 12:00 AM", "May 21 12:00 AM", "May 22 12:00 AM", "May 23 12:00 AM", "May 24 12:00 AM", "May 25 12:00 AM", "May 26 12:00 AM", "May 27 12:00 AM", "May 28 12:00 AM", "May 29 12:00 AM", "May 30 12:00 AM", "May 31 12:00 AM", "Jun 1 12:00 AM", "Jun 2 12:00 AM", "Jun 3 12:00 AM", "Jun 4 12:00 AM", "Jun 5 12:00 AM", "Jun 6 12:00 AM", "Jun 7 12:00 AM", "Jun 8 12:00 AM", "Jun 9 12:00 AM", "Jun 10 12:00 AM", "Jun 11 12:00 AM", "Jun 12 12:00 AM", "Jun 13 12:00 AM", "Jun 14 12:00 AM", "Fri 12:00 AM", "Sat 12:00 AM", "Sun 12:00 AM", "Mon 12:00 AM", "yst 12:00 AM", "12:00 AM"],
    datasets: [{
      label: 'Answered Percentage',
      backgroundColor: window.chartColors.red,
      borderColor: window.chartColors.red,
      data: ["90.0229", "90.0027", "89.9802", "89.9748", "89.9868", "89.9923", "89.9685", "89.9861", "90.0081", "89.9973", "89.9849", "90.0006", "89.9790", "89.9817", "89.9911", "89.9979", "90.0127", "90.0040", "90.0019", "90.0069", "89.9896", "89.9731", "89.9829", "89.9804", "89.9781", "89.9643", "89.9741", "89.9783", "89.9808", "89.9760", "89.9895", "90.0025", "90.0084", "89.9738", "89.9727", "90.0107", "90.0017", "90.0162", "90.0069", "90.0036", "90.0080", "90.0120", "90.0086", "90.0099", "90.0004", "90.0032", "89.9888", "89.9899", "89.9688", "89.9671", "89.9620", "89.9778", "89.9668", "89.9790", "89.9968", "90.0233", "90.0313", "90.0419", "90.0261", "90.0170", "90.0119", "90.0255", "90.0359", "90.0234", "90.0302", "90.0006", "90.0009", "89.9923", "89.9883", "90.0026", "90.0047", "89.9938", "90.0006", "89.9940", "89.9750", "89.9927", "90.0197", "90.0032", "90.0248", "90.0182", "90.0248", "90.0349", "90.0272", "90.0409", "90.0289", "90.0345", "90.0425", "90.0421", "90.0376", "90.0264", "90.0095", "89.9877", "89.9987", "89.9850", "89.9815", "89.9815", "89.9681", "89.9765", "89.9815", "89.9830", "89.9776"],
      fill: false
    }]
  },
  options: {
    title: {
      display: true,
      text: 'Answered percentage in recent months'
    },
    scales: {
      xAxes: [{
        display: true,
        scaleLabel: {
          display: true,
          labelString: 'Date'
        }
      }],
      yAxes: [{
        display: true,
        scaleLabel: {
          display: true,
          labelString: 'Percentage'
        }
      }]
    }
  }
};
document.addEventListener('DOMContentLoaded', _ => {
  var ctx = document.getElementById('canvas').getContext('2d');
  window.myLine = new Chart(ctx, config);
});
<script src="//www.chartjs.org/dist/2.7.2/Chart.bundle.js"></script>
<script src="//www.chartjs.org/samples/latest/utils.js"></script>
<div style="width:75%;">
  <div class="chartjs-size-monitor" style="position: absolute; left: 0px; top: 0px; right: 0px; bottom: 0px; overflow: hidden; pointer-events: none; visibility: hidden; z-index: -1;">
    <div class="chartjs-size-monitor-expand" style="position:absolute;left:0;top:0;right:0;bottom:0;overflow:hidden;pointer-events:none;visibility:hidden;z-index:-1;">
      <div style="position:absolute;width:1000000px;height:1000000px;left:0;top:0"></div>
    </div>
    <div class="chartjs-size-monitor-shrink" style="position:absolute;left:0;top:0;right:0;bottom:0;overflow:hidden;pointer-events:none;visibility:hidden;z-index:-1;">
      <div style="position:absolute;width:200%;height:200%;left:0; top:0"></div>
    </div>
  </div>
  <canvas id="canvas" width="481" height="240" class="chartjs-render-monitor" style="display: block; width: 481px; height: 240px;"></canvas>
</div>

\$\endgroup\$

1 Answer 1

Reset to default
10
\$\begingroup\$

When I ran your script and started digging through the dom, I noticed that Apr 21 12:00 AM was missing -- I wonder why that was.

Anyhow, I wanted to encourage you to adjust the timestamp values because they are irregularly formatted. Then I started to play with iterated strtotime() calls to generate consistent Y-m-d stamps, but some of the strings needed to be prepared, so I wrote some conditionals and it just felt like hacky bloat. If you are happy with the time strings in your graph, I'll just leave that bit alone.

As a matter of directness, I'll recommend:

$dd->loadHTMLFile('https://chat.stackexchange.com/search?q=%22RELOAD!+There+are%22&user=125580&room=8595&pagesize=150');

Rather than the two step:

$html = file_get_contents('https://chat.stackexchange.com/search?q=%22RELOAD!+There+are%22&user=125580&room=8595&pagesize=150');
$dd->loadHtml($html);

The regex block can be tightened up slightly. Write the preg_match() in a conditional so that you don't need to count(). A capture group is not necessary and a literal dot is more accurate than an [any character] dot.

if (preg_match('#\d{2}\.\d{4}#', (string)$messages[$index], $match)) {
    $labels[] = (string)$timestamp;
    $data[] = $match[0];
}

These are very small adjustments, so I'd say your work is fine.

p.s. I did also toy with //div[@class="messages"] so that I was guaranteed to be processing pairs of data (by making subsequent xpath calls to isolate the content and timestamp child elements). However, I abandoned that process, because it was adding too much complexity to an originally simple task and the effort (mine & php's) didn't seem to be worth the validation gains.

\$\endgroup\$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.