3
\$\begingroup\$

I have a Django project that has an HTML page with a table on it.

The table can have thousands of rows in it. The problem is that, as a customer requirement, each row has 2 <select>'s with options for the user to choose from:

<table>
  <thead>
    <tr>
      <th>Column 1</th>
      <th>Column 2</th>
      <th>Column 3</th>
      <th>Column 4</th>
    </td>
  </thead>
  <tbody>
    {% for row in my_model.rows.all %}
      <tr>
        <td>{{ row.column_1 }}</td>
        <td>{{ row.column_2 }}</td>
        <td>
          <select autocomplete="off" data-id="{{ row.id }}">
            <option value="">--------</option>
            {% for short_name, long_name in column_3_choices %}
              <option value="{{ short_name }}" {% if short_name == row.column_3 %}selected{% endif %}>{{ long_name }}</option>
            {% endfor %}
          </select>
        </td>
        <td>
          <select autocomplete="off" data-id="{{ row.id }}">
            <option value="">--------</option>
            {% for short_name, long_name in column_4_choices %}
              <option value="{{ short_name }}" {% if short_name == row.column_4 %}selected{% endif %}>{{ long_name }}</option>
            {% endfor %}
          </select>
        </td>
    {% endfor %}
  </tbody>
</table>

Each <select> has a javascript event handler attached to it so that when it's changed, an AJAX request is sent to the server to update the database that the row should now have a different value for that column.

The code itself works, but because each table row contains so much extra HTML defining the possible options in the <select>'s, and which option is currently chosen, just the selects alone more than double the size of the table in terms of filesize.

Without the two <select> columns, when my HTML page loads, the browser's dev tools tells me the html page is about 1.5MB. With the two <select> columns, it's closer to 5MB.

Because there are multiple tables on the page, this effect compounds, and my final HTML page is about 10MB in size, and takes several seconds for the browser to download.

I use datatables to make the table with the thousands of rows manageable for the user, but the entire table still needs to be downloaded before datatables can make it presentable. There is a way to make datatables use server-side processing, so that the page doesn't need to load the entire table all at once, and I've used that successfully before on other tables, but that comes with its own difficulties.

Is there a simple way to stop this html size explosion? Or is server-side processing my only path?

\$\endgroup\$

1 Answer 1

4
\$\begingroup\$

Since your page already relies on Javascript, you could populate the combo boxes in Javascript as well. Basically you need to define two sets of key/value pairs:

  • column_3_choices
  • column_4_choices

You can have these embedded inside your HTML page, then use the onload event to populate the combo boxes on the browser.

This would reduce the amount of generated HTML browser-side. However performance could be a problem if you have so many rows but it's something you can try. To be honest I doubt that performance will be so much better but that is easy to try and doesn't take a lot of effort.

Regarding page size, I assume your webserver is sending gzipped content but if in doubt verify. Even if the final page is 10 Mb in size, it doesn't mean that 10 Mb were transferred or have to be transferred. It should be much less than that.

My gut feeling is that rendering a large page with forms is inevitably going to burden the browser (and the speed of rendering can vary a lot from one browser to another, so be sure to test with different browsers). It could even be the bottleneck in your application. You could test your site with cURL to simply measure the response time from your server, without any rendering of HTML on screen or Javascript code being executed. See how it does.

In my opinion, a table with thousands of rows is problematic. This is the thing I would want to change. Does the user really need all to see all that information at once? I would try a different approach: refresh the table with Ajax as per user filters. I am not sure this is what you call server-side processing, but maybe the datatable can be tweaked in that direction, at least you can add some handlers to improve it.

In the meantime, try to identify the bottleneck to decide where your efforts should go first:

  • how long it takes for your Django app to fetch the data (presumably from a table using SQL or ORM) => test the SQL individually or run an execution plan on your DBMS
  • how long it takes for the browser to fetch the page from the server => use curl for that
  • finally how long it takes for the browser to render the page after it's fetched => your browser should have some profiling tools, for example the developer tools in Firefox and probably Chrome.

Bonus: datatable has caching features of its own, maybe you could capitalize on them for your need?

\$\endgroup\$
1
  • \$\begingroup\$ When I enabled gzip compression, it did significantly reduce the download size of the page, but the loading time didn't change. Now it makes sense - my problem is likely the rendering time - not the download time. I'll do some more investigating to confirm using your suggestions. Thanks! \$\endgroup\$
    – John
    Mar 19, 2022 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.