Good morning everyone,

I am trying to figure out how deep learning works. My approach is mainly theoretical but I have decided to code a few deep learning projects to get a better feel of the kind of work involved.

Most courses and textbooks out there recommend that your first project as an aspiring AI engineer should be a linear neural network that solves some simple problem. So, I have coded one such network with a single layer and I have trained it to solve a linear regression problem. I have used Dive into deep learning as a reference, but I have decided against developing the whole network from scratch as they do because I don't want to get bogged down in the details, and I'd much rather have a bird's-eye view of the field for the time being.

Thus, I tried to use the high level abstract tools provided by the Pytorch library. I hope you can tell me whether I have put such tools to good use or not. In particular, I would like to get feedback on the general architecture of the program and the OOP aspects of it. All advice is welcome, though.

The project is divided into 4 files.

  1. model.py defines the model architecture:
"""One layer linear neural network model to solve regression problems."""
import torch
from torch import nn

class LinearRegressionModel(nn.Module):
    """Linear neural network model used to solve linear regression problems."""

    def __init__(self, number_of_features: int):
        Constructor for LinearRegressionModel.

        :param number_of_features: number of features used to compute a label.

        # Randomly initialize as many weights as there are feature and randomly initialize a bias.
        self.weights = nn.Parameter(torch.rand(number_of_features), requires_grad=True)
        self.bias = nn.Parameter(torch.rand(1), requires_grad=True)

    def forward(self, instances: torch.Tensor) -> torch.Tensor:
        """Forward function for the neural network.

        Pass the input instances through a linear layer: return 'instances*weights + bias'.
        return torch.matmul(instances, self.weights) + self.bias
  1. data.py handles the synthetic data to train the model (I am using synthetic data for simplicity's sake):
"""Synthetic linear data generation."""
import torch

class SyntheticLinearDataset(torch.utils.data.Dataset):
    """Create synthetic linear data polluted by noise according to a normal distribution."""

    def __init__(self, weights: torch.Tensor, bias: torch.Tensor, noise: float, number_of_instances: int):
        Constructor for SyntheticLinearDataset.

        Store weights, bias, and noise as attributes. Generate the linear data and pollute it with noise.

        :param weights: True weights used to generate linear data. 'weights.size(0)' will be taken as the number of
            features in the generated data.
        :param bias: True bias used to generate linear data.
        :param noise: Standard deviation of the noise.
        :param number_of_instances: number of instances that are to be generated.
        # Store the parameters.
        self.weights = weights.unsqueeze(-1)
        self.bias = bias
        self.noise = noise

        # Generate the linear data and pollute it with noise.
        self.instances = torch.rand(number_of_instances, self.weights.size(0))
        self.labels = torch.matmul(self.instances, self.weights) + self.bias
        self.labels += (torch.randn(number_of_instances, self.weights.size(0))*self.noise)

    def __getitem__(self, item):
        """Overload indexing to allow DataLoader to retrieve the data."""
        return self.instances[item], self.labels[item]

    def __len__(self):
        """Return the length of the dataset."""
        return len(self.instances)
  1. visualization.py contains a function that allows me to draw a scatter plot to visualize the model's predictions:
"""Visualization tools to compare actual data with neural network predictions."""
import matplotlib.pyplot as plt
import torch
from torch import nn
from torch.utils.data import DataLoader

def plot_predictions(model: nn.Module, dataloader: DataLoader, title: str = None):
    """Take a random batch of data and plot the model-predicted labels against the actual labels.

    :param model: neural network.
    :param dataloader: dataloader used to load the random batch.
    :param title: title of the plot.
    :returns: None
    instances, labels = next(iter(dataloader))
    with torch.inference_mode():
        predictions = model(instances)

    plt.scatter(instances, labels, c='b', s=4, label="Actual data")
    plt.scatter(instances, predictions, c='r', s=4, label="Model predictions")
  1. main.py uses the classes and functions defined in the other files to actually generate the data, create the model, specify the hyperparameters, train the model, and finally plot the data:
"""Create a one layer linear neural network and train it to solve a linear regression problem with synthetic data."""
import torch
from torch import nn
from torch.utils.data import DataLoader, random_split
from data import SyntheticLinearDataset
from model import LinearRegressionModel
from visualization import plot_predictions

def training_loop(model: nn.Module, dataloader: DataLoader, number_of_epochs: int, loss_function, optimizer):
    """raining loop for LinearRegressionModel.

    Train the model using backpropagation and minibatch stochastic gradient descent (MSGD).

    :param model: model to be trained.
    :param dataloader: dataloader for the training data.
    :param number_of_epochs: number of training epochs.
    :param loss_function: loss function.
    :param optimizer: optimizer.
    :returns: None
    for epoch in range(number_of_epochs):
        # Apply MSGD
        for instances, labels in dataloader:
            predictions = model(instances).unsqueeze(-1)
            loss = loss_function(predictions, labels)

# Create and split dataset. Use an 80% vs 20% split for training data vs testing data.
number_of_features = 1
number_of_instances = 10000
weights = torch.rand(number_of_features)
bias = torch.rand(1)
noise = 0.01

dataset = SyntheticLinearDataset(weights, bias, noise, number_of_instances)
training_length = int(number_of_instances*0.8)
testing_length = number_of_instances - training_length
training_dataset, testing_dataset = random_split(dataset, [training_length, testing_length])

# Create model.
regression_model = LinearRegressionModel(number_of_features)

# Create hyperparameters.
learning_rate = 0.1
batch_size = 100
num_epochs = 100

# Create optimizer and loss function.
mse_loss_function = nn.MSELoss()
sgd_optimizer = torch.optim.SGD(params=regression_model.parameters(), lr=learning_rate)

# Create data loaders.
training_dataloader = DataLoader(training_dataset, batch_size, shuffle=True)
testing_dataloader = DataLoader(testing_dataset, batch_size, shuffle=True)

# Visualize predictions before training.
plot_predictions(regression_model, testing_dataloader, title="Before training")

# Execute training loop.
training_loop(regression_model, training_dataloader, num_epochs, mse_loss_function, sgd_optimizer)

# Visualize predictions after training.
plot_predictions(regression_model, testing_dataloader, title="After training")

A couple of questions that come to mind are:

  1. Should I import torch and its submodules in every file? Is it good practice to import a library so many times?
  2. Should I create a class to train my model, or is having a training_loop function a good approach?

Once again, thank you all for your priceless advice. As a self-learning student of programming, this forum is helping me so much! I hope I can get good enough to be in a position where I'm able to give back to this community.


1 Answer 1

  1. Should I import torch and its submodules in every file? Is it good practice to import a library so many times?

Yes. Yes.

If you need it, import it.

Don't worry, there's zero cost if you already imported it somewhere else in this process -- you'll just get a cache hit.

  1. Should I create a class to train my model, or is having a training_loop function a good approach?

I feel you're asking the wrong question.

Better to ask: Is training_loop() sufficient for our business needs?

Looks like the answer is "yes!" So we're done.

OTOH we might have needed several collaborating functions, which turn out to be passing (x,y,z), and (x,y,z), and always (x,y,z) around amongst themselves. Which suggests that those three parameters should perhaps be lumped together as a single object. Or perhaps we'd like to have a class whose objects can conveniently have methods refer to

  • self.x
  • self.y
  • self.z


Your main module defines nearly twenty global variables before it gets around to invoking training_loop(). Humans can maybe hold seven or nine items in their head at once, fewer if the items are on the abstract side. It would be useful to bury that code within def main():, if only so the local variables disappear once we return and then they will go out of scope.

Having done that, you might see opportunities to Extract Helper functions. Three dozen lines of source fits in a screenful and is not necessarily Too Long. But as written, the narrative is not very clear, it wanders between high- and low-level details, it does not remain at a single level of abstraction. What I'm looking for is a business problem to be solved, steps to accomplish that, and then decompositions into smaller problems that eventually are so trivial they are one-liners.

Often the way to get to that is to read each # comment, and decide whether it suggests that several lines should be bundled up as some def _helper function.

  • \$\begingroup\$ Thank you! I will implement your suggestions as soon as I have time to do so. \$\endgroup\$ Commented Apr 5 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.