2
\$\begingroup\$

I am working on a multiple classification problem and after dabbling with multiple neural network architectures, I settled for a stacked LSTM structure as it yields the best accuracy for my use-case. Unfortunately the network takes a long time (almost 48 hours) to reach a good accuracy (~1000 epochs) even when I use GPU acceleration.

At this point, I would like some feedback about my code and my results, did I miss something? and hopefully some suggestions on how to further speed-up and improve the training process/code.

My code is the following:

# -*- coding: utf-8 -*-
import keras
import numpy as np
from time import time
from utils import dmanip, vis
from keras.models import Sequential
from keras.layers import LSTM, Dense
from keras.utils import to_categorical
from keras.callbacks import TensorBoard
from sklearn.preprocessing import LabelEncoder
from tensorflow.python.client import device_lib
from sklearn.model_selection import train_test_split

###############################################################################
####################### Extract the data from .csv file #######################
###############################################################################
# get data
data, column_names = dmanip.get_data(file_path='../data_one_outcome.csv')

# split data
X = data.iloc[:, :-1]
y = data.iloc[:, -1:].astype('category')

###############################################################################
########################## init global config vars ############################
###############################################################################
# check if GPU is used
print(device_lib.list_local_devices())

# init
n_epochs = 1500
n_comps = X.shape[1]

###############################################################################
################################## Keras RNN ##################################
###############################################################################
# encode the classification labels
le = LabelEncoder()
yy = to_categorical(le.fit_transform(y))

# split the dataset
x_train, x_test, y_train, y_test = train_test_split(X, yy, test_size=0.35,
                                                    random_state=True,
                                                    shuffle=True)

# exapand dimensions
x_train = np.expand_dims(x_train, axis=2)
x_test = np.expand_dims(x_test, axis=2)

# define model
model = Sequential()
model.add(LSTM(units=n_comps, return_sequences=True,
               input_shape=(x_train.shape[1], 1),
               dropout=0.2, recurrent_dropout=0.2))
model.add(LSTM(64, return_sequences=True, dropout=0.2, recurrent_dropout=0.2))
model.add(LSTM(32, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(4 ,activation='softmax'))

# print model architecture summary
print(model.summary())

# compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Create a TensorBoard instance with the path to the logs directory
tensorboard = TensorBoard(log_dir='./logs/rnn/{}'.format(time()))

# fit the model
history = model.fit(x_train, y_train, epochs=n_epochs, batch_size=100,
                    validation_data=(x_test, y_test), callbacks=[tensorboard])

# plot results
vis.plot_nn_stats(history=history, stat_type="accuracy", fname="RNN-accuracy")
vis.plot_nn_stats(history=history, stat_type="loss", fname="RNN-loss")

The resulting accuracy and loss functions are:

My data is essentially a big matrix (38607, 150), where 150 is the number of features and 38607 is the number of samples, with a target vector including 4 classes.

\$\endgroup\$
  • \$\begingroup\$ Hi, if you have an NVIDIA GPU card then you could use the cudnnLSTM to speed up your network training time. However, cudnnLSTM has fewer options compared to the LSTM. \$\endgroup\$ – Nestoras Chalkidis Dec 19 '19 at 10:31
  • \$\begingroup\$ You'd be better off posting this exact same question in the Keras subreddit, for example. I'm not sure you'd get the advice you need in this general code review site. \$\endgroup\$ – Zchpyvr Dec 31 '19 at 21:22
1
\$\begingroup\$

If I understand your problem, your samples are 150-dimensional vectors which is to be classified into one of four categories. If so, an LSTM architecture is a very poor fit because there are no relations between your samples. I.e what is in the n:th sample doesn't impact what is in the n + 1:th sample.

Try something like this instead:

from keras.layers import Dense
from keras.models import Sequential
from keras.utils import to_categorical
import numpy as np

n = 38607
d = 150
k = 4

# Generate data
X = np.random.randn(n, d)
Y = to_categorical(np.random.randint(k, size = n), k)

model = Sequential()
model.add(Dense(128, input_dim = d, activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(4, activation = 'softmax'))
model.compile(
    loss = 'binary_crossentropy',
    optimizer='adam', metrics=['accuracy'])
model.summary()

model.fit(X, Y, epochs = 100, batch_size = 128, verbose = 1)

I haven't used any test data so the model quickly overfits (accuracy > 25% implies overfitting). Which brings me to my next point. On your graphs, you get better performance on the test data than on the training data which is very suspect. I suggest you try and train your network without any dropout first to see if it behaves as expected before adding it back.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.