I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use stochastic gradient descent I'm getting really poor accuracy so I'm not sure if it's my learning rate, epochs or just my code is incorrect. Also I'm wondering how would I add regularization to both of these? Do I add a variable lambda and multiply it by the learning rate or is the more to it?
def batch_gradient(df, weights, bias, lr, epochs): X = df.values y = X[:,:1] X = X[:,1:] length = X.shape for i in range(epochs): output = (sigmoid((np.dot(weights, X.T)+bias))) weights_tmp = (1/length) * (np.dot(X.T, (output - y.T).T)) bias_tmp = (1/length) * (np.sum(output - y.T)) weights -= (lr * (weights_tmp.T)) bias -= (lr * bias_tmp) return weights, bias
def stochastic_gradient(df, weights, bias, lr, epochs): x_matrix = df.values for i in range(epochs): np.random.shuffle(x_matrix) x_instance = x_matrix[np.random.choice(x_matrix.shape, 1, replace=True)] y = x_instance[:,:1] output = sigmoid(np.dot(weights, x_instance[:,1:].T) + bias) weights_tmp = lr * np.dot(x_instance[:,1:].T, ((output - y))) weights = (weights - weights_tmp.T) bias -= lr * (output - y) return weights, bias