1
\$\begingroup\$

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use stochastic gradient descent I'm getting really poor accuracy so I'm not sure if it's my learning rate, epochs or just my code is incorrect. Also I'm wondering how would I add regularization to both of these? Do I add a variable lambda and multiply it by the learning rate or is the more to it?

BGD:

def batch_gradient(df, weights, bias, lr, epochs):
    X = df.values
    y = X[:,:1]
    X = X[:,1:]
    length = X.shape[0]
    for i in range(epochs):
        output = (sigmoid((np.dot(weights, X.T)+bias)))
        weights_tmp = (1/length) * (np.dot(X.T, (output - y.T).T))
        bias_tmp = (1/length) * (np.sum(output - y.T)) 

        weights -= (lr * (weights_tmp.T))
        bias -= (lr * bias_tmp)

    return weights, bias

SGD:

def stochastic_gradient(df, weights, bias, lr, epochs):
    x_matrix = df.values
    for i in range(epochs):
        np.random.shuffle(x_matrix)
        x_instance = x_matrix[np.random.choice(x_matrix.shape[0], 1, replace=True)]
        y = x_instance[:,:1]

        output = sigmoid(np.dot(weights, x_instance[:,1:].T) + bias)    
        weights_tmp = lr * np.dot(x_instance[:,1:].T, ((output - y)))

        weights = (weights - weights_tmp.T)
        bias -= lr * (output - y)

    return weights, bias
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.