I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use stochastic gradient descent I'm getting really poor accuracy so I'm not sure if it's my learning rate, epochs or just my code is incorrect. Also I'm wondering how would I add regularization to both of these? Do I add a variable lambda and multiply it by the learning rate or is the more to it?


def batch_gradient(df, weights, bias, lr, epochs):
    X = df.values
    y = X[:,:1]
    X = X[:,1:]
    length = X.shape[0]
    for i in range(epochs):
        output = (sigmoid((np.dot(weights, X.T)+bias)))
        weights_tmp = (1/length) * (np.dot(X.T, (output - y.T).T))
        bias_tmp = (1/length) * (np.sum(output - y.T)) 

        weights -= (lr * (weights_tmp.T))
        bias -= (lr * bias_tmp)

    return weights, bias


def stochastic_gradient(df, weights, bias, lr, epochs):
    x_matrix = df.values
    for i in range(epochs):
        x_instance = x_matrix[np.random.choice(x_matrix.shape[0], 1, replace=True)]
        y = x_instance[:,:1]

        output = sigmoid(np.dot(weights, x_instance[:,1:].T) + bias)    
        weights_tmp = lr * np.dot(x_instance[:,1:].T, ((output - y)))

        weights = (weights - weights_tmp.T)
        bias -= lr * (output - y)

    return weights, bias

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.