Linear Regression in Scikit_learn

Question

I have 2 datasets (one for training and the other for testing) containing information about days temperature and humidity; My programm should process the training dataset and find a relation between both, and then predicts humidity values from test dataset, after processing temperature values.

My training dataset has almost 97.000 rows of examples, but I only got 42% of accuracy. Maybe because weather is so complex to measure, or it's the program. Any tips for improvement are very welcome.

Training Dataset: https://drive.google.com/file/d/1d-jGkFlM6_Wf01UUZGH1mDbDAEyypgnL/view?usp=sharing

Testing Dataset: https://drive.google.com/file/d/1wRb-rufT046q7hR83l2IKcCB-raZYhLW/view?usp=sharing

import numpy as np
import pandas as pd
import sklearn as sk
from sklearn import linear_model


df_train = pd.read_csv("path\\weather_train.csv")
df_test = pd.read_csv("path\\weather_test.csv")


x_train = np.array(df_train['Temperature (C)']).reshape(-1, 1)
y_train = np.array(df_train['Humidity']).reshape(-1, 1)

x_test = np.array(df_test['Temperature (C)']).reshape(-1, 1)
y_test = np.array(df_test['Humidity']).reshape(-1, 1)

#The Model
algorithm = linear_model.LinearRegression()
algorithm.fit(x_train,y_train)

#Here it will predict humidity (Y values) from Temperature (X values) of test dataset and get precision%
print(algorithm.predict(x_test))
accu = algorithm.score(x_test, y_test)

print("==============================")
print(f"Accuracy: {accu * 100}%")
print("==============================")

Answer 1

To test whether the libraries do what you would expect simply create a very simple dataset (e.g. y = 2x + normalerror) which OLS should deal with without any issues.

From my understanding of the documentation, what you call "accuracy" here (sklearn.linear_model.LinearRegression score method) is R² calculated using data out of the training set. With this in mind, it would be worth thinking about what R² should you expect.

R² = 0.42 for regressing stock returns on information from previous time period is incredible and would make you a billionaire in no time.

R² = 0.42 for regressing Force on Acceleration given some object of fixed mass is questionable and there is something wrong either with your data or model.

Is there a linear relationship between temperature and humidity? From some very light googling I found a claim that they are inversely proportional (although the source seems questionable). If that's true then OLS cannot capture their relationship (unless you transform them). Given that you have only one regressor, you can plot your data using scatterplot to gain some intuition about their relationship.