1
\$\begingroup\$

I have 2 datasets (one for training and the other for testing) containing information about days temperature and humidity; My programm should process the training dataset and find a relation between both, and then predicts humidity values from test dataset, after processing temperature values.

My training dataset has almost 97.000 rows of examples, but I only got 42% of accuracy. Maybe because weather is so complex to measure, or it's the program. Any tips for improvement are very welcome.

Training Dataset: https://drive.google.com/file/d/1d-jGkFlM6_Wf01UUZGH1mDbDAEyypgnL/view?usp=sharing

Testing Dataset: https://drive.google.com/file/d/1wRb-rufT046q7hR83l2IKcCB-raZYhLW/view?usp=sharing

import numpy as np
import pandas as pd
import sklearn as sk
from sklearn import linear_model


df_train = pd.read_csv("path\\weather_train.csv")
df_test = pd.read_csv("path\\weather_test.csv")


x_train = np.array(df_train['Temperature (C)']).reshape(-1, 1)
y_train = np.array(df_train['Humidity']).reshape(-1, 1)

x_test = np.array(df_test['Temperature (C)']).reshape(-1, 1)
y_test = np.array(df_test['Humidity']).reshape(-1, 1)

#The Model
algorithm = linear_model.LinearRegression()
algorithm.fit(x_train,y_train)

#Here it will predict humidity (Y values) from Temperature (X values) of test dataset and get precision%
print(algorithm.predict(x_test))
accu = algorithm.score(x_test, y_test)

print("==============================")
print(f"Accuracy: {accu * 100}%")
print("==============================")
\$\endgroup\$
4
\$\begingroup\$

To test whether the libraries do what you would expect simply create a very simple dataset (e.g. y = 2x + normalerror) which OLS should deal with without any issues.

From my understanding of the documentation, what you call "accuracy" here (sklearn.linear_model.LinearRegression score method) is R2 calculated using data out of the training set. With this in mind, it would be worth thinking about what R2 should you expect.

R2 = 0.42 for regressing stock returns on information from previous time period is incredible and would make you a billionaire in no time.

R2 = 0.42 for regressing Force on Acceleration given some object of fixed mass is questionable and there is something wrong either with your data or model.

Is there a linear relationship between temperature and humidity? From some very light googling I found a claim that they are inversely proportional (although the source seems questionable). If that's true then OLS cannot capture their relationship (unless you transform them). Given that you have only one regressor, you can plot your data using scatterplot to gain some intuition about their relationship.

\$\endgroup\$
7
  • \$\begingroup\$ Great answer. Yes it seems that they are inversely proportional, so I will test on some simple problems (4x + 50 or something like that) to get a better idea where's the problem. But you said that maybe I should transform the data. What do you mean by that? \$\endgroup\$
    – irtexas19
    Jan 3 at 13:20
  • \$\begingroup\$ What I mean by that is that if you believe that the relationship is inversely proportional, transform x to 1/x. \$\endgroup\$ Jan 3 at 13:29
  • 1
    \$\begingroup\$ You need to make a scatter plot and not a line plot \$\endgroup\$
    – Reinderien
    Jan 3 at 13:46
  • 1
    \$\begingroup\$ Yes, it looks linear. But it's to "fat",variable, specially on center. Could linear regression really give something like 80% precision? \$\endgroup\$
    – irtexas19
    Jan 3 at 14:36
  • 1
    \$\begingroup\$ I've never heard anyone refer to heteroskedasticity as "fat" lol. Incredible stuff. I recommend looking into theory behind OLS. This channel is very good: youtube.com/… \$\endgroup\$ Jan 3 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.