2
\$\begingroup\$

I'm currently working on a project involving time series analysis and have written the following code for the train-test split section. I'm particularly concerned about the correctness of the test_data variable. To generate the initial test sequences, I included some data points from the train data by using -prediction_days. This ensures that there is enough data to create these sequences.

Is this approach correct? The code works, but I want to know whether it is a correct approach especially in the test-data variable.

data = df[['Close']]
prediction_days = 60

train_data_len = int(len(data) * 0.8)
train_data = data[:train_data_len]
test_data = data[train_data_len - prediction_days:]

# Scale the data
scaler = MinMaxScaler(feature_range=(0, 1))

scaled_train_data = scaler.fit_transform(train_data)
scaled_test_data = scaler.transform(test_data)

for i in range(prediction_days, train_data_len):
    x_train.append(scaled_train_data[i-prediction_days:i, 0])
    y_train.append(scaled_train_data[i, 0])
    
x_train, y_train = np.array(x_train), np.array(y_train)
y_train_original = data[prediction_days:train_data_len]

# Prepare the test data
x_test = []

for i in range(prediction_days, len(test_data)):
    x_test.append(scaled_test_data[i-prediction_days:i, 0])

x_test = np.array(x_test)
y_test = data[train_data_len:].values
\$\endgroup\$
3
  • \$\begingroup\$ Welcome. Code is working, that's a good thing but it's up to you to find the correct approach. \$\endgroup\$ Commented Jul 7 at 18:21
  • 1
    \$\begingroup\$ After doing some research, there are people that use the same approach i have used, and some of them don't include the perdiction_days in the test_data. I'm not sure whether including it will create a data leakage. \$\endgroup\$
    – user284332
    Commented Jul 7 at 18:33
  • 5
    \$\begingroup\$ Please do not edit the question, especially the code, after an answer has been posted. Changing the question may cause answer invalidation. Everyone needs to be able to see what the reviewer was referring to. What to do after the question has been answered. \$\endgroup\$
    – pacmaninbw
    Commented Jul 14 at 16:01

1 Answer 1

2
\$\begingroup\$

model

There is some context missing. You didn't tell us anything about the inference model you're training. The details matter -- some models handle missing values more flexibly than others.

leakage

You are quite right to express doubts about this approach.

concerned about the correctness of the test_data variable. ... Is this approach correct?

No.

"Test" data is examples which the model never saw during training. We evaluate a trained model against such test data to estimate how well the model generalizes to out-of-sample instances. We're hoping that there is structure in the real world generative process, which the model successfully captured.

You didn't tell us about the class of hypotheses you're considering, but based on the 60-day rolling window I will assume the underlying process is well approximated by a smoothly differentiable function. For example, knowing recent history of temperatures would put us in a good position to predict tomorrow's daily high, or tomorrow's humidity. We find variation in values, without sudden wide swings.

test_data = data[train_data_len - prediction_days:]
...
    x_test.append(scaled_test_data[i-prediction_days:i, 0])

Those don't make sense. We have nearly 60 rows of information leakage there. We call this "cheating".

Put your paranoid hat on. Assume the model, in addition to maybe estimating some linear coefficients, contains a database. That is, it memorizes every row of training data, and has perfect recall of that data at inference time. And by hypothesis, these two results will be very very similar:

  • f(x[1000 : 1060]) --> y1
  • f(x[1001 : 1061]) --> y2

y1 was offered during training, and we ask that y2 be inferred during testing. We have discarded one historic row and introduced a single novel row, which will have limited impact on the target value. The sane thing for any model to do during testing would be start with first order prediction y1, and then maybe modify it according to the deleted and the added row.

We can measure accuracy across any test rows, with information leakage or no. But it's only a valid estimate of production accuracy if there was no leakage, and if the distribution of test data resembles production data.

creating datasets

The OP code seems to make the assumption that we always need a window of 60 valid examples available at inference time. If that's a solid assumption, then we should discard a window's worth of Y values, and have test data pick up after that, so that all test instances can have a 60-day lookback.

imputing

But perhaps your model is amenable to variable-length lookback windows. For example, instead of relying on a sum of 60 values, it might normalize by length and use the mean of 60 or fewer values. You might be able to specify the number of values, or it might suffice to give a prefix of some NaNs followed by observed values.

Notice that you could augment the training dataset. For example, in addition to \$N\$ 60-day instances, you could also supply \$N\$ rows of 30 NaNs plus 30 days' worth of values. This isn't cheating; it is exposing the model to the cruel fact that sometimes a portion of the data will be missing, and asking it to learn a robust model under those constraints.

There are other approaches to imputing. Given a "missing" prefix of some days, we might simply fill them all in with a replicated value from the first available day.

\$\endgroup\$
2
  • \$\begingroup\$ Please remove the post as the comments here didn't answer my question, perhaps they misunderstood it as another problem, it is misleading. \$\endgroup\$
    – user284332
    Commented Jul 16 at 19:11
  • \$\begingroup\$ Your question expressed particular concern “about the correctness of the test_data variable.“ I responded to that concern. I’m sorry that my perspective on it differs from your own. One of the functions of this site is to share viewpoints so folks may come to appreciate a greater perspective. \$\endgroup\$
    – J_H
    Commented Jul 16 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.