2
\$\begingroup\$

I'm implementing KNN algorithm to predict outliers in network data which contains the following columns: source IP address, source port number, protocol and total bytes transferred. To achieve this, I'm using the Python library sklearn to predict response variable (total bytes) with regards to explanatory variables (source IP address, source port number, protocol).

However, the accuracy of prediction on test data is extremely low (0.21). Any recommendations to improve the accuracy score will be much appreciated.

import pandas as pd
from sklearn.neighbors import KNeighborsRegressor
import random
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from numpy.random import permutation
from sklearn.metrics import accuracy_score
import sys
import math
import netaddr as nt 
from netaddr import IPAddress

#Import csv into dataframe
df = pd.read_csv('data.csv')
df = df[['sourceaddress','sourceport','protocol','total_bytes']]

df_arr = df.as_matrix()

for i in range(df_arr.shape[0]):
    df_arr[i,0] = int(nt.IPAddress(df_arr[i,0]))

df = pd.DataFrame(df_arr)
df.columns = ['sourceaddress','sourceport','protocol','total_bytes']
df = df.astype('str')

# Design matrix X and target vector y
X = np.array(df.ix[:, 0:3])     
Y = np.array(df['total_bytes']) 

# split into train and test
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33, random_state=42)


knn = KNeighborsClassifier(n_neighbors=24)
# fitting the model
knn.fit(X_train, Y_train)

# prediction of the response variable
pred = knn.predict(X_test)
print pred

# evaluate accuracy
print accuracy_score(Y_test, pred)
\$\endgroup\$
1
  • 1
    \$\begingroup\$ Considering the code does not produce the intended result (a high enough accuracy), the code is not ready for review. On Code Review, we only review code that already works the way it should (producing the output it should). Please take a look at the help center. You already got a question, hopefully it helped you out, but please comply with the scope of the site next time. \$\endgroup\$
    – Mast
    Sep 10 '19 at 7:43
2
\$\begingroup\$

You mention accuracy, but talk about the number of bytes, so I'm going to assume that you apply a cut-off point to number of bytes and convert it to binary variable behind the screen. Now the answer:

The reason why the performance is so low is most probably lack of feature engineering. You should definitely inject as much domain knowledge as possible. Let's go over it step by step:

  1. You are, if I understand correctly one-hot encoding IP addressees - however, it could be done in more concise and pythonic way without casting dataframe to matrix:

    dummies = pd.get_dummies(df['sourceaddress'], prefix='ip')  # also called one-hot encoding
    df = df.merge(dummies, left_index=True)
    

    Protocol should be one-hot encoded as well, as it is a categorical variable.

    Now you might want to group IPs by geographic region - this will allow the algorithm to pick up any correlations between not only particular IP but a country or continent. That's how you can add such a feature:

    df['region'] = determine_region([df['sourceaddress'])
    

    If you write determine_region(ip) in a way that allows array-wide operations (in other words using numpy), it will be applied to whole dataset at once. This vectorization tends to be faster than .apply, especially for complex functions. And you should also one-hot encode the regions.

    Same probably applies to sourceport (not an expert on web technology, so making assumptions here) - there are certain ports that are used in general for some thing and not for others, so another feature that reflects that could be crafted and then one-hot encoded (all your features are categorical so far, tough luck).

    Now, speculating here, but maybe combination of region and sourceport is meaningful as well? Really think about anything and everything that might explain the value of your target and create a feature out of that. Then you could choose the most relevant features, using one of the feature selection techniques, but that's not nearly as important as creating features in the first place.

  2. Next, I expect that your dataset is highly imbalanced, which means that your algorithm will be biased towards majority class. What is typically done is rebalancing. Depending on the size of your dataset, you either upsample the minority class (with replacement obviosuly), or downsample the majority class (without replacement, that's important). An example of downsampling when majority class is ~total_bytes - so normal amount of bytes:

    df_majority = df[~df['total_bytes']]
    df_minority = df[df['total_bytes']]
    df_majority_downsampled = resample(df_majority,
                                   replace=False,
                                   n_samples=len(df_minority),
                                   random_state=0)  
    df_downsampled = pd.concat([df_majority_downsampled, df_minority])
    

    Now the amount of positive and negative examples is equal, so no bias.

    Then you should try to tune the hyper parameters. Good way to do that is gridsearch with crossvalidation. Docs are excellent, so no need to explain further here.

Doing all this should improve the performance of your model.

\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.