0
\$\begingroup\$

I've been trying to create a toxic comment 'rater' from this Kaggle challenge. The training data consists of a number of comments with scorers on whether it was toxic, severely toxic, obscene, a threat, an insult, or identity hate.

I first created my target 'overall_negativity' which is a cumulative sum of different ratings of toxicity in comments given in the dataframe. I also pre-process the dataframe (this part is not problematic).

import pandas as pd
import os
import tensorflow as tf

train= pd.read_csv('files/jigsaw-toxic-comment-classification-challenge/train.csv')

negative_columns = ['toxic','severe_toxic','obscene','threat','insult','identity_hate']

train['overall_negativity'] = 0

train['overall_negativity'] = train[negative_columns].sum(axis = 1)

from preprocessing.preprocess_text import pre_process_text
from tqdm import tqdm

train['comment_text'] = [pre_process_text(text) for text in tqdm(train['comment_text'])]

The training data looks like this after the comments are preprocessed: enter image description here

I then split up the training and validation sets, and try to reduce the number of 0 overall negativity comments to help with target balance a little bit and hedge against overfitting:

import numpy as np
train['overall_negativity'] = train['overall_negativity'] / np.max(train['overall_negativity'])

train = train.sample(frac=1)

import matplotlib.pyplot as plt
train = pd.concat([train[train['overall_negativity']>0] , 
                train[train.overall_negativity==0].sample(int(len(train[train.overall_negativity>0]))) ], axis=0).sample(frac=1)

x_train = train['comment_text'][:round(0.8 * len(train))]
y_train = train['overall_negativity'][:round(0.8 * len(train))]

x_val = train['comment_text'][round(0.8 * len(train)):]
y_val = train['overall_negativity'][round(0.8 * len(train)):]

Then the model. I was following this guide, just trying to see if it could apply to my usage case. I mostly just copied the setup for the neural net here in hopes that given my input is the same and I want to accomplish a similar task, I should maybe see encouraging results.

from tensorflow.keras.models import Sequential
from tensorflow.keras import Input

model = Sequential()
model.add(Input(shape=(1,), dtype="string"))

from tensorflow.keras.layers.experimental.preprocessing import TextVectorization


max_tokens = 10000
max_len = 200
vectorize_layer = TextVectorization(
  # Max vocab size. Any words outside of the max_tokens most common ones
  # will be treated the same way: as "out of vocabulary" (OOV) tokens.
  max_tokens=max_tokens,
  # Output integer indices, one per string token
  output_mode="int",
  # Always pad or truncate to exactly this many tokens
  output_sequence_length=max_len,
)

vectorize_layer.adapt(x_train)

model.add(vectorize_layer)

from tensorflow.keras.layers import Embedding

# Note that we're using max_tokens + 1 here, since there's an
# out-of-vocabulary (OOV) token that gets added to the vocab.
model.add(Embedding(max_tokens + 1, 128))

from tensorflow.keras.layers import LSTM

# 64 is the "units" parameter, which is the
# dimensionality of the output space.
model.add(LSTM(64, dropout=0.25))

from tensorflow.keras.layers import Dense

model.add(Dense(64, activation="relu"))
model.add(Dense(1, activation="relu"))

model.compile(
  optimizer='adam',
  loss='binary_crossentropy',
  metrics=['accuracy'],
)

model.fit(x_train,y_train,validation_data = (x_val, y_val), epochs=25, callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_loss',patience=3, mode = 'max',verbose = 1, baseline = None))

However, after testing two values which should have definitely different results:

print(model.predict([
  "i loved it! highly recommend it to anyone and everyone looking for a great movie to watch.",

[[0.1817187]]

print(model.predict([
  "I thought this kinda sucked tbh. I thought it was completely bad",
]))

[[0.1817187]]

They have the exact same toxicity score, despite the model fitting without giving me an error. How did this happen?

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.