I've been trying to create a toxic comment 'rater' from this Kaggle challenge. The training data consists of a number of comments with scorers on whether it was toxic, severely toxic, obscene, a threat, an insult, or identity hate.

I first created my target 'overall_negativity' which is a cumulative sum of different ratings of toxicity in comments given in the dataframe. I also pre-process the dataframe (this part is not problematic).

import pandas as pd
import os
import tensorflow as tf

train= pd.read_csv('files/jigsaw-toxic-comment-classification-challenge/train.csv')

negative_columns = ['toxic','severe_toxic','obscene','threat','insult','identity_hate']

train['overall_negativity'] = 0

train['overall_negativity'] = train[negative_columns].sum(axis = 1)

from preprocessing.preprocess_text import pre_process_text
from tqdm import tqdm

train['comment_text'] = [pre_process_text(text) for text in tqdm(train['comment_text'])]

The training data looks like this after the comments are preprocessed: enter image description here

I then split up the training and validation sets, and try to reduce the number of 0 overall negativity comments to help with target balance a little bit and hedge against overfitting:

import numpy as np
train['overall_negativity'] = train['overall_negativity'] / np.max(train['overall_negativity'])

train = train.sample(frac=1)

import matplotlib.pyplot as plt
train = pd.concat([train[train['overall_negativity']>0] , 
                train[train.overall_negativity==0].sample(int(len(train[train.overall_negativity>0]))) ], axis=0).sample(frac=1)

x_train = train['comment_text'][:round(0.8 * len(train))]
y_train = train['overall_negativity'][:round(0.8 * len(train))]

x_val = train['comment_text'][round(0.8 * len(train)):]
y_val = train['overall_negativity'][round(0.8 * len(train)):]

Then the model. I was following this guide, just trying to see if it could apply to my usage case. I mostly just copied the setup for the neural net here in hopes that given my input is the same and I want to accomplish a similar task, I should maybe see encouraging results.

from tensorflow.keras.models import Sequential
from tensorflow.keras import Input

model = Sequential()
model.add(Input(shape=(1,), dtype="string"))

from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

max_tokens = 10000
max_len = 200
vectorize_layer = TextVectorization(
  # Max vocab size. Any words outside of the max_tokens most common ones
  # will be treated the same way: as "out of vocabulary" (OOV) tokens.
  # Output integer indices, one per string token
  # Always pad or truncate to exactly this many tokens



from tensorflow.keras.layers import Embedding

# Note that we're using max_tokens + 1 here, since there's an
# out-of-vocabulary (OOV) token that gets added to the vocab.
model.add(Embedding(max_tokens + 1, 128))

from tensorflow.keras.layers import LSTM

# 64 is the "units" parameter, which is the
# dimensionality of the output space.
model.add(LSTM(64, dropout=0.25))

from tensorflow.keras.layers import Dense

model.add(Dense(64, activation="relu"))
model.add(Dense(1, activation="relu"))


model.fit(x_train,y_train,validation_data = (x_val, y_val), epochs=25, callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_loss',patience=3, mode = 'max',verbose = 1, baseline = None))

However, after testing two values which should have definitely different results:

  "i loved it! highly recommend it to anyone and everyone looking for a great movie to watch.",


  "I thought this kinda sucked tbh. I thought it was completely bad",


They have the exact same toxicity score, despite the model fitting without giving me an error. How did this happen?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.