For a project I am using the yelp dataset (found here: https://www.yelp.com/dataset) to create a Hashset of all verbs, nouns and adjectives found in the restaurant reviews. I have it up and running using the stanford nlp pipeline, however it is quite slow (takes about 1 hour to process 10000 reviews) and the dataset contains a few million reviews. I am not an advanced programmer, I usually barely get it working so I really need help increasing the performance of my program. General coding advice is very much appreciated as well!
My code is as structured as follows: I have a MyCorpus class that has a function review_loader(). This function loads one review ( a json object) and puts the relevant data in a class named review. review contains a function that performs the pipeline operation and returns all nouns, verbs and adjectives of the review as a HashSet, I then add this hashset to a global hashset which will contain all nouns, verbs and adjectives of the yelp dataset.
Code for the relevant functions can be seen below:
Review.java
public class review {
private String text;
private String business_id;
private int stars;
private ArrayList<String> listOfSentences = new ArrayList<String>();
private ArrayList<String> pos_tags = new ArrayList<String>();
private HashSet<String> all_terms = new HashSet<String>();
public review() {
}
public HashSet<String> find_terms(StanfordCoreNLP pipeline) {
CoreDocument doc = new CoreDocument(text);
pipeline.annotate(doc);
for(int f = 0; f <doc.sentences().size(); f++) {
for (int d = 0; d < doc.sentences().get(f).tokens().size(); d++) {
String tag = doc.sentences().get(f).posTags().get(d);
CoreLabel word = doc.sentences().get(f).tokens().get(d);
if (tag.contains("VB") == true|| tag.contains("JJ") == true || tag.contains("NN") == true);{
String pattern ="[\\p{Punct}&&[^@',&]]";
// Create a Pattern object
Pattern r = Pattern.compile(pattern, Pattern.CASE_INSENSITIVE);
// Now create matcher object.
Matcher m = r.matcher(word.originalText());
if (m.find() || word.originalText() == "") {
} else {
all_terms.add(word.originalText());
}
}
}
}
return all_terms;
}
MyCorpus.java
public class MyCorpus{
private String filelocation_review;
private String filelocation_business;
private String filelocation_pos;
private ArrayList<String> restaurants = new ArrayList<String>();
private Set<String> allTerms = new HashSet<String>();
public MyCorpus(String filelocation_review, String filelocation_business, String filelocation_pos) {
this.filelocation_review = filelocation_review;
this.filelocation_business = filelocation_business;
this.filelocation_pos = filelocation_pos;
}
public void review_loader() throws FileNotFoundException, UnsupportedEncodingException {
int counter = 0;
Properties props = new Properties();
// set the list of annotators to run
props.setProperty("annotators", "tokenize,ssplit,pos,parse");
// set a property for an annotator, in this case the coref annotator is being
// set to use the neural algorithm
props.setProperty("coref.algorithm", "neural");
// build pipeline
StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
MaxentTagger tagger = new MaxentTagger(filelocation_pos);
InputStream is_r = new FileInputStream(filelocation_review);
Reader r_r = new InputStreamReader(is_r, "UTF-8");
Gson gson_r = new GsonBuilder().create();
JsonStreamParser p = new JsonStreamParser(r_r);
while (p.hasNext()) {
counter += 1;
JsonElement e = p.next();
if (e.isJsonObject()) {
review review = gson_r.fromJson(e, review.class);
// This if statement checks if the review belongs to a restaurant by matching the business id to a list of all business_id's of a restaurant created previously
if (restaurants.contains(review.get_id())) {
HashSet<String> review_terms = review.find_terms(pipeline);
allTerms.addAll(review_terms);
System.out.println("size:" + allTerms.size() + "reviews processed: " + counter);
}
}
}
public static void main(String args[]) throws IOException {
// WHEN YOU RUN THE FILE CHANGE THE 3 FILELOCATIONS OF THE MYCORPUS CLASS!
MyCorpus yelp_dataset = new MyCorpus("E:\\review.json", "E:\\business.json", "C:\\Users\\Ruben\\git\\Heracles\\stanford-postagger-2018-10-16\\models\\english-bidirectional-distsim.tagger");
ArrayList<String> restaurants = yelp_dataset.business_identifier();
yelp_dataset.review_loader();
}
If there is anything that is unclear or seems weird, please do ask and thank you for taking the time to read this question.
Kind regards, Ruben