Pulling thread data from Invision power board from an external Java application

Question

What I am planning on doing is pulling the first post of my news section. This was made for VBulletin and now I need to add it for IPb.

package com.rigory.loader.main.services;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import com.rigory.loader.main.gui.MainFrame;
import com.rigory.loader.main.gui.SplashScreen;
import com.rigory.loader.main.services.data.ThreadData;
public class LatestThread {

String to VBulletin:

private final String WEBSITE_URL ="http://rigory.com/forums/forum/8-news-and-announcements/";

Regex search thread name:

private final String REGEX_THREAD_NAMES = "<font color=\"#0099FF\"><strong>(.*?)</a> ";

Regex search thread URL:

private final String REGEX_THREAD_URL = "border=\"0\" /> <a href=\"(.*)\">1</a>";

Regex search thread content (this used to be a developer font color in an HTML element, now we have it stored as a CSS element):

div > div.post_body > div.group_4.post.entry-content > p:nth-child(1)

private final String REGEX_THREAD_CONTENT ="<div class=\"blizzquote\">(.*)";

This variable is the index of your latest news post:

private final int THREADINDEX = 3;

private SplashScreen splash;

public LatestThread(SplashScreen splash){
    this.splash = splash;
}
public List<ThreadData> getData(){
    List<ThreadData> out = new ArrayList<ThreadData>();
    try{
        URL url = new URL(WEBSITE_URL);
        BufferedReader reader =  new BufferedReader(new InputStreamReader(url.openStream()));
        String inputLine;
        int count = 0;
        while ((inputLine = reader.readLine()) != null  && count < THREADINDEX) {
            ThreadData in = getDetails(inputLine);
            if(in != null){
                out.add(in);
                count++;
                this.splash.getProgressBar().setValue(this.splash.getProgressBar().getValue() + 33);
                this.splash.getDescLbl().setText("Loading News Data.");
                this.splash.repaint();
                this.splash.validate();
            }
        }
        reader.close();
    } catch (Exception e){
        e.printStackTrace();
    }

    return out;
}

Create a new ThreadData object from our regex strings. This method should be rewritten, rushed job.

@param line - our line from the URL

@return new ThreadData object

    private ThreadData getDetails(String line){
        Pattern pattern = Pattern.compile(REGEX_THREAD_NAMES);
        Matcher matcher = pattern.matcher(line);
        if(matcher.find()){
              String title = matcher.group(1);
              pattern = Pattern.compile(REGEX_THREAD_URL);
              matcher = pattern.matcher(line);
              if(matcher.find()){
                  String urlLink = matcher.group(1);
                    return new ThreadData(title,urlLink,getContent(urlLink)); 
              }
        }
        return null;
    }
    private String getContent(String urlLink){
          String output = null;
          try {
        URL url = new URL(urlLink);
        BufferedReader reader =  new BufferedReader(new InputStreamReader(url.openStream()));
        String inputLine;
        while ((inputLine = reader.readLine()) != null) {
               Pattern pattern = Pattern.compile(REGEX_THREAD_CONTENT);
               Matcher matcher = pattern.matcher(inputLine);
                while (matcher.find()) {
                    output = matcher.group();
                    output = output.replace("<div class=\"blizzquote\">","");
                    try {
                        output = output.substring(0, output.indexOf("</div>")-1);
                    } catch (Exception e){ }
                }
        }
        reader.close();
          } catch (Exception e){
              e.printStackTrace();
          }
        return output;
    }

}

This is the full file:

package com.rigory.loader.main.services;

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import com.rigory.loader.main.gui.MainFrame;
import com.rigory.loader.main.gui.SplashScreen;
import com.rigory.loader.main.services.data.ThreadData;

public class LatestThread {
    /** String to VBulletin */
    private final String WEBSITE_URL = "http://rigory.com/forums/forum/8-news-and-announcements/";
    /** Regex search Thread Name */
    private final String REGEX_THREAD_NAMES = "<font color=\"#0099FF\"><strong>(.*?)</a> ";
    /** Regex search Thread URL */
    private final String REGEX_THREAD_URL = "border=\"0\" /> <a href=\"(.*)\">1</a>";
    /** Regex search Thread Content */
    private final String REGEX_THREAD_CONTENT ="<div class=\"blizzquote\">(.*)";
    /** This variable is the index of your latest news post **/
    private final int THREADINDEX = 3;

    private SplashScreen splash;

    public LatestThread(SplashScreen splash){
        this.splash = splash;
    }
    public List<ThreadData> getData(){
        List<ThreadData> out = new ArrayList<ThreadData>();
        try{
            URL url = new URL(WEBSITE_URL);
            BufferedReader reader =  new BufferedReader(new InputStreamReader(url.openStream()));
            String inputLine;
            int count = 0;
            while ((inputLine = reader.readLine()) != null  && count < THREADINDEX) {
                ThreadData in = getDetails(inputLine);
                if(in != null){
                    out.add(in);
                    count++;
                    this.splash.getProgressBar().setValue(this.splash.getProgressBar().getValue() + 33);
                    this.splash.getDescLbl().setText("Loading News Data.");
                    this.splash.repaint();
                    this.splash.validate();
                }
            }
            reader.close();
        } catch (Exception e){
            e.printStackTrace();
        }

        return out;
    }
    /** 
     *  Create a new ThreadData Object from our regex strings
     *  This method should be rewritten, rushed job.
     * @param line - our line from the URL
     * @return new ThreadData object
     * 
     */
    private ThreadData getDetails(String line){
        Pattern pattern = Pattern.compile(REGEX_THREAD_NAMES);
        Matcher matcher = pattern.matcher(line);
        if(matcher.find()){
              String title = matcher.group(1);
              pattern = Pattern.compile(REGEX_THREAD_URL);
              matcher = pattern.matcher(line);
              if(matcher.find()){
                  String urlLink = matcher.group(1);
                    return new ThreadData(title,urlLink,getContent(urlLink)); 
              }
        }
        return null;
    }
    private String getContent(String urlLink){
          String output = null;
          try {
        URL url = new URL(urlLink);
        BufferedReader reader =  new BufferedReader(new InputStreamReader(url.openStream()));
        String inputLine;
        while ((inputLine = reader.readLine()) != null) {
               Pattern pattern = Pattern.compile(REGEX_THREAD_CONTENT);
               Matcher matcher = pattern.matcher(inputLine);
                while (matcher.find()) {
                    output = matcher.group();
                    output = output.replace("<div class=\"blizzquote\">","");
                    try {
                        output = output.substring(0, output.indexOf("</div>")-1);
                    } catch (Exception e){ }
                }
        }
        reader.close();
          } catch (Exception e){
              e.printStackTrace();
          }
        return output;
    }

}

This is the ThreadData.java file:

package com.rigory.loader.main.services.data;

public class ThreadData {
    private String title;
    private String url;
    private String post;

    public ThreadData(String title, String url, String post) {
        super();
        this.title = title;
        this.url = url;
        this.post = post;
    }

    public String getTitle() {
        return this.title;
    }
    public String getUrl() {
        return this.url;
    }
    @Override
    public String toString() {
        return "ThreadData [title=" + title + ", url=" + url + ", post=" + post
                + "]";
    }

    public String getPost() {
        return this.post;
    }
    public void setTitle(String title) {
        this.title = title;
    }
    public void setUrl(String url) {
        this.url = url;
    }
    public void setPost(String post) {
        this.post = post;
    }

}

Heslacher · Accepted Answer · 2015-01-21 08:24:29Z

ThreadData

There is no need to call super() inside the constructor. You should remove it.

LatestThread

In the getData() method

you are doing too many things. You should extract the manipulation of the SplashScreen to a separate method.
you should add finally to close the reader.
by reverting the if condition and using continue you can save horizontal space which makes your code more readable.
if you are using java 1.7 or above you can use the diamond operator like List<ThreadData> out = new ArrayList<>();.

implementing the mentioned points above leads to

public List<ThreadData> getData() {
    List<ThreadData> out = new ArrayList<>();
    BufferedReader reader = null;
    try {
        URL url = new URL(WEBSITE_URL);
        reader = new BufferedReader(new InputStreamReader(url.openStream()));
        String inputLine;
        int count = 0;
        while ((inputLine = reader.readLine()) != null && count < THREADINDEX) {
            ThreadData in = getDetails(inputLine);
            if (in == null) {
                continue;
            }

            out.add(in);
            count++;
            incrementProgressbar(33);
        }
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        if (reader != null) {
            reader.close();
        }
    }

    return out;
}

private void incrementProgressbar(int value) {
    this.splash.getProgressBar().setValue(this.splash.getProgressBar().getValue() + value);
    this.splash.getDescLbl().setText("Loading News Data.");
    this.splash.repaint();
    this.splash.validate();
}

In the getContent() method

you should add finally to close the reader.
you should fix the indention of the code
you are returning only the manipulated last match of the last line of the stream. If you wanted to do this, you should add a comment because it is not obvious.

ratchet freak · Accepted Answer · 2015-01-21 11:52:20Z

0

Patterns should be pre-compiled and held as a static field to avoid recompilation.

Using reluctant qualifiers we can match only up until the first closing tag.

private final static String REGEX_THREAD_CONTENT ="<div class=\"blizzquote\">(.*?)</div>";
private final static Pattern pattern = Pattern.compile(REGEX_THREAD_CONTENT);


Matcher matcher = pattern.matcher(inputLine);
while (matcher.find()) {
    output = matcher.group(1);
}

answered Jan 21, 2015 at 11:52

ratchet freak

12.8k21 silver badges46 bronze badges

\$\begingroup\$ Okay thank you both a lot, however I no longer need the blizzquote element and I'm attempting to pull the first posts in a certain news section, prior this was the method used for Vbulletin forums, I'm not exactly aiming to rewrite the code completely, but I am lost on figuring out how to do this on Invision power board, if anyone has any experience with IPB that would be awesome as I am only learning \$\endgroup\$
– Ezra Polter
Jan 21, 2015 at 17:53

Add a comment |

Stack Exchange Network

Pulling thread data from Invision power board from an external Java application

2 Answers 2

Your Answer

Not the answer you're looking for? Browse other questions tagged
java
multithreading
regex
or ask your own question.

Hot Network Questions

Pulling thread data from Invision power board from an external Java application

2 Answers 2

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged javamultithreadingregex or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
java
multithreading
regex
or ask your own question.