2
\$\begingroup\$

According to the first part: Forum crawler - counts statistics for words in chosen forum topic I take into account the review of Janos and created Iterator for my classes.

This is part of the whole App. I have a problem with dividing it into classes, I'm not sure about this architecture which I've provided.

Descirption:

Topic is the specific topic of the forum Page is the page of the forum The aim of this code is to iterate through pages in Topic and create Page for every page.

The code

// example of use:
public class AppMain{
public static void main(String[] args){
                String url = "https://4programmers.net/Forum/Newbie/210891-od_czego_zaczac_nauke_programowania";
        Topic topic = new Topic(url);

        PageIterator it = topic.iterator();
        //it.next();
        Page page = it.next();
}
}

...

/*
 * to consider: what if the view of the post will be changed from 10 to 50?
 */
public class Topic implements Iterable<Page> {

    private String urlBase;    // URL of the topic on forum
    private String pagePattern; // pattern whcich change while changing page number of the topic, for example: ?page=
    private PageIteratorEnum pageIteratorType; // type of the forum 4programmers? Codercity? etc.

    // to consider: put these variables into abstract class?

    public Topic(String url, String pagePattern) {
        this.urlBase = url;
        this.pagePattern = pagePattern;
    }

    public Topic(String url) {
        this.urlBase = url;
        this.pagePattern = "?page=";
    }

    public void setUrlBase(String html) {
        this.urlBase = html;
    }

    public String getUrlBase() {
        return urlBase;
    }

    public void setPagePattern(String pattern) {
        pagePattern = pattern;
    }

    public String getPagePattern() {
        return pagePattern;
    }

    public String toString() {
        return getUrlBase();
    }

    public void setIterator(PageIteratorEnum type){
        pageIteratorType = type;
    }

    @Override
    public PageIterator iterator() {
        if(pageIteratorType == null){
            throw new RuntimeException("First set iterator type by setIterator() method.");
        }
        return new PageIteratorFactoryImp1(this).produceIterator(pageIteratorType);
    }
}

...

import org.jsoup.nodes.Document;

public class Page {

    Document doc;

    public Page(Document doc){
        this.doc = doc;
    }
}

...

import java.util.Iterator;

public interface PageIterator extends Iterator<Page>{

    public boolean hasNext();

    public Page next();
}

...

public enum PageIteratorEnum {

    it4PROGRAMMERS, itCODERCITY;
}

...

public interface PageIteratorFactory {

    public PageIterator produceIterator(PageIteratorEnum type);
}

...

public class PageIteratorFactoryImp1 implements PageIteratorFactory{

    Topic topic;

    public PageIteratorFactoryImp1(Topic topic){
        this.topic = topic;
    }

    public PageIterator produceIterator(PageIteratorEnum type){
        if(type.equals("it4PROGRAMMERS")){
            return new TopicPageIterator4programmers(topic);
        }
        return null;
    }
}

...

import java.io.IOException;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class TopicPageIterator4programmers implements PageIterator {

    private Topic topic;
    private int count = 0;
    String urlBase;
    String pagePattern;
    String browser = "Mozilla/5.0";
    NavigatePanel navigation;

    public TopicPageIterator4programmers(Topic topic) {
        this.topic = topic;
        this.urlBase = topic.getUrlBase();
        this.pagePattern = topic.getPagePattern();
        this.navigation = new NavigatePanel4programmers(urlBase, pagePattern);
    }

    @Override
    public boolean hasNext() {
        return ((count + 1) <= navigation.getMaxPageNumber());
    }

    @Override
    public Page next() {
        Page result = null;
        String newUrl = urlBase + pagePattern + String.valueOf(++count);
        Document doc = null;
        try {
            doc = Jsoup.connect(newUrl).userAgent(browser).get();
            result = new Page4programmers(doc, count);
        } catch (IOException e) {
            System.out.println("something went wrong with downloading the page: " + newUrl);
            e.printStackTrace();
        }
        return result;

    }

}
\$\endgroup\$
1
\$\begingroup\$

If this code is on github, it would be useful to link to it.

Your main does much less than I would expect. The advantage of implementing iterable is so you can concisely express something like this:

for (Page page : new Topic(url)) {
    // do stuff with page
}

It seems reasonable to me that you might pass Topic both a website url and a class that knows about structure of that site's urls.

I wouldn't expose setters & getters for urlBase & pagePattern until some unit test forces you to do so. Based on the for loop above, it's not obvious to me that caller would ever need to interact with Topic, since it's just a means to an end, which is obtaining Pages. Oh, I see, down in the iterator you're wishing for lots of access, sigh.

Exposing setIterator() does not seem like a wise API choice, as the ctor should have dealt with that. But it doesn't, so surely broken main throws an exception? Making this CR post off-topic?

It seems that Page is such a thin layer over Document that we could dispense with it and just have Topic directly interact with jsoup.

PageIterator could include a better explanation of what code is actually responsible for downloads.

This is perverse:

    if (type.equals("it4PROGRAMMERS")) {

Having gone to the trouble of creating an identifier, you may as well supply it, rather than a string.

TopicPageIterator4programmers does not obey the robots.txt protocol. You might consult https://github.com/crawler-commons/crawler-commons and use their support for parsing robots.txt. You should definitely sleep for a second or so between GETs.

These lines that vacuum up the contents of a topic

    this.topic = topic;
    this.urlBase = topic.getUrlBase();
    this.pagePattern = topic.getPagePattern();

suggest you may have wanted to initially communicate parameters to another class.

For the temp variable newUrl, a name of url would suffice. That one line of code, which knows how to construct well-formed URLs for the site, appears to be all that is unique about this class, compared with a class you might write tomorrow that scrapes some other site.

        e.printStackTrace();

You swallowed the exception. It would be better to let a checked exception bubble up the stack, or to re-throw e as a RuntimeException, so it's easy for callers to use your API without fear of accidental NPE.

I recommend you add support for a 2nd website, and then ruthlessly refactor so you have as few lines of source code as possible. Write a javadoc comment at the top of each class explaining what the class is responsible for. Significantly, this will help you to keep random lines of code out of the class, lines or attributes that clearly don't fall under the sentence explaining the class' responsibility. Also, if you write "this class is responsible for (1) FOO and also for (2) BAR" then you'll know it's time to break it apart. Single responsibility, do one thing well.

\$\endgroup\$
  • \$\begingroup\$ Yes, this will be doing something like this whaa ya've written. \$\endgroup\$ – wBacz Sep 9 '17 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.