2
\$\begingroup\$

I'm developing a simple application that crawls web pages to obtain some information. For this I used and tested some libraries, like crawler4j, jsoup, jaunt and htmlunit. I change the crawler APIs several times because sometimes perceived that one served me better in a certain feature than the one I was using. I may have to do it again, and every time I do so I have to go around the code making various refactorings.

So I decide to separate the calls for these APIs in a kind of encapsulated class that holds all the operations I have to do with these API. So if I decide to change the crawler API I just point to another implementation of CrawlerApiHandler.

I have troubles to identify one design pattern to mitigate this problem.

Below the 2 classes and the interface I created so far:

I named with suffix handler, but I don't know if it is it's appropriate.

/**
 * Created by alexandre on 29/01/2016.
 */
public interface CrawlerApiHandler {
    String visit (String url);
}

public class JauntApiHandlerImpl implements CrawlerApiHandler {
    UserAgent userAgent;

    public JauntApiHandlerImpl(UserAgent userAgent) {
        this.userAgent = userAgent;
    }


    @Override
    public String visit(String url) {
        try {
            return userAgent.visit(url).toString();
        } catch (ResponseException e) {
            return "";
        }
    }
}

public class JsoupCrawlerApiHandler implements CrawlerApiHandler {
    public static final String USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36";

    @Override
    public String visit(String url) {
        try {
            return Jsoup.connect(url).timeout(20000)
                    .userAgent(USER_AGENT).get().toString();
        } catch (IOException e) {
            //LOG
            return "";
        }
    }
}
\$\endgroup\$
  • \$\begingroup\$ Is the code you posted working correctly right now? You stated you are having a problem with it, but it is not clear whether it is working or not. \$\endgroup\$ – Phrancis Jan 30 '16 at 1:02
  • \$\begingroup\$ Yes, it compiles and runs if you have linked the libraries. My problem is to identify a design pattern to deal with API changes. \$\endgroup\$ – alexpfx Jan 30 '16 at 1:07
  • \$\begingroup\$ Okay, thanks for clarifying that, maybe you should mention in your post more details about the problem you are working to solve \$\endgroup\$ – Phrancis Jan 30 '16 at 1:08
1
\$\begingroup\$

Honestly, I think you have it just right with your current code: simple enough, yet powerful enough.

In your case, you need to support multiple implementations of Web crawlers API. To be able to abstract your code from all those implementations, an interface needs to be introduced. This interface will define the contract that your Web crawlers will satisfy. When this interface is introduced, the rest of your code can depend on that interface only, and not on the implementation.

Right now, your contract defines a methid visit(url) that takes a URL and returns the HTML content at that URL. This could be changed at a later time (e.g. adding time-outs or error handlers) but it's a good start.


The next problem to tackle is how to use the right implementation? This is typically done by using dependency injection. To put it simply, this is a design pattern where a class lets its dependencies be injected through constructor or setters instead of constructing them itself. For your example, consider a class MyClass that would depend on a Web crawler:

public class MyClass {

    private CrawlerApiHandler crawler;

    // the crawler to use in being injected into this class
    public BodyExtractor(CrawlerApiHandler crawler) {
        this.crawler = crawler;
    }

    // later on, use crawler to do some fancy things

}

You could then centralize the real implementation in a configuration class / file. The great advantage is that you now need to define the real implementation class in a single place and the rest of the code is completely decoupled from it by just depending on the interface.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Hi, thanks for your answer. Actually I made another implementation but keep the same structure. I created another question for that. Now I found the design pattern I looking for: Bridge Pattern. \$\endgroup\$ – alexpfx Jan 30 '16 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.