I've been trying to learn Java for the last day or two. This is the first project I am working on, so please bear with me. I worked on a multithreaded web crawler. It is fairly simple but I'd like to ask for some advice.
Purpose / Running
The program starts at one web address (in this code, http://google.com) and looks for all valid URLs inside the response given. All URLs found in the response will be added to the queue. The crawler will then continue crawling through the URLs in the queue. To stop the crawler, type exit
in the input
Http.java
package com.janchr;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
/**
* Created by Jan on 8/20/2016.
*/
public class Http {
public static BufferedReader Get(URL url) throws IOException {
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setRequestMethod("GET");
// pretend that we are a new-ish browser. current user agent is actually from 2015.
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36");
con.setInstanceFollowRedirects(true);
int statusCode = con.getResponseCode();
// https://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/
boolean redirect = false;
if (statusCode != HttpURLConnection.HTTP_OK) {
if (statusCode == HttpURLConnection.HTTP_MOVED_TEMP
|| statusCode == HttpURLConnection.HTTP_MOVED_PERM
|| statusCode == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
if (redirect) {
// get redirect url from "location" header field
String newUrl = con.getHeaderField("Location");
// get the cookie if need
String cookies = con.getHeaderField("Set-Cookie");
return Http.Get(new URL(newUrl));
}
return new BufferedReader(new InputStreamReader(con.getInputStream()));
}
}
Crawler.java
package com.janchr;
import java.io.BufferedReader;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Created by Jan on 8/20/2016.
*/
class CrawlThread implements Runnable {
final static Pattern urlPat = Pattern.compile("(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]");
Crawler c;
int num;
boolean stop;
public Thread t;
public CrawlThread(Crawler c, int num) {
this.c = c;
this.num = num;
this.t = new Thread(this, "CrawlThread");
t.start();
}
private LinkedList<String> parse(BufferedReader r) {
String lineBuf = "";
LinkedList<String> urls = new LinkedList<String>();
do {
try {
lineBuf = r.readLine();
} catch (IOException e) {
System.out.println("(" + this.num + ") error parsing: " + e);
return urls;
}
if (lineBuf == null) {
return urls;
}
Matcher m = urlPat.matcher(lineBuf);
while(m.find()) {
//System.out.println("(" + this.num + ") match: " + m.group(0));
urls.add(m.group(0));
}
} while(lineBuf != null);
return urls;
}
public void run() {
// pop_front the next URL and get it
do {
String surl = c.next();
//System.out.println("(" + this.num + ") getting " + surl);
URL url;
try {
url = new URL(surl);
} catch (MalformedURLException e) {
System.out.println("(" + this.num + ") bad url " + surl + ": " + e);
continue;
}
BufferedReader r;
try {
r = Http.Get(url);
} catch (IOException e) {
System.out.println("(" + this.num + ") IOException Http.Get " + surl + ": " + e);
continue;
}
c.done(surl);
for (String newUrl: this.parse(r)) {
c.addURL(newUrl);
}
} while(!this.stop);
}
}
class VisitedURL {
public String url;
public int visits;
VisitedURL(String url) {
this.url = url;
}
}
public class Crawler {
private List<String> queue = Collections.synchronizedList(new LinkedList<>());
private Map<String, VisitedURL> visited = Collections.synchronizedMap(new LinkedHashMap<>());
private ArrayList<CrawlThread> threads = new ArrayList<>();
private int maxThreads;
public Crawler(int maxThreads) {
this.maxThreads = maxThreads;
}
public void start(String entryPoint) {
this.queue.add(entryPoint);
for (int i = 0; i < this.maxThreads; i++) {
this.threads.add(new CrawlThread(this, i));
}
}
public synchronized void stop() {
for(CrawlThread t: this.threads) {
// interrupting the thread should be fine for us in our use-case.
t.stop = true;
t.t.interrupt();
}
}
public synchronized String next() {
// I got IndexOutOfBoundsException here when starting up the crawler.
// the only way to fix it for me was this loop. I don't know what would
// be a better way to fix it. A mutex didn't work for me.
do {
if (this.queue.size() == 0) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
} while(this.queue.size() == 0);
synchronized (this.queue) {
if (this.queue.size() == 1) {
System.out.println("QUEUE EMPTY NOW");
}
return this.queue.remove(0);
}
}
public void done(String url) {
final VisitedURL obj = this.visited.putIfAbsent(url, new VisitedURL(url));
if (obj == null) {
this.visited.get(url).visits++;
}
}
public synchronized void addURL(String url) {
// TODO: we might want to ignore the URLs query
if (this.queue.contains(url)) {
return;
}
if (this.visited.containsKey(url)) {
this.visited.get(url).visits++;
return;
}
this.queue.add(url);
notifyAll();
}
public Map<String, VisitedURL> getVisitedUrls() {
return visited;
}
}
Main.java
package com.janchr;
import java.util.Scanner;
public class Main {
public static void main(String[] args) {
Crawler c = new Crawler(8);
System.out.println("starting crawler");
c.start("http://google.com");
Scanner s = new Scanner(System.in);
while (!s.next().equals("exit"));
c.stop();
synchronized (c) {
System.out.println("\n\n---------------------------------------------------------------------");
for (VisitedURL u : c.getVisitedUrls().values()) {
System.out.println(u.visits + "x " + u.url);
}
System.out.println("---------------------------------------------------------------------");
System.out.println("visited " + c.getVisitedUrls().size() + " unique urls");
}
}
}
Some questions:
- As seen in Crawler::next, I ran into a few problems when having a lot of threads running. I haven't exactly figured out why I can't just put one of the
wait
calls there, but I guess it is because I usenotifyAll
later on(?) - Would it be better if I created the threads "on demand" rather than creating them all on startup and then waiting for new URLs to be in the queue? Would it eliminate the issue I have above?
- What is the "usual Java way" to work with multiple threads?
- By design, the crawler will get slower and slower over time because the Lists/Maps get bigger. I can't really think of a way to increase performance there. What can I improve here?
Edit
I've updated my code to use the threads how as I specified in one of my questions. Is this a better solution?
Crawler.java
package com.janchr;
import java.io.BufferedReader;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Created by Jan on 8/20/2016.
*/
class CrawlThread implements Runnable {
final static Pattern urlPat = Pattern.compile("https?://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]");
Crawler c;
String url;
public Thread t;
public CrawlThread(Crawler c, String url) {
this.c = c;
this.url = url;
this.t = new Thread(this, "CrawlThread");
t.start();
}
private LinkedList<String> parse(BufferedReader r) {
String lineBuf = "";
LinkedList<String> urls = new LinkedList<String>();
do {
try {
lineBuf = r.readLine();
} catch (IOException e) {
System.out.println("error parsing: " + e);
return urls;
}
if (lineBuf == null) {
return urls;
}
Matcher m = urlPat.matcher(lineBuf);
while(m.find()) {
urls.add(m.group(0));
}
} while(lineBuf != null);
return urls;
}
public void run() {
URL url;
try {
url = new URL(this.url);
} catch (MalformedURLException e) {
System.out.println("bad url " + this.url + ": " + e);
c.done(this, this.url);
return;
}
BufferedReader r;
try {
r = Http.Get(url);
} catch (IOException e) {
System.out.println("IOException Http.Get " + this.url + ": " + e);
c.done(this, this.url);
return;
}
for (String newUrl: this.parse(r)) {
c.addURL(newUrl);
}
c.done(this, this.url);
}
}
class VisitedURL {
public String url;
public int visits;
VisitedURL(String url) {
this.url = url;
}
}
public class Crawler {
private List<String> queue = Collections.synchronizedList(new LinkedList<>());
private Map<String, VisitedURL> visited = Collections.synchronizedMap(new LinkedHashMap<>());
private ArrayList<CrawlThread> threads = new ArrayList<>();
private int maxThreads;
public Crawler(int maxThreads) {
this.maxThreads = maxThreads;
}
public void start(String entryPoint) {
this.queue.add(entryPoint);
this.tryNext();
}
public synchronized void stop() {
for(CrawlThread t: this.threads) {
// interrupting the thread should be fine for us in our use-case.
t.t.interrupt();
}
}
public synchronized boolean hasNext() {
return this.queue.size() > 0;
}
public synchronized String next() {
if (this.queue.size() == 0) {
return null;
}
return this.queue.remove(0);
}
private void tryNext() {
if (!this.hasNext() || this.threads.size() == this.maxThreads) {
return;
}
String next = this.next();
if (next == null) {
System.out.println("invalid next string");
return;
}
this.threads.add(new CrawlThread(this, next));
}
public void done(CrawlThread t, String url) {
final VisitedURL obj = this.visited.putIfAbsent(url, new VisitedURL(url));
if (obj == null) {
this.visited.get(url).visits++;
}
this.threads.remove(t);
this.tryNext();
}
public synchronized void addURL(String url) {
// TODO: we might want to ignore the URLs query
if (this.queue.contains(url)) {
return;
}
if (this.visited.containsKey(url)) {
this.visited.get(url).visits++;
return;
}
this.queue.add(url);
this.tryNext();
}
public Map<String, VisitedURL> getVisitedUrls() {
return visited;
}
}
Now I am not able to .interrupt
my threads anymore though. How do I fix this?
String cookies = con.getHeaderField("Set-Cookie");
and never usecookies
again do you need it? \$\endgroup\$