Java classes for downloading all in-coming/out-going links of an article in the Wikipedia article graph

Question

(The entire project is in GitHub.)

Introduction

This project provides facilities for generating in-coming or out-going links in a given Wikipedia page.

Code

com.github.coderodde.wikipedia.graph.expansion.AbstractWikipediaGraphNodeExpander.java:

package com.github.coderodde.wikipedia.graph.expansion;

import com.github.coderodde.graph.pathfinding.delayed.AbstractNodeExpander;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.Charset;
import java.util.Arrays;
import java.util.Locale;
import org.apache.commons.io.IOUtils;

/**
 * This abstract class specifies the facilities shared by both forward and 
 * backward node expanders.
 * 
 * @version 1.0.0 (Mar 20, 2024)
 */
public abstract class AbstractWikipediaGraphNodeExpander
extends AbstractNodeExpander<String> {
   
    /**
     * The script URL template for expanding forward.
     */
    protected static final String FORWARD_REQUEST_API_URL_SUFFIX = 
            "?action=query" +
            "&titles=%s" + 
            "&prop=links" + 
            "&pllimit=max" + 
            "&format=json"; 
    
    /**
     * The script URL template for expanding backwards.
     */
    protected static final String BACKWARD_REQUEST_API_URL_SUFFIX = 
            "?action=query" +
            "&list=backlinks" +
            "&bltitle=%s" + 
            "&bllimit=max" + 
            "&format=json";
    
    protected final String languageLocaleName;
    
    protected static final String API_URL_FORMAT = 
            "https://%s.wikipedia.org/w/api.php";
    
    /**
     * Caches the textual representation of the URL pointing to the 
     * <a href="https://www.mediawiki.org/wiki/API:Main_page">Wikipedia API</a>.
     */
    protected final String apiUrl;

    /**
     * Constructs a graph node expander for the language subgraph specified in
     * the input URL.
     */
    protected AbstractWikipediaGraphNodeExpander(String languageLocaleName) {
        checkLanguageLocaleName(languageLocaleName);
        this.languageLocaleName = languageLocaleName;
        this.apiUrl = constructAPIURL(languageLocaleName);
    }
    
    /**
     * {@inheritDoc }
     */
    @Override
    public boolean isValidNode(final String node) {
        return !generateSuccessors(node).isEmpty();
    }
    
    /**
     * The actual implementation of the method producing the neighbour JSON data
     * of the article graph node.
     * 
     * @param articleName the node to expand.
     * @param forward specifies the direction of the node expansion operation.
     *                if {@code forward} is {@code true}, generates the child
     *                nodes of {@code node}. Otherwise, generates the parent 
     *                nodes of {@code node}.
     * @return the JSON data describing the forward- or backward-links.
     */
    protected String downloadJson(final String articleName,
                                  final boolean forward) {
        String jsonDataUrl;
        
        try {
            jsonDataUrl =
                    apiUrl + String.format(forward ?
                                                FORWARD_REQUEST_API_URL_SUFFIX :
                                                BACKWARD_REQUEST_API_URL_SUFFIX,
                                           URLEncoder.encode(articleName,
                                                             "UTF-8"));
            
        } catch (final UnsupportedEncodingException ex) {
            throw new IllegalStateException(ex.getMessage(), ex);
        }
        
        String jsonText;
        
        try {
            jsonText = IOUtils.toString(new URL(jsonDataUrl),
                                        Charset.forName("UTF-8"));
        } catch (final IOException ex) {
            throw new IllegalStateException(
                    "[I/O ERROR] Failed loading the JSON data from the " +
                    "Wikipedia API: " + ex.getMessage(), ex);
        }
        
        return jsonText;
    }
    
    /**
     * Constructs the full Wikipedia article URL with the input title.
     * 
     * @param title the title of the target article URL.
     * @return the full URL to the article.
     */
    protected String constructFullWikipediaLink(String title) {
        return String.format("https://%s.wikipedia.org/wiki/%s",
                             languageLocaleName,
                             title);
    }
    
    /**
     * Constructs a Wikipedia API URL from the raw {@code wikipediaUrl}. The
     * input {@code wikipediaUrl} is of the form <tt>en.wikipedia.org</tt>. The
     * idea here is that the search may be applied to article subgraphs with 
     * different languages.
     * 
     * @param wikipediaUrl the Wikipedia URL.
     * @return full URL to Wikipedia API.
     */
    private String constructAPIURL(final String languageLocaleName) {
        return String.format(API_URL_FORMAT, languageLocaleName);
    }
    
    /**
     * Checks that the input language ISO code is in the list of existing codes.
     * 
     * @param languageLocaleName the language country code to check.
     */
    private void checkLanguageLocaleName(String languageLocaleName) {
        if (!Arrays.asList(Locale.getISOLanguages())
                   .contains(languageLocaleName)) {
            
            throw new IllegalArgumentException(
                    String.format("Unknown language locale name: %s.", 
                                  languageLocaleName));
        }
    }
}

com.github.coderodde.wikipedia.graph.expansion.ForwardWikipediaGraphNodeExpander.java:

package com.github.coderodde.wikipedia.graph.expansion;

import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Set;

/**
 * This class implements a forward node expander in the Wikipedia article graph.
 * If article <tt>A</tt> has a link to <tt>B</tt>, this expander will generate
 * <tt>B</tt> whenever asked to process <tt>A</tt>. We can say that this 
 * expander traverses each directed arc from tail to head.
 * 
 * @version 1.0.0 (Mar 19, 2024)
 */
public class ForwardWikipediaGraphNodeExpander 
extends AbstractWikipediaGraphNodeExpander {

    public ForwardWikipediaGraphNodeExpander(final String languageLocaleName) {
        super(languageLocaleName);
    }
    
    @Override
    public List<String> generateSuccessors(final String articleTitle) {
        List<String> linkNameList = new ArrayList<>();
        String jsonText = downloadJson(articleTitle, true);
        
        try {
            Gson gson = new Gson();
            
            JsonObject root = gson.fromJson(jsonText, JsonObject.class);
            JsonElement queryElement = root.get("query");
            JsonObject queryObject = queryElement.getAsJsonObject();
            JsonObject pagesObject = queryObject.getAsJsonObject("pages");
            
            Set<Map.Entry<String, JsonElement>> set = pagesObject.entrySet();
            
            JsonObject idObject = 
                    set
                        .iterator()
                        .next()
                        .getValue()
                        .getAsJsonObject();
            
            JsonArray linkArray = idObject.getAsJsonArray("links");
            
            for (JsonElement titleElement : linkArray) {
                int namespace = 
                        titleElement.getAsJsonObject().get("ns").getAsInt();
                
                if (namespace != 0) {
                    continue;
                }
                
                String title = 
                        titleElement
                                .getAsJsonObject()
                                .get("title")
                                .getAsString();
                try {
                    title = URLEncoder.encode(
                            title,
                            StandardCharsets.UTF_8.toString())
                            .replace("+", "_");
                    
                } catch (UnsupportedEncodingException ex) {
                    System.err.printf(
                            "Could not URL encode \"%s\". Omitting.\n", 
                            title);
                    
                    return Collections.<String>emptyList();
                }
                
                linkNameList.add(constructFullWikipediaLink(title));
            }
        } catch (NullPointerException ex) {
            return linkNameList;
        }
        
        return linkNameList;
    }
}

com.github.coderodde.wikipedia.graph.expansion.BackwardWikipediaGraphNodeExpander.java:

package com.github.coderodde.wikipedia.graph.expansion;

import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

/**
 * This class implements a backward node expander in the Wikipedia article 
 * graph. If article <tt>A</tt> has a link to <tt>B</tt>, this expander will generate
 * <tt>A</tt> whenever asked to process <tt>B</tt>. We can say that this 
 * expander traverses each directed arc from head to tail.
 * 
 * @author Rodion "rodde" Efremov
 * @version 1.0.0 (Mar 19, 2024)
 */
public class BackwardWikipediaGraphNodeExpander 
extends AbstractWikipediaGraphNodeExpander {
    
    public BackwardWikipediaGraphNodeExpander(final String languageLocaleName) {
        super(languageLocaleName);
    }
    
    @Override
    public List<String> generateSuccessors(final String articleTitle) {
        List<String> linkNameList = new ArrayList<>();
        String jsonText = downloadJson(articleTitle, false);

        try {
            Gson gson = new Gson();
            JsonObject root = gson.fromJson(jsonText, JsonObject.class);
            JsonElement queryElement = root.get("query");
            JsonElement backlinksElement = 
                    queryElement.getAsJsonObject().get("backlinks");
            
            JsonArray pagesArray = backlinksElement.getAsJsonArray();
            
            for (JsonElement element : pagesArray) {
                int namespace = element.getAsJsonObject().get("ns").getAsInt();
                
                if (namespace == 0) {
                    String title = element.getAsJsonObject().get("title")
                            .getAsString();
                    try {
                        title = URLEncoder.encode(
                                title,
                                StandardCharsets.UTF_8.toString())
                                .replace("+", "_");
                        
                    } catch (UnsupportedEncodingException ex) {
                        System.err.printf(
                                "Could not URL encode \"%s\". Omitting.\n", 
                                title);

                        return Collections.<String>emptyList();
                    }
                    
                    linkNameList.add(constructFullWikipediaLink(title));
                }
            }
        } catch (NullPointerException ex) {
            return linkNameList;
        }

        return linkNameList;
    }
}

com.github.coderodde.wikipedia.graph.expansion.ForwardWikipediaGraphNodeExpanderTest.java:

package com.github.coderodde.wikipedia.graph.expansion;

import org.junit.Test;
import static org.junit.Assert.*;

// Tests in this class require that there is connection to the Wikipedia API!
public class ForwardWikipediaGraphNodeExpanderTest {
    
    private ForwardWikipediaGraphNodeExpander generateSuccessorser;
    
    @Test
    public void generateSuccessors() {
        generateSuccessorser = 
                new ForwardWikipediaGraphNodeExpander("en");
        
        assertTrue(generateSuccessorser.isValidNode("Disc_jockey"));
        assertTrue(!generateSuccessorser.generateSuccessors("Disc_jockey").isEmpty());
        
        assertFalse(generateSuccessorser.isValidNode("Disc_jfdsfsockey"));
        assertTrue(generateSuccessorser.generateSuccessors("Disc_jockeyfdsfsd").isEmpty());
    }
    
    @Test(expected = IllegalArgumentException.class)
    public void testOnBadWikipediaCountryCode() {
        new ForwardWikipediaGraphNodeExpander("shit");
    }
}

com.github.coderodde.wikipedia.graph.expansion.BackwardWikipediaGraphNodeExpanderTest.java:

package com.github.coderodde.wikipedia.graph.expansion;

import org.junit.Test;
import static org.junit.Assert.*;

// Tests in this class require that there is connection to the Wikipedia API!
public class BackwardWikipediaGraphNodeExpanderTest {
    
    private BackwardWikipediaGraphNodeExpander generateSuccessorser;
    
    @Test
    public void generateSuccessors() {
        generateSuccessorser = 
                new BackwardWikipediaGraphNodeExpander("en");
        
        assertTrue(generateSuccessorser.isValidNode("Disc_jockey"));
        assertTrue(!generateSuccessorser.generateSuccessors("Disc_jockey").isEmpty());
        
        assertFalse(generateSuccessorser.isValidNode("Disc_jfdsfsockey"));
        assertTrue(generateSuccessorser.generateSuccessors("Disc_jockeyfdsfsd").isEmpty());
    }
    
    @Test(expected = IllegalArgumentException.class)
    public void testOnBadWikipediaCountryCode() {
        new BackwardWikipediaGraphNodeExpander("shit");
    }
}

com.github.coderodde.wikipedia.graph.expansion.benchmark.Benchmark.java:

package com.github.coderodde.wikipedia.graph.expansion.benchmark;

import java.util.List;
import com.github.coderodde.wikipedia.graph.expansion.BackwardWikipediaGraphNodeExpander;
import com.github.coderodde.wikipedia.graph.expansion.ForwardWikipediaGraphNodeExpander;

public final class Benchmark {
    
    public static void main(String[] args) {
        System.out.println("[Forward expansion]");
        
        long start = System.currentTimeMillis();
        
        List<String> forwardLinks = 
                new ForwardWikipediaGraphNodeExpander("fi")
                        .generateSuccessors("Bugatti");
        
        long end = System.currentTimeMillis();
        
        System.out.printf("Forward request in %d milliseconds.\n", end - start);
        
        forwardLinks.forEach(System.out::println);
        
        System.out.println("[Backward expansion]");
        
        start = System.currentTimeMillis();
        
        List<String> backwardLinks = 
                new BackwardWikipediaGraphNodeExpander("fi")
                        .generateSuccessors("Bugatti");
        
        end = System.currentTimeMillis();
        
        System.out.printf("Backward request in %d milliseconds.\n", end - start);
        
        backwardLinks.forEach(System.out::println);
    }
}

Benchmark output

Looks like this:

[Forward expansion]
Forward request in 2176 milliseconds.
https://fi.wikipedia.org/wiki/Alsace
https://fi.wikipedia.org/wiki/Audi
https://fi.wikipedia.org/wiki/Audi_Sport
https://fi.wikipedia.org/wiki/Auto_Union
https://fi.wikipedia.org/wiki/Autoteollisuus
https://fi.wikipedia.org/wiki/Bentley
...
https://fi.wikipedia.org/wiki/Volkswagen_do_Brasil
https://fi.wikipedia.org/wiki/Wanderer_%28yritys%29
https://fi.wikipedia.org/wiki/Wikimedia_Commons
https://fi.wikipedia.org/wiki/%C5%A0koda_Auto
[Backward expansion]
Backward request in 624 milliseconds.
https://fi.wikipedia.org/wiki/Volkswagen
https://fi.wikipedia.org/wiki/NSU
...
https://fi.wikipedia.org/wiki/Pariisin_Grand_Prix_1945
https://fi.wikipedia.org/wiki/Nethercutt_Collection
https://fi.wikipedia.org/wiki/Mullin_Automotive_Museum
https://fi.wikipedia.org/wiki/Coupe_des_Alpes
https://fi.wikipedia.org/wiki/Mauro_Forghieri

Critique request

As always, I am eager to receive any constructive commentary on my work.

Answer 1

AbstractWikipediaGraphNodeExpander.java

• Locale.getISOLanguages() is an intuitive way to check if a language code is valid, but not all Wikipedias have a valid ISO 639-1 code as their language - this fails for "ceb.wikipedia.org", which has the second highest page count of all Wikipedias. Expanding to ISO 639-2, or even -3 won't entirely solve this problem either - neither includes "simple", as in the well-known "simple.wikipedia.org". Unfortunately, I don't know of a better way to check whether Wikipedia uses a given language code than to ask Wikipedia directly. We could either do that with a dedicated query early on (like perhaps ?action=help&format=none), or we could just let the program keep going until one of the later API calls fail.

• I question the way downloadJson wraps exceptions inside (unchecked) IllegalStateExceptions. First off, with no mention of them in the javadoc, a caller has no way of knowing to expect IllegalStateExceptions in the first place. And second, since the caller still needs to handle the exceptions, what are we gaining by wrapping the perfectly fine IOExceptions inside a different exception type? It seems to me like it just makes it harder for the caller to tell whether we failed to encode the page title or whether something went wrong with the network request (which they may want to know!). Simply declaring the method as throws IOException seems like it'd make life easier both for us and for anyone else using this code.

• downloadJson feels like it uses inheritance, but does not take advantage of it. There are fields in the base class where each subclass is exected to use one or the other, and this method can work in two ways but each subclass should only take advantage of one or the other. It would arguably be cleaner to just let each subclass provide its own definition of downloadJson, but we could also keep the core logic in the base class while letting the subclass be responsible for the query string. However, since most of the rest of the method is exception handling I already advocated for moving elsewhere, we'd be left with

  protected abstract String getQueryParameters();
  
  protected final String downloadJson(final String articleName) throws UnsupportedEncodingException, IOException {
      String jsonDataUrl = apiUrl + getQueryParameters(articleName);
  
      return IOUtils.toString(new URL(jsonDataUrl), Charset.forName("UTF-8"));
  }
  

  which seems a bit redundant.

• While perhaps out of scope, this code could also be useful for non-Wikipedia MediaWiki wikis, had "wikipedia.org" not been hardcoded into it. Perhaps someone may wish to use it to navigate sites like "minecraft.wiki" or "gardenology.org"? It seems like adding the option to specify a different site root, even if keeping Wikipedia as the default, would be relatively simple.

• I do not, personally, like the naming of isValidNode. Surely a page with no outgoing links, while not very useful to this program, is still valid?

ForwardWikipediaGraphNodeExpander.java/BackwardWikipediaGraphNodeExpander.java

• I find the exception handling strategy a bit confusing here. The fact that practically the entire method is wrapped in a single try {} catch(NullPointerException) leaves me wondering where, and why, we might expect to see a NullPointerException thrown. And if one is thrown (for whatever reason) we seem to return a list of some of the outgoing links, which strikes me as questionable. I'd probably try and move any code we don't expect to throw an NPE out of this try block, and (if that doesn't clarify things) add a comment clarifying what we're doing about the exception and why.
• I also mislike the handling of UnsupportedEncodingExceptions. Sure, encoding to UTF-8 is unlikely to fail, so it probably won't matter. But if it does, we either want the caller to know something went wrong (in which case we shouldn't swallow the exception), or we want to handle the issue and carry on (in which case we should surely return any page titles we did manage to encode, including any we have yet to process?). It also seems we're introducing a semipredicate problem by using an empty list to signify both "there was something wrong with the response" and "everything went fine but the page has no outgoing links", which seem like two different situations that a caller might want to handle differently? If we really don't want our methods to throw exceptions (a strange decision in my eyes, given that that's what exceptions are for), might it not still be preferable to return null or perhaps Optional.empty()?
• Also, pllimit=max/bllimit=max still isn't guaranteed to fetch all links - there's still a limit to how many results you can get from a single query. On en.wikipedia.org, we aren't getting more than 500 links in a single request, and there are pages with well over 500 links - List of canonically crowned images seems to have just over 1700. Which means we'll want to add continuing logic to our queries.
• For forward links, unless we need to support wikis using MediaWiki 1.24 and older (for context, 1.25 was released in 2014, and all the Wikipedia wikis use far later versions), we can use &formatversion=2 to get the responses as an array instead of an object. This could let the forward expander access linkArray without having to mess about with sets and iterators simply using queryObject.getAsJsonArray("pages").get(0).getAsJsonArray() or something similar, much like we already do for the reverse expander.