So as a starter Java project I decided to web scrape some data (specifically all historically No. 1 ranked players for weeks starting from 1973) from the ATP website, and do something with it (IPR). I'm in the process of refactoring my working web scraper and wanted some feedback.

• Currently my scraper retrieves the No.1s - or so it seems. I haven't tested it apart from just printing it to my console and verifying it that way. One thing I feel is that I can tighten some of the exception handling, but I wasn't sure how what test cases to develop in JUnit for that. Any tips?

• More importantly, feedback on the code style would be really appreciated! The bulk of my code is in Scraper (duh), but I'm not sure I'm too comfortable with having various static methods. That being said, a sprawling main function is not ideal either, especially when there are separable pieces of the logic that the scraper performs. Does this indicate I need to somehow break the Scraper design into smaller objects? What be a good design practice?

• Any other feedback, especially related to best practices and idioms in Java would be appreciated (I come from a primarily C & C++ background)

Here's my code:

Scraper:

package Scraper;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;

public class Scraper {
    public static void main() {
        final String ATP_URL_PREFIX = "https://www.atptour.com/en/rankings/singles?";
        final String ATP_URL_SUFFIX = "&rankRange=0-100";
        // get the list of historical ranking weeks - basically from 1973-present.
        ArrayList<String> weeks = new ArrayList<String>();
        weeks = getWeeksForRankings(ATP_URL_PREFIX, weeks);
        // weeks might be null if no valid HTML
        if (weeks.size() == 0) {
            System.out.println("Please provide a historical time range! Cannot rank otherwise!");
            return;
        }
        getPlayerNames(ATP_URL_PREFIX, ATP_URL_SUFFIX, weeks);
    }
    
    static ArrayList getWeeksForRankings(String url, ArrayList<String> weeks) {
        try {
            final Document document = Jsoup.connect(url).get();
            // extract the series of list items corresponding to the ranking weeks, from the dropdown menu
            Elements rankingWeeksList = document.getElementsByAttributeValue("data-value", "rankDate").select("ul li");
            for (Element li : rankingWeeksList) {
                // for accessing the relevant week's ranking page later, the rankDate= param in the URL takes '-'s
                // instead of dots so we replace the characters here and then add them to out list.
                String week = li.text().replaceAll("\\.", "-");
                weeks.add(week);
            }
        } catch (IOException e) {
            System.out.println("Error while connecting and parsing HTML: " + e);
            System.exit(1);
        } catch (Exception e) {
            System.out.println("Fatal Error: " + e);
            System.exit(1);
        }
        Collections.reverse(weeks); // start from 1973.
        return weeks;
    }

    static void getPlayerNames(String url_prefix, String url_suffix, ArrayList<String> weeks) {
        // dynamically update a player's ranking and animate his status
        for (String week : weeks) {
            String url = url_prefix+"rankDate="+week+url_suffix;
            try {
                final int SECONDS_TO_MILLISECONDS = 1000;
                // time out is an issue. ideally, try mutliple times to get the data??
                final Document document = Jsoup.connect(url).timeout(180 * SECONDS_TO_MILLISECONDS).get();
                Element player = document.getElementsByClass("player-cell").first();
                if (player == null) {
                    continue;
                } else {
                    System.out.println("Week: " + week + " No.1: "+ player.text());
                }
            } catch (IOException e) {
                System.out.println("Error while connecting and parsing HTML: " + e);
                System.exit(1);
            }
        }
    }
}

Main Driver:

package tennisProject;

import Scraper.Scraper;

public class TennisProject {
    public static void main(String[] args) {
        Scraper.main();
    }
}
```