2
\$\begingroup\$

BookstoreScraper is application based on Spring Boot. It allows to scrape data from two biggest polish online bookstores - EMPIK and MERLIN. You can scrap books within 3 options:

  • bestsellers
  • most precise book (you simply give title and it looks for most precise book)
  • categorized book (currently 5 categories of books are available: BIOGRAPHY, CRIME, GUIDES, FANTASY, ROMANCES)

There is ranking option. It is comparing books from each bookstore and if title repeats the book is higher in the ranking.

There is also history system which tracks every action of logged user as there is provided simply security.

There are a lot of classes so I will paste just the most important classes, but if you can I will paste GitHub link so you can check it out so you can tell me also about structure and other unit tests that are not posted here.

Let's start from the class responsible for scraping data from the site:

MerlinSource which implements BookServiceSource interface. I'm not gonna paste EmpikSource as it looks really similiar.

package bookstore.scraper.book.booksource.merlin;

import bookstore.scraper.book.Book;
import bookstore.scraper.book.booksource.BookServiceSource;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.JSoupConnector;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

@Service
public class MerlinSource implements BookServiceSource {

    private static final int CATEGORIZED_BOOKS_NUMBER_TO_FETCH = 16;
    private static final int BESTSELLERS_NUMBER_TO_FETCH = 6;

    private final JSoupConnector jSoupConnector;
    private final MerlinUrlProperties merlinUrlProperties;

    @Autowired
    public MerlinSource(JSoupConnector jSoupConnector, MerlinUrlProperties merlinUrlProperties) {
        this.jSoupConnector = jSoupConnector;
        this.merlinUrlProperties = merlinUrlProperties;
    }

    @Override
    public Bookstore getName() {
        return Bookstore.MERLIN;
    }

    @Override
    public List<Book> getBooksByCategory(CategoryType categoryType) {
        Document document = jSoupConnector.connect(merlinUrlProperties.getCategory(categoryType));

        return IntStream.range(1, CATEGORIZED_BOOKS_NUMBER_TO_FETCH)
                .mapToObj(iterator -> getBestSellerOrCategorizedBook(document, iterator))
                .collect(Collectors.toList());
    }

    @Override
    public Book getMostPreciseBook(String givenTitle) {
        String concatedUrl = concatUrlWithTitle(merlinUrlProperties.getCategory(CategoryType.MOST_PRECISE_BOOK), givenTitle);

        Document document = jSoupConnector.connect(concatedUrl);

        String title = document.select("div.b-products-list__title-holder").select("a").first().text();
        String price = document.select("div.b-products-list__price-holder > a").first().text();
        String author = document.select("div.b-products-list__manufacturer-holder").select("a").first().text();
        String productID = document.select("div.grid__col.grid__col--20-80-80.b-products-wrap > ul > li:nth-child(1)").first().attr("data-ppc-id");
        String bookUrl = createBookUrl(title, productID);

        return Book.builder()
                .author(author)
                .price(price)
                .title(title)
                .productID(productID)
                .bookURL(bookUrl)
                .build();
    }

    @Override
    public List<Book> getBestSellers() {
        Document document = jSoupConnector.connect(merlinUrlProperties.getCategory(CategoryType.BESTSELLER));

        return IntStream.range(1, BESTSELLERS_NUMBER_TO_FETCH)
                .mapToObj(iterator -> getBestSellerOrCategorizedBook(document, iterator))
                .collect(Collectors.toList());
    }

    private Book getBestSellerOrCategorizedBook(Document document, int iteratedBook) {
        Elements siteElements = document.select("div.grid__col.grid__col--20-80-80.b-products-wrap > ul > li:nth-child(" + iteratedBook + ")");
        String author = siteElements.select(" > div > div.b-products-list__desc-wrap > div > div.b-products-list__main-content > div.b-products-list__desc-prime > div.b-products-list__manufacturer-holder").select("a").first().text();
        String title = siteElements.select(" > div > div.b-products-list__desc-wrap > div > div.b-products-list__main-content > div.b-products-list__desc-prime > div.b-products-list__title-holder > a").first().text();
        String price = siteElements.select(" div.b-products-list__price-holder > a").first().text();
        String productID = siteElements.first().attr("data-ppc-id");
        String bookUrl = createBookUrl(title, productID);

        return Book.builder()
                .author(author)
                .price(price)
                .title(title)
                .productID(productID)
                .bookURL(bookUrl)
                .build();
    }

    private String createBookUrl(String title, String productID) {
        return String.format(merlinUrlProperties.getConcreteBook(), title, productID);
    }

    private String concatUrlWithTitle(String url, String title) {
        return String.format(url, title);
    }
}

BookService - it fetches result from scraping data classes and wrap it into Map.

package bookstore.scraper.book;

import bookstore.scraper.account.LoggedAccountService;
import bookstore.scraper.book.booksource.BookServiceSource;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import bookstore.scraper.historysystem.ActionType;
import bookstore.scraper.historysystem.HistorySystemService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

@Service
public class BookService {

    private final List<BookServiceSource> sources;
    private final HistorySystemService historySystemService;
    private final LoggedAccountService loggedAccountService;

    @Autowired
    public BookService(List<BookServiceSource> sources, HistorySystemService historySystemService, LoggedAccountService loggedAccountService) {
        this.sources = sources;
        this.historySystemService = historySystemService;
        this.loggedAccountService = loggedAccountService;
    }

    public Map<Bookstore, List<Book>> getBooksByCategory(CategoryType category) {
        historySystemService.saveAccountHistory(
                loggedAccountService.getLoggedAccountID(), ActionType.CATEGORIZED_BOOK.toString());

        return sources.stream()
                .collect(Collectors.toMap(BookServiceSource::getName,
                        source -> source.getBooksByCategory(category)));
    }

    public Map<Bookstore, Book> getMostPreciseBOok(String title) {
        historySystemService.saveAccountHistory(
                loggedAccountService.getLoggedAccountID(), ActionType.MOST_PRECISE_BOOK.toString());

        return sources.stream()
                .collect(Collectors.toMap(BookServiceSource::getName,
                        source -> source.getMostPreciseBook(title)));
    }

    public Map<Bookstore, List<Book>> getBestsellers() {
        historySystemService.saveAccountHistory(
                loggedAccountService.getLoggedAccountID(), ActionType.BEST_SELLERS.toString());

        return sources.stream()
                .collect(Collectors.toMap(BookServiceSource::getName,
                        BookServiceSource::getBestSellers));
    }
}

CategorizedBookRankingService - as I said this is service responsible for comparing books and creating ranking.

package bookstore.scraper.book.rankingsystem;

import bookstore.scraper.account.LoggedAccountService;
import bookstore.scraper.book.Book;
import bookstore.scraper.book.BookService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import bookstore.scraper.historysystem.ActionType;
import bookstore.scraper.historysystem.HistorySystemService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Stream;

import static java.util.stream.Collectors.*;

@Slf4j
@Component
public class CategorizedBooksRankingService {

    private final BookService bookService;
    private final HistorySystemService historySystemService;
    private final LoggedAccountService loggedAccountService;

    @Autowired
    public CategorizedBooksRankingService(BookService bookService, HistorySystemService historySystemService, LoggedAccountService loggedAccountService) {
        this.bookService = bookService;
        this.historySystemService = historySystemService;
        this.loggedAccountService = loggedAccountService;
    }

    public Map<String, Integer> getRankingForCategory(CategoryType category) {
        Map<Bookstore, List<Book>> bookstoreWith15CategorizedBooks = bookService.getBooksByCategory(category);

        List<Book> merlinBooks = bookstoreWith15CategorizedBooks.get(Bookstore.MERLIN);
        List<Book> empikBooks = bookstoreWith15CategorizedBooks.get(Bookstore.EMPIK);

        Map<String, List<String>> purifiedTitleWithOriginalTitles = getPurifiedTitleWithAccordingOriginalTitles(merlinBooks, empikBooks);
        Map<String, Integer> bookTitleWithOccurrencesNumber = getTitleWithOccurrences(purifiedTitleWithOriginalTitles);

        historySystemService.saveAccountHistory
                (loggedAccountService.getLoggedAccountID(), ActionType.CATEGORIZED_BOOKS_RANKING.toString());

        return getSortedLinkedHashMapByValue(bookTitleWithOccurrencesNumber);
    }

    private Map<String, List<String>> getPurifiedTitleWithAccordingOriginalTitles(List<Book> list1, List<Book> list2) {
        return Stream.concat(list1.stream(), list2.stream())
                .collect(
                        groupingBy(Book::getPurifiedTitle, mapping(Book::getTitle, toList())));
    }

    private Map<String, Integer> getTitleWithOccurrences(Map<String, List<String>> map) {
        return map.values().stream().collect(toMap(list -> list.get(0), List::size));
    }

    private Map<String, Integer> getSortedLinkedHashMapByValue(Map<String, Integer> mapToSort) {
        return mapToSort.entrySet()
                .stream()
                .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
                .collect(
                        toMap(Map.Entry::getKey, Map.Entry::getValue, (e1, e2) -> e2,
                                LinkedHashMap::new));
    }
}

BookController

package bookstore.scraper.controller;

import bookstore.scraper.book.Book;
import bookstore.scraper.book.BookService;
import bookstore.scraper.book.rankingsystem.CategorizedBooksRankingService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;
import java.util.Map;

@RestController
public class BookController {

    private final CategorizedBooksRankingService categorizedBooksRankingService;
    private final BookService bookService;

    @Autowired
    public BookController(CategorizedBooksRankingService categorizedBooksRankingService, BookService bookService) {
        this.categorizedBooksRankingService = categorizedBooksRankingService;
        this.bookService = bookService;
    }

    @GetMapping("/bestsellers")
    public Map<Bookstore, List<Book>> getBestSellers() {
        return bookService.getBestsellers();
    }

    @GetMapping("/book/{title}")
    public Map<Bookstore, Book> getBookByTitle(@PathVariable String title) {
        return bookService.getMostPreciseBOok(title);
    }

    @GetMapping("/{categoryType}")
    public Map<Bookstore, List<Book>> get15RomanticBooks(@PathVariable CategoryType categoryType) {
        return bookService.getBooksByCategory(categoryType);
    }

    @GetMapping("/ranking/{categoryType}")
    public Map<String, Integer> getRankingForCategory(@PathVariable CategoryType categoryType) {
        return categorizedBooksRankingService.getRankingForCategory(CategoryType.forName(categoryType));
    }
}

Let's go further to Account staff.

AccountService - create useres.

package bookstore.scraper.account;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.crypto.bcrypt.BCrypt;
import org.springframework.stereotype.Service;

@Service
public class AccountService {

    private final AccountRepository accountRepository;

    @Autowired
    public AccountService(AccountRepository accountRepository) {
        this.accountRepository = accountRepository;
    }

    public Account createAccount(Account account) {
        if (accountRepository.existsByNickname(account.getNickname())) {
            throw new IllegalArgumentException("Account with that nickname already exists");
        }

        Account encryptedAccount = Account.builder()
                .nickname(account.getNickname())
                .password(BCrypt.hashpw(account.getPassword(), BCrypt.gensalt()))
                .build();

        return accountRepository.save(encryptedAccount);
    }
}

LoggedAccountService - retrieve logged account id

package bookstore.scraper.account;

import bookstore.scraper.account.security.AccountPrincipal;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Service;

@Service
public class LoggedAccountService {

    public int getLoggedAccountID() {
        AccountPrincipal accountPrincipal = (AccountPrincipal) SecurityContextHolder.getContext().getAuthentication().getPrincipal();
        return accountPrincipal.getId();
    }
}

MerlinUrlPropeties - I got .yml file with corresponding URLs.

package bookstore.scraper.urlproperties;

import bookstore.scraper.enums.CategoryType;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

import java.util.Objects;

@Getter
@Setter
@Component
@ConfigurationProperties("external.library.url.merlin")
public class MerlinUrlProperties {

    private String mostPreciseBook;
    private String bestSellers;
    private String concreteBook;
    private String romances;
    private String biographies;
    private String crime;
    private String guides;
    private String fantasy;

    public String getCategory(CategoryType categoryType) {

        switch (Objects.requireNonNull(categoryType)) {
            case CRIME:
                return getCrime();
            case BESTSELLER:
                return getBestSellers();
            case BIOGRAPHY:
                return getBiographies();
            case FANTASY:
                return getFantasy();
            case GUIDES:
                return getGuides();
            case MOST_PRECISE_BOOK:
                return getMostPreciseBook();
            case ROMANCES:
                return getRomances();
            default:
                throw new IllegalArgumentException("Unexpected categoryType: " + categoryType);
        }
    }
}

JSoupConnector - create connection to URL and return document

package bookstore.scraper;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.springframework.stereotype.Component;

import java.io.IOException;

@Component
public class JSoupConnector {

    public Document connect(String url) {
        try {
            return Jsoup.connect(url).get();
        } catch (IOException e) {
            throw new IllegalArgumentException("Cannot connect to" + url);
        }
    }
}

HistorySystemService - service responsible for fetching account history and saving account history.

package bookstore.scraper.historysystem;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class AccountHistorySystemService {

    private final HistoryRepository historyRepository;

    @Autowired
    public AccountHistorySystemService(HistoryRepository historyRepository) {
        this.historyRepository = historyRepository;
    }

    public List<AccountHistory> getHistoryOfAccount(int accountID) {
        if (!historyRepository.existsAccountHistoriesByAccountID(accountID)) {
            throw new IllegalArgumentException("Account with that ID does not exist");
        }

        return historyRepository.findAccountHistoriesByAccountID(accountID);
    }

    public void saveAccountHistory(int accountID, String action) {
        historyRepository.save(AccountHistory.builder().accountID(accountID).actionName(action).build());
    }
}

TEST SECTION

AccountServiceTEst

package bookstore.scraper.account;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;
import org.springframework.security.crypto.bcrypt.BCrypt;

import static org.junit.Assert.*;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;

@RunWith(MockitoJUnitRunner.class)
public class AccountServiceTest {

    @Mock
    AccountRepository accountRepository;

    @InjectMocks
    AccountService accountService;

    @Test(expected = IllegalArgumentException.class)
    public void createAccountWhenNicknameAlreadyExists() {
        Account dummyAccount = createDummyAccount();

        when(accountRepository.existsByNickname(dummyAccount.getNickname())).thenReturn(true);

        Account actualAccount = accountService.createAccount(dummyAccount);

        assertEquals(dummyAccount, actualAccount);
    }

    @Test
    public void createAccountWhenNicknameNotExistsInDB() {
        Account dummyAccount = createDummyAccount();

        when(accountRepository.existsByNickname(dummyAccount.getNickname())).thenReturn(false);
        when(accountRepository.save(any(Account.class))).thenReturn(dummyAccount);

        Account actualAccount = accountService.createAccount(dummyAccount);

        assertEquals(dummyAccount, actualAccount);
    }

    private Account createDummyAccount() {
        return Account
                .builder()
                .nickname("Piotr")
                .password(BCrypt.hashpw("123", BCrypt.gensalt()))
                .build();
    }
}

MerlinSourceTest

package bookstore.scraper.book.booksource.merlin;

import bookstore.scraper.book.Book;
import bookstore.scraper.enums.CategoryType;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.JSoupConnector;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.List;

import static bookstore.scraper.dataprovider.MerlinBookProvider.prepare15CrimeBooks;
import static bookstore.scraper.dataprovider.MerlinBookProvider.prepare5Bestsellers;
import static bookstore.scraper.dataprovider.MerlinBookProvider.prepareMostPreciseBook;
import static org.junit.Assert.assertEquals;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.when;

@RunWith(MockitoJUnitRunner.class)
public class MerlinSourceTest {

    @Mock
    JSoupConnector jSoupConnector;
    @Mock
    MerlinUrlProperties merlinUrlProperties;

    @InjectMocks
    MerlinSource merlinSource;

    @Test
    public void getBooksByCategory() throws IOException {
        File in = getFile("/merlin/CrimeCategoryMerlin.html");
        Document empikDocument = Jsoup.parse(in, "UTF-8");

        when(jSoupConnector.connect(any())).thenReturn(empikDocument);
        when(merlinUrlProperties.getConcreteBook()).thenReturn(anyString());

        List<Book> actualBooks = merlinSource.getBooksByCategory(CategoryType.CRIME);
        List<Book> expectedBooks = prepare15CrimeBooks();

        assertEquals(expectedBooks, actualBooks);
    }

    @Test
    public void getMostPreciseBook() throws IOException {
        File in = getFile("/merlin/MostPreciseBookMerlin.html");
        Document empikDocument = Jsoup.parse(in, "UTF-8");

        when(jSoupConnector.connect(any())).thenReturn(empikDocument);
        when(merlinUrlProperties.getCategory(CategoryType.MOST_PRECISE_BOOK)).thenReturn("https://merlin.pl/catalog/ksiazki-m10349074/?q=%s");
        when(merlinUrlProperties.getConcreteBook()).thenReturn(anyString());

        Book actualBooks = merlinSource.getMostPreciseBook("W pustyni i w puszczy. Lektura z opracowaniem - Henryk Sienkiewicz");
        Book expectedBooks = prepareMostPreciseBook();

        assertEquals(expectedBooks, actualBooks);
    }

    @Test
    public void getBestSellers() throws IOException {
        File in = getFile("/merlin/BestsellersMerlin.html");
        Document empikDocument = Jsoup.parse(in, "UTF-8");

        when(jSoupConnector.connect(any())).thenReturn(empikDocument);
        when(merlinUrlProperties.getConcreteBook()).thenReturn(anyString());

        List<Book> actualBooks = merlinSource.getBestSellers();
        List<Book> expectedBooks = prepare5Bestsellers();

        assertEquals(expectedBooks, actualBooks);
    }

    private File getFile(String resourceName) {
        try {
            return new File(MerlinSourceTest.class.getResource(resourceName).toURI());
        } catch (URISyntaxException e) {
            throw new IllegalStateException(e);
        }
    }
}

CategorizedBookRankingService

package bookstore.scraper.book.rankingsystem;

import bookstore.scraper.account.LoggedAccountService;
import bookstore.scraper.book.Book;
import bookstore.scraper.book.BookService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import bookstore.scraper.historysystem.AccountHistorySystemService;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.util.List;
import java.util.Map;

import static bookstore.scraper.dataprovider.MergedMapProvider.prepareCrimeBooksMap;
import static bookstore.scraper.dataprovider.MergedMapProvider.prepareExpectedRankingMap;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.when;

@RunWith(MockitoJUnitRunner.class)
public class CategorizedBooksRankingServiceTest {

    @Mock
    AccountHistorySystemService historySystemService;
    @Mock
    BookService bookService;
    @Mock
    LoggedAccountService loggedAccountService;

    @InjectMocks
    CategorizedBooksRankingService categorizedBooksRankingService;

    @Before
    public void setUp() {
        when(loggedAccountService.getLoggedAccountID()).thenReturn(1);
    }

    @Test
    public void getRankingForCrimeCategory() {
        Map<Bookstore, List<Book>> bookstoreWith15CrimeBooks = prepareCrimeBooksMap();

        when(bookService.getBooksByCategory(CategoryType.CRIME)).thenReturn(bookstoreWith15CrimeBooks);

        Map<String, Integer> actualMap = categorizedBooksRankingService.getRankingForCategory(CategoryType.CRIME);
        Map<String, Integer> expectedMap = prepareExpectedRankingMap();

        assertEquals(expectedMap, actualMap);
    }
}

BookServiceTest

package bookstore.scraper.book;

import bookstore.scraper.account.LoggedAccountService;
import bookstore.scraper.book.booksource.BookServiceSource;
import bookstore.scraper.dataprovider.EmpikBookProvider;
import bookstore.scraper.dataprovider.MerlinBookProvider;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import bookstore.scraper.historysystem.AccountHistorySystemService;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import static bookstore.scraper.dataprovider.MergedMapProvider.prepareCrimeBooksMap;
import static bookstore.scraper.dataprovider.MergedMapProvider.prepareExpectedMergedBestSellerMap;
import static bookstore.scraper.dataprovider.MergedMapProvider.prepareExpectedMergedMostPreciseBookMap;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

@RunWith(MockitoJUnitRunner.class)
public class BookServiceTest {

    @Mock
    AccountHistorySystemService historySystemService;
    @Mock
    LoggedAccountService loggedAccountService;

    @Before
    public void setUp() {
        when(loggedAccountService.getLoggedAccountID()).thenReturn(1);
    }

    @Test
    public void getBooksByCategory() {
        List<Book> merlinBestsellers = MerlinBookProvider.prepare15CrimeBooks();
        List<Book> empikBestsellers = EmpikBookProvider.prepare15CrimeBooks();

        BookServiceSource empikSource = mock(BookServiceSource.class);
        when(empikSource.getName()).thenReturn(Bookstore.EMPIK);
        when(empikSource.getBooksByCategory(CategoryType.CRIME)).thenReturn(empikBestsellers);

        BookServiceSource merlinSource = mock(BookServiceSource.class);
        when(merlinSource.getName()).thenReturn(Bookstore.MERLIN);
        when(merlinSource.getBooksByCategory(CategoryType.CRIME)).thenReturn(merlinBestsellers);

        List<BookServiceSource> sources = new ArrayList<>();
        sources.add(empikSource);
        sources.add(merlinSource);
        BookService service = new BookService(sources, historySystemService, loggedAccountService);

        Map<Bookstore, List<Book>> actualMap = service.getBooksByCategory(CategoryType.CRIME);
        Map<Bookstore, List<Book>> expectedMap = prepareCrimeBooksMap();

        assertEquals(expectedMap, actualMap);
    }

    @Test
    public void getMostPreciseBOok() {
        Book merlinBook = MerlinBookProvider.prepareMostPreciseBook();
        Book empikBook = EmpikBookProvider.prepareMostPreciseBook();

        BookServiceSource empikSource = mock(BookServiceSource.class);
        when(empikSource.getName()).thenReturn(Bookstore.EMPIK);
        when(empikSource.getMostPreciseBook("")).thenReturn(empikBook);

        BookServiceSource merlinSource = mock(BookServiceSource.class);
        when(merlinSource.getName()).thenReturn(Bookstore.MERLIN);
        when(merlinSource.getMostPreciseBook("")).thenReturn(merlinBook);

        List<BookServiceSource> sources = new ArrayList<>();
        sources.add(empikSource);
        sources.add(merlinSource);
        BookService service = new BookService(sources, historySystemService, loggedAccountService);

        Map<Bookstore, Book> expectedMap = prepareExpectedMergedMostPreciseBookMap();
        Map<Bookstore, Book> actualMap = service.getMostPreciseBOok("");

        assertEquals(expectedMap, actualMap);
    }

    @Test
    public void getBestsellers() {
        List<Book> merlinBestsellers = MerlinBookProvider.prepare5Bestsellers();
        List<Book> empikBestsellers = EmpikBookProvider.prepare5Bestsellers();

        BookServiceSource empikSource = mock(BookServiceSource.class);
        when(empikSource.getName()).thenReturn(Bookstore.EMPIK);
        when(empikSource.getBestSellers()).thenReturn(empikBestsellers);

        BookServiceSource merlinSource = mock(BookServiceSource.class);
        when(merlinSource.getName()).thenReturn(Bookstore.MERLIN);
        when(merlinSource.getBestSellers()).thenReturn(merlinBestsellers);

        List<BookServiceSource> sources = new ArrayList<>();
        sources.add(empikSource);
        sources.add(merlinSource);
        BookService service = new BookService(sources, historySystemService, loggedAccountService);

        Map<Bookstore, List<Book>> actualMap = service.getBestsellers();
        Map<Bookstore, List<Book>> expectedMap = prepareExpectedMergedBestSellerMap();

        assertEquals(expectedMap, actualMap);
    }
}

AccountHistoryServiceTest

package bookstore.scraper.historysystem;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.util.Collections;
import java.util.List;

import static org.hamcrest.collection.IsCollectionWithSize.hasSize;
import static org.junit.Assert.assertThat;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.Mockito.*;

@RunWith(MockitoJUnitRunner.class)
public class AccountHistorySystemServiceTest {

    @Mock
    HistoryRepository historyRepository;

    @InjectMocks
    AccountHistorySystemService historySystemService;

    @Test(expected = IllegalArgumentException.class)
    public void getHistoryOfUserWhenItDoesNotHaveAny() {
        AccountHistory dummyAccountHistory = createDummyAccountHistory();

        when(historyRepository.existsAccountHistoriesByAccountID(anyInt())).thenReturn(false);

        historySystemService.getHistoryOfAccount(dummyAccountHistory.getAccountID());
    }

    @Test
    public void getHistoryOfUserWhenItsExists() {
        AccountHistory dummyAccountHistory = createDummyAccountHistory();

        when(historyRepository.existsAccountHistoriesByAccountID(anyInt())).thenReturn(true);
        when(historyRepository.findAccountHistoriesByAccountID(anyInt())).thenReturn(Collections.singletonList(dummyAccountHistory));

        List<AccountHistory> actualAccountHistory = historySystemService.getHistoryOfAccount(dummyAccountHistory.getAccountID());

        assertThat(actualAccountHistory, hasSize(1));
    }

    @Test
    public void saveAccountHistory() {
        AccountHistory dummyAccountHistory = createDummyAccountHistory();
        historySystemService.saveAccountHistory(dummyAccountHistory.getAccountID(), dummyAccountHistory.getActionName());

        verify(historyRepository).save(dummyAccountHistory);
        verify(historyRepository, times(1)).save(createDummyAccountHistory());
    }

    private AccountHistory createDummyAccountHistory() {
        return AccountHistory
                .builder()
                .accountID(0)
                .actionName(ActionType.BEST_SELLERS.toString())
                .build();
    }

}

MerlinUrlPropertiesTest

package bookstore.scraper.urlproperties;

import bookstore.scraper.enums.CategoryType;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import static org.junit.Assert.assertEquals;

@SpringBootTest
@RunWith(SpringRunner.class)
public class MerlinUrlPropertiesTest {

    @Autowired
    MerlinUrlProperties merlinUrlProperties;

    @Test
    public void getCrimeCategory() {
        String actual = merlinUrlProperties.getCategory(CategoryType.BESTSELLER);
        String expected = "https://merlin.pl/bestseller/?option_80=10349074";

        assertEquals(expected, actual);
    }

    @Test
    public void getBiographyCategory() {
        String actual = merlinUrlProperties.getCategory(CategoryType.BIOGRAPHY);
        String expected = "https://merlin.pl/catalog/ksiazki-m10349074/biografie-c100115/";

        assertEquals(expected, actual);
    }

    @Test
    public void getRomancesCategory() {
        String actual = merlinUrlProperties.getCategory(CategoryType.ROMANCES);
        String expected = "https://merlin.pl/catalog/ksiazki-m10349074/romanse-c1774/";

        assertEquals(expected, actual);
    }

    @Test
    public void getFantasyCategory() {
        String actual = merlinUrlProperties.getCategory(CategoryType.FANTASY);
        String expected = "https://merlin.pl/catalog/ksiazki-m10349074/fantastyka-c467/";

        assertEquals(expected, actual);
    }
}

That's all for now. Looking forward to hear your opinions. Thanks.

\$\endgroup\$
1
\$\begingroup\$

The naming seems consistent, that's great. Also overall I don't have much to complain about, for Java it reads pretty well.


MerlinSource: I'd be a little bit concerned with the very hardcoded and very specific selectors, especially since something like grid__col--20-80-80 seems like it could very easily change without notice! Maybe some safeguards would make sense, like first checking if the requested element even existed (or by all means, checking for empty strings)?

The builder pattern looks good. For the whole IntStream mapping, of course it's good that the document is only loaded once and is then queried multiple times. Is there a benefit of going from index to li:nth-child, instead of e.g. finding all elements at that level and returning an iterator? Either way, checking that the book exists first would probably be good?

Ah, also the URLs are configuration, but to change the selectors the code will have to be recompiled - that could easily be abstracted out though and also made configuration ... but I guess that happens rarely enough.

Same for the constants, perhaps it would be easy enough to make them configuration values too?


BookService: Why's the ActionType passed as a String? I'd consider being more type safe and passing in objects directly, then calling toString if the database layer can't do that itself.

Apart from that the three methods are largely the same, if they're more coming, I'd consider some shared method here.


CategorizedBookRankingService: The name bookstoreWith15CategorizedBooks seems like it could easily be 14 or 16 books too?

I'm torn on getSortedLinkedHashMapByValue: The name is basically exactly what the implementation does. But it also tells me nothing what the purpose here is. Also, it looks like getTitleWithOccurrences already creates a map - maybe they could just be fused and getTitleWithOccurrences explicitly returns a sorted map result directly?


AccountService: Only nitpick is that encryptedAccount should at most be hashedAccount, it's not encrypting the password. But good that it does indeed hash it. Consider scrypt too while you're at it.


JSoupConnector: Is the wrapping into an IllegalArgumentException useful? It's not like the URL argument was necessarily invalid, there might have been a number of network problems instead that can prevent getting the document.


The tests look good to me, especially since they're not actually accessing the network.

\$\endgroup\$
  • 1
    \$\begingroup\$ I didn't expect any answer after such a long time. Thanks a lot. It's hard to create shared method from these 3 as getBestsellers has slightly different ending stream + finding name for just one method (after flatting 3 into 1) is really hard. About Jsoupconnector, how would you do it, then? \$\endgroup\$ – mara122 Sep 9 '19 at 13:40
  • \$\begingroup\$ Sure, that's fine then. JSoupConnector - ah I see you need it to be unchecked, right? Then I guess new RuntimeException("Cannot connect to" + url, e) would be preferable, possibly even making your own exception type if you want to catch and process it later. Wrapping the original exception at least gives you a chance at looking at the stacktrace etc. \$\endgroup\$ – ferada Sep 9 '19 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.