DO NOT REVIEW IT. I PROVIDED BIG CHANGES, SO THIS CODE IS NOT UPDATED
I'm developing my bookscraper to improve myself in coding and I think I have solid structure that I want to be reviewed by more experienced people than me. It is app based on SpringBoot which fetches data from 2 different bookstores. For now it has 3 options of fetching books.
1) Bestsellers
2) Book with given title e.g: You say you want to get book with title "Great boy" and it gives you book from each bookstore with title, price, link to the book
3) Categorized books (currently for 5 categories e.g.: crimes, biogrpahies etc)
I have created ranking for categorized books which takes 15 books from each bookstore, then it counts how many given title was repeated (if for example two, then it is higher in the ranking) and returns it in the map Map<String,Integer>
which goes for title and number of occurrences.
I have integration and unit tests. In my test resources I put 3 files for each bookstore which contains .html
files for each type of fetching (bestsellers,categorized book,most precise book).
Here is the link to github, because I'm not gonna put all the classes here as there is a lot of them. https://github.com/must1/BookstoreScraper
I added also CI with Travis and SonarCloud. JsoupConnector (responsible for connecting to the given url)
package bookstore.scraper.utilities;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.springframework.stereotype.Component;
import java.io.IOException;
@Component
public class JSoupConnector {
public Document connect(String url) {
try {
return Jsoup.connect(url).get();
} catch (IOException e) {
throw new IllegalArgumentException("Cannot connect to" + url);
}
}
}
MerlinUrlProperties (all links are stored in .yml
file), same class for second bookstore
package bookstore.scraper.urlproperties;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
@Getter
@Setter
@Component
@ConfigurationProperties("external.library.url")
public class MerlinUrlProperties {
private Merlin merlin = new Merlin();
@Getter
@Setter
public static class Merlin {
private String mostPreciseBook;
private String bestSellers;
private String concreteBook;
private String romances;
private String biographies;
private String crime;
private String guides;
private String fantasy;
}
}
MerlinFetchingBookService (fetches data from the net)
package bookstore.scraper.fetcher.merlin;
import bookstore.scraper.book.Book;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@Service
public class MerlinFetchingBookService {
private static final int BESTSELLERS_NUMBER_TO_FETCH = 6;
private static final int CATEGORIZED_BOOKS_NUMBER_TO_FETCH = 16;
private final MerlinUrlProperties merlinUrlProperties;
@Autowired
public MerlinFetchingBookService(MerlinUrlProperties merlinUrlProperties) {
this.merlinUrlProperties = merlinUrlProperties;
}
public Book getMostPreciseMerlinBook(Document document) {
String title = document.select("div.b-products-list__title-holder").select("a").first().text();
String price = document.select("div.b-products-list__price-holder > a").first().text();
String author = document.select("div.b-products-list__manufacturer-holder").select("a").first().text();
String productID = document.select("div.grid__col.grid__col--20-80-80.b-products-wrap > ul > li:nth-child(1)").first().attr("data-ppc-id");
String bookUrl = createBookUrl(title, productID);
return new Book.BookBuilder()
.withAuthor(author)
.withPrice(price)
.withTitle(title)
.withProductID(productID)
.withBookUrl(bookUrl)
.build();
}
public List<Book> get5BestSellersMerlin(Document document) {
return IntStream.range(1, BESTSELLERS_NUMBER_TO_FETCH)
.mapToObj(iterator -> getBestSellerOrCategorizedBook(document, iterator))
.collect(Collectors.toList());
}
public List<Book> get15BooksFromCategory(Document document) {
return IntStream.range(1, CATEGORIZED_BOOKS_NUMBER_TO_FETCH)
.mapToObj(iterator -> getBestSellerOrCategorizedBook(document, iterator))
.collect(Collectors.toList());
}
//Todo
//need to think about better name for the method as it can be used for fetching bestseller and categorized book
private Book getBestSellerOrCategorizedBook(Document document, int iteratedBook) {
Elements siteElements = document.select("div.grid__col.grid__col--20-80-80.b-products-wrap > ul > li:nth-child(" + iteratedBook + ")");
String author = siteElements.select(" > div > div.b-products-list__desc-wrap > div > div.b-products-list__main-content > div.b-products-list__desc-prime > div.b-products-list__manufacturer-holder").select("a").first().text();
String title = siteElements.select(" > div > div.b-products-list__desc-wrap > div > div.b-products-list__main-content > div.b-products-list__desc-prime > div.b-products-list__title-holder > a").first().text();
String price = siteElements.select(" div.b-products-list__price-holder > a").first().text();
String productID = siteElements.first().attr("data-ppc-id");
String bookUrl = createBookUrl(title, productID);
return new Book.BookBuilder()
.withAuthor(author)
.withPrice(price)
.withTitle(title)
.withProductID(productID)
.withBookUrl(bookUrl)
.build();
}
private String createBookUrl(String title, String productID) {
return String.format(merlinUrlProperties.getMerlin().getConcreteBook(), title, productID);
}
}
CategorizedBookRankingService (responsible for ranking for given category)
package bookstore.scraper.rankingsystem;
import bookstore.scraper.book.Book;
import bookstore.scraper.book.scrapingtypeservice.CategorizedBookService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.util.*;
import java.util.stream.Stream;
import static java.util.stream.Collectors.*;
@Slf4j
@Component
public class CategorizedBooksRankingService {
private final CategorizedBookService categorizedBookService;
@Autowired
public CategorizedBooksRankingService(CategorizedBookService categorizedBookService) {
this.categorizedBookService = categorizedBookService;
}
public Map<String, Integer> getRankingForCategory(CategoryType category) {
Map<Bookstore, List<Book>> bookstoreWith15CategorizedBooks = chooseGetterImplementationByCategory(category);
List<Book> merlinBooks = bookstoreWith15CategorizedBooks.get(Bookstore.MERLIN);
List<Book> empikBooks = bookstoreWith15CategorizedBooks.get(Bookstore.EMPIK);
Map<String, List<String>> purifiedTitleWithOriginalTitles = getPurifiedTitleWithAccordingOriginalTitles(merlinBooks, empikBooks);
Map<String, Integer> bookTitleWithOccurrencesNumber = getTitleWithOccurrences(purifiedTitleWithOriginalTitles);
return getSortedLinkedHashMapByValue(bookTitleWithOccurrencesNumber);
}
private Map<String, List<String>> getPurifiedTitleWithAccordingOriginalTitles(List<Book> list1, List<Book> list2) {
return Stream.concat(list1.stream(), list2.stream())
.collect(
groupingBy(Book::getPurifiedTitle, mapping(Book::getTitle, toList())));
}
private Map<String, Integer> getTitleWithOccurrences(Map<String, List<String>> map) {
return map.values().stream().collect(toMap(list -> list.get(0), List::size));
}
private Map<String, Integer> getSortedLinkedHashMapByValue(Map<String, Integer> mapToSort) {
return mapToSort.entrySet()
.stream()
.sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
.collect(
toMap(Map.Entry::getKey, Map.Entry::getValue, (e1, e2) -> e2,
LinkedHashMap::new));
}
private Map<Bookstore, List<Book>> chooseGetterImplementationByCategory(CategoryType categoryType) {
Map<Bookstore, List<Book>> map = new EnumMap<>(Bookstore.class);
if (categoryType.equals(CategoryType.CRIME))
map = categorizedBookService.get15BooksFromCrimeCategory();
if (categoryType.equals(CategoryType.ROMANCES))
map = categorizedBookService.get15BooksFromRomanceCategory();
if (categoryType.equals(CategoryType.FANTASY))
map = categorizedBookService.get15BooksFromFantasyCategory();
if (categoryType.equals(CategoryType.GUIDES))
map = categorizedBookService.get15BooksFromGuidesCategory();
if (categoryType.equals(CategoryType.BIOGRAPHY))
map = categorizedBookService.get15BooksFromBiographiesCategory();
return map;
}
}
CategorizedBookService
package bookstore.scraper.book.scrapingtypeservice;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.EnumMap;
import java.util.List;
import java.util.Map;
@Service
@Slf4j
public class CategorizedBookService {
private final EmpikFetchingBookService empikBookService;
private final MerlinFetchingBookService merlinFetchingBookService;
private final EmpikUrlProperties empikUrlProperties;
private final MerlinUrlProperties merlinUrlProperties;
private final JSoupConnector jSoupConnector;
@Autowired
public CategorizedBookService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinFetchingBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
this.empikBookService = empikBookService;
this.merlinFetchingBookService = merlinFetchingBookService;
this.empikUrlProperties = empikUrlProperties;
this.merlinUrlProperties = merlinUrlProperties;
this.jSoupConnector = jSoupConnector;
}
public Map<Bookstore, List<Book>> get15BooksFromRomanceCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getRomances(), merlinUrlProperties.getMerlin().getRomances());
}
public Map<Bookstore, List<Book>> get15BooksFromFantasyCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getFantasy(), merlinUrlProperties.getMerlin().getFantasy());
}
public Map<Bookstore, List<Book>> get15BooksFromCrimeCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getCrime(), merlinUrlProperties.getMerlin().getCrime());
}
public Map<Bookstore, List<Book>> get15BooksFromGuidesCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getGuides(), merlinUrlProperties.getMerlin().getGuides());
}
public Map<Bookstore, List<Book>> get15BooksFromBiographiesCategory() {
return get15BooksFrom(empikUrlProperties.getEmpik().getBiographies(), merlinUrlProperties.getMerlin().getBiographies());
}
private Map<Bookstore, List<Book>> get15BooksFrom(String bookStoreEmpikURL, String bookStoreMerlinURL) {
Map<Bookstore, List<Book>> bookstoreWith15CategorizedBooks = new EnumMap<>(Bookstore.class);
bookstoreWith15CategorizedBooks.put(Bookstore.EMPIK, empikBookService
.get15BooksFromCategory(jSoupConnector.connect(bookStoreEmpikURL)));
bookstoreWith15CategorizedBooks.put(Bookstore.MERLIN, merlinFetchingBookService
.get15BooksFromCategory(jSoupConnector.connect(bookStoreMerlinURL)));
return bookstoreWith15CategorizedBooks;
}
}
MostPreciseBookService
package bookstore.scraper.book.scrapingtypeservice;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.EnumMap;
import java.util.Map;
@Service
public class MostPreciseBookService {
private final EmpikFetchingBookService empikBookService;
private final MerlinFetchingBookService merlinBookService;
private final EmpikUrlProperties empikUrlProperties;
private final MerlinUrlProperties merlinUrlProperties;
private final JSoupConnector jSoupConnector;
@Autowired
public MostPreciseBookService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
this.empikBookService = empikBookService;
this.merlinBookService = merlinBookService;
this.empikUrlProperties = empikUrlProperties;
this.merlinUrlProperties = merlinUrlProperties;
this.jSoupConnector = jSoupConnector;
}
public Map<Bookstore, Book> getBookByTitle(String title) {
Map<Bookstore, Book> bookstoreWithMostPreciseBook = new EnumMap<>(Bookstore.class);
bookstoreWithMostPreciseBook.put(Bookstore.MERLIN,
merlinBookService.getMostPreciseMerlinBook(
jSoupConnector.connect(concatUrlWithTitle(merlinUrlProperties.getMerlin().getMostPreciseBook(), title))));
bookstoreWithMostPreciseBook.put(Bookstore.EMPIK,
empikBookService.getMostPreciseEmpikBook(
jSoupConnector.connect(concatUrlWithTitle(empikUrlProperties.getEmpik().getMostPreciseBook(), title))));
return bookstoreWithMostPreciseBook;
}
private String concatUrlWithTitle(String url, String title) {
return String.format(url, title);
}
}
BestsellersService
package bookstore.scraper.book.scrapingtypeservice;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.EnumMap;
import java.util.List;
import java.util.Map;
@Service
public class BestSellersService {
private final EmpikUrlProperties empikUrlProperties;
private final MerlinUrlProperties merlinUrlProperties;
private final EmpikFetchingBookService empikBookService;
private final MerlinFetchingBookService merlinBookService;
private final JSoupConnector jSoupConnector;
@Autowired
public BestSellersService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
this.empikBookService = empikBookService;
this.merlinBookService = merlinBookService;
this.empikUrlProperties = empikUrlProperties;
this.merlinUrlProperties = merlinUrlProperties;
this.jSoupConnector = jSoupConnector;
}
public Map<Bookstore, List<Book>> getBestSellers() {
Map<Bookstore, List<Book>> bookstoreWithBestSellers = new EnumMap<>(Bookstore.class);
bookstoreWithBestSellers.put(Bookstore.EMPIK, empikBookService
.get5BestSellersEmpik(jSoupConnector.connect(empikUrlProperties.getEmpik().getBestSellers())));
bookstoreWithBestSellers.put(Bookstore.MERLIN, merlinBookService
.get5BestSellersMerlin(jSoupConnector.connect(merlinUrlProperties.getMerlin().getBestSellers())));
return bookstoreWithBestSellers;
}
}
TESTS
CategorizedBooksRankingServiceTest
package bookstore.scraper.rankingsystem;
import bookstore.scraper.book.Book;
import bookstore.scraper.book.scrapingtypeservice.CategorizedBookService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;
import java.util.List;
import java.util.Map;
import static bookstore.scraper.dataprovider.MerlinBookProvider.prepareExpectedRankingMap;
import static bookstore.scraper.dataprovider.MerlinBookProvider.prepareMapWithBookstoreAndCrimeBooks;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.when;
@RunWith(MockitoJUnitRunner.class)
public class CategorizedBooksRankingServiceTest {
@Mock
CategorizedBookService categorizedBookService;
@InjectMocks
CategorizedBooksRankingService categorizedBooksRankingService;
@Test
public void getRankingForCrimeCategory() {
Map<Bookstore, List<Book>> bookstoreWith15CrimeBooks = prepareMapWithBookstoreAndCrimeBooks();
when(categorizedBookService.get15BooksFromCrimeCategory()).thenReturn(bookstoreWith15CrimeBooks);
Map<String, Integer> actualMap = categorizedBooksRankingService.getRankingForCategory(CategoryType.CRIME);
Map<String, Integer> expectedMap = prepareExpectedRankingMap();
assertEquals(expectedMap, actualMap);
}
}
MerlinFetchingBookService
package bookstore.scraper.fetcher.merlin;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookServiceTest;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.List;
import static bookstore.scraper.dataprovider.MerlinBookProvider.*;
import static org.junit.Assert.assertEquals;
@SpringBootTest
@RunWith(SpringRunner.class)
public class MerlinFetchingBookServiceTest {
@Autowired
private MerlinFetchingBookService merlinFetchingBookService;
@Test
public void getMostPreciseMerlinBook() throws IOException {
File in = getFile("/merlin/MostPreciseBookMerlin.html");
Document doc = Jsoup.parse(in, "UTF-8");
Book actualBooks = merlinFetchingBookService.getMostPreciseMerlinBook(doc);
Book expectedBooks = prepareMostPreciseBook();
assertEquals(expectedBooks, actualBooks);
}
@Test
public void get5BestSellersMerlin() throws IOException {
File in = getFile("/merlin/BestsellersMerlin.html");
Document doc = Jsoup.parse(in, "UTF-8");
List<Book> actualBooks = merlinFetchingBookService.get5BestSellersMerlin(doc);
List<Book> expectedBooks = prepare5Bestsellers();
assertEquals(expectedBooks, actualBooks);
}
@Test
public void get15BooksFromCategory() throws IOException {
File in = getFile("/merlin/CrimeCategoryMerlin.html");
Document doc = Jsoup.parse(in, "UTF-8");
List<Book> actualBooks = merlinFetchingBookService.get15BooksFromCategory(doc);
List<Book> expectedBooks = prepare15CrimeBooks();
assertEquals(expectedBooks, actualBooks);
}
private File getFile(String resourceName) {
try {
return new File(EmpikFetchingBookServiceTest.class.getResource(resourceName).toURI());
} catch (URISyntaxException e) {
throw new IllegalStateException(e);
}
}
}
BestsellersServiceTest
package bookstore.scraper.book.scrapingtypeservice;
import bookstore.scraper.book.Book;
import bookstore.scraper.dataprovider.EmpikBookProvider;
import bookstore.scraper.dataprovider.MerlinBookProvider;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import org.jsoup.nodes.Document;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;
import java.util.List;
import java.util.Map;
import static bookstore.scraper.dataprovider.MergedBestsellersMapProvider.prepareExpectedMergedBestSellerMap;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@RunWith(MockitoJUnitRunner.class)
public class BestsellersServiceTest {
@Mock
private EmpikFetchingBookService empikBookService;
@Mock
private MerlinFetchingBookService merlinBookService;
@Mock
private EmpikUrlProperties empikUrlProperties;
@Mock
private MerlinUrlProperties merlinUrlProperties;
@Mock
private EmpikUrlProperties.Empik empikMock;
@Mock
private MerlinUrlProperties.Merlin merlinMock;
@Mock
JSoupConnector jSoupConnector;
@InjectMocks
private BestSellersService bestSellersService;
@Test
public void getBestSellers() {
List<Book> merlinBestsellers = MerlinBookProvider.prepare5Bestsellers();
List<Book> empikBestsellers = EmpikBookProvider.prepare5Bestsellers();
Document empikDocument = mock(Document.class);
Document merlinDocument = mock(Document.class);
when(jSoupConnector.connect("https://www.empik.com/bestsellery/ksiazki")).thenReturn(empikDocument);
when(empikUrlProperties.getEmpik()).thenReturn(empikMock);
when(empikMock.getBestSellers()).thenReturn("https://www.empik.com/bestsellery/ksiazki");
when(empikBookService.get5BestSellersEmpik(empikDocument)).thenReturn(empikBestsellers);
when(jSoupConnector.connect("https://merlin.pl/bestseller/?option_80=10349074")).thenReturn(merlinDocument);
when(merlinMock.getBestSellers()).thenReturn("https://merlin.pl/bestseller/?option_80=10349074");
when(merlinUrlProperties.getMerlin()).thenReturn(merlinMock);
when(merlinBookService.get5BestSellersMerlin(merlinDocument)).thenReturn(merlinBestsellers);
Map<Bookstore, List<Book>> actualMap = bestSellersService.getBestSellers();
Map<Bookstore, List<Book>> expectedMap = prepareExpectedMergedBestSellerMap();
assertEquals(expectedMap, actualMap);
}
}
Those are main classes that I want to be reviewed. Thanks a lot for each suggestion/opinion!