2
\$\begingroup\$

I have made a program that sort of mimics YouTube trending, but only with videos from channels that I am subscribed to. It looks at each Subscription, converts the subscription to a Channel, gets the channel's uploads PlayList, converts each PlayListItem to a Video, and finally filters the videos to only include new uploads and then sorts by views.

It works, I guess, but is so painfully slow that the app is unusable. Manually looping through everything takes too much time. I am subscribed to too many channels, and each channel has too many uploads. I tried to shed time by replacing loops with streams where I could, but it doesn't really help.

The primary class in the program, and the only one I will be posting here, is APICaller.java.

import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.extensions.java6.auth.oauth2.AuthorizationCodeInstalledApp;
import com.google.api.client.extensions.jetty.auth.oauth2.LocalServerReceiver;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.youtube.YouTube;
import com.google.api.services.youtube.model.*;

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.security.GeneralSecurityException;
import java.time.LocalDate;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

public class APICaller {

    private static final String CLIENT_SECRETS = "client_secret.json";
    private static final Collection<String> SCOPES =
            Collections.singletonList("https://www.googleapis.com/auth/youtube.readonly");

    private static final String APPLICATION_NAME = "Better YouTube Trending";
    private static final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();

    public static List<Video> generateVideos() throws IOException, GeneralSecurityException {
        YouTube youtubeService = getService();
        List<Subscription> subscriptions = getSubscriptions(youtubeService);
        List<String> playLists = getPlayLists(youtubeService, subscriptions);
        List<PlaylistItem> playlistItems = getPlayListItems(youtubeService, playLists);
        List<Video> videos = getVideos(youtubeService, playlistItems);

        videos.sort(APICaller::comparison);

        return videos;
    }

    private static YouTube getService() throws GeneralSecurityException, IOException {
        final NetHttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
        Credential credential = authorize(httpTransport);
        return new YouTube.Builder(httpTransport, JSON_FACTORY, credential)
                .setApplicationName(APPLICATION_NAME)
                .build();
    }

    private static Credential authorize(final NetHttpTransport httpTransport) throws IOException {
        // Load client secrets.
        InputStream in = APICaller.class.getResourceAsStream(CLIENT_SECRETS);
        GoogleClientSecrets clientSecrets =
                GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));
        // Build flow and trigger user authorization request.
        GoogleAuthorizationCodeFlow flow =
                new GoogleAuthorizationCodeFlow.Builder(httpTransport, JSON_FACTORY, clientSecrets, SCOPES)
                        .build();
        return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");
    }

    private static List<Subscription> getSubscriptions(YouTube youtubeService) throws IOException {
        List<Subscription> subscriptions = new ArrayList<>();
        YouTube.Subscriptions.List request = youtubeService.subscriptions().list("snippet");
        SubscriptionListResponse response;
        String nextPageToken = "";
        for (boolean first = true; nextPageToken != null; first = false) {
            if (first) {
                response = request.setMine(true)
                        .setMaxResults(50L)
                        .execute();
            } else {
                response = request.setMine(true)
                        .setPageToken(nextPageToken)
                        .setMaxResults(50L)
                        .execute();
            }
            subscriptions.addAll(response.getItems());
            nextPageToken = response.getNextPageToken();
        }
        return subscriptions;
    }

    private static List<String> getPlayLists(YouTube youtubeService, List<Subscription> subscriptions) throws IOException {
        List<String> playLists = new ArrayList<>();
        YouTube.Channels.List request = youtubeService.channels().list("contentDetails");
        subscriptions.forEach(s -> {
            try {
                String id = s.getSnippet().getResourceId().getChannelId();
                ChannelListResponse response = request.setId(id).execute();
                playLists.add(response.getItems().get(0).getContentDetails().getRelatedPlaylists().getUploads());
            } catch (IOException e) {
                e.printStackTrace();
            }
        });
        return playLists;
    }

    private static List<PlaylistItem> getPlayListItems(YouTube youtubeService, List<String> playLists) throws IOException {
        List<PlaylistItem> videos = new ArrayList<>();
        YouTube.PlaylistItems.List request = youtubeService.playlistItems().list("snippet");
        playLists.forEach(playlist -> {
            try {
                PlaylistItemListResponse response;
                String nextPageToken = "";
                for (boolean first = true; nextPageToken != null; first = false) {
                    if (first) {
                        response = request.setPlaylistId(playlist)
                                .setMaxResults(50L)
                                .execute();
                    } else {
                        response = request.setPlaylistId(playlist)
                                .setPageToken(nextPageToken)
                                .setMaxResults(50L)
                                .execute();
                    }
                    videos.addAll(response.getItems());
                    nextPageToken = response.getNextPageToken();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        });
        return videos;
    }

    private static List<Video> getVideos(YouTube youtubeService, List<PlaylistItem> playlistItems) throws IOException {
        List<Video> videos = new ArrayList<>();
        YouTube.Videos.List request = youtubeService.videos().list("snippet,statistics");
        playlistItems.forEach(playlistItem -> {
            String id = playlistItem.getSnippet().getResourceId().getVideoId();
            try {
                VideoListResponse response = request.setId(id).execute();
                if (isNewVideo(response.getItems().get(0))) {
                    videos.add(response.getItems().get(0));
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        });
        return videos;
    }

    private static boolean isNewVideo(Video video) {
        LocalDate publishedAt = LocalDate.parse(video.getSnippet().getPublishedAt().toString().substring(0, 10));
        return ChronoUnit.DAYS.between(publishedAt, LocalDate.now()) < 7;
    }

    private static int comparison(Video video1, Video video2) {
        return (int)(video1.getStatistics().getViewCount().longValue()) - (int)(video2.getStatistics().getViewCount().longValue());
    }
}

The program is run by calling the one public method in the class: generateVideos(). You would need to use your own client_secret.json file, which you can download here. The method returns returns a List of videos, which would then be used elsewhere in the program to populate the GUI. I haven't included all that because it isn't really relevant to my question.

So, how would you improve this code? Are there any glaring inefficiencies, obvious mistakes, extraneous steps, and the like?

\$\endgroup\$
  • 2
    \$\begingroup\$ I don't know youtube api, but it seems me you are making distinct requests limited to the first 50 results, so if you have a great number of items to retrieve this could explain why it is so slow. \$\endgroup\$ – dariosicily Oct 2 '20 at 9:16
  • 1
    \$\begingroup\$ If I read this correctly, you page throug all the results, never stopping until the very last page. How many results and how many calls for 50 items is this in total? How long does a single list() call take? \$\endgroup\$ – mtj Oct 2 '20 at 9:52
  • 1
    \$\begingroup\$ @dariosicily The Youtube API only allows a maximum of 50 results with each response. You have to paginate through the whole thing in order to get all the results. \$\endgroup\$ – a p Oct 2 '20 at 12:49
  • 1
    \$\begingroup\$ I see, then if you have a great number of items you are obliged to do several calls of the API. \$\endgroup\$ – dariosicily Oct 2 '20 at 13:42
  • 1
    \$\begingroup\$ @dariosicily I initially set it to return some absurdly large number of results with one call, but it always returned a maximum of 50 anyway. That led me to this on Stack Overflow: Youtube API V3; Maximum number of videos only 50? \$\endgroup\$ – a p Oct 2 '20 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.