3
\$\begingroup\$

This is a quick question. I am reading a mass amount of JSON from a variety of text files. The JSONs are tweets, in this format:

{"filter_level":"medium","retweeted_status":{"contributors":null,"text":"Daily Mail: Vincent Kompany says Manchester United are favourites http://t.co/NB2p1cqlRs #mufc","geo":null,"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"truncated":false,"lang":"en","entities":{"symbols":[],"urls":[{"expanded_url":"http://dailym.ai/15y9p6X","indices":[66,88],"display_url":"dailym.ai/15y9p6X","url":"http://t.co/NB2p1cqlRs"}],"hashtags":[{"text":"mufc","indices":[89,94]}],"user_mentions":[]},"in_reply_to_status_id_str":null,"id":364326884395323393,"source":"<a href=\"http://twitterfeed.com\" rel=\"nofollow\">twitterfeed<\/a>","in_reply_to_user_id_str":null,"favorited":false,"in_reply_to_status_id":null,"retweet_count":9,"created_at":"Mon Aug 15 10:07:52 +0000 2013","in_reply_to_user_id":null,"favorite_count":0,"id_str":"364326884395323393","place":null,"user":{"location":"Old Trafford, Manchester, M16 ","default_profile":false,"profile_background_tile":true,"statuses_count":20839,"lang":"en","profile_link_color":"BD0015","profile_banner_url":"https://pbs.twimg.com/profile_banners/16557902/1347990848","id":16557902,"following":null,"protected":false,"favourites_count":0,"profile_text_color":"333333","description":"Tweeting the latest Manchester United news from the BBC and a number of other sources. Unofficial. Created by @gavreilly.","verified":false,"contributors_enabled":false,"profile_sidebar_border_color":"FFFFFF","name":"BBC Manchester Utd","profile_background_color":"BD0015","created_at":"Thu Oct 02 11:24:52 +0000 2008","default_profile_image":false,"followers_count":56086,"profile_image_url_https":"https://si0.twimg.com/profile_images/61130289/manutd_normal.png","geo_enabled":true,"profile_background_image_url":"http://a0.twimg.com/profile_background_images/662809119/lj2ujpnj7iwwvvxxpoor.jpeg","profile_background_image_url_https":"https://si0.twimg.com/profile_background_images/662809119/lj2ujpnj7iwwvvxxpoor.jpeg","follow_request_sent":null,"url":"http://news.bbc.co.uk/sport2/hi/football/teams/m/man_utd/default.stm","utc_offset":3600,"time_zone":"London","notifications":null,"profile_use_background_image":true,"friends_count":54572,"profile_sidebar_fill_color":"FFC8C7","screen_name":"bbcmanutd","id_str":"16557902","profile_image_url":"http://a0.twimg.com/profile_images/61130289/manutd_normal.png","listed_count":789,"is_translator":false},"coordinates":null},"contributors":null,"text":"RT @bbcmanutd: Daily Mail: Vincent Kompany says Manchester United are favourites http://t.co/NB2p1cqlRs #mufc","geo":null,"retweeted":false,"in_reply_to_screen_name":null,"truncated":false,"lang":"en","entities":{"symbols":[],"urls":[],"hashtags":[{"text":"mufc","indices":[104,109]}],"user_mentions":[{"id":16557902,"name":"BBC Manchester Utd","indices":[3,13],"screen_name":"bbcmanutd","id_str":"16557902"}]},"in_reply_to_status_id_str":null,"id":364329399950540800,"source":"<a href=\"http://blackberry.com/twitter\" rel=\"nofollow\">Twitter for BlackBerry®<\/a>","in_reply_to_user_id_str":null,"favorited":false,"in_reply_to_status_id":null,"retweet_count":0,"created_at":"Mon Aug 15 10:17:52 +0000 2013","in_reply_to_user_id":null,"favorite_count":0,"id_str":"364329399950540800","place":null,"user":{"location":"","default_profile":false,"profile_background_tile":false,"statuses_count":3520,"lang":"en","profile_link_color":"2FC2EF","profile_banner_url":"https://pbs.twimg.com/profile_banners/561283874/1375643576","id":561283874,"following":null,"protected":false,"favourites_count":234,"profile_text_color":"666666","description":"A curious case. An enigma. ..","verified":false,"contributors_enabled":false,"profile_sidebar_border_color":"181A1E","name":"vi̲̅ƞci̲̅ thE EnIgma","profile_background_color":"1A1B1F","created_at":"Mon Apr 23 16:41:58 +0000 2012","default_profile_image":false,"followers_count":195,"profile_image_url_https":"https://si0.twimg.com/profile_images/378800000241332248/569425827c3cc3ad8ff44958bad5307a_normal.jpeg","geo_enabled":false,"profile_background_image_url":"http://a0.twimg.com/images/themes/theme9/bg.gif","profile_background_image_url_https":"https://si0.twimg.com/images/themes/theme9/bg.gif","follow_request_sent":null,"url":null,"utc_offset":7200,"time_zone":"Amsterdam","notifications":null,"profile_use_background_image":true,"friends_count":339,"profile_sidebar_fill_color":"252429","screen_name":"Dinnydavinci","id_str":"561283874","profile_image_url":"http://a0.twimg.com/profile_images/378800000241332248/569425827c3cc3ad8ff44958bad5307a_normal.jpeg","listed_count":0,"is_translator":false},"coordinates":null}

I am reading them with the following code:

public void performAnalysis() {
    File tweets;
    String[] categories;
    tweets = new File("//home//andrew//Python//Tweets//");
    categories = tweets.list();

    for (int i = 0; i < categories.length; ++i) {
        String category = categories[i];
        File file = new File(tweets, category);
        try {
            Scanner sc = new Scanner(file);
            while (sc.hasNextLine()) {
                String rawTweet = sc.nextLine().trim();
                if (rawTweet.equals(""))
                    continue;
                Tweet parsedTweet = new Gson().fromJson(rawTweet, Tweet.class);
                System.out.println(parsedTweet.getText());
            }
            sc.close();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }
    }
}

I have another class called Tweet, with one instance variable: text. The Google Gson library allows me to strip the field "text" from the JSON which I am importing.

My problem is regarding the if statement in the while loop. It seems to be necessary. I didn't realise I had whitespace after my JSONs, but the program crashes at the end of each file it reads if I don't include this.

Does anybody have any suggestions on how to check the text file for white space or get rid of it without this if statement? Maybe I'm totally wrong, but it just looks horrible to me - especially since I'm saving the line, trimming it, THEN checking if it's empty. I was going to check first, but I don't know how without using sc.nextLine() twice (thus skipping a line).

\$\endgroup\$

closed as off-topic by Jamal Nov 21 '13 at 20:46

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking for code to be written to solve a specific problem are off-topic here as there is no code to review." – Jamal
If this question can be reworded to fit the rules in the help center, please edit the question.

  • \$\begingroup\$ Some suggestions not related to your question : you can use for-each instead of a normal for loop. And if I'm not mistaken, you don't need to create a new Gson object all the time (not too sure about that). \$\endgroup\$ – Marc-Andre Aug 12 '13 at 13:21
  • \$\begingroup\$ Maybe the problem is when you're receiving or saving the JSON. Cause if the problem is a blank space at the end of the file when you receive it or save it you should fix the problem there and not in the parsing phase. \$\endgroup\$ – Marc-Andre Aug 12 '13 at 15:22
  • \$\begingroup\$ I am downloading the tweets using Flume and streaming them into Hadoop. After some operation, I output them as JSON by saving them to text directly as they are. \$\endgroup\$ – Andrew Martin Aug 12 '13 at 15:25
  • 1
    \$\begingroup\$ If you receive it with a blank space at the end and save it with the space, then you should trim it before saving, that would simplify the parsing. That is if the white space is at the end of the file. \$\endgroup\$ – Marc-Andre Aug 12 '13 at 15:32
1
\$\begingroup\$

You're skipping a line that contains spaces. That's what the trim() does, removes spaces from the beginning and the end of a line.

There is this alternative:

    if (!rawTweet.equals("")) {
        Tweet parsedTweet = new Gson().fromJson(rawTweet, Tweet.class);
        System.out.println(parsedTweet.getText());
    }
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.