# Reading in millions of lines of AOL query logs for query suggestion/expansion

I am working on a query suggestion program, and to faciliate this, I am reading in millions of AOL queries logs. The logs have this format:

AnonID  Query   QueryTime
479 family guy  2006-03-01 16:01:20
479 family guy movie references 2006-03-03 22:37:46
479 top grossing movies of all time 2006-03-03 22:42:42
479 car decals  2006-03-03 23:20:12
479 car window decals   2006-03-03 23:24:05
479 car window sponsor decals   2006-03-03 23:27:17
479 car sponsor decals  2006-03-03 23:28:59
...


Where each "column" is separated by a tab character.

This following code reads in the AOL Query logs into a Guava MultiMap (to allow duplicates) and then puts those into an Apache-Collections PatriciaTrie so that prefixes can be grabbed efficiently for future query suggestion/expansion.

private void readInAolQueries(String directory)
{
try (Stream<Path> paths = Files.walk(Paths.get(directory)))
{
Multimap<String, QueryLog> multimap = TreeMultimap.create();
paths
.filter(Files::isRegularFile)
.filter(path -> path.toString().endsWith(".txt"))
.filter(path -> path.toString().contains("Clean-Data-"))
.forEach(path ->
{
String fileName = path.toString();
List<QueryLog> queryLogs = new AolQueryLogsProcessor(fileName).getQueryLogs();
// Read logs into a multimap to preserve duplicates
multimap.putAll(Multimaps.index(queryLogs, QueryLog::getQueryString));
});
//Put the multimap into the trie. It now also has duplicates.
}
catch (IOException e)
{
e.printStackTrace();
}
}


The AOL logs files are in my resources directory (which is passed into the readInAolQueries method), and the files all have the form Clean-Data-XX.txt where XX is a number.

Here is the code for the AolQueryLogsProcessor class from the code above:

public class AolQueryLogsProcessor
{
private List<QueryLog> queryLogs;

public AolQueryLogsProcessor(String fileName)
{
queryLogs = new ArrayList<>();
try
{
.stream()
.filter(line -> !line.isEmpty())
.map(line -> line.split("\t"))
.collect(Collectors.toCollection(ArrayList<String[]>::new));

//Dates looks like this 2006-03-28 20:39:58
final DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");

//The first line is the header. Skip it.
for (int i = 1; i < lines.size(); ++i)
{
String anonId = lines.get(i)[0];
String[] query = lines.get(i)[1].split("\\s");
LocalDateTime timeStamp = LocalDateTime.parse(lines.get(i)[2], formatter);
QueryLog log = new QueryLog(anonId, query, timeStamp);
}
}
catch (IOException e)
{
e.printStackTrace();
}
}

private List<String> readFile(String fileName) throws IOException
{
}

public List<QueryLog> getQueryLogs()
{
return Collections.unmodifiableList(queryLogs);
}
}


I'm using a lot of Java 8 Streams, lambdas, and the like because I am trying to familiarize myself with them.

I'm looking for a better and more efficient way to read in the millions of query logs. On my reasonably fast computer, it takes 45-50 seconds to read in all of the logs.

Is there something I can do to speed up this whole process?

Maybe some of my answers and thoughts from this thread (Scanning through logs (tail -f fashion) parsing and sending to a remote server) can help you.

What I want to point out: It's very important, to measure. If you just post a piece of code and tell, it takes that amount of time and ask, how to increase 'performance', it's usually ghost hunting. If you can reduce the time of a routine by 99% but it only takes 1% of the overall time - you just wasted your time.

My main concern about the app is the use case: You're reading millions of lines into memory into a tree structure for further processing/querying - at least that's my interpretation. Have you thought about using an indexing library, such as Apache Lucene? Usually, people invest more time in better response times, instead of "indexing times" - which is, of course, also important, if you have a real-time or near-real-time requirement. But then I would analyze the software architecture first.

Other than that, I can't identify any major problem with the code. One problem could be the garbage collector, since millions of lines are using a lot of memory. And more memory means more memory managament. But that has to measured too and also analyzed in detail. Playing around with collectors, heap sizes, collector threads and so on, can improve throughput significantly.

Hope this helps,

slowy

• I should mention that I am doing this for a school project, so something that does all of the work for me (like Lucene) would kind of defeat the purpose. That being said, I did do some basic profiling of my code. Almost all of the time spent running the program was in reading in the logs. When I used it to suggest queries (based on an input query), it generally returned in real time, so I wasn't really concerned about that in terms of performance. I did as you suggested though, and played with some different structures, and that seemed to help a bit. I'll accept the answer. Jun 7 '17 at 15:07
• With real time I meant, how long does it take for a log entry to be available in a search result. So if you analyze every daily rolled log file, which will be 24 hours later, analyzing the file in a minute isn't big of a deal. But if a requirement would be, that an entry needs to be found after a minute into the logfile - well, that's quite the difference and will need a different approach. Jun 7 '17 at 15:29
• Ah. I understand. Jun 7 '17 at 15:32