2
\$\begingroup\$

I want to write a python script to sort a huge file, say 2 GB in size, which contains logs in the following format -

Jan 1 02:32:40 other strings but may or may not unique in all those lines
Jan 1 02:32:40 other strings but may or may not unique in all those lines
Mar 31 23:31:55 other strings but may or may not unique in all those lines
Mar 31 23:31:55 other strings but may or may not unique in all those lines
Mar 31 23:31:55 other strings but may or may not unique in all those lines
Mar 31 23:31:56 other strings but may or may not unique in all those lines
Mar 31 23:31:56 other strings but may or may not unique in all those lines
Mar 31 23:31:56 other strings but may or may not unique in all those lines
Mar 31 23:31:57 other strings but may or may not unique in all those lines
Mar 31 23:31:57 other strings but may or may not unique in all those lines
Mar 31 23:31:57 other strings but may or may not unique in all those lines
Mar 31 23:31:57 other strings but may or may not unique in all those lines
Feb 1 03:52:26 other strings but may or may not unique in all those lines
Feb 1 03:52:26 other strings but may or may not unique in all those lines
Jan 1 02:46:40 other strings but may or may not unique in all those lines
Jan 1 02:44:40 other strings but may or may not unique in all those lines
Jan 1 02:40:40 other strings but may or may not unique in all those lines
Feb 10 03:52:26 other strings but may or may not unique in all those lines

I want to sort them basted on the timestamp.

I was able to get this working but for my code to succeed, I need to load the WHOLE file into a list.. this means it will be extremely inefficient from a memory utilization point of view.

Can you please suggest if there is a more efficient way where I can sort this by reading the file line by line or perhaps some other approach that I am not aware of ?

Here is my code -

# convert the log into a list of strings
with open("log.txt", 'r') as f:
    lines = f.read().splitlines()

# writing the method which will be fed as a key for sorting
def convert_time(logline):
    # extracting hour, minute and second from each log entry
    h, m, s = map(int, logline.split()[2].split(':'))
    time_in_seconds = h * 3600 + m * 60 + s
    return time_in_seconds


sorted_log_list = sorted(lines, key=convert_time)

''' sorted_log_list is a "list of lists". Each list within it is a representation of one log entry. We will use print and join to print it out as a readable log entry'''
for lines in sorted_log_list:
    print lines
\$\endgroup\$
  • \$\begingroup\$ Is sort -M <logfile> not working in this case? \$\endgroup\$ – yuri May 11 at 11:00
  • \$\begingroup\$ Hi @yuri ! Sorry forgot to mention that I am trying to write a python script for this. Updated the title and description. \$\endgroup\$ – user168115 May 11 at 11:07
  • 1
    \$\begingroup\$ @BadAtGeometry - Won't the list take the same amount of memory as the file data? My code at present is doing what you are suggesting already :/ \$\endgroup\$ – user168115 May 11 at 19:09
  • 2
    \$\begingroup\$ @OhMyGoodness - 2 GB is a random size picked up. The problem statement is "how to sort a HUGE file in python in the most efficient way". It could be something with millions of records or more. I found this article that so far seems to be the closest match for what I am looking to do . It is written in python 3 - neopythonic.blogspot.com/2008/10/… \$\endgroup\$ – user168115 May 12 at 1:31
  • 2
    \$\begingroup\$ the most efficient way is to load the file into memory and sort it there. If that's infeasible, you'll get better answers if you provide concrete details, like how big the log files are, how long the records are, and how much memory is available. \$\endgroup\$ – Oh My Goodness May 12 at 2:03
1
\$\begingroup\$

You're ignoring the Date-part of the timestamps; it doesn't sound like that's on purpose. (Also, the year is missing altogether, which should make us quite nervous.) Also, let's use explicit datetime utilities and regexes.

import datetime
import re

timestamp_regex = re.compile("[^:]+:\d\d:\d\d")

def convert_time(logline):
    stamp = timestamp_regex.match(logline).group() #this will error if there's no match.
    d = datetime.strptime(stamp, "%b %e %H:%M:%S")
    return int(d.timestamp())

As for the rest, the comments are right that we can't do much unless we know exactly what it would mean for the solution to be improved.

If the concern is just to handle the biggest file with the least ram, something like this might work:

def save_where_we_can_find_it(line, temp_file):
    retval = temp_file.tell()
    temp_file.write(line)
    return retval

def fetch_line(location, temp_file):
    temp_file.seek(location)
    return temp_file.readline()

items = []

with open("log.txt", 'r') as original, open(".temp.log.txt", 'w') as temp:
    for line in original:
        items.append((convert_time(line), save_where_we_can_find_it(line, temp)))

items.sort(key = lambda pair: pair[0]) #sort-in-place isn't necessarily a good idea; whatever.

with open(".temp.log.txt", 'r') as temp:
    for (stamp, location) in items:
        print(fetch_line(location, temp))

import os
os.remove(".temp.log.txt")

But this is just a really inefficient way using a scratch-file. Better to register scratch-space in the OS, and then do your file manipulation "in memory".

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.