6
\$\begingroup\$

I have some text file in following format (network traffic collected by tcpdump):

1505372009.023944 00:1e:4c:72:b8:ae > 00:23:f8:93:c1:af, ethertype IPv4 (0x0800), length 97: (tos 0x0, ttl 64, id 5134, offset 0, flags [DF], proto TCP (6), length 83)
    192.168.1.53.36062 > 74.125.143.139.443: Flags [P.], cksum 0x67fd (correct), seq 1255996541:1255996572, ack 1577943820, win 384, options [nop,nop,TS val 356377 ecr 746170020], length 31
    0x0000:  0023 f893 c1af 001e 4c72 b8ae 0800 4500  .#......Lr....E.
    0x0010:  0053 140e 4000 4006 8ab1 c0a8 0135 4a7d  .S..@.@......5J}
    0x0020:  8f8b 8cde 01bb 4adc fc7d 5e0d 830c 8018  ......J..}^.....
    0x0030:  0180 67fd 0000 0101 080a 0005 7019 2c79  ..g.........p.,y
    0x0040:  a6a4 1503 0300 1a00 0000 0000 0000 04d1  ................
    0x0050:  c300 9119 6946 698c 67ac 47a9 368a 1748  ....iFi.g.G.6..H
    0x0060:  1c                                       .

and want to change it to:

1505372009.023944 
    000000:  00 23 f8 93 c1 af 00 1e 4c 72 b8 ae 08 00 45 00  .#......Lr....E.
    000010:  00 53 14 0e 40 00 40 06 8a b1 c0 a8 01 35 4a 7d  .S..@.@......5J}
    000020:  8f 8b 8c de 01 bb 4a dc fc 7d 5e 0d 83 0c 80 18  ......J..}^.....
    000030:  01 80 67 fd 00 00 01 01 08 0a 00 05 70 19 2c 79  ..g.........p.,y
    000040:  a6 a4 15 03 03 00 1a 00 00 00 00 00 00 00 04 d1  ................
    000050:  c3 00 91 19 69 46 69 8c 67 ac 47 a9 36 8a 17 48  ....iFi.g.G.6..H
    000060:  1c                                               .

Here is what I have done:

import re
regexp_time =re.compile("\d\d\d\d\d\d\d\d\d\d.\d\d\d\d\d\d+")
regexp_hex = re.compile("(\t0x\d+:\s+)([0-9a-f ]+)+  ")

with open ('../Traffic/traffic1.txt') as input,open ('../Traffic/txt2.txt','w') as output:
    for line in input:
        if regexp_time.match(line):
            output.write ("%s\n" % (line.split()[0]))
        elif regexp_hex.match(line):
            words = re.split(r'\s{2,}', line)
            bytes=""
            for byte in words[1].split():
                if len(byte) == 4:
                    bytes += "%s%s %s%s "%(byte[0],byte[1],byte[2],byte[3])
                elif len(byte) == 2:
                    bytes += "%s%s "%(byte[0],byte[1])
            output.write ("%s  %s %s \n" % (words[0].replace("0x","00"),"{:<47}".format (bytes),words[2].replace("\n","")))

input.close()
output.close()

Could some one help me in speed up?

\$\endgroup\$
  • \$\begingroup\$ Two things: 1. Are you always going to have 8 groups of 4 characters in the hex lines? 2. Will that last line always be just 1 group of 2 characters? \$\endgroup\$ – pstatix Sep 14 '17 at 11:20
  • \$\begingroup\$ @pstatix, All except last line is 8 group of 4 characters, but the last line differ from 1 group to 8 and also just 2 characters. \$\endgroup\$ – user3933607 Sep 14 '17 at 11:50
6
\$\begingroup\$

Regex

The time regular expression can be simplified to

regexp_time = re.compile(r'\d{10}\.\d{6}')

which is a bit more readable. The second one does not acount for letters in hexadecimal numbers so a line starting with 0x00a0: won't be parsed… Since you are using a file that is extremely regularly formatted, I suggest you drop this regular expression and rely on the fact that, if regexp_time does not match, then you have a line of data. (At least on my tcpdump output, I only get a single line.) But if you want to use a second regular expression to filter out lines that may be between the timestamp and the first line of data, I'd suggest simply using

regexp_hex = re.compile(r'\t0[xX][\da-fA-F]+:')

and not worry about the rest of the line.

Matches

When using the match method on a compiled regular expression, you receive a match object or None. This match object let you easily extract out the matched bits using its group method. So there is little use of splitting a line to parse stuff:

timestamp = regexp_time.match(line):
if timestamp:
    output.write("%s\n" % timestamp.group())

In fact, you can so easily extract out the groups that I would use a second regex for the hexadecimal data anyway:

regexp_hex = re.compile(r'\t0x([\da-fA-F]+):  ([\da-fA-F ]+?)\s{2,}(.*)')

So that, the first group is the hexadecimal address, the second one is the hexadecimal payload and the third one is the ASCII payload. You then just need to format the output:

payload = regexp_hex.match(line)
if payload:
    address = payload.group(1)
    hex_data = payload.group(2).replace(' ', '')
    ascii_data = payload.group(3)
    output.write('\t{0>6}:  {:<47}  {}\n'.format(address, hex_data, ascii_data))

The only thing left is to split the hex_data in groups of 2 digits. This can be done using the itertools recipe grouper.

Proposed improvements

import re
import itertools


TIMESTAMP = re.compile(r'\d{10}\.\d{6}')
PAYLOAD = re.compile(r'\t0x([\da-fA-F]+):  ([\da-fA-F ]+?)\s{2,}(.*)')


def grouper(iterable, count, fillvalue=None):
    args = [iter(iterable)] * count
    return itertools.zip_longest(*args, fillvalue=fillvalue)


def convert_tcpdump_to_text2pcap(in_filename, out_filename):
    with open(in_filename) as input, open(out_filename, 'w') as output:
        for line in input:
            time = TIMESTAMP.match(line)
            if time:
                output.write('{}\n'.format(time.group()))
                continue
            payload = PAYLOAD.match(line)
            if payload:
                address = payload.group(1)
                hex_data = payload.group(2).replace(' ', '')
                ascii_data = payload.group(3)

                hex_data = ' '.join(''.join(part) for part in grouper(hex_data, 2, ' '))
                output.write('\t{:0>6}:  {:<47}  {}\n'.format(address, hex_data, ascii_data))


if __name__ == '__main__':
    convert_tcpdump_to_text2pcap('../Traffic/traffic1.txt', '../Traffic/txt2.txt')
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Thanks, It fixed but still need more speed up. \$\endgroup\$ – user3933607 Sep 14 '17 at 18:18
1
\$\begingroup\$

For best performance, you would be much better off loading tcpdump's binary libpcap output format rather than parsing a human-friendly textual representation of it. That's what computers are good at — handling binary data. In fact, that page suggests using the python-libpcap for this purpose.

We can tweak your implementation, even though it will likely not make any substantial difference to the performance.

My main criticism is that it's hard to figure out what your code does just by inspection. There is a lot of .split()ting, indexing, and filling in of %s placeholders.

I suggest taking better advantage of regexes to parse the input:

  • Use named capture groups for readability. That's smarter and more readable than using the regexes just as a test, then extracting the same information again using .split().
  • Combine the two regexes into one.
  • Avoid being too picky about the timestamp format. Any decimal number at the beginning of the line should be acceptable.
  • It would be better to write regexes using r'raw strings', especially if they contain backslashes.

For better flexibility, I suggest putting most of the code into a generator function. That way, it would be relatively easy to accept input from a different source (piped from sys.stdin, for example), or to redirect the output (to sys.stdout, maybe) — without touching the format translation logic.

Suggested solution

As I said, you should probably rewrite the code using python-libpcap. But if I had to tweak your implementation, I'd write it like this:

import re

TCPDUMP_RE = re.compile(
    r'^(?P<timestamp>\d+\.\d*)|'
    r'^\t0x(?P<offset>[0-9a-f]+):  (?P<words>[0-9a-f ]{39})  (?P<text>.*)'
)

def tcpdump_to_text2pcap(lines):
    for line in lines:
        match = TCPDUMP_RE.match(line)
        if not match:
            continue
        elif match['timestamp']:
            yield match['timestamp']
        elif match['offset']:
            # Reformat words into bytes
            yield '    {offset:06x}:  {bytes}  {text}'.format(
                offset=int(match['offset'], 16),
                bytes=re.sub('(..)(..)( ?)', r'\1 \2\3', match['words']),
                text=match['text']
            )

with open('traffic1.txt') as tcpdump_in, \
     open('traffic2.txt', 'w') as text2pcap_out:
    for line in tcpdump_to_text2pcap(tcpdump_in):
        print(line, file=text2pcap_out)

There is no need to call input.close() and output.close() explicitly, since that will be done for you already when exiting the with block.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Thanks for your explanation, I use this for capturing traffic 'tcpdump -nnvvvSettXXU -s 0 -i eth1 > traffic.txt', because of some issue I couldn't use -w traffic.pcap so I came up with using > instead. Also time-stamp of each packet is important. I get following error: elif match['timestamp']: TypeError: '_sre.SRE_Match' object is not subscriptable \$\endgroup\$ – user3933607 Sep 15 '17 at 5:46
  • \$\begingroup\$ The match['timestamp'] syntax was introduced in Python 3.6. For earlier versions, use match.group('timestamp'). \$\endgroup\$ – 200_success Sep 15 '17 at 5:55
  • \$\begingroup\$ "Because of some issue"? I am intrigued. \$\endgroup\$ – 200_success Sep 15 '17 at 5:58
  • \$\begingroup\$ Here is the story behind it: I have multiple Genymotion devices on host, which is assigned static IP, So I couldn't set network mode of the devices to bridge. Therefor all of them set to NAT mode and then get same IP but different MAC addresses. So to capturing traffic I came up with capturing each devices traffic by running tcpdump on each devices. But As I have other running processes in each device and also limited space on each of them and also no write permission. Continue in next comment .... \$\endgroup\$ – user3933607 Sep 15 '17 at 6:01
  • \$\begingroup\$ I couldn't use -w which lead to save file on emulators. But when I use > it will write on host machine which I don't have any limitation. \$\endgroup\$ – user3933607 Sep 15 '17 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.