6
\$\begingroup\$

I have been making a script to interact with a library I created before. I have ended up with a working script, but feel that the argparsing could be done a lot cleaner, and better.

First of all, some details about the script. These are the different processes that I try to support, using * to show it's required:

  • Login - target*, username*, password*

  • Query - target*, username*, password*, query*, starttime, endtime, discoverfields, summaryfields, fieldssummary, localsearch, searchtype, timeout, wait

  • Status - target*, username*, password*, status*, searchid*, wait

  • Histogram - target*, username*, password*, histogram*, searchid*

  • Drilldown - target*, username*, password*, drilldown*, searchid*, starttime, endtime

  • Event - target*, username*, password*, event*, searchid*, dir, fields, length, offset

  • Raw event - target*, username*, password*, rawevent*, searchid*, rowid*

  • Chart data - target*, username*, password*, chartdata*, searchid*, fields, length, offset

So my questions are these:

  1. Would using sub parsers be better, if so should I define the same sub parsers for each "parent" parser? So if two or more requires searchid then should I define it on all the parents?
  2. Is there a better way to require to use one, and only one of the parents when running the script? So they cannot be combined, and they can't be missing.
  3. Is there a better way to write help for sub parsers? For example help query starttime.
  4. Is using .format still useful in Python 3?
  5. Currently if a parent argument is called, it will use all optional arguments in the calls, even if empty, is this okay? Or you would end up with unlimited amounts of if statements.
  6. Any coding standards around parsers like this that can be followed?

Here is my code, currently print is used for output, but will support CSV and JSON in the future:

#!/usr/bin/env python3

"""Script to generate searches on the ArcSight Logger"""

import arcsightrest
import argparse

parser = argparse.ArgumentParser(description='Script used to send search '
                                             'queries to ArcSight Logger API')

"""
Login information
"""

parser.add_argument('-t', '--target',
                    help='IP Address of the Loggger', required=True)
parser.add_argument('-u', '--username',
                    help='Username to access the logger', required=True)
parser.add_argument('-p', '--password',
                    help='Password to access the logger', required=True)

"""
SSL Settings
"""

parser.add_argument('-ussl', '--unsecuressl', action='store_true',
                    help='Disable ssl warnings', )

"""
Query settings
"""

parser.add_argument('--query',
                    help='Query to be used in the search')
parser.add_argument('--starttime',
                    help='From which time the query should look')
parser.add_argument('--endtime',
                    help='To which time the query should look')
parser.add_argument('--discoverfields',
                    help='Try to discover fields in the events found')
parser.add_argument('--summaryfields',
                    help='The list of fields')
parser.add_argument('--fieldssummary',
                    help='Use fields summary')
parser.add_argument('--localsearch',
                    help='Indicates the search is local only')
parser.add_argument('--searchtype',
                    help='Interactive search or not')
parser.add_argument('--timeout',
                    help='The number of milliseconds to keep the search '
                         'after it has finished running')

"""
Define search id if it is not a new search
"""

parser.add_argument('--searchid',
                    help='Search id of a currently running search')

"""
Status settings
"""

parser.add_argument('--status',
                    help='Status of running search')
parser.add_argument('--wait', action='store_true',
                    help='Wait for search to finish', )

"""
Histogram Settings
"""

parser.add_argument('--histogram',
                    help='Get histogram overview of specific earch')

"""
Drilldown Settings
"""

parser.add_argument('--drilldown',
                    help='Gets drilldown of specific search')

"""
Event Settings
"""

parser.add_argument('--event',
                    help='Get all information from a finished search')
parser.add_argument('--dir',
                    help='Sort direction based on event time')

"""
Raw Event Settings
"""

parser.add_argument('--rawevent',
                    help='Get the raw event results from a search')
parser.add_argument('--rowid',
                    help='Specific row id for the raw event')

"""
Chart Data Settings
"""

parser.add_argument('--chartdata',
                    help='Returns data in a chart format')

"""
Store all arguments
"""

args = (parser.parse_args())

"""
Sets the target Logger Server
"""

arcsightrest.ArcsightLogger.TARGET = args.target

"""
Gets login token from the Logger API
"""

arc = arcsightrest.ArcsightLogger(args.username, args.password,
                                  args.unsecuressl)

"""
Checks if query is used, and starts a search
"""

if args.query:
    search_id, response = arc.search(args.query, start_time=args.starttime,
                                     end_time=args.endtime,
                                     discover_fields=args.discoverfields,
                                     summary_fields=args.summaryfields,
                                     fields_summary=args.fieldssummary,
                                     local_search=args.localsearch,
                                     search_type=args.searchtype,
                                     timeout=args.timeout,
                                     )
    if args.wait:
        arc.wait(search_id)
        if args.rawevents:
            print(arc.raw_events(search_id, row_ids=args.rowids))
        print(arc.events(search_id, dir=args.dir, fields=args.fields,
                         length=args.length, offset=args.offset))
    print('The search id is {}'.format(search_id))


if args.histogram:
    print(arc.histogram(search_id=args.searchid))

if args.drilldown:
    print(arc.drilldown(search_id=args.searchid, start_time=args.starttime,
                        end_time=args.endtime))

if args.event:
    print(arc.events(search_id=args.searchid, dir=args.dir, fields=args.fields,
                     length=args.length, offset=args.offset))

if args.chartdata:
    print(arc.events(search_id=args.searchid, length=args.length,
                     offset=args.offset))

if args.status:
    print(arc.search_complete(search_id=args.searchid))
\$\endgroup\$
  • 1
    \$\begingroup\$ Your question could be less daunting if you changed the information about your eight parent processors to something simpler, like "Login -- target*, username*, password*", and state that * is required. (I didn't do this is my last edit as I think it should be your choice) \$\endgroup\$ – Peilonrayz Dec 2 '16 at 11:14
  • \$\begingroup\$ @Peilonrayz is this better? :) \$\endgroup\$ – Marius Dec 2 '16 at 11:59
  • \$\begingroup\$ I changed it so that you're not using code blocks, as it shouldn't be code, ;P \$\endgroup\$ – Peilonrayz Dec 2 '16 at 12:16
5
\$\begingroup\$

Yes, subparsers can improve a lot of things. You could also use if __name__ == '__main__': even though it is not necessary for this kind of "executable only" script.

The important thing being that, using subparsers, you can limit the amount of available options to the action you are performing.

Moreover, you should consider using positional arguments rather than optional ones + required=True. It feels cleaner on the help message.


Looking at your commands, you have two possibilities:

  • use only one subparser that will provide a parser for each of your commands (login, query, status, histogram, …);
  • or use two subparsers: one providing the login, query and search commands and a second one under search to provide the various kind of searches.

Usage would be:

$ python script.py <target> <username> <password> [-s]  # login
$ python script.py <target> <username> <password> [-s] query <query> [options]
$ python script.py <target> <username> <password> [-s] status <search_id> [-w]
$ python script.py <target> <username> <password> [-s] histogram <search_id> 
etc…

The disadvantage being that you need to specify the search id for each search parser. Or

$ python script.py <target> <username> <password> [-s]  # login
$ python script.py <target> <username> <password> [-s] query <query> [options]
$ python script.py <target> <username> <password> [-s] search <search_id> status [-w]
$ python script.py <target> <username> <password> [-s] search <search_id> histogram
etc…

where you can factorize the definition of search_id to a common search parser but you will put the burden on the user by having they to provide a longer command line.


I’ll show the second approach, though, as it is easy to get back to the first one from there:

#!/usr/bin/env python3

"""Script to generate searches on the ArcSight Logger"""

import arcsightrest
import argparse

def parse_command_line():
    parser = argparse.ArgumentParser(
            description='Script used to send search queries '
            'to ArcSight Logger API')

    # General informations
    parser.add_argument('target', help='IP Address of the Loggger')
    parser.add_argument('username', help='Username to access the logger')
    parser.add_argument('password', help='Password to access the logger')
    parser.add_argument('-s', '--unsecuressl', action='store_true',
                        help='Disable ssl warnings')


    # Commands
    command_subparser = parser.add_subparsers(dest='action', metavar='action')

    # Query
    query = command_subparser.add_parser('query', help='search informations')
    query.add_argument('query', help='Query to be used in the search')
    query.add_argument(
        '--starttime', help='From which time the query should look')
    query.add_argument('--endtime', help='To which time the query should look')
    query.add_argument(
        '--discoverfields', help='Try to discover fields in the events found')
    query.add_argument('--summaryfields', help='The list of fields')
    query.add_argument('--fieldssummary', help='Use fields summary')
    query.add_argument(
        '--localsearch', help='Indicates the search is local only')
    query.add_argument('--searchtype', help='Interactive search or not')
    query.add_argument(
        '--timeout', help='The number of milliseconds to keep the '
        'search after it has finished running')
    query.add_argument(
        '--wait', action='store_true', help='Wait for search to finish')


    # Past search
    search = command_subparser.add_parser(
        'search', help='actions on previous search')
    search.add_argument(
        'search_id', help='Search id of a currently running search')
    search_subparser = search.add_subparsers(
        dest='search_kind', metavar='kind')

    # Status settings
    status = search_subparser.add_parser(
        'status', help='Status of running search')
    status.add_argument(
        '--wait', action='store_true', help='Wait for search to finish')

    # Histogram Settings
    histogram = search_subparser.add_parser(
        'histogram', help='Get histogram overview of specific earch')

    # Drilldown Settings
    drilldown = search_subparser.add_parser(
        'drilldown', help='Gets drilldown of specific search')
    drilldown.add_argument(
        '--starttime', help='From which time the search should look')
    drilldown.add_argument(
        '--endtime', help='To which time the search should look')

    # Event Settings
    event = search_subparser.add_parser(
        'event', help='Get all information from a finished search')
    event.add_argument(
        '--dir', help='Sort direction based on event time')
    event.add_argument('--field')
    event.add_argument('--length')
    event.add_argument('--offset')

    # Raw Event Settings
    raw_event = search_subparser.add_parser(
        'rawevent', help='Get the raw event results from a search')
    raw_event.add_argument(
        'row_id', help='Specific row id for the raw event')

    # Chart Data Settings
    chart_data = search_subparser.add_parser(
        'chartdata', help='Returns data in a chart format')
    chart_data.add_argument('--field')
    chart_data.add_argument('--length')
    chart_data.add_argument('--offset')

    # Retrieve all arguments
    args = parser.parse_args()

    if args.action is None:
        # Not sure if this one is mandatory
        parser.error('you should provide an action to execute')
    elif args.action == 'search' and args.search_kind is None:
        parser.error(
            "you should provide the kind of search "
            "when performing the 'search' action")
    else:
        return args


def query(arc, args):
    search_id, response = arc.search(
        args.query,
        start_time=args.starttime,
        end_time=args.endtime,
        discover_fields=args.discoverfields,
        summary_fields=args.summaryfields,
        fields_summary=args.fieldssummary,
        local_search=args.localsearch,
        search_type=args.searchtype,
        timeout=args.timeout)
    if args.wait:
        arc.wait(search_id)
    return search_id


def search(arc, search_id, kind, args):
    if kind == 'histogram':
        return arc.histogram(search_id=search_id)

    if kind == 'drilldown':
        return arc.drilldown(
            search_id=search_id,
            start_time=args.starttime,
            end_time=args.endtime)

    if kind == 'event':
        return arc.events(
            search_id=search_id,
            dir=args.dir,
            fields=args.fields,
            length=args.length,
            offset=args.offset)

    if kind == 'rawevent':
        return arc.raw_events(search_id, row_ids=args.rowids)

    if kind == 'chartdata':
        return arc.events(
            search_id=search_id,
            length=args.length,
            offset=args.offset)

    if kind == 'status':
        if args.wait:
            arc.wait(search_id)
        return arc.search_complete(search_id=search_id)


if __name__ == '__main__':
    args = parse_command_line()

    # Sets the target Logger Server
    arcsightrest.ArcsightLogger.TARGET = args.target
    arc = arcsightrest.ArcsightLogger(
        args.username,
        args.password,
        args.unsecuressl)

    if args.action == 'query':
        print(query(arc, args))
    elif args.action == 'search':
        print(search(arc, args.search_id, args.search_kind, args))

I changed a bit how the wait parameter is used, as you described it being an option of status. And the fact that, on a query, it should use information from other actions will make the parser a bit more messy. The drawback being that the user can't perform a wait with one command, they will need 2:

$ python script.py <target> <username> <password> query ... --wait
$ python script.py <target> <username> <password> search 12345 rawevent 12

Lastly, please do not use strings as comments, but real comments instead: i.e. lines starting with a #.

\$\endgroup\$
  • \$\begingroup\$ It tok a while to go through all the code, but this totally changes my view on how to design my argparsers, this is much cleaner, i thank you very much for this! I will use a few days to test out a few things, to see if it all works. \$\endgroup\$ – Marius Dec 3 '16 at 14:44
  • \$\begingroup\$ One error i found in this, is that you wanted to check args.command, while setting a dest= action, it would complain about missing argument in namespace, setting it to args.action instead fixed that. I also moved status to it's own function, in case i want to add more to it later. Changed search to events, as this is gathering events based on past searches. One thing i do notice though, is that the help menu is very "strange" when trying to display help for event, while on status and query it is just fine.. For some reason the word "kind" is in there, and i cant run help on the sub parser. \$\endgroup\$ – Marius Dec 5 '16 at 9:04
  • \$\begingroup\$ I also wanted to add (there was no space on the other comment), is that would it be "okay" to add a def for each option instead? And write my own helper for each? Like histogram, drilldown, raw_event etc? Or would that be a nono? \$\endgroup\$ – Marius Dec 5 '16 at 9:12
  • \$\begingroup\$ @Marius Changed command to action, thanks. kind is there from search_subparser = search.add_subparsers(dest='search_kind', metavar='kind'), you can use type if it makes more sense to you. To get the help, you may want to use python script.py search histogram --help for instance (or python script.py event histogram --help if I understand your changes correctly). \$\endgroup\$ – Mathias Ettinger Dec 5 '16 at 9:13
  • \$\begingroup\$ @Marius using a function for each kind of search would let you build a dictionnary out of them like lookup_table = {'histogram': hist_event, 'drilldown': drill_event, ...} and call it in one step using lookup_table[args.search_kind]() \$\endgroup\$ – Mathias Ettinger Dec 5 '16 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.