Average row id volume

Question

After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's not possible to further analyse the data. As I'm learning python I decided to write a script to do the job for me, this is my second python script, but I've been a programmer for quite a few year and I used quite a few different languages, so I'd like to understand if I'm writing python or "I'm just translating code from another language to python".

There is no test for the data because it's copied from the LoadRunner Analysis tool. It uses tab as column separator, and it look like this

Vuser ID    Group Name  Transaction End Status  Location Name   Script Name Transaction Hierarchical Path   Host Name   Scenario Elapsed Time   Transaction Response Time   Transaction Name
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   11,484  7,526   AR00_Homepage_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   11,512  7,525   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   48,607  35,334  AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   52,098  39,043  AR01_Login_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   70,698  0,048   AR07_Logout_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   70,768  0,009   AR07_Logout_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   74,466  2,021   AR00_Homepage_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   75,752  2,199   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   78,169  1,825   AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   79,203  1,096   AR01_Login_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   85,963  0,01    AR07_Logout_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   86,571  0,009   AR07_Logout_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   123,846 1,933   AR00_Homepage_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   125,49  1,939   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   125,58  1,25    AR01_Login_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   128,174 1,598   AR01_Login_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   128,715 1,67    AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   132,251 0,325   AR07_Logout_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   134,641 0,016   AR07_Logout_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   135,899 0,011   AR07_Logout_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   136,427 2,002   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   137,611 1,954   AR00_Homepage_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   139,254 1,944   AR00_Homepage_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   139,437 1,239   AR01_Login_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   140,738 1,947   AR01_Login_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   141,829 2,194   AR00_Homepage_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   142,508 1,096   AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   145,244 0,026   AR07_Logout_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   145,841 1,855   AR01_Login_AR

My code is here

"""
Script to calculate the average time per Vugen volume from the raw data.
LoadRunner Analysis creates the graph but don't shows the base value, that make
it difficoult to do further analisys.

Parameters
----------
-l, --log : str, default 'WARNING'
    the log level
-i, --input : str, default 'raw.txt'
    the filepath of the input
-o, --output : str, default 'out.txt'
    the filepath for the output
"""
import pandas

import argparse
import logging

MAX_TIME_WINDOW = 10
"""
The max time, in second, to count the change of Vugen volume as a single event
"""


def read_and_prepare(file):
    """
    Convert the input file to a table

    Read the input file and convert it to a DataFrame for future work.
    To reduce the memory load the columns that will not be used are deleted.

    Parameters
    ----------
    file : str
        the filepath of the data

    Returns
    -------
    A DataFrame with the readed data
    """
    df_raw = pandas.read_table(file, index_col='Scenario Elapsed Time',
                               decimal=',', thousands='.')
    df_raw.columns = df_raw.columns.str.strip().str.replace(' ', '_')

    logging.debug('Imported column list: %s', df_raw.columns)
    script_name = df_raw.iat[1, 4]
    logging.debug('Script name: %s', script_name)

    del df_raw['Group_Name']
    del df_raw['Transaction_End_Status']
    del df_raw['Location_Name']
    del df_raw['Host_Name']
    del df_raw['Script_Name']

    df_raw.sort_index(inplace=True)

    logging.debug('Removed not needed column, current columns: %s',
                  df_raw.columns)

    logging.debug('Imported data sample:\n %s', df_raw)

    return df_raw


def generate_empty_row(input_df):
    """
    Generate the base row for the work DataFrame

    Generate a dictionary with a cell for each distinct value of the
    'Transaction_Name' column in the input DataFrame.

    Parameters
    ----------
    input_df : DataFrame
        The DataFrame with all the data

    Returns
    -------
    dict
        A dictionary with a default value for all the columns.
    """
    transactions = input_df['Transaction_Name'].unique()
    transactions.sort(axis=0)
    transactions = transactions.tolist()

    logging.debug('Transactions: %s', transactions)

    base_row = dict(zip(transactions, [0] * len(transactions)))
    return base_row


def __average_and_append(data_totals, data_counter, out_data, new_index):
    """
    Average the value and add the values to out_data with index new_index

    Divides the totals by the relative counters to get the average for the
    transaction, then add the new data to the out_data DataFrame with index
    new_index

    Parameters
    ----------
    data_totals : dict
        A dict with the totals for each transaction, must have the same fields
        of data_counter
    data_counter : dict
        A dict that count the number of items added to get the totals, must
        have the same fields of data_totals
    out_data : pandas.DataFrame
        The DataFrame to append the new row to
    new_index : int
        The index for the new row in the DataFrame

    Returns
    -------
    pandas.DataFrame
        The out_data DataFrame with the new row appended
    """
    data_avg = {key: total/max(data_counter[key], 1)
                for (key, total)
                in data_totals.items()}
    row = pandas.DataFrame(data=data_avg, index={new_index})
    out_data = out_data.append(row)

    return out_data


def calculate_response_time(input_data, base_row):
    """
    Calculate the response time for each transaction for each Vugen load

    Read each row of the input data, for each transaction add the response time
    to an accumulator and the number of added values to a counter.
    Every time the load counter increase outside the time window of
    MAX_TIME_WINDOW the data will be averaged and wrote to the output DataFrame

    Parameters
    ----------
    input_data : DataFrame
        The DataFrame with all the data
    base_row : dict
        A dictionary with a cell for each transaction in the data

    Returns
    -------
    DataFrame
        A DataFrame with the calculated average for each transaction, with the
        vugen volume as index
    """

    data_counter = base_row.copy()
    data_adder = base_row.copy()
    seen_id_set = set()
    # Set time_last_added_id base at the time of the first row, i.e. it's index
    time_last_added_id = input_data.index[0]
    out_data = pandas.DataFrame()
    vugen_volume = 0

    for index, row in input_data.iterrows():
        is_new_step = row['Vuser_ID'] not in seen_id_set
        is_in_window = ((index - time_last_added_id) < MAX_TIME_WINDOW)

        if (is_new_step and is_in_window):
            # Put the new Vugen ID in the set without changing the last time
            seen_id_set.add(row['Vuser_ID'])
            vugen_volume = len(seen_id_set)
            logging.debug("New ID %s found within %i second" %
                          (row['Vuser_ID'], MAX_TIME_WINDOW))
        elif (is_new_step):
            # Calculate the averages and append the new row to output
            logging.debug("New ID %s found outside the %i second window, "
                          "adding a row to the output" %
                          (row['Vuser_ID'], MAX_TIME_WINDOW))
            out_data = __average_and_append(data_adder, data_counter,
                                            out_data, vugen_volume)

            # Set the last time a Vugen ID was added and put it in the set
            time_last_added_id = index
            seen_id_set.add(row['Vuser_ID'])

            # Reset the counter and the adder
            data_counter = base_row.copy()
            data_adder = base_row.copy()

        data_adder[row['Transaction_Name']] += row['Transaction_Response_Time']
        data_counter[row['Transaction_Name']] += 1

    # Add the data of the last step
    out_data = __average_and_append(data_adder, data_counter,
                                    out_data, vugen_volume)

    return out_data


def response_time_under_load(input_file, output_file):
    """
    Read the input file, calculate the response under load and save it in the
    output file

    Main procedure of the module, it uses the other function to import the raw
    data from LoadRunner, calculate the average response under load for each
    transaction, as the step in the Vugen volume are not perfectly aligned
    if the volume is changed multiple times in a window of XX second they will
    be counted as a single step. The number of second is defined as a constant

    Parameters
    ----------
    input_file : string
        The file path of the data file, the data must be response time raw data
        file generated by the LoadRunner Analysis tool

    output_file : string
        The file path where to save the data calculated
    """

    input_data = read_and_prepare(args.input_file)
    base_row = generate_empty_row(input_data)
    working_data = calculate_response_time(input_data, base_row)

    working_data.to_csv(path_or_buf=args.output_file, sep=';',
                        index_label='Number of VUser')


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
              description='Tool that gets raw data from LoadRunner Analysis '
              'and calculate the average for transaction and Vugen running.'
              'LoadRunner generate the graph but don''t share the data.')
    parser.add_argument('-l', '--log', dest='log_level', default='WARNING',
                        help='Livello di log da usare')
    parser.add_argument('-i', '--input', dest='input_file', default='raw.txt',
                        help='Nome del file da caricare')
    parser.add_argument('-o', '--output', dest='output_file',
                        default='out.txt',
                        help='Nome del file su cui salvare i dati elaborati')
    args = parser.parse_args()

    log_level = getattr(logging, args.log_level.upper(), None)
    if not isinstance(log_level, int):
        raise ValueError('Invalid log level: %s' % args['log_level'])

    logging.basicConfig(filename='reader.log', filemode='w', level=log_level,
                        format='%(asctime)s - %(levelname)s - %(message)s')

    logging.debug('Script called with parameters: -l %s, -i %s, -o %s' %
                  (args.log_level, args.input_file, args.output_file))

    response_time_under_load(args.input_file, args.output_file)

I'm not sure if pandas is really needed, in the end I just used it to read and write csv files and remove columns that I don't need, still using a new module is fun :)

Answer 1

Unfortunately, I've never used pandas before (I've been meaning to try it for awhile), so I can't comment on its usage. Looking at how it's being used here though, it may help, but parsing a list of lines would still be fairly simple.

---

Honestly, this is some clean looking code. You actually format things fairly similarly to how I like, so I don't have much to say in that regard.

To nit pick though, here

data_avg = {key: total/max(data_counter[key], 1)
            for (key, total)
            in data_totals.items()}

I wouldn't break up the for...in. It's not that long of a line.

I also wouldn't use a double underscore prefix for __average_and_append. If your intent was to mark it as "private", just use one leading underscore.

---

The one suggestion I can make though is to try out type hints. Right now, you're indicating the type in the docstring, and I don't think that it's in a format that IDEs can read readily. Type hints allow some type errors to be caught as you're writing, and show up in a more readable way in docs.

For an example of their use, you could annotate response_time_under_load as:

def response_time_under_load(input_file: str, output_file: str) -> None:

This does a few things:

• The types show up in the docs in the signature instead of buried in the doc string
• If you accidentally pass something of the wrong type, a good IDE will warn you
• From within the function, it knows input_file is a string, so it can give better autocomplete suggestions

• -> None means that the function doesn't return anything (AKA, it implicitly returns None). If you attempt to do

  some_var = response_time_under_load(inp, out)
  

  You'll get a warning that response_time_under_load doesn't return anything.

You can also annotate the types that dictionaries and lists hold. This allows it to know the types when you do a lookup. For example, __average_and_append takes two dictionaries, a Dataframe, and an int. You don't say what the dictionary is holding though. The values are numbers, but I can't tell what the keys are. Pretend for the example that the keys and values are both integers.

def _average_and_append(data_totals, data_counter, out_data, new_index):

Could be changed to

from typing import Dict

def _average_and_append(data_totals: Dict[int, int],
                        data_counter: Dict[int, int],
                        out_data: Dataframe,  # Assuming Dataframe is imported
                        new_index: int
                        ) -> Dataframe:

Yes, this is quite verbose, but it conveys a lot of useful information. Dict[int, int] can also be aliased to reduce redundancy and neaten up:

Data = Dict[int, int]  # Type alias

def _average_and_append(data_totals: Data,
                        data_counter: Data,
                        out_data: Dataframe,
                        new_index: int
                        ) -> Dataframe:

Dataframe may be generic like Dict is, so you may be able to specify the types that it holds as well. The docs for the class should mention that.