# Average row id volume

After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's not possible to further analyse the data. As I'm learning python I decided to write a script to do the job for me, this is my second python script, but I've been a programmer for quite a few year and I used quite a few different languages, so I'd like to understand if I'm writing python or "I'm just translating code from another language to python".

There is no test for the data because it's copied from the LoadRunner Analysis tool. It uses tab as column separator, and it look like this

Vuser ID    Group Name  Transaction End Status  Location Name   Script Name Transaction Hierarchical Path   Host Name   Scenario Elapsed Time   Transaction Response Time   Transaction Name
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   11,484  7,526   AR00_Homepage_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   11,512  7,525   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   48,607  35,334  AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   52,098  39,043  AR01_Login_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   70,698  0,048   AR07_Logout_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   70,768  0,009   AR07_Logout_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   74,466  2,021   AR00_Homepage_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   75,752  2,199   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   78,169  1,825   AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   79,203  1,096   AR01_Login_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   85,963  0,01    AR07_Logout_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   86,571  0,009   AR07_Logout_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   123,846 1,933   AR00_Homepage_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   125,49  1,939   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   125,58  1,25    AR01_Login_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   128,174 1,598   AR01_Login_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   128,715 1,67    AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   132,251 0,325   AR07_Logout_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   134,641 0,016   AR07_Logout_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   135,899 0,011   AR07_Logout_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   136,427 2,002   AR00_Homepage_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   137,611 1,954   AR00_Homepage_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   139,254 1,944   AR00_Homepage_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   139,437 1,239   AR01_Login_AR
Vuser1  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   140,738 1,947   AR01_Login_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   141,829 2,194   AR00_Homepage_AR
Vuser4  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   142,508 1,096   AR01_Login_AR
Vuser2  VC_AR   Pass    N/A VC_AR_Nuova Highest Level   localhost   145,244 0,026   AR07_Logout_AR
Vuser3  VC_AR   Pass    N/A VC_AR_Nuova Global_AREA_RISERVATA   localhost   145,841 1,855   AR01_Login_AR


My code is here

"""
Script to calculate the average time per Vugen volume from the raw data.
LoadRunner Analysis creates the graph but don't shows the base value, that make
it difficoult to do further analisys.

Parameters
----------
-l, --log : str, default 'WARNING'
the log level
-i, --input : str, default 'raw.txt'
the filepath of the input
-o, --output : str, default 'out.txt'
the filepath for the output
"""
import pandas

import argparse
import logging

MAX_TIME_WINDOW = 10
"""
The max time, in second, to count the change of Vugen volume as a single event
"""

"""
Convert the input file to a table

Read the input file and convert it to a DataFrame for future work.
To reduce the memory load the columns that will not be used are deleted.

Parameters
----------
file : str
the filepath of the data

Returns
-------
A DataFrame with the readed data
"""
df_raw = pandas.read_table(file, index_col='Scenario Elapsed Time',
decimal=',', thousands='.')
df_raw.columns = df_raw.columns.str.strip().str.replace(' ', '_')

logging.debug('Imported column list: %s', df_raw.columns)
script_name = df_raw.iat[1, 4]
logging.debug('Script name: %s', script_name)

del df_raw['Group_Name']
del df_raw['Transaction_End_Status']
del df_raw['Location_Name']
del df_raw['Host_Name']
del df_raw['Script_Name']

df_raw.sort_index(inplace=True)

logging.debug('Removed not needed column, current columns: %s',
df_raw.columns)

logging.debug('Imported data sample:\n %s', df_raw)

return df_raw

def generate_empty_row(input_df):
"""
Generate the base row for the work DataFrame

Generate a dictionary with a cell for each distinct value of the
'Transaction_Name' column in the input DataFrame.

Parameters
----------
input_df : DataFrame
The DataFrame with all the data

Returns
-------
dict
A dictionary with a default value for all the columns.
"""
transactions = input_df['Transaction_Name'].unique()
transactions.sort(axis=0)
transactions = transactions.tolist()

logging.debug('Transactions: %s', transactions)

base_row = dict(zip(transactions, [0] * len(transactions)))
return base_row

def __average_and_append(data_totals, data_counter, out_data, new_index):
"""
Average the value and add the values to out_data with index new_index

Divides the totals by the relative counters to get the average for the
transaction, then add the new data to the out_data DataFrame with index
new_index

Parameters
----------
data_totals : dict
A dict with the totals for each transaction, must have the same fields
of data_counter
data_counter : dict
A dict that count the number of items added to get the totals, must
have the same fields of data_totals
out_data : pandas.DataFrame
The DataFrame to append the new row to
new_index : int
The index for the new row in the DataFrame

Returns
-------
pandas.DataFrame
The out_data DataFrame with the new row appended
"""
data_avg = {key: total/max(data_counter[key], 1)
for (key, total)
in data_totals.items()}
row = pandas.DataFrame(data=data_avg, index={new_index})
out_data = out_data.append(row)

return out_data

def calculate_response_time(input_data, base_row):
"""
Calculate the response time for each transaction for each Vugen load

Read each row of the input data, for each transaction add the response time
to an accumulator and the number of added values to a counter.
Every time the load counter increase outside the time window of
MAX_TIME_WINDOW the data will be averaged and wrote to the output DataFrame

Parameters
----------
input_data : DataFrame
The DataFrame with all the data
base_row : dict
A dictionary with a cell for each transaction in the data

Returns
-------
DataFrame
A DataFrame with the calculated average for each transaction, with the
vugen volume as index
"""

data_counter = base_row.copy()
seen_id_set = set()
# Set time_last_added_id base at the time of the first row, i.e. it's index
out_data = pandas.DataFrame()
vugen_volume = 0

for index, row in input_data.iterrows():
is_new_step = row['Vuser_ID'] not in seen_id_set
is_in_window = ((index - time_last_added_id) < MAX_TIME_WINDOW)

if (is_new_step and is_in_window):
# Put the new Vugen ID in the set without changing the last time
vugen_volume = len(seen_id_set)
logging.debug("New ID %s found within %i second" %
(row['Vuser_ID'], MAX_TIME_WINDOW))
elif (is_new_step):
# Calculate the averages and append the new row to output
logging.debug("New ID %s found outside the %i second window, "
"adding a row to the output" %
(row['Vuser_ID'], MAX_TIME_WINDOW))
out_data, vugen_volume)

# Set the last time a Vugen ID was added and put it in the set

# Reset the counter and the adder
data_counter = base_row.copy()

data_counter[row['Transaction_Name']] += 1

# Add the data of the last step
out_data, vugen_volume)

return out_data

"""
Read the input file, calculate the response under load and save it in the
output file

Main procedure of the module, it uses the other function to import the raw
transaction, as the step in the Vugen volume are not perfectly aligned
if the volume is changed multiple times in a window of XX second they will
be counted as a single step. The number of second is defined as a constant

Parameters
----------
input_file : string
The file path of the data file, the data must be response time raw data
file generated by the LoadRunner Analysis tool

output_file : string
The file path where to save the data calculated
"""

base_row = generate_empty_row(input_data)
working_data = calculate_response_time(input_data, base_row)

working_data.to_csv(path_or_buf=args.output_file, sep=';',
index_label='Number of VUser')

if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='Tool that gets raw data from LoadRunner Analysis '
'and calculate the average for transaction and Vugen running.'
'LoadRunner generate the graph but don''t share the data.')
help='Livello di log da usare')
help='Nome del file da caricare')
default='out.txt',
help='Nome del file su cui salvare i dati elaborati')
args = parser.parse_args()

log_level = getattr(logging, args.log_level.upper(), None)
if not isinstance(log_level, int):
raise ValueError('Invalid log level: %s' % args['log_level'])

format='%(asctime)s - %(levelname)s - %(message)s')

logging.debug('Script called with parameters: -l %s, -i %s, -o %s' %
(args.log_level, args.input_file, args.output_file))



I'm not sure if pandas is really needed, in the end I just used it to read and write csv files and remove columns that I don't need, still using a new module is fun :)

• High-level tip: iterrows() has some big disadvantages (see the docs), which similar methods like itertuples() do not suffer from. Of course, the best is to avoid explicit iteration altogether, whenever possible. I will take a complete look at your code tomorrow :) – AMC Nov 15 '19 at 3:54

Unfortunately, I've never used pandas before (I've been meaning to try it for awhile), so I can't comment on its usage. Looking at how it's being used here though, it may help, but parsing a list of lines would still be fairly simple.

Honestly, this is some clean looking code. You actually format things fairly similarly to how I like, so I don't have much to say in that regard.

To nit pick though, here

data_avg = {key: total/max(data_counter[key], 1)
for (key, total)
in data_totals.items()}


I wouldn't break up the for...in. It's not that long of a line.

I also wouldn't use a double underscore prefix for __average_and_append. If your intent was to mark it as "private", just use one leading underscore.

The one suggestion I can make though is to try out type hints. Right now, you're indicating the type in the docstring, and I don't think that it's in a format that IDEs can read readily. Type hints allow some type errors to be caught as you're writing, and show up in a more readable way in docs.

For an example of their use, you could annotate response_time_under_load as:

def response_time_under_load(input_file: str, output_file: str) -> None:


This does a few things:

• The types show up in the docs in the signature instead of buried in the doc string
• If you accidentally pass something of the wrong type, a good IDE will warn you
• From within the function, it knows input_file is a string, so it can give better autocomplete suggestions
• -> None means that the function doesn't return anything (AKA, it implicitly returns None). If you attempt to do

some_var = response_time_under_load(inp, out)


You'll get a warning that response_time_under_load doesn't return anything.

You can also annotate the types that dictionaries and lists hold. This allows it to know the types when you do a lookup. For example, __average_and_append takes two dictionaries, a Dataframe, and an int. You don't say what the dictionary is holding though. The values are numbers, but I can't tell what the keys are. Pretend for the example that the keys and values are both integers.

def _average_and_append(data_totals, data_counter, out_data, new_index):


Could be changed to

from typing import Dict

def _average_and_append(data_totals: Dict[int, int],
data_counter: Dict[int, int],
out_data: Dataframe,  # Assuming Dataframe is imported
new_index: int
) -> Dataframe:


Yes, this is quite verbose, but it conveys a lot of useful information. Dict[int, int] can also be aliased to reduce redundancy and neaten up:

Data = Dict[int, int]  # Type alias

def _average_and_append(data_totals: Data,
data_counter: Data,
out_data: Dataframe,
new_index: int
) -> Dataframe:


Dataframe may be generic like Dict is, so you may be able to specify the types that it holds as well. The docs for the class should mention that.

• Thanks I'll try the type hints. Yes __average_and_append is a private function, it started as part of calculate_response_time and was refactored out to get some more columns to write the code, that's the reason why the for..in is in two lines, initially that's the space I had. The two dict parameters are copies of base_row so the keys are the distinct values of the 'Transaction Name' columns of the input files, in data_totals the values are the sum of the response time so dict[str,float](?), in data_counter the values are the number of rows added so dict[str,int](?) – Serpiton Nov 14 '19 at 23:52
• @Serpiton Yes, those would be correct, except you need Dict instead of dict. Unfortunately, the built-in types aren't yet generic themselves. I think I read that this is going to be fixed in future updates though. Type hints are still fairly new. – Carcigenicate Nov 15 '19 at 0:01