After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's not possible to further analyse the data. As I'm learning python I decided to write a script to do the job for me, this is my second python script, but I've been a programmer for quite a few year and I used quite a few different languages, so I'd like to understand if I'm writing python or "I'm just translating code from another language to python".
There is no test for the data because it's copied from the LoadRunner Analysis tool. It uses tab as column separator, and it look like this
Vuser ID Group Name Transaction End Status Location Name Script Name Transaction Hierarchical Path Host Name Scenario Elapsed Time Transaction Response Time Transaction Name
Vuser1 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 11,484 7,526 AR00_Homepage_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 11,512 7,525 AR00_Homepage_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 48,607 35,334 AR01_Login_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 52,098 39,043 AR01_Login_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 70,698 0,048 AR07_Logout_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 70,768 0,009 AR07_Logout_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 74,466 2,021 AR00_Homepage_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 75,752 2,199 AR00_Homepage_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 78,169 1,825 AR01_Login_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 79,203 1,096 AR01_Login_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 85,963 0,01 AR07_Logout_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 86,571 0,009 AR07_Logout_AR
Vuser4 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 123,846 1,933 AR00_Homepage_AR
Vuser3 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 125,49 1,939 AR00_Homepage_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 125,58 1,25 AR01_Login_AR
Vuser4 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 128,174 1,598 AR01_Login_AR
Vuser3 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 128,715 1,67 AR01_Login_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 132,251 0,325 AR07_Logout_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 134,641 0,016 AR07_Logout_AR
Vuser4 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 135,899 0,011 AR07_Logout_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 136,427 2,002 AR00_Homepage_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 137,611 1,954 AR00_Homepage_AR
Vuser4 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 139,254 1,944 AR00_Homepage_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 139,437 1,239 AR01_Login_AR
Vuser1 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 140,738 1,947 AR01_Login_AR
Vuser3 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 141,829 2,194 AR00_Homepage_AR
Vuser4 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 142,508 1,096 AR01_Login_AR
Vuser2 VC_AR Pass N/A VC_AR_Nuova Highest Level localhost 145,244 0,026 AR07_Logout_AR
Vuser3 VC_AR Pass N/A VC_AR_Nuova Global_AREA_RISERVATA localhost 145,841 1,855 AR01_Login_AR
My code is here
"""
Script to calculate the average time per Vugen volume from the raw data.
LoadRunner Analysis creates the graph but don't shows the base value, that make
it difficoult to do further analisys.
Parameters
----------
-l, --log : str, default 'WARNING'
the log level
-i, --input : str, default 'raw.txt'
the filepath of the input
-o, --output : str, default 'out.txt'
the filepath for the output
"""
import pandas
import argparse
import logging
MAX_TIME_WINDOW = 10
"""
The max time, in second, to count the change of Vugen volume as a single event
"""
def read_and_prepare(file):
"""
Convert the input file to a table
Read the input file and convert it to a DataFrame for future work.
To reduce the memory load the columns that will not be used are deleted.
Parameters
----------
file : str
the filepath of the data
Returns
-------
A DataFrame with the readed data
"""
df_raw = pandas.read_table(file, index_col='Scenario Elapsed Time',
decimal=',', thousands='.')
df_raw.columns = df_raw.columns.str.strip().str.replace(' ', '_')
logging.debug('Imported column list: %s', df_raw.columns)
script_name = df_raw.iat[1, 4]
logging.debug('Script name: %s', script_name)
del df_raw['Group_Name']
del df_raw['Transaction_End_Status']
del df_raw['Location_Name']
del df_raw['Host_Name']
del df_raw['Script_Name']
df_raw.sort_index(inplace=True)
logging.debug('Removed not needed column, current columns: %s',
df_raw.columns)
logging.debug('Imported data sample:\n %s', df_raw)
return df_raw
def generate_empty_row(input_df):
"""
Generate the base row for the work DataFrame
Generate a dictionary with a cell for each distinct value of the
'Transaction_Name' column in the input DataFrame.
Parameters
----------
input_df : DataFrame
The DataFrame with all the data
Returns
-------
dict
A dictionary with a default value for all the columns.
"""
transactions = input_df['Transaction_Name'].unique()
transactions.sort(axis=0)
transactions = transactions.tolist()
logging.debug('Transactions: %s', transactions)
base_row = dict(zip(transactions, [0] * len(transactions)))
return base_row
def __average_and_append(data_totals, data_counter, out_data, new_index):
"""
Average the value and add the values to out_data with index new_index
Divides the totals by the relative counters to get the average for the
transaction, then add the new data to the out_data DataFrame with index
new_index
Parameters
----------
data_totals : dict
A dict with the totals for each transaction, must have the same fields
of data_counter
data_counter : dict
A dict that count the number of items added to get the totals, must
have the same fields of data_totals
out_data : pandas.DataFrame
The DataFrame to append the new row to
new_index : int
The index for the new row in the DataFrame
Returns
-------
pandas.DataFrame
The out_data DataFrame with the new row appended
"""
data_avg = {key: total/max(data_counter[key], 1)
for (key, total)
in data_totals.items()}
row = pandas.DataFrame(data=data_avg, index={new_index})
out_data = out_data.append(row)
return out_data
def calculate_response_time(input_data, base_row):
"""
Calculate the response time for each transaction for each Vugen load
Read each row of the input data, for each transaction add the response time
to an accumulator and the number of added values to a counter.
Every time the load counter increase outside the time window of
MAX_TIME_WINDOW the data will be averaged and wrote to the output DataFrame
Parameters
----------
input_data : DataFrame
The DataFrame with all the data
base_row : dict
A dictionary with a cell for each transaction in the data
Returns
-------
DataFrame
A DataFrame with the calculated average for each transaction, with the
vugen volume as index
"""
data_counter = base_row.copy()
data_adder = base_row.copy()
seen_id_set = set()
# Set time_last_added_id base at the time of the first row, i.e. it's index
time_last_added_id = input_data.index[0]
out_data = pandas.DataFrame()
vugen_volume = 0
for index, row in input_data.iterrows():
is_new_step = row['Vuser_ID'] not in seen_id_set
is_in_window = ((index - time_last_added_id) < MAX_TIME_WINDOW)
if (is_new_step and is_in_window):
# Put the new Vugen ID in the set without changing the last time
seen_id_set.add(row['Vuser_ID'])
vugen_volume = len(seen_id_set)
logging.debug("New ID %s found within %i second" %
(row['Vuser_ID'], MAX_TIME_WINDOW))
elif (is_new_step):
# Calculate the averages and append the new row to output
logging.debug("New ID %s found outside the %i second window, "
"adding a row to the output" %
(row['Vuser_ID'], MAX_TIME_WINDOW))
out_data = __average_and_append(data_adder, data_counter,
out_data, vugen_volume)
# Set the last time a Vugen ID was added and put it in the set
time_last_added_id = index
seen_id_set.add(row['Vuser_ID'])
# Reset the counter and the adder
data_counter = base_row.copy()
data_adder = base_row.copy()
data_adder[row['Transaction_Name']] += row['Transaction_Response_Time']
data_counter[row['Transaction_Name']] += 1
# Add the data of the last step
out_data = __average_and_append(data_adder, data_counter,
out_data, vugen_volume)
return out_data
def response_time_under_load(input_file, output_file):
"""
Read the input file, calculate the response under load and save it in the
output file
Main procedure of the module, it uses the other function to import the raw
data from LoadRunner, calculate the average response under load for each
transaction, as the step in the Vugen volume are not perfectly aligned
if the volume is changed multiple times in a window of XX second they will
be counted as a single step. The number of second is defined as a constant
Parameters
----------
input_file : string
The file path of the data file, the data must be response time raw data
file generated by the LoadRunner Analysis tool
output_file : string
The file path where to save the data calculated
"""
input_data = read_and_prepare(args.input_file)
base_row = generate_empty_row(input_data)
working_data = calculate_response_time(input_data, base_row)
working_data.to_csv(path_or_buf=args.output_file, sep=';',
index_label='Number of VUser')
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='Tool that gets raw data from LoadRunner Analysis '
'and calculate the average for transaction and Vugen running.'
'LoadRunner generate the graph but don''t share the data.')
parser.add_argument('-l', '--log', dest='log_level', default='WARNING',
help='Livello di log da usare')
parser.add_argument('-i', '--input', dest='input_file', default='raw.txt',
help='Nome del file da caricare')
parser.add_argument('-o', '--output', dest='output_file',
default='out.txt',
help='Nome del file su cui salvare i dati elaborati')
args = parser.parse_args()
log_level = getattr(logging, args.log_level.upper(), None)
if not isinstance(log_level, int):
raise ValueError('Invalid log level: %s' % args['log_level'])
logging.basicConfig(filename='reader.log', filemode='w', level=log_level,
format='%(asctime)s - %(levelname)s - %(message)s')
logging.debug('Script called with parameters: -l %s, -i %s, -o %s' %
(args.log_level, args.input_file, args.output_file))
response_time_under_load(args.input_file, args.output_file)
I'm not sure if pandas
is really needed, in the end I just used it to read and write csv files and remove columns that I don't need, still using a new module is fun :)
iterrows()
has some big disadvantages (see the docs), which similar methods likeitertuples()
do not suffer from. Of course, the best is to avoid explicit iteration altogether, whenever possible. I will take a complete look at your code tomorrow :) \$\endgroup\$ – Alexander Cécile Nov 15 at 3:54