# Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use.

# IMPORTS
import numpy as np
import pylab as py
import matplotlib.pyplot as plt
from scipy import optimize
from scipy.optimize import curve_fit

# THE INEFFICIENT PARSING: SELECTING VALUES FROM SEVERAL TABLES ONE BY ONE

skip_bottom_lines = 5 # currently i skip all lines but one at the time, cumbersome

skip_top_lines = (7-skip_bottom_lines) # always number of picked picks + 1

residue_id = np.genfromtxt("321.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 0, dtype = str, delimiter = "-")

a_value00 = np.genfromtxt("321.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 5, unpack = True)
a_value01 = np.genfromtxt("322.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 5, unpack = True)
a_value02 = np.genfromtxt("323.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 5, unpack = True)
...
a_valueXX

a_noise00 = np.genfromtxt("321.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 6, unpack = True)
a_noise01 = np.genfromtxt("322.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 6, unpack = True)
a_noise02 = np.genfromtxt("323.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 6, unpack = True)
...
a_noiseXX

# NORMALIZATION OF SAID PARSED VALUES

a_norm00 = a_value00/a_value00
a_norm01 = a_value01/a_value00
a_norm02 = a_value02/a_value00
...
a_normXX

a_noiseval00 = a_norm00/a_noise00/2
a_noiseval01 = a_norm01/a_noise01/2
a_noiseval02 = a_norm02/a_noise02/2
...
a_noisevalXX

# FITTING AND PLOTTING

spin_lock_durations = (0.005, 0.010, 0.020, 0.050, 0.100)

y_signal_a = (a_norm00, a_norm01, a_norm02,..., a_normXX)

e_signal_a = (a_noiseval00, a_noiseval01, a_noiseval02,..., a_noisevalXX)

new_x = np.linspace(0, 0.105)

def exp_decay():
x = spin_lock_durations

y_signal_a = (a_norm00, a_norm01, a_norm02,..., a_normXX)

x = np.array(x, dtype = float)

y_signal_a = np.array(y_signal_a, dtype = float)

plt.errorbar(x, y_signal_a, yerr = e_signal_a, fmt = ".k", markersize = 8, capsize = 3)

def exp(x, a, b):
return a * np.exp(-b * x)

popt, pcov = curve_fit(exp, x, y_signal_a)

par_err = np.sqrt(np.diag(pcov))

print(par_err)
plt.plot(new_x, exp(new_x, *popt))

plt.tight_layout()
plt.show()
exp_decay()


All the tables (321.list, 322.list, etc.) are formatted in exactly the same manner, so I am confident the process can be improved via a for loop of some kind. I am currently trying to import all the files and parse them via the following:

for file in os.listdir('.'):                                  # loop through all the files in your current folder
if file.endswith('.list'):                                # find .list files
file_name, file_extension = os.path.splitext(file)    # split file name and extension
print(tot_data)


But then, I fail at selecting the correct columns and proceed with the fitting plotting.

One file looks like:

     Assignment         w1         w2     w1 (Hz)    w2 (Hz)  Data Height       S/N

X1H-H      9.482      9.482    7588.56    7588.56       926279        129
C4H7-H7      9.419      9.419    7538.63    7538.59      2029781        282



Any help would be enormously appreciated! Thanks for your time!

• I am failing at making it better. The original, inefficient code works. I am trying to parse all the relevant entries at the same time without calling them one by one the way I do now. – Shawn Marion fan Jul 10 at 13:38
• so to clarify, a_value00 = np.genfromtxt("321.list", skip_header = skip_top_lines, skip_footer = skip_bottom_lines, usecols = 5, unpack = True) reads 1 value from a file – Maarten Fabré Jul 10 at 14:14
• @MaartenFabré exactly! by changing skip_top_lines (and thus skip_bottom_lines) I select a specific entry in each of the files. It would be much better if I can write the script so that it would do this automatically. – Shawn Marion fan Jul 10 at 14:20
• A problem I just realized, but is very relevant, it that the files have mixed delimiters (distinct number of white spaces from column to column, which is the reason why I am unable to generate a proper df.DataFrame. I am currently focusing on this issue. – Shawn Marion fan Jul 10 at 14:40