I am trying to create a robust, generic way to parse a bar delimited usr
files, now I can read the file in and separate it by |
then index with integers.
However, this always feels very rigid in its design and I want to try to avoid it.
What I would like is a way to map any bar delimited file to JSON or at least a Python dict
. I'm looking for some from of Factory method I think.
Say if the file like this:
Header|Header1|Header2|Header3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
Footer|Footer1|Footer2|Footer3
It would be relatively straight forward. However, it would become undesirable when you get files like this:
Header|Header1|Header2|Header3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
Footer|Footer1|Footer2|Footer3
This represents a Header
, Tail
(which are always the same in every file) and 2 entries (2 sets of Group1
and Group2
)
So I need to also retain the fact that, files have groups and each set of group has to be 'scooped' up together. I.E: File X
may have two groups (A
and B
) - if File X
had one entry it would look like this:
Header|Header1|Header2|Header3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
Footer|Footer1|Footer2|Footer3
Two entries would look like this:
Header|Header1|Header2|Header3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
Footer|Footer1|Footer2|Footer3
All the key names for File X
are known, so I can use a lookup structure
At the moment I have a Pandas implementation looks like this:
df = pd.read_csv('file1.usr', sep='|')
header_names = ["HeaderKey", "HeaderKey1", "HeaderKey2", "HeaderKey3"]
footer_names = ["FooterKey", "FooterKey1", "FooterKey2", "FooterKey3"]
groups = {'A': ['AValueKey', 'A2ValueKey', 'A3ValueKey'],
'B': ['BValueKey', 'B2ValueKey', 'B3ValueKey']}
first_group_name = 'A'
df1 = df.iloc[:-1]
s = df1.iloc[:, 0].eq(first_group_name).cumsum()
for i, x in df1.groupby(s):
group = {}
for k, v in x.set_index(x.columns[0]).T.to_dict('l').items():
group[k] = dict(zip(groups[k], v))
header = dict(zip(header_names, df.columns))
footer= dict(zip(footer_names, df.iloc[-1]))
file = {'header': header, 'groups': group, 'footer': footer}
print(file)
{
'groups': {
'A': {
'AValueKey': 'Entry1', 'A2ValueKey': 'Entry2', 'A3ValueKey': 'Entry3'
},
'B': {
'BValueKey': 'Entry1', 'B2ValueKey': 'Entry2', 'B3ValueKey': 'Entry3'}
},
'header': {
'HeaderKey': 'Header'
'HeaderKey1': 'Header1',
'HeaderKey2': 'Header2',
'HeaderKey3': 'Header3',
},
'footers': {
'FooterKey': 'Footer',
'FooterKey1': 'Footer1',
'FooterKey2': 'Footer2',
'FooterKey3': 'Footer3',
}
}
So it relies on having the structure:
header_names = ["HeaderKey", "HeaderKey1", "HeaderKey2", "HeaderKey3"]
trailer_names = ["FooterKey", "FooterKey1", "FooterKey2", "FooterKey3"]
groups = {'A': ['AValueKey', 'A2ValueKey', 'A3ValueKey'],
'B': ['BValueKey', 'B2ValueKey', 'B3ValueKey']}
first_group_name = 'A'
Are there any other ways that would be more efficient?
EDIT based on @Reinderien answer
- Updated data format
Header|Header1|Header2|Header3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
Footer|Footer1|Footer2|Footer3
Firstly, thanks for going out on a limb even though evidently I haven't provided a clear scope.
To address your points;
Suggestions on global code, cap constants, tuples over lists and tail/trailer all noted, thanks :)
Indication of scale:
Each file is up <5KB, with a volume of between 10,000-100,000/day. I.E this script would need to parse and load up to 100,000 5KB files daily.
- Case of repeated groups:
File would look like this:
Header|Header1|Header2|Header3
A|Entry1|Entry2|Entry3
B|Entry1|Entry2|Entry3
A|Entry2|Entry3|Entry4
B|Entry2|Entry3|Entry4
Footer|Footer1|Footer2|Footer3
I take full responsibility for not being clearer in my question, but this is undesirable behavior. In the case of repeated groups, we would need to retain all the data, but split it into two separate payloads. Header and Footers:) will be the same for both however the group
part of the payload would contain the corresponding data.
The first entry in the group line is always the same, but the data leading from that can differ. I hope that clears things up, please let me know.
foo
,MyClass
, ordoSomething()
) leaves too much to the imagination. \$\endgroup\$ – Mast Apr 29 '20 at 9:48'B': ['BValueKey', 'B2ValueKey', 'A3ValueKey']
have anA
in it? \$\endgroup\$ – Reinderien Apr 29 '20 at 17:11