I have a list of ~27000 objects. Each object represents a line from a file where each line is one record of a measurement from some instrument.
The most import aspects of the object are:
- Instrument name: int (49099, 89, …)
- Station Name: string (K-900, MK-45, …)
- Time: datetime object (13:34:17 01/02/2017)
All these objects are going to be used in creating a Hierarchical Data Format file where the top most "layer" is a measurement. Each measurement contains multiple line objects which have the same name and have a difference in time within some duration (30 minutes).
One major problem is that the data files I read to create the line objects are very unstructured. I cannot assume that subsequent lines in one file have anything to do with each other, so I cannot compare each line to the previous line to have some filtering logic in the reading part. Even files that have been generated on the same date SHOULD look similar, just with different instrument name, but this is not the case for this problem.
That is why I am reading them all in and THEN comparing all lines to each other. But it is taking a very long time and is not scalable at all.
The code provided is what I am currently doing and I would love to hear any improvements I could make or different ways to tackle my problem.
new =  for i, r in enumerate(self.records): x = (y for y in self.records if y.compare_record_same_name(r)) if any(r in x for x in new): continue else: new.append(x) class Record(): def compare_record_same_name(self, other): duration = abs(self.date_time - other.date_time) duration = duration.total_seconds() return (self.name == other.name and duration < TIME_SEPERATOR and duration > 0)