Comparing objects in a list against each other

Question

I have a list of ~27000 objects. Each object represents a line from a file where each line is one record of a measurement from some instrument.

The most import aspects of the object are:

• Instrument name: int (49099, 89, …)
• Station Name: string (K-900, MK-45, …)
• Time: datetime object (13:34:17 01/02/2017)

All these objects are going to be used in creating a Hierarchical Data Format file where the top most "layer" is a measurement. Each measurement contains multiple line objects which have the same name and have a difference in time within some duration (30 minutes).
 One major problem is that the data files I read to create the line objects are very unstructured. I cannot assume that subsequent lines in one file have anything to do with each other, so I cannot compare each line to the previous line to have some filtering logic in the reading part. Even files that have been generated on the same date SHOULD look similar, just with different instrument name, but this is not the case for this problem.
 That is why I am reading them all in and THEN comparing all lines to each other. But it is taking a very long time and is not scalable at all.

The code provided is what I am currently doing and I would love to hear any improvements I could make or different ways to tackle my problem.

new = []
for i, r in enumerate(self.records):
    x = (y for y in self.records if  y.compare_record_same_name(r))
    if any(r in x for x in new):
        continue
    else:
        new.append(x)

class Record():
    def compare_record_same_name(self, other):
        duration = abs(self.date_time - other.date_time)
        duration = duration.total_seconds()
        return (self.name == other.name and duration < TIME_SEPERATOR
                and duration > 0)

Answer 1

I find the problem you are trying to solve to be as hard to grasp from the code presented as from the description:
- document your code, using the conventions of the language (&environment) of choice.
- use telling names

• In the for-loop, you don't use i: don't use enumerate().
• I think the first comprehension easier to understand with reading (? "r") mentioned first:
  measurement = (y for y in self.records if reading.<predicate>(y))
• I think it confusing to use the same identifier for unrelated objects in one and the same statement - suggesting if not any(reading in n for n in new):
• instead of if <condition>: <pass> else: <act> do
  if not <condition>: <act>
• compare_record_same_name() is a horrible name, as it mentions only half of what the function checks. It seems it is there to check whether the other record belongs to the same "measurement" - name it for that: same_measurement().
• instead of converting duration to seconds each and very time, make TIME_SEPARATOR a timedelta

I'd try a different approach:

• readings with different instrument names do not belong to the same measurement, anyway:
  use a separate collection for each name
• sort each collection by time (hoping the method used makes good use of what order is bound to exist in input)
• any gaps exceeding TIME_SEPARATOR separate measurements
• if the time covered by any group g is too big for one measurement, split at large internal gaps or evenly by number of readings or time range or …