4
\$\begingroup\$

I have a list of ~27000 objects. Each object represents a line from a file where each line is one record of a measurement from some instrument.

The most import aspects of the object are:

  • Instrument name: int (49099, 89, …)
  • Station Name: string (K-900, MK-45, …)
  • Time: datetime object (13:34:17 01/02/2017)

All these objects are going to be used in creating a Hierarchical Data Format file where the top most "layer" is a measurement. Each measurement contains multiple line objects which have the same name and have a difference in time within some duration (30 minutes).
 One major problem is that the data files I read to create the line objects are very unstructured. I cannot assume that subsequent lines in one file have anything to do with each other, so I cannot compare each line to the previous line to have some filtering logic in the reading part. Even files that have been generated on the same date SHOULD look similar, just with different instrument name, but this is not the case for this problem.
 That is why I am reading them all in and THEN comparing all lines to each other. But it is taking a very long time and is not scalable at all.

The code provided is what I am currently doing and I would love to hear any improvements I could make or different ways to tackle my problem.

new = []
for i, r in enumerate(self.records):
    x = (y for y in self.records if  y.compare_record_same_name(r))
    if any(r in x for x in new):
        continue
    else:
        new.append(x)

class Record():
    def compare_record_same_name(self, other):
        duration = abs(self.date_time - other.date_time)
        duration = duration.total_seconds()
        return (self.name == other.name and duration < TIME_SEPERATOR
                and duration > 0)
\$\endgroup\$
  • \$\begingroup\$ How exactly does the time factor into your classification? Because it sounds like you'll need some means of normalization in order to group things cleanly \$\endgroup\$ – Edward Minnix Jul 2 at 18:09
  • \$\begingroup\$ Why don't you sort the records? \$\endgroup\$ – vnp Jul 2 at 18:45
  • \$\begingroup\$ Is this your real code? At the moment it does not look like valid Python code. \$\endgroup\$ – AlexV Jul 2 at 20:00
  • \$\begingroup\$ @vnp, that is a pretty good suggestion, will try it thanks. Didn't think abou that... \$\endgroup\$ – Megaloming Jul 3 at 7:27
  • \$\begingroup\$ @EdwardMinnix, Not sure I follow. The time is when they did a measurement and saved it to file, so all measurements within the same timeframe (30 mins) and same name should be grouped together. \$\endgroup\$ – Megaloming Jul 3 at 7:30
1
\$\begingroup\$

I find the problem you are trying to solve to be as hard to grasp from the code presented as from the description:
- document your code, using the conventions of the language (&environment) of choice.
- use telling names

  • In the for-loop, you don't use i: don't use enumerate().
  • I think the first comprehension easier to understand with reading (? "r") mentioned first:
    measurement = (y for y in self.records if reading.<predicate>(y))
  • I think it confusing to use the same identifier for unrelated objects in one and the same statement - suggesting if not any(reading in n for n in new):
  • instead of if <condition>: <pass> else: <act> do
    if not <condition>: <act>
  • compare_record_same_name() is a horrible name, as it mentions only half of what the function checks. It seems it is there to check whether the other record belongs to the same "measurement" - name it for that: same_measurement().
  • instead of converting duration to seconds each and very time, make TIME_SEPARATOR a timedelta

I'd try a different approach:

  • readings with different instrument names do not belong to the same measurement, anyway:
    use a separate collection for each name
  • sort each collection by time (hoping the method used makes good use of what order is bound to exist in input)
  • any gaps exceeding TIME_SEPARATOR separate measurements
  • if the time covered by any group g is too big for one measurement, split at large internal gaps or evenly by number of readings or time range or …
\$\endgroup\$
  • \$\begingroup\$ thanks for the feedback. I been messing around with these algorithms for so long, changing this and that, forgetting about style and not removing stuff I dont use anymore. Appreciate it. I will try out your last two suggestions also. As for the two first suggestions, I managed to improve it, by sorting the full collection by name and date and comparing each line to their previous line. Seems to be working \$\endgroup\$ – Megaloming Jul 4 at 9:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.