Summarizing downtime statistics

I am not satisfied with the performance and especially the time my piece of code takes to complete.

The surrounding environment for better understanding is the following:

A statistic for Downtime has to be created. Downtime means that some machines are not working and standing still. These have to be captured and my part is to collect this data and display it on request.

For this I created a structure of a POJO called Downtime which holds the basic information like when it started and when it finished. And a POJO called DowntimeSpecification which holds more detailed information as where in the machine the problem happened and such things.

The connection between these is that 1 Downtime can have N Specifications and 1 Specification can have only 1 Downtime it is associated to.

The data I get looks like this:

"data": [
[
"2016-05-01T14:35:01+02:00", // Downtime
1,
0,
900000,
"2016-05-01T14:50:01+02:00",
0,
"2016-05-01T14:35:01+02:00", // specification starts here
6,
56,
0,
1,
40,
0,
0,
0,
0
],
[
"2016-05-01T14:35:01+02:00",
1,
0,
900000,
"2016-05-01T14:50:01+02:00",
0,
"2016-05-01T14:35:01+02:00",
5,
93,
0,
1,
31,
0,
15,
16029,
1
],
[
"2016-05-01T15:55:01+02:00",
1,
0,
900000,
"2016-05-01T16:10:01+02:00",
0,
"2016-05-01T15:55:01+02:00",
6,
95,
0,
1,
40,
0,
0,
0,
0
],
[
"2016-05-01T15:55:01+02:00",
1,
0,
900000,
"2016-05-01T16:10:01+02:00",
0,
"2016-05-01T15:55:01+02:00",
6,
103,
0,
1,
111,
0,
15,
16030,
1
],
[
"2016-05-01T23:45:01+02:00",
1,
0,
900000,
"2016-05-02T00:00:01+02:00",
0,
"2016-05-01T23:45:01+02:00",
6,
95,
0,
1,
40,
0,
0,
0,
0
],
[
"2016-05-01T23:45:01+02:00",
1,
0,
900000,
"2016-05-02T00:00:01+02:00",
0,
"2016-05-01T23:45:01+02:00",
6,
79,
0,
1,
111,
0,
15,
16031,
1
]
]


The data in this JSON is redundant as the first bit of data in each json part is the Downtime Pojo and the other data is for the specifications.

Here is the piece of code that takes too long and is generally not satisfying. What I would like it to look like:

private List<Downtime> createDowntimes( Department department, TQLResult queryTQL ) throws CustomException
{
List<Downtime> result = new ArrayList<>( queryTQL.getData().size() );
Downtime savedDowntime = null;
Downtime savedSummary = null;

for ( Object[] row : queryTQL.getData() ) // The above JSON is in this Object[]
{
OffsetDateTime time = OffsetDateTime.parse( (String) row[COLINDEX_TIME_FROM] ); // this is the first data in the JSON ( 2016-05-01T14:35:01+02:00 )
ZonedDateTime downtimeFrom = time.atZoneSimilarLocal( ZoneId.systemDefault() );

Downtime downtime = new Downtime( row, department );
downtime.setSummary( false );
DowntimeSpecification specification = new DowntimeSpecification( row, department, false );

if ( previousDateTime != null ) // first run in loop?
{
if ( savedDowntime != null ) // did I find a Downtime already?
{
if ( downtimeFrom.equals( previousDateTime ) ) // is the first bit the same? Only the specification is interesting then
{
log.trace( "found specification for the same downtime {0}", previousDateTime );
savedDowntime.addSpecifications( new DowntimeSpecification( row, department, false ) );
}
else
{
if ( savedSummary != null && savedSummary.getDowntimeTo().getDayOfMonth() <= savedDowntime.getDowntimeFrom().getDayOfMonth()
&& savedDowntime.getDowntimeFrom().getHour() >= 6 ) // it is a new day after 6AM so I have to add the summary
{
log.trace( "Adding summary {0} to the result", savedSummary );
savedSummary = null;
}
log.trace( "adding finished downtime {0} to the result", savedDowntime );

ZonedDateTime tempPrevious = previousDateTime.truncatedTo( ChronoUnit.DAYS ).plusHours( 6 );
ZonedDateTime tempFrom = downtimeFrom.truncatedTo( ChronoUnit.DAYS ).plusHours( 6 );

if ( tempFrom.isAfter( tempPrevious ) ) // day change? create summary
{
log.trace( "Day change detected. Building Summary" );
if ( savedSummary != null )
{
}
savedSummary = createSummary( department );
}
previousDateTime = downtimeFrom;
savedDowntime = downtime;
}
}
else
{
savedDowntime = downtime;
}
}
else
{
previousDateTime = downtimeFrom;
savedDowntime = downtime;
}
}

if ( !result.contains( savedDowntime ) && savedDowntime != null ) // to not swallow the last downtime
{
}

if ( savedSummary != null ) // to create a summary even if the last downtimes don't change the day for completion reasons
{
}

log.debug( "setting the states for the downtimes" );
setStates( result ); // only takes >1 sec. So its ok
return result;
}

private Downtime createSummary( Deparment department ) throws CustomException
{
ZonedDateTime dayStart = previousDateTime.truncatedTo( ChronoUnit.DAYS );
dayStart = dayStart.plusHours( 6 );

ZonedDateTime dayEnd = previousDateTime.truncatedTo( ChronoUnit.DAYS );
dayEnd = dayEnd.plusDays( 1 );
dayEnd = dayEnd.plusHours( 6 );

Percent availability = downtimeCalculationService.getAvailability( department, dayStart, dayEnd ); // DB query
BigDecimal downtimeSummary = downtimeCalculationService.getDowntimeMinutes( department, dayStart, dayEnd ); // DB query
Downtime summaryDowntime = new Downtime( availability, downtimeSummary, dayStart, dayEnd );
summaryDowntime.setSummary( true );
return summaryDowntime;
}


What I do in this code is the following: I iterate over the result I get from some other system. Each Object[] therefore is a downtime with a specification included. So I need to recognize if the data is for the same downtime and I only have to add the bit of data for the specification. Also I need to know if a day change happened to build the summary.

I know that the function createSummary is part of the culprit since this has two calls that make db queries. Meaning I query the db for each day change I detect. However I can't change that for now, since this is done by someone else and I have to wait until they give me the ability to do that once for a span of days.

This piece of code takes around 3-6 seconds for data that represents about 130 downtimes.

I would like to reduce this time and also, if possible, find a better solution for the code in the for loop. Less if-else would be really appreciated since it would make it more readable.

• That's quite the ghost hunt, since we do not know what parts take how much time. How much you want to reduce the time? From 5 to 4 seconds? To 10 ms? How long does the sql operations take compared to the total time - since that's the culprit, as you mentioned? You're also writing ">1 sec. So its ok", but that's 33% from 3 seconds, which is... a third from the total time. Which is quite a lot... Jun 6 '17 at 14:19
• The sql operation take about 100ms which seems fine to me for a big load of data. The culprit is part of the problem. I refactored some other places to come down to 4-5 seconds. I guess when the service I rely on get's the new feature, I will be able to come down to 3-4 seconds. The biggest part of time is spent inside the for loop mentioned above. ( takes about 4,126 - 5,241 seconds ) ( with logging enabled for me to tell this times it slows down a bit ) Where as it seems to be about how I try to notice all the changes I have to pay attention to. I would like to get down to 2-3 seconds.
– Nico
Jun 6 '17 at 14:47
• The downtime data come from a database? If so, then it would probably make more sense to perform the computation in the database query instead. Sep 7 '17 at 8:29
• @200_success The scenario is a little bit complicated since I get part of data from an external system and have to lookup the references in our internal database. I did some modifications by now and it got a bit faster. A month takes only 700ms now but a year takes about 11s which is ok for me right now. I cannot perform computation in the databse sadly.
– Nico
Sep 7 '17 at 10:51

I happened to reduce the time to a great ~2 seconds. ( ~1,90sec a lot of times ). So heres what I did to increase the performance:

1. I changed the creation of the summaries to one function that does all instead of creating each summary when I noticed one is needed.

private Map<String, Downtime> createSummaries( StringBuilder summaryTql ) throws CustomException
{
Map<String, Downtime> result = new HashMap<>();

Connection connection = tsdbService.connect( getDataSourceName() );
TQLResult queryTQL = connection.queryTQL( summaryTql.toString() );

for ( Object[] row : queryTQL.getData() )
{
BigDecimal downtimeSummary = BigDecimal.valueOf( Long.parseLong( row[0].toString() ) / 60000.0 ); // downtime of that day in minutes
ZonedDateTime summaryTime = StringUtils.convertString2ZonedDateTime( row[2].toString() ); // timestamp of the day the summary is for
ZonedDateTime summaryEnd = summaryTime.plusDays( 1 ); // simply the end :D
Percent availability = downtimeCalculationService.getAvailability( summaryTime, summaryEnd, row ); // percentage of day without an active downtime
Downtime summaryDowntime = new Downtime( availability, downtimeSummary, summaryTime, summaryEnd );
summaryDowntime.setSummary( true );
result.put( StringUtils.convertZonedDateTimeToISO8601String( summaryTime.truncatedTo( ChronoUnit.DAYS ) ), summaryDowntime );
}
return result;


}

2. Changed the logic inside the function createDowntimes like this:

  if ( downtimeFrom.equals( previousDateTime ) ) // same downtime, only add specification
{
log.trace( "found specification for the same downtime {0}", previousDateTime );
savedDowntime.addSpecifications( new JassDowntimeSpecDTO( row, paperMachine, convertedCleaning, false ) );
}
else
{
if ( downtime.getDowntimeFrom().truncatedTo( ChronoUnit.DAYS ).isAfter( savedDowntime.getDowntimeFrom().truncatedTo( ChronoUnit.DAYS ) ) ) // simply check if the day changed. Ignore the time for now
{
if ( !dayChanged ) // did the day change without adding the previous detected day change? ( Happens if multiple days between downtimes )
{
dayChanged = true;
summaryKey = StringUtils.convertZonedDateTimeToISO8601String( savedDowntime.getDowntimeFrom().truncatedTo( ChronoUnit.DAYS ) );
}
else
{
addSummary( summaries, result, savedDowntime, summaryKey, downtime );
dayChanged = false;
summaryKey = StringUtils.convertZonedDateTimeToISO8601String( savedDowntime.getDowntimeFrom().truncatedTo( ChronoUnit.DAYS ) );
}
}
if ( summaries.containsKey( summaryKey ) && dayChanged && downtime.getDowntimeFrom().getHour() >= 6
|| publishSummary( StringUtils.convertISO8601StringToZonedDateTime( summaryKey ), downtime.getDowntimeFrom() ) ) // finally check if it is the time to add the summary
{
addSummary( summaries, result, savedDowntime, summaryKey, downtime );
dayChanged = false;
}
log.trace( "adding finished downtime {0} to the result", savedDowntime );
previousDateTime = downtimeFrom;
savedDowntime = downtime;
}


In combination with some other changes at places I am not responsible for and just use I was finally able to get down to my wished ~2 seconds. Now the only thing that would be nice to have is having far less if-else in the function createDowntimes but I guess that will not be answered here unless I provide more information of what data comes in and how the POJO looks like. Something that is not possible for me to do.

Hope this helps someone one day.

• 1. Is it okay to call this in the for loop? queryTQL.getData() Jul 21 '18 at 15:24
• 2. Isn't it possible to cache this: downtimeCalculationService.getAvailability Jul 21 '18 at 15:24
• 3. Does result capacity need to be as big as the query data? result = new ArrayList<>( queryTQL.getData().size() ); Jul 21 '18 at 15:25