Java 8 spliterator for paged results

Question

Summary: I am using an API which returns paged results. I want to have these results as Java 8 Stream and implemented a Spliterator for this purpose.

---

I am using AWS S3 Java API to list objects in the S3 bucket. The API returns paged results: when I call client.listObjects(bucketName, rootKey) for the first time, I get an ObjectListing instance which may be complete or truncated i.e. return just one "page" of results.

If ObjectListing is truncated I have to request further "page" via client.listNextBatchOfObjects(objectListing) (providing current "page" as marker) and so on until I get an ObjectListing which is not truncated.

I want to use Java 8 Stream APIs to work with ObjectListings. Ideally, I want to hide querying pages of ObjectListings behind some facility which would just give me a Stream<ObjectListing>. For this I've implemented a Spliterator<ObjectListing>:

import java.util.Objects;
import java.util.Spliterator;
import java.util.function.Consumer;

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ObjectListing;

public class ObjectListingSpliterator implements Spliterator<ObjectListing> {

    private final AmazonS3 client;
    private ObjectListing objectListing;
    private volatile boolean split = false;
    private volatile boolean currentObjectListingWasConsumed = false;

    public ObjectListingSpliterator(AmazonS3 client, ObjectListing objectListing) {
        Objects.requireNonNull(client, "client must not be null.");
        Objects.requireNonNull(objectListing, "objectListing must not be null.");
        this.client = client;
        this.objectListing = objectListing;
    }

    @Override
    public boolean tryAdvance(Consumer<? super ObjectListing> action) {
        if (!currentObjectListingWasConsumed) {
            action.accept(objectListing);
            currentObjectListingWasConsumed = true;
            if (!split && objectListing.isTruncated()) {
                objectListing = client.listNextBatchOfObjects(objectListing);
                currentObjectListingWasConsumed = false;
            }
            return true;
        } else {
            return false;
        }
    }

    @Override
    public Spliterator<ObjectListing> trySplit() {
        if (split) {
            // Already split, cannot be split once again
            return null;
        } else if (objectListing.isTruncated()) {
            split = true;
            final ObjectListing nextObjectListing = client
                    .listNextBatchOfObjects(objectListing);
            return new ObjectListingSpliterator(client, nextObjectListing);
        } else {
            return null;
        }
    }

    @Override
    public long estimateSize() {
        if (objectListing.isTruncated()) {
            return Long.MAX_VALUE;
        } else {
            return currentObjectListingWasConsumed ? 0 : 1;
        }
    }

    @Override
    public int characteristics() {
        return Spliterator.ORDERED | Spliterator.IMMUTABLE;
    }
}

Usage example - list all keys from the bucket and root key:

// Get the original objectListing
ObjectListing objectListing = client.listObjects(bucketName,
    rootKey);
// Create a stream of objectListings
Stream<ObjectListing> objectListings = StreamSupport.stream(
    new ObjectListingSpliterator(client, objectListing), false);

List<String> keys = objectListings
    .map(ObjectListing::getObjectSummaries)
    .flatMap(Collection::stream)
    .map(S3ObjectSummary::getKey);

What I'm interested in:

• Is my spliterator OK?
• I'm not sure if there might be a concurrency issue between tryAdvance and trySplit.
• Not sure what characteristics should be.

The question is actually not specific to AWS API, it is generally about streaming paged result where:

• there is some way to retrieve the first page;
• with every page you know if it is complete or not;
• you can get the next page using the previous page.

Answer 1

First of all, splitting the spliterator only makes sense, if both the remainder of the current spliterator and the returned spliterator still have work pending. In your case, this is (almost) not true, as you operate on the complete batches and the current spliterator at most returns its current batch after split. Thus, I'd replace the trySplit() with a simple return null. This also addresses any potential concurrency issues (which I have not looked into in depth.)

Characteristics basically tell the caller the - for lack of a better word - characteristics of your spliterator. :-)

I think for the batch approach you take, ORDERED, NONNULL, IMMUTABLE should be OK.

Apart from that, for more direct utility, I'd rather take the approach not to iterate over the batches, but over their contents, i.e. create a Spliterator which gets initialized with the first ObjectListing batch and then transparently goes through the underlying collections element-wise and fetches the next batch as needed. This would eliminate the need to flatMap on the result stream and feel more natural for a streaming approach. (In fact, this sounds so useful that I'd like to have it :-))