20
\$\begingroup\$

I'm using the Amazon S3 Java SDK to fetch a list of files in a (simulated) sub-folder. This code is rather standard (AWSConfiguration is a class that contains a bunch of account specific values):

String prefix = "/images/cars/";
int prefix_size = prefix.length();
AmazonS3 s3 = new AmazonS3Client(new AWSConfiguration());
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest().
    withBucketName(AWSConfiguration.BUCKET).
    withPrefix(prefix));

Now this list will include objects like /images/cars/default.png as well as /images/cars/ford/Default.png (because they both contain the same prefix). To list only the objects that are directly inside the /images/cars/ "folder" I have the following function (in a class called S3Asset)

public static boolean isInsideFolder(int root_size, String key) {
  return (key.substring(root_size).indexOf("/") == -1);
}

This looks at the full key for any / after the prefix as a clue that it is inside a sub-folder. This lets me iterate over the objects with the following code (I'm trimming the prefix for clarity):

for(S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
  if(S3Asset.isInsideFolder(prefix_size, objectSummary.getKey())) {
    System.out.println(objectSummary.getKey().substring(prefix_size));
  }
}

As near as I can tell this is the cleanest way to do this but it has one characteristic that I don't like. If I'm looking a root level folder I'm requesting the names of all files in all sub-folders only to iterate over them and learn that there is only one object in the actual root level folder. I've considered associating a key with the value being the full path of the folder, which would allow me to request objects with a predictable key instead of the prefix, but the major downside to this is that the key would have to be generated in code and therefor assets uploaded directly in to the S3 Bucket (through the management console) would not have this key. Anyone have a better idea?

\$\endgroup\$
22
\$\begingroup\$

In the ListObjectsRequest javadoc there is a method called withDelimiter(String delimiter). Adding .withDelimiter("/") after the .withPrefix(prefix) call then you will receive only a list of objects at the same folder level as the prefix (avoiding the need to filter the returned ObjectListing after the list was sent over the wire).

Some notes about the code:

1, I'd extract out to a local variable for the ListObjectsRequest instance:

final ListObjectsRequest listObjectRequest = new ListObjectsRequest().
    withBucketName(AWSConfiguration.BUCKET).
    withPrefix(prefix);
final ObjectListing objectListing = s3.listObjects(listObjectRequest);

It's easier to read.

2, root_size should be rootSize. (Regarding to the Java Coding Conventions.)

3, I would use String.contains instead of indexOf. It's more meaningful, easier to read since you don't have to use the -1 magic number.

4, In the last snippet I'd create a local variable for the key:

for (final S3ObjectSummary objectSummary: objectListing.getObjectSummaries()) {
    final String key = objectSummary.getKey();
    if (S3Asset.isImmediateDescendant(prefix, key)) {
        final String relativePath = getRelativePath(prefix, key);
        System.out.println(relativePath);
    }
}

5, Furthermore, I'd move the length call inside the helper method:

public String getRelativePath(final String parent, final String child) {
    if (!child.startsWith(parent)) {
        throw new IllegalArgumentException("Invalid child '" + child 
            + "' for parent '" + parent + "'");
    }
    // a String.replace() also would be fine here
    final int parentLen = parent.length();
    return child.substring(parentLen);
}

public boolean isImmediateDescendant(final String parent, final String child) {
    if (!child.startsWith(parent)) {
        // maybe we just should return false
        throw new IllegalArgumentException("Invalid child '" + child 
            + "' for parent '" + parent + "'");
    }
    final int parentLen = parent.length();
    final String childWithoutParent = child.substring(parentLen);
    if (childWithoutParent.contains("/")) {
        return false;
    }
    return true;
}

Note the input check. (Effective Java, Second Edition, Item 38: Check parameters for validity)

The multiple calls of length could look redundant and slow but premature optimization is not a good thing (see Effective Java, Second Edition, Item 55: Optimize judiciously). If you check the source of java.lang.String, you will find this:

/** The count is the number of characters in the String. */
private final int count;

...

public int length() {
    return count;
}

String is immutable, so it's easy to cache its length and JDK does it for you.

\$\endgroup\$
  • 4
    \$\begingroup\$ If this is the kind of input that you can expect from this community then I am 100% behind supporting it's promotion to a full fledged site! Thank you for such a complete answer. \$\endgroup\$ – Jason Sperske Dec 14 '11 at 22:11
  • 1
    \$\begingroup\$ Point 4 is very close to how this code actually looks in my actual app, and for point 5, in my S3Asset class I have 2 methods (one that takes 2 strings and does the length check on the parent, and one that I referenced in my question). It's a premature optimization, but calling a string length function on each iteration when the value will always be the same just makes me feel bad. Now what will I do with all the extra nanoseconds I'm saving? \$\endgroup\$ – Jason Sperske Dec 14 '11 at 22:19
  • 1
    \$\begingroup\$ Check the update, there is no extra nanosecond :-) \$\endgroup\$ – palacsint Dec 14 '11 at 22:32
  • 1
    \$\begingroup\$ It looks like the missing API call is "withDelimiter" which is clearly described on this page aws.amazon.com/releasenotes/213, if you update your answer I'll mark it as correct. \$\endgroup\$ – Jason Sperske Dec 17 '11 at 0:05
  • 1
    \$\begingroup\$ Great answer, I think it also can be improved by mentioned that at the ObjectListing level, the resulting list is truncated for more than 1000 acording to API Doc this docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html, The ObjectListing has a method objectListing.isTruncated() to indicate if there are more results, objectListing.getNextMarker() and objectListing.setNextMarker() are used to control the paging. A do-while() is suggested for the iteration, it would contain the "for (final S3ObjectSummary objectSummary: objectListing.getObjectSummaries())" loop. \$\endgroup\$ – le0diaz Nov 1 '18 at 19:29
5
\$\begingroup\$

That's an old question, but I just've find out that with a new version of amazonaws "1.9.22" you can call "getCommonPrefixes" to get a list of all available prefixes. You need to use delimiter either.

val listObjectRequest = new ListObjectsRequest().
  withBucketName("bucket").
  withPrefix("prefix").
  withDelimiter("/")

s3Client.listObjects(listObjectRequest).getCommonPrefixes

So, for a structure like "prefix/aaa/111" it will return "prefix/aaa" level items.

UPDATE:

And a java version (original one uses scala)

AmazonS3 s3 = new AmazonS3Client(new AWSConfiguration());
ListObjectsRequest listObjectRequest = new ListObjectsRequest().
    withBucketName(AWSConfiguration.BUCKET).
    withPrefix(prefix).
    withDelimiter("/");
ObjectListing objectListing = s3.listObjects(listObjectRequest).getCommonPrefixes();
\$\endgroup\$
  • \$\begingroup\$ Welcome to CR! That's great, but good answers on this site directly relate to the OP's code. Perhaps you could edit and add a bit of context to say where this snippet fits in? \$\endgroup\$ – Mathieu Guindon Mar 3 '15 at 17:09
  • \$\begingroup\$ Thanks @cyrillk! This community continues to pleaantly surprise me :) The original answer has become a deep part of a lot of my projects, I'm going to revisit the video see if your answer can make the intent cleaner :) \$\endgroup\$ – Jason Sperske Mar 3 '15 at 18:02
3
\$\begingroup\$

I got nothing returned when I used withDelimiter("/"). It turned out that in order to use delimiter, I also needs to end my prefix with /. In other words, withPrefix("prefix/abc") will get all objects under prefix/abc, but for delimiter to work, use withPrefix("prefix/abc/") instead.

\$\endgroup\$

protected by Jamal Jul 28 '17 at 4:10

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.