The following are details and guidance for a complex example, how to paginate (list ordered subsets of) the search results on objects matching specific metadata.

This walkthrough shows how and why to combine the use of three related https://perifery.atlassian.net/wiki/spaces/public/pages/2443821835 : size, marker, and sort.

How to Count Objects in a Bucket

This query returns an empty set (size=0), focus on the header output alone:

Check the value for Castor-Object-Count to determine how many objects are associated with the search performed. The number of objects in the "public" bucket under domain "jdoe.cloud.acme.com" is 62 per above.

How to Count Filtered Objects

Drill down further and focus on items matching a metadata characteristic. Filter for a specific kind of content (application, audio, image, text, video) being stored in the object, which is recorded in the Content-Type metadata header. Note: filter objects by custom metadata as well.

This search filters for objects holding MP4 video content:

Filtering the "public" bucket in domain "jdoe.cloud.acme.com" for MP4 content (content-type=video/mp4) produces a count of 39 videos (Castor-Object-Count: 39).

How to Limit (Page) the Results

Limit the size of the search results when a portion of the search results is needed or the entire set of objects is too large to be displayed in full. Combining three search query arguments provides the control needed:

• size - Controls the size of the result set, unrelated to object size (content-length). Set it to 0 when the actual listing is not needed.

• marker - Used with size to paginate large result sets. Use an empty key to begin a new search, then use the last sort key value of the results on the next request to continue pagination.

• sort - Sorts the results on one or more fields, in the order listed. Sorting defaults to ascending, so add descending (:desc) as needed. Sorting is computationally intensive, so sort output when necessary.

• Skip getting the return headers of the request: this is performed because total object account above for objects filtering for is already determined.

• "marker=", set to empty, starts at the beginning of the result set.

• "size=5" returns the first 5 of our filtered objects.

• "sort=etag:desc" sorts the objects in descending order from the "hash" value (ETag) associated with the object which is covered in detail below.

How to Pull the Next Result Set

Select a marker for the subsequent query to get the next five results in the set. Subsequent requests can be selected (marked) by a characteristic (metadata field) returned for the last object in the set. There are many to choose from:

The best practice is to use the "hash" field:

The "hash" is the object's ETag (entity tag), which is guaranteed to be unique across the entire cluster. It supports queries spanning multiple buckets and domains.

How to Use the Hash as Marker

It takes two steps to page through result sets using the hash value as the marker:

1. Parse the hash value out of the output for the last object in the previous set.

2. Set the marker argument to be the hash string.

The hash value listed for the last object in the result above is "e7e4a3d4cd8ee0df2894520d0624ceca", so start our next search for results after the object as follows:

This returns the next set of 5 objects in descending ETag value ordering (sort=etag:desc).

For the next set, parse out the hash for the last object listed (b6e556acd26d43f052490afd0fe42e4f) and continue until walking through all objects returned.