Changes

This release of Content Gateway includes these improvements:

• Support for Elasticsearch 5.6: This release of Content Gateway now completes support for Elasticsearch 5.6, so the migration away from Elasticsearch 2.3.3 can be finished and reclaim those resources. With Gateway 6.0, the newly indexed Elasticsearch 5.6 feed is made the primary search feed, which includes Swarm's new atime (access time) metadata for tracking content usage, if enabled. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443809821.

• Upgraded Logging Format: As part of support for Elasticsearch 5.6, the logging system for Content Gateway has been upgraded to log4j2, which offers more flexibility and hierarchical control. The configuration file, which was logging.cfg, is now logging.yaml. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443810201. (CLOUD-3070)

• Graceful Reboots: Encryption keys are now persisted so rebooting of the Gateway no longer disrupts active Swarm UI browser sessions. (CLOUD-3027)

• Health Reporting: The cluster health report providing proactive support from Swarm now includes information about the Gateway installation, including the version and configured components. This feature requires valid entries for managementUser and managementPassword in the [storage_cluster] configuration section. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443810201. (CLOUD-2753)

This release includes these fixes: 

• The gateway error "Unable to create phone home data" was erroneously logged at startup. (CLOUD-3051)

• A rare race condition can result in a duplicate domain being created when enabling versioning in the Content UI. (CLOUD-3030)

• The service may not automatically start after a system reboot after upgrading. (CLOUD-2819)

• The cloudgateway_server.log had invalid SCSP warnings reporting 'Failed requests will not be retried' and 'Failed querying cluster for name and version'. (CLOUD-2663)

• S3 Gateway reports it as a failure to find the bucket when an EC write fails because of too few nodes to erasure-code a large object. (CLOUD-1452)

Upgrade Impacts

Watch Items and Known Issues

These are known operational limitations and watch items that exist for Gateway.

• Traffic to the Gateway is blocked unless action is taken to disable IPTABLES or to enable inbound traffic to the front-end protocol port(s) when using the default RHEL/CentOS configuration of IPTABLES.

• Gateway is not compatible with the fingerprint scanner module for Linux PAM. Remove it by running the following if it is installed: yum remove fprintd-pam

• SCSP reading operations that request a Content-MD5 hash validation and for which there is a hash mismatch causing a storage node to be temporarily removed for the Gateway's connection pool due to the way Swarm reports a hash validation failure.

• Swarm Integrity Seal upgrades cannot be performed through Gateway. They may be done directly to the back-end Swarm cluster.

• If the HTTP cache control headers If-Modified-Since and If-Unmodified-Since are used, review the discussion of these in the .

The following are known issues in this release:

• Quota states may not be properly evaluated at all times. (CLOUD-3079)

• When buckets are created, the x-amz-storage-class header is not preserved. (CLOUD-3062)

• The Gateway error "Failed reading from client" on a PUT due to "EofException: Early EOF" may occur when clients do not send the full body. This may point to a bug in the client's retry logic, such as not resetting the position marker to the beginning of the file or part. (CLOUD-3010)

• During new object creation as part of renaming with ?newname, Gateway does not verify the user has permission to create the new object name (although it is highly likely, because it is a write within the same context). (CLOUD-2966)

• An s3cmd or rclone server-side copy request may time out on a multipart copy for >5GB objects (s4cmd performs it correctly). Workaround: Increase the client timeout after verifying it is not the HTTPS proxy timing out: set s3mcd socket_timeout = 600 in ~/.s3cfg or use rclone copy --timeout=10m --contimeout=2m caringo:mybucket/5gb caringo:mybucket/subfolder/. (CLOUD-2949)

• Listings with max-keys may be shorter than expected because CommonPrefixes are included in the count of keys returned. (CLOUD-2917)

• Uploading files / photos using Panic's Transmit app on iOS fails due to a 403 Invalid Signature error. (CLOUD-2886)

• Gateway 5.2.2 and earlier do not output the NextMarker field in S3 listings, which can cause some S3 clients such as Caringo Drive, rclone, and Transmit to show only 1000 files in a directory or to miss some subdirectories. (CLOUD-2871)

• Usernames are case-insensitive, but listings exclude a token if the username (myadmin) does not match the case used when the token was created (myAdmin). (CLOUD-2837)

• Multipart PUT requests via recent Cyberduck versions fail with 403 SignatureDoesNotMatch when using AWS Signature Version 4. Install the Caringo .cyberduckprofiles from Using the Cyberduck application with Content Gateway S3 which force V2 signatures. (CLOUD-2799)

• The policy fails to take effect without warning if a policy document includes a Principal that has plural "users" or "groups" instead of "user" or "group". (CLOUD-2783)

• Versioning-enabled buckets with large numbers of objects may generate Gateway server.log warnings that can be safely ignored: "S3BucketRequestHandler: WARNING: problem with versioned bucket listing. Number of CommonPrefix (2000) exceeds max-size limit (1000)." (CLOUD-2643)

• 403 S3 V4 Signature mismatch errors may result when using Cyberduck with the "pound" proxy in front of Gateway S3. Workaround: Disable the Expect header in the Cyberduck preferences, or (recommended) use a different proxy such as HAProxy. (CLOUD-2628)

• When Gateway cannot connect to Elasticsearch nodes, the errors may erroneously report this as being related to Storage nodes. (CLOUD-2595)

• Because of issues with Range and ETag header handling, video playback of .mp4 streams may not work correctly when served via the Gateway S3 port. It does work when served via the Gateway SCSP port. (CLOUD-1964)

• Gateway caches the Swarm version from the "Server:" response header, so after upgrading Swarm Gateway must be restarted to consistently see the new version. (CLOUD-1271)

• Gateway responds with a 500 (Internal Server Error) instead of 400 (Bad Request) if the size of the metadata headers sent to Swarm is too large. (CLOUD-800)

• The S3 bucket listing StorageClass response element always reports STANDARD. (CLOUD-766)

• The Gateway audit log escapes the "%" characters used by the client as escape characters if an S3 client escapes URI path characters such as "/". URI audit log processing for S3 clients require double-unescaping when this occurs. (CLOUD-703)