S3 Protocol Architecture
The S3 protocol personality is a front-end storage protocol for client applications. It runs within the Gateway itself. All Gateway deployment and scaling features apply when using the S3 front-end protocol.
An administrator can configure the Gateway to run several ways:
the SCSP protocol
the S3 protocol
both protocols at the same time
Gateways can be scaled horizontally with any combination of front-end protocols as needed for a particular deployment. This allows for a heterogeneous mix of client types utilizing the same Swarm storage cluster and sharing content with each other. Performing this allows an administrator to provide a unified object storage platform.
Sharing Storage across S3 and SCSP
Content Gateway provides the mechanism to unify the back-end Swarm object storage so S3 applications and SCSP applications can share content. In a unified object storage deployment, a device like a load balancer is used to route incoming client traffic through the appropriate port number or pool of Gateway servers.
Routing Methods
These are some routing methods usable for a unified object storage front-end:
IP Address | Listen to multiple virtual IP addresses and distribute traffic based upon the IP address used by the client. | |
|---|---|---|
X-Forwarded-Host | Use Layer 7 inspection of the requests to distribute traffic based upon headers contained in the client requests. | The X-Forwarded-Host header can be used by a proxy or load balancer to allow client applications to use a different host name for each supported HTTP storage protocol while sharing the same storage domain for the content. Consider the storage domain The load balancer needs to add the header to direct storage requests to the shared storage domain: X-Forwarded-Host: castor.example.com |
DNS | Cause the DNS name resolution to be different for the clients using one storage protocol than for the clients using another protocol. | Clients using S3 may resolve the storage domain castor.example.com to 10.100.100.81 while clients using SCSP resolve the same storage domain to 10.100.100.82. This method requires the administrator has control of the hosts where the client applications run so as to allow alteration of the DNS/host resolution but does not require in-line modification of the HTTP requests. |
OPTIONS | Route request method OPTIONS with "Origin" to the S3 port rather than default to the SCSP port (which happens when an Authorization header does not exist or have "AWS"). | S3 must handle "bucket in Host" style requests because SCSP reports the domain was not found. curl -i
-H 'Origin: http://www.example.com'
-H 'Access-Control-Request-Method: PUT'
-X OPTIONS https://mycorsbucket.elsewhere.com/ |
Pattern-Matching | Use Layer 7 inspection of the requests to switch incoming traffic based on the object storage protocol. See below: | This is performed by looking for the distinctive S3 Authorization header pattern or one of the query string arguments for authenticated S3 requests. The pattern-matching rules for these authenticated S3 requests are as follows. |
| Use pattern-matching on the request header. | Authorization ~= ^AWS.* |
| Use pattern-matching on the query string arguments. | |
| Use pattern-matching to test for the absence of all AWS request patterns. | The request is either an anonymous S3 request or an SCSP request. Since anonymous S3 requests do not create, update, or delete content, they are most likely GET requests, and it is safe to allow SCSP to handle these. Using SCSP for video (e.g., mp4) objects also avoids an issue with If-Range handling on the Gateway S3 port. |
© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.