Swarm 9 Highlights

Swarm 9.6 — launched June 2018

• Usability Enhancements for UI — The Content UI has significant new capability around object renaming and version management. From the UI, you can now rename an object, including any pseudo path. Renaming a versioned object has the effect of marking the existing object as deleted and creating a new object using the new name. The rename dialog alerts you to this behavior if the object being renamed is versioned.
  
  

  Open

  

  
  
  The Content UI also allows precise control over managing and deleting versioned objects so the full storage API expressiveness is incorporated into UI actions. This allows for deleting the object so it can be restored later, deleting a specific version of the object from the version history, or completely deleting the object along with its entire version history. The UI also clearly denotes special delete markers in the version history.
  
  

  Open

  

• Next-Generation Replication Method — Swarm Storage now supports a new replication method, replication by direct POST. Swarm's new replication technology offers better performance and flow management between source and target clusters, so the rate can be increased (for faster hydration) or lowered, to prevent swamping a smaller target. Now, when you define a replication feed, you specify which replication mode to use: either the legacy bidirectional GET method of replication (which you may need for specific application compatibility or network requirements) or the new direct POST method. 
  
  

  
  The Swarm Storage UI supports this new direct POST method of replication. Each replication feed shows which type of replication is in force and what speed (number of concurrent threads) is set for direct POST.
  
  

• Platform CLI Expansion — The set of CLI commands for Swarm Platform server is extensively expanded and simplified to improve system administration. In addition to UEFI boot support and many other enhancements, Platform now implements an API to orchestrate rolling reboots of live Swarm storage clusters. The CLI provides broad control over the rolling reboot process with commands to pause, resume, skip specific nodes, and cancel rolling reboots altogether.     

• Documentation Reorganization — In response to client requests for more role-focused documentation and to unify similar topics across Swarm components, the Swarm documentation is fundamentally reorganized into three main phases and audiences: Deployment, Administration, and Development:
  
  

• Deployment Planning Documentation — To better support planning for and implementing Swarm storage solutions, new guidance and numerous architectural diagrams are available in the new Swarm Deployment section of the documentation:

  • Migrating from Traditional Storage

  • Use Cases and Architectures

  • Deployment Planning

    • Capacity Planning

    • Hardware Selection

    • Deployment Best Practices

  • Deployment Process

• SwarmNFS Configuration UI Updates — The Swarm Storage UI has several improvements to support the latest releases of SwarmNFS.

Swarm 9.5 — launched February 2018

* Mandatory update for those using erasure coding (EC) with 9.3.0 or higher.

• Efficient Hardware Refreshes — Swarm Storage has significantly increased the efficiency of refreshing cluster hardware and retiring whole chassis. The processes for retiring volumes and cleaning up over-replication are optimized.

• Filtering Response Headers — For MSPs and other direct-to-Internet deployments, Swarm Storage now offers response metadata filtering, for managing bandwidth and information flow. Swarm provides cluster settings for restricting which headers are returned for GET and HEAD responses, using either a whitelist or blacklist approach. Filtering the response headers best serves those needing to conserve bandwidth (eliminating as many bytes as possible when serving content) and those needing enhanced security (to reveal as little as possible about the content and its context).

• Updating Custom Headers — Swarm allows management of an object's custom headers in bulk during updates. Adding the preserve query argument to PUT or APPEND allows keeping the existing custom headers as well as add new ones; adding replace to an APPEND allows stripping off the existing custom headers as well as add new ones on the request.

• Improved Tracking — Swarm now keeps track of EC conversions and consolidations that are potentially and actually performed during HP cycles, and it publishes these statistics via SNMP. In addition, the gradual cleanup of the orphaned contents of removed bucket and domain objects is now tracked and reported in health report summaries.

• Streamlined Metrics — To minimize the impact of metrics indices on Elasticsearch clusters, Swarm Metrics now has fewer shards in its schemas. After upgrade, it starts creating the new, improved indices.

• Interactive Metadata Editing — The Content UI now allows users to edit object metadata directly in the UI. While applications have always been able to edit object metadata, this new interface enables users to correct and extend metadata from the convenience of a browser.
  

• SwarmNFS 2.0 — Swarm 9.5 supports SwarmNFS 2.0, the second major release of SwarmNFS, which is a lightweight protocol converter that integrates Swarm object storage with NFS v4. SwarmNFS provides a traditional file interface to Swarm object storage for content generators who use native NFS-based applications to create, access, and manage that content while allowing for the same content to be created, accessed, and managed through modern cloud and object APIs such as S3 and SCSP. SwarmNFS integrates with the Storage UI for configuration of exports that map disks to buckets in Swarm domains.
  
  
  

• SwarmNFS Interface through Gateway — The SwarmNFS storage interface now supports Content Gateway in addition to Direct to Swarm. The Cloud Security section of export configuration allows setting up the method best fitting the situation: Session Token (token admin credentials with expiration), Single User (user, password, and token), or Pass-through.
   

Swarm 9.4 — launched December 2017

• New Access Policy Editor — The Content UI includes a interactive policy editor for creating and updating access control policies for a provided tenant, domain, or bucket, greatly simplifying the process. The editor presents selections for quickly adding the most commonly needed policies (such as public read-only and authorized user full access) as well as options for designing granular access for users and groups. Safeguards help protect from unintended consequences, such as denying access to All Authorized Users.
  
  
  

• Dynamic Subcluster Reassignment — Swarm Storage now supports dynamic reassigning of nodes to subclusters, so the nodes do not need to be restarted for subcluster assignments to take effect.

• Elasticsearch Config Automation — Implementing Elasticsearch for Swarm requires making a set of configuration and settings file changes consistently across all nodes of the Elasticsearch cluster. The Elasticsearch package now provides a configuration script that automates these changes.

• OSS Updates — Swarm is now based on the Debian 8 (jessie) distribution, which brings numerous operating system bug fixes and improvements. Many drivers and services are also updated: Linux Kernel to 4.4.84, Linux Firmware to 1.167, NTP to 4.2.8-p10, and Intel drivers ixgbe 5.2.3, igb 5.3.5.10, ixgbevf 4.2.1, e1000e 3.3.5.10, i40e 2.1.26, and i40evf 3.0.8.

• S3 Compatibility — Gateway includes several new S3 protocol enhancements to track with Amazon S3 changes.

• Swarm Hybrid Cloud — Swarm Hybrid Cloud is a new Microsoft Azure Marketplace solution for seamlessly converting files from an S3 object, Swarm object, or NFS-accessible file to Azure's native Blob format. It allows implementing hands-free replication of content from on-premises Swarm Storage into an Azure storage account, using a standard Swarm replication feed.

Swarm 9.3 — launched August 2017

• Swarm Platform Server — Swarm Platform Server 9.0 is the next generation version of CSN 8.3, unifying and centralizing the services and management information needed to install, upgrade, and monitor Swarm Storage clusters. Installed on a dedicated node, Platform simplifies the network services and Swarm product installation process for Swarm administrators. The Platform Server extends the capabilities of the Storage UI, such as allowing you to reconfigure the cluster settings dynamically without SNMP:
  

• Multiple Filters for Replication — Swarm provides finer granularity and greater control when replicating content between clusters. Replication feeds support multiple filters to specify which domains to include, which ones to exclude, or both, and provide individual control for replicating unnamed objects with no domain. Additionally, the domain filters support wildcard matching for uncomplicated rule specifications. 

• Dynamic Stale Volume Handling — Administrators now have an easier mechanism for handling stale (offline for over two weeks) volumes without rebooting the cluster. Stale volumes can now be dynamically mounted or reformatted while retaining the volume's encryption status. 

• Faster Random Reads of Large Objects — Range read performance of large files is improved via caching of erasure coded segments.

• Storage UI Enhancements — Storage UI has several improvements in the behavior, formatting, messaging, and localization of its web pages.

• S3 Compatibility — Numerous improvements in Content Gateway support Amazon S3 compatibility, including the new setting enhancedListingConsistency.

Swarm 9.2 — launched May 2017

• Metadata Typing Expanded — In addition to indexing content metadata as string types, Swarm now supports alternative representations — such as numeric, time, and geo-location — for the same metadata. This provides applications with powerful query capabilities when performing searching operations. For example, a metadata field value of "30.3675, -97.7878" can simultaneously be searched for as a string value or as a latitude/longitude geographic coordinate. Swarm automatically adds any supported type representations for metadata as it is indexed.

• S3 Multipart Listing Compatibility Improvements — Some S3 applications naively codify formatting validations for multipart operations contrary to AWS S3 API guidance. Swarm mimics AWS S3 behavior with regard to the formatting of the opaque ETag string on multipart upload and listing operations to support these applications. 

• Volume Upgrades and Management Simplification — Swarm volumes that hold mostly small objects and are formatted by Swarm 7.5 or earlier can be automatically upgraded in-place to better handle objects <100KB. This upgrade expands the journal space and eliminates a "full" volume due to journal exhaustion. An administrator had to manually retire and reformat the old volume to expand the journal space prior to this improvement. 

• Performance and Stability Improvements — Load balancing of small objects (<100KB) onto newly added volumes is improved. Hot-plug add/remove of volumes is improved for rapidly occurring events on multiple volumes in a chassis. When using a replication feed, the target cluster's replicate-on-write policy is honored, which both reduces the load impact to the cluster and eliminates any time before full protection for the replicated objects.

• Live Changes to Cluster Settings — The Cluster Settings page of Storage UI allows making dynamic changes to persisted cluster settings. These dynamic settings are previously only modified using SNMP.

Swarm 9.1 — launched January 2017

• Automatic EC Segment Optimization — Because design constraints may require applications to write EC objects with excessive segmentation, you can now configure the health processor to automatically consolidate these segments into an optimal number in the background.

• Automatic Computation of Content-MD5 — For better SCSP and S3 application compatibility, a new Swarm setting automatically adds the Content-MD5 header on all writes. Previously, SCSP applications had to request this behavior on a per-transaction basis, and it can cause problems when sharing content with some S3 applications if not computed.

• Composite Content-MD5 for Multipart Writes — Swarm computes and saves a composite MD5 header with an object to improve S3 multipart upload compatibility and to provide SCSP clients with end-to-end transfer validation with parallel writes. This complements the use of the Content-MD5 request header on the pieces of a multipart object so transmission integrity is validated by both parties during all phases of a multipart (parallel) write.

• Synchronous Remote Object Replication — In some deployments, an application needs proof that critical content is replicated across independent clusters prior to completing a transaction. A new SCSP method is added that allows an application to request the immediate remote replication of an object and to receive positive confirmation that the object exists in the remote cluster.

• Storage UI hosted on CSN — CSN includes the redesigned storage interface so transitioning from the legacy Swarm Admin Console while using the familiar CSN software stack is possible.
  The new UI is includes the ability to view, configure, and manage the storage cluster. 

• Historical Metrics hosted on CSN — CSN includes all components needed to allow the CSN to collect and store the operational time-series data produced by the storage cluster and used by the new UI. Deployments not making use of a separate Elasticsearch cluster for metadata searching can collect these metrics.

• New Service Proxy — The Service Proxy (ServiceProxyServlet) works as a special instance of Gateway configured with additions and overrides to its configuration. The Service Proxy provides a proxy to a Swarm cluster, leveraging the same IDSYS authorization and authentication as the Content Gateway. With the Service Proxy, host the Storage UI and API from a server accessible to administrators and have Service Proxy manage communication with the storage cluster. 

Swarm 9.0 — launched October 2016

• New User Interface — Provides both hardware and content management. Administrators and end-users can use the new HTML5-responsive UI to monitor usage, specify data protection policies, and search the content with unified access across desktop, laptop, tablet and mobile devices.

• Quota Management — Manages storage and bandwidth control at the tenant, storage domain and bucket levels via the new UI or REST API, enabling administrative control, end-user self servicing, and dev-ops process automation—making it even easier to provide public or private cloud storage.

• Historical Metrics and Trends — Presents Swarm metrics within new UI and through REST APIs so clients can leverage third-party data analysis tools such as Kibana.

• Encryption at Rest — Provides industry-standard AES-256 whole volume encryption, eliminating the need to scrub retired or failed hardware before replacement.

• Metadata Annotation — Allows users to attach structured or unstructured metadata to existing objects without altering the base object.

• Elasticsearch 2.x — Provides an analytics engine designed for horizontal scalability and open access to Swarm’s operational metrics, metering, and metadata. Used through the new UI or with third-party big data tools.

• Integrated SwarmNFS Management — Offers complete management for SwarmNFS with the ability to define shares for domains and buckets.