Swarm 10 Highlights

Swarm 10.2 — launched April 2019

• Swarm 10 performance — The rate at which nodes retire is now improved over both version 10.1 and version 9.6 of Swarm Storage. Swarm has also boosted the performance of erasure-coded range reads under high loads. 

• Prometheus Node Exporter — The Prometheus Node Exporter preview has configuration enhancements. The service is now enabled by default (metrics.enableNodeExporter=True), which makes basic hardware queries across nodes available without reboot. A new setting, metrics.nodeExporterFrequency, allows you prevent or control how frequently to refresh Swarm-specific metrics in Elasticsearch.

• Swarm management — Several improvements help you upgrade and work with Swarm. To ease upgrades to Swarm 10, the cluster-wide setting ec.protectionLevel is now a persisted setting, so you can change it on demand via Swarm UI or SNMP; the setting is no longer managed within and across config files, requiring consistency and cluster restarts. Swarm raises alerts on objects with persistent feed-related failures, such as objects that cannot be indexed in Elasticsearch or be remotely replicated. The versions query argument on listing queries now accepts versions=previous, which allows you limit results to only the past versions of an object. Swarm also now allows you write named objects with names that looks like a UUID (32-character hexadecimal).

• Swarm UI settings expansion — This release improves and extends the handling of Swarm Storage settings in the UI. Swarm settings include both cluster-wide and node-specific options, which can vary from chassis to chassis. On the Cluster Settings page, the Swarm UI now prevents erroneous changes by hiding the node-specific settings when Platform Server is not implemented and handling those nodes. The page also now includes an option to Show advanced settings. This option reveals all dynamic (persisted) advanced (unpublished) settings when enabled. These settings can be updated on a running cluster without a reboot.
  
  

  Open

  

• Docker and tool support — Content Gateway now uses the Version 2 implementation of AWS S3 GET Bucket (List Objects). With this change, Gateway supports the upcoming Docker Distribution 2.7 (registry). Gateway has also added infrastructural support for future dynamic features and extensions, such as an upcoming video clip creation tool.

Swarm 10.1 — launched February 2019

• Self-service migration from CSN to Platform Server — Swarm 10 now includes tools and documentation for upgrading from CSN 8.3 to the new generation of Platform server. These enable you to extract the live configuration and run-time information from the CSN Platform 8.3.x server and perform a live migration to a Platform 10 server without downtime for your storage cluster. 

• Offline Platform installs and multi-subnets — With this release, Platform Server supports offline usage (for "dark" sites): it can install and run Swarm Storage with no connection to the outside world. It also allows you to configure Platform to work across multiple subnets, an architecture in which each subnet uses its own PXE boot server.

• Single-IP architecture performance — The performance both for writes and for erasure-coded object reads is improved for Swarm 10's density-friendly single-IP architecture, the result of optimizations in how Swarm nodes write to volumes under the new design. Swarm also has improved memory handling, especially with bursts and high loads.

• Elasticsearch 5.6 for Gateway — This release of Content Gateway completes support for Elasticsearch 5.6, so the migration away from Elasticsearch 2.3.3 can be finished and those resources can be reclaimed. With Gateway 6.0, the newly indexed Elasticsearch 5.6 feed is made the primary search feed, which includes Swarm's new atime (access time) metadata for tracking content usage, if enabled. As part of support for Elasticsearch 5.6, Gateway's logging system is upgraded to log4j2, which offers more flexibility and hierarchical control.
  
  
  

• Hardware diagnostics with Prometheus — This release includes a preview of the Prometheus Node Exporter, which offers enhanced monitoring and diagnostics on the machines in your Swarm cluster. Prometheus is an open-source systems monitoring and alerting toolkit that allows you view what statistics are available for your system, even under failure conditions. Grafana and other API consumers can allow visualizing the collected data. As a preview, the settings and implementation are subject to change; for more about this preview, contact DataCore Support.

Swarm 10.0 — launched December 2018

• Density-friendly machine addressing — The internal architecture of storage cluster nodes can leverage very dense servers containing several CPU cores and disks. The visible effect of having a single IP per physical or virtual machine is that far fewer IP addresses are needed for deployment, which simplifies network administration, monitoring, and architecture. A new Settings Checker tool identifies configuration changes needed to support upgrading to the new architecture as well as each new version of Swarm, going forward.

• Multiple VLANs and subnets — Swarm supports multiple VLANs (multiple L3 subnets) within the Platform server and UI to support larger deployments where storage clusters are spread across multiple racks, rooms, and locations. This support allows for growing a cluster from dozens of pieces of hardware to hundreds, and simplifies the network architecture and administration.

• Remote replication without VPN tunnels — The push-style (direct POST) replication protocol introduced in 9.6 offers better performance and flow control. With 10.0, direct POST now supports SSL/TLS network encryption and standard proxy servers for replication feeds, which eliminates the need for separate VPN tunnels between clusters. This capability streamlines deployments where encrypted communications are needed over wide-area, untrusted networks, and it streamlines how you can dynamically mirror remote sites:
  
  

  Open

  

• Elasticsearch major upgrade, atime tracking — Swarm now ships Elasticsearch 5.6, which extends Swarm's built-in metadata searching capabilities and allows you to integrat with readily available off-the-shelf tools (such as the ELK stack) that use Elasticsearch for data analytics and monitoring. Once you have reindexed on the new ES schema, you can enable Swarm's new atime (access time) feature, which allows you track content usage and determine candidates for tiering to cold storage. 

• Hardware reporting and tools — This new generation of the Swarm UI builds out deep support for hardware management, whether physical or virtual machines. Drilling down to a given machine within Cluster > Hardware, you have new tabs for troubleshooting the chassis: Driver message (Dmesg), Hardware info (Hwinfo), Memory usage, and Statistics detail reports (health processor, Swarm communications, and more). The Advanced tab allows you to dynamically change machine-level logging levels and also work with Swarm's management API, both through a hands-on HAL browser and a Swagger visualizer.
  
  

  
  Elasticsearch clusters also have advanced, detailed troubleshooting help through the Elasticsearch Reports, which span nodes, thread pools, indices, and shards:
  
  

• UI enhancements: rolling reboots, UX improvements — Both the Storage and Content UIs have extensive UX improvements for storage cluster administration and content management. A highly requested feature of an orchestrated, unattended rolling reboot of the storage cluster has been included.
  
  

• SwarmNFS update for performance and scaling — The SwarmNFS 2.1 release provides greater single-client performance, security, and overall scaling, supporting parallel ingest of 3 PB+ per month/per instance without special hardware.

• S3 compatibility tracking — Content Gateway continues to keep pace with the evolving S3 protocol changes to maintain best-in-class compatibility with applications written for the AWS S3 protocol.

• FileFly 3 — FileFly offloads Windows and NetApp data to all major clouds and writes to multiple destinations simultaneously. 3.0 adds support for the latest Windows and NetApp versions and introduces a free Community Edition (up to 25TB).