Swarm 7.5.3 Upgrade Doesn't Accept scsp.autoRecursiveDelete Parameter Change (CloudScaler)

In the course of upgrading a Swarm cluster to v7.5.3, you may find it necessary to make the following parameter change in order to accommodate requirements for running with CloudScaler 3.x:

scsp.autoRecursiveDelete = True (or "1")

Per the Swarm 7.5.3 release notes, this is a new parameter that was introduced to enforce proper function of domain and bucket deletes under CloudScaler:

When using Swarm 7.5.3 with CloudScaler 3.x, you must set the "scsp.autoRecursiveDelete=True" parameter or else domain and bucket deletes will fail.

It's very important that this parameter be set in the appropriate node configuration file (cluster.cfg, node.cfg) before Swarm 7.5.3 is started for the first time. This allows the desired value to be placed in the persistent settings stream (PSS) on startup. Specifically, if the value is set to "True" this will override the 7.5.3 default of setting the value to "False" on initial startup.

If this parameter requirement is overlooked in CloudScaler deployments, and Swarm 7.5.3 is started without it being set by the config to a non-default value, the default value of "False" will persist in spite of the parameter being set to something different in the configuration file(s). As a symptom of this, you will see Swarm log messages similar to the following:

10.20.30.85 <177>2015-09-23 18:35:06,873 CONFIG ANNOUNCE: READ NEW VALUE OF FALSE FOR SCSP.AUTORECURSIVEDELETE FROM CLUSTER SETTINGS STREAM

This may also be seen in conjunction with the following log message:

10.20.30.85 <179>2015-09-24 16:29:45,458 CONFIG ERROR: FAILED TO UPDATE PERSISTENT SETTINGS: HTTP UPDATE RETURNED 412

Once the persistent settings stream is in this state, the only way to set and persist the desired scsp.autoRecursiveDelete value of "True" or "1" is by use of SNMP. An 'snmpset' command must be issued to at least one of the nodes in the cluster in the following manner:

snmpset -v2c -c<snmp write password> <Swarm cluster node IP address> .1.3.6.1.4.1.24659.1.35.44 i 1

Note that there is no space between the "-c" flag and the SNMP write password to be used. The output for the above will look like the following on success:

SNMPv2-SMI::enterprises.24659.1.35.44 = INTEGER: 1

You should also see evidence in the cluster logs that this setting has taken hold as follows:

10.20.30.85 <177>2015-09-24 17:04:36,987 CONFIG ANNOUNCE: READ NEW VALUE OF TRUE FOR SCSP.AUTORECURSIVEDELETE FROM CLUSTER SETTINGS STREAM

At this point, your Swarm 7.5.3 cluster should have the proper scsp.autoRecursiveDelete value in place to support CloudScaler 3.x. Furthermore, this value will now be set in the PSS, so it will be maintained across node and/or cluster restarts.