System: Loss of functionality when publisher node goes down in multi-node deployments

In the multi-node deployment scenario, all scheduler functions run on the publisher node. Consequently, if the publisher node goes down, jobs that are scheduled to execute during the downtime period do not run. When the publisher comes back up and rejoins the cluster, jobs that are scheduled to run after the restart of the publisher node get executed at the scheduled time, but the jobs scheduled to execute between the crash time and restart time are lost forever.

In addition, when the publisher is down, new schedules can be created by clients logged on to the subscriber nodes, but existing schedules cannot be edited.

Also, clicking the Run Now button on any screen of CUIC when the publisher node is down will result in an error.

Cluster Membership Changes When a new CUIC node is added to the cluster, or an existing node is required to be removed from the cluster (say, for maintenance activities), cluster membership change CLI commands need to be first executed on the publisher in order to prepare the CUIC cluster for the addition/deletion of a cluster node. If the publisher node is down, it is not possible to add/delete nodes from the CUIC cluster.