How to Clear Emergency Shutdown Flags on Zenoss

When you encounter "404 Not Found", "424 Failed Dependency" and "502 Bad Gateway" tsdb errors in Grafana, this means that a Service died.

In this documentation, I am using Zenoss as an example of service.
So, "ServiceID" = Zenoss

How can this issue be solved?
1. We need to clear the emergency-shutdown flags on Zenoss
2. Restart Zenoss

The following steps will guide you through the process in solving this issue and what statuses should be checked:

1. Log in to the Control Center master host as root.
i.e.
root@mon1:~()#

2. Use this command to check the status of Zenoss:

root@mon1:~()# serviced healthcheck

All service status should be "passed".
If status is "failed", this needs to clear the emergency flag.

3. Use this command to clear the emergency-shutdown flag for a service (Zenoss):

root@mon1:~()# serviced service clear-emergency Zenoss

Sample output:
Cleared emergency status for 45 services

4. This command will check services for the emergency shutdown flag:

root@mon1:~()# for s in $(serviced service list --show-fields Zenoss); \
  do serviced service list $s | grep EmergencyShutdown\"; done

Sample output:
"EmergencyShutdown": true,

Note:
"True" indicates an emergency shutdown.
Therefore, you need to restart Zenoss.

Sample output:
"EmergencyShutdown": false,

Note:
"False" indicates that the flags have been cleared.
Therefore, there is no need to restart Zenoss.

5. Use this command to restart Zenoss:

root@mon1:~()# serviced service start Zenoss

Sample output:
Scheduled 41 service(s) to start

Then, check your Grafana if data are showing back again.