Skip to main content

Posts

Showing posts from February, 2019

How to Clear Emergency Shutdown Flags on Zenoss

When you encounter "404 Not Found", "424 Failed Dependency" and "502 Bad Gateway" tsdb errors in Grafana, this means that a Service died. In this documentation, I am using Zenoss as an example of service. So, "ServiceID" = Zenoss How can this issue be solved? 1. We need to clear the emergency-shutdown flags on Zenoss 2. Restart Zenoss The following steps will guide you through the process in solving this issue and what statuses should be checked: 1. Log in to the Control Center master host as root. i.e. root@mon1:~()# 2. Use this command to check the status of Zenoss: root@mon1:~()# serviced healthcheck All service status should be "passed". If status is "failed", this needs to clear the emergency flag. 3. Use this command to clear the emergency-shutdown flag for a service (Zenoss): root@mon1:~()# serviced service clear-emergency Zenoss Sample output: Cleared emergency status for 45 services 4. This command will check services

How to Setup Alerts in Grafana?

Here's a sample process on how to create an alert on Grafana: 1. Go to your desired graph where you wanted to add an alert. 2. Click Edit 3. Go to the Alert tab. 4. Click " Create Alert " 5. In the " Alert Config " section, change the following values: a. Name - this will appear on the alert notification b. Evaluate every - change this to 5s. This means that the graph will update its data every 5 seconds. c. For - change this to 0m (zero minutes) 6. Under Conditions , set the following: a. When - click on "avg() to change to min(). There are also other options to choose from – depending on what you need. b. Of - change to query(A,1h,now) Query parameters: 1st parameter is the metric. 2nd parameter is the FROM value. 3rd parameter is the TO value. This is the condition being set to validate metrics data. c. Is Above - change this to " IS BELOW " and put the value " 0.01 ". There are a