How to configure the Ceph Grafana dashboards manually

I did not have Grafana dashboards that worked in my Ceph installation. Hence, I had to install Prometheus and Grafana manually and then configure the dashboards to show up in my Ceph cluster dashboard screens.

NODE

Setting up prometheus exporter in ceph.

sudo ceph mgr module enable prometheus
sudo ceph config set mgr mgr/prometheus/server_addr 0.0.0.0
sudo ceph config set mgr mgr/prometheus/server_port 9283

Installing node exporter for operating system statistics.

wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
tar xvf node_exporter-1.2.2.linux-amd64.tar.gz
sudo cp node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin/

Prepare user before installing prometheus.

sudo apt-get install gosu
sudo useradd -M prometheus
sudo usermod -L prometheus

Starting the node exporter and setting premissions correctly.

sudo chown prometheus:prometheus /usr/local/bin/node_exporter
sudo vi /etc/init.d/node_exporter
sudo chmod +x /etc/init.d/node_exporter
sudo update-rc.d node_exporter defaults
sudo service node_exporter status
sudo vi /etc/default/node_exporter
sudo service node_exporter start
ps auxww | grep node_exporter
cat /var/log/prometheus/node_exporter.log

Enable the grafana views without ssl in Ceph.

sudo ceph dashboard set-grafana-api-ssl-verify False
sudo ceph dashboard set-grafana-api-url http://node1:3000?orgId=1

GRAFANA

Installing Grafana.

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_8.1.2_arm64.deb
sudo dpkg -i grafana-enterprise_8.1.2_arm64.deb
sudo vi /etc/grafana/grafana.ini
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server

Reset the admin password for Grafana.

sudo grafana-cli --homepath "/usr/share/grafana" admin reset-admin-password admin

PROMETHEUS

Prepare user for Prometheus installation.

sudo apt-get install gosu
sudo useradd -M prometheus
sudo usermod -L prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Download and unpack Prometheus.

wget https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-arm64.tar.gz
tar xvf prometheus-2.29.2.linux-arm64.tar.gz

Copy binaries to a path location on your system.

sudo cp prometheus-2.29.2.linux-arm64/prometheus /usr/local/bin/
sudo cp prometheus-2.29.2.linux-arm64/promtool /usr/local/bin/
sudo cp -r prometheus-2.29.2.linux-arm64/consoles /etc/prometheus
sudo cp -r prometheus-2.29.2.linux-arm64/console_libraries /etc/prometheus
sudo vi /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus/

Preparing service and starting.

sudo vi /etc/init.d/prometheus
sudo chmod +x /etc/init.d/prometheus
sudo update-rc.d prometheus defaults
sudo service prometheus status
sudo vi /etc/default/prometheus
sudo service prometheus start
ps auxww | grep prometheus
cat /var/log/prometheus/prometheus.log

Adding some plugins in Grafana required for Ceph statistics.

sudo grafana-cli plugins install vonage-status-panel
sudo grafana-cli plugins install grafana-piechart-panel

Preparing directory for Ceph dashboards.

sudo mkdir -p /etc/grafana/dashboards/ceph-dashboard/
cd /etc/grafana/dashboards/ceph-dashboard/

Download all the json files defining different views in Ceph.

sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/ceph-cluster.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/cephfs-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/host-details.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/hosts-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/osd-device-details.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/osds-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/pool-detail.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/pool-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/radosgw-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/rbd-overview.json

Preparing the dashboards in Grafana, this can be done by importing the views in the GUI.

sudo mkdir -p /etc/grafana/provisioning/dashboards
sudo vi ceph-dashboard.yml

Restart to enable.

sudo /bin/systemctl restart grafana-server
sudo tail -f /var/log/grafana/grafana.log