How to monitor a Ceph cluster

In this article I will show a few examples on how to monitor your ceph cluster. I will be using following monitoring tools:

  • Ceph tools
  • Ceph dashboard
  • Prometheus
  • ELK

Ceph tools

Ceph monitoring tools are easiest and the fastest in terms of getting an overview of your cluster health. You will not need additional modules to get you started. I will show some useful commands that will help you along with ceph.

First check version of ceph cluster, execute the following:

> ceph -v

Console output:

ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)

This is my current ceph version.

To find out ceph cluster status execute the following command:

> ceph -s

Console output:

  cluster:
    id:     15e7ff89-5ebc-49c9-b154-8e893b740109
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ip-10-0-0-226.eu-west-1.compute.internal,ip-10-0-1-53.eu-west-1.compute.internal,ip-10-0-1-120.eu-west-1.compute.internal
    mgr: ip-10-0-0-226.eu-west-1.compute.internal(active), standbys: ip-10-0-1-120.eu-west-1.compute.internal
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   1 pools, 128 pgs
    objects: 0 objects, 0 bytes
    usage:   323 MB used, 1184 GB / 1184 GB avail
    pgs:     128 active+clean

So, from cluster section we know id and health is HEALTH_OK which is awesome. From services section we can see the list of services which are running in our ceph cluster: mon, mgr and osd. We have 3 Monitors, 2 MGR and 3 OSD which is up and in. From data section we get information about pools, objects, usage and pgs

To monitor the ceph cluster and see all changes in real time, execute the following:

>ceph -w

Console output:

cluster:
    id:     15e7ff89-5ebc-49c9-b154-8e893b740109
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ip-10-0-0-226.eu-west-1.compute.internal,ip-10-0-1-53.eu-west-1.compute.internal,ip-10-0-1-120.eu-west-1.compute.internal
    mgr: ip-10-0-0-226.eu-west-1.compute.internal(active), standbys: ip-10-0-1-120.eu-west-1.compute.internal
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   1 pools, 128 pgs
    objects: 0 objects, 0 bytes
    usage:   323 MB used, 1184 GB / 1184 GB avail
    pgs:     128 active+clean
 

...
2018-06-26 08:28:15.185452 mon.ip-10-0-1-53.eu-west-1.compute.internal [WRN] Health check failed: 1 osds down (OSD_DOWN)
2018-06-26 08:28:15.185482 mon.ip-10-0-1-53.eu-west-1.compute.internal [WRN] Health check failed: 3 hosts (1 osds) down (OSD_HOST_DOWN)
...

So, we can plainly see what's going on with the cluster. For example: we see that 1 OSD went down.

To get OSD status, execute the following:

> ceph osd stat

Console output:

3 osds: 2 up, 2 in

So, just 2 OSD is up and in now.

To get OSD CRUSH tree, execute the following:

ceph osd tree

Console output:


ID  CLASS WEIGHT  TYPE NAME          STATUS REWEIGHT PRI-AFF 
 -1       1.17000 root default                                           
-13       0.39000     host ip-10-0-0-226                                 
  2   ssd 0.39000         osd.2                    down        0 1.00000 
-11             0     host ip-10-0-0-226.eu-west-1.compute.internal      
 -9       0.39000     host ip-10-0-1-120                                 
  1   ssd 0.39000         osd.1                      up  1.00000 1.00000 
 -5             0     host ip-10-0-1-120.eu-west-1.compute.internal      
 -7       0.39000     host ip-10-0-1-53                                  
  0   ssd 0.39000         osd.0                      up  1.00000 1.00000 
 -2             0     host ip-10-0-1-53.eu-west-1.compute.internal   
 bgfgh

To just check cluster health, execute the following:

> ceph health

Console output:

HEALTH_OK

To check utilization of pools and cluster, execute the following:

> ceph df

Console output:

GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED 
    789G      789G         216M          0.03 
POOLS:
    NAME     ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd      1         0         0          562G           0 

From GLOBAL section we get information about the amount of storage your cluster uses for your data.

From POOLS section we get information about pools and the notional usage of each pool.

Ceph dashboard

The ceph dashboard has very good WebUI with convenient functionality. It's easy to enable and use.

Ceph Luminous has ready to use dashboard module in the ceph-mgr package.
All that's left for you to do is to enable it.

To enable ceph dashboard, execute the following:

> ceph mgr module enable dashboard

or use ceph.conf

[mon]
        mgr initial modules = dashboard

or kubernetes helm chart, set next values to values.yaml

ceph_mgr_enabled_modules:
  - dashboard

The ceph dashboard runs on port 7000 by default. Browse it by http://< active mgr host >:7000/.

ceph-dashdoard-general-2

To get list of enabled modules, execute the following:

> ceph mgr module ls

Console output:

{
    "enabled_modules": [
        "dashboard",
        "prometheus",
        "restful",
        "status"
    ],
    "disabled_modules": [
        "balancer",
        "influx",
        "localpool",
        "selftest",
        "zabbix"
    ]
}

The dashboard gives you a lot information about ceph cluster. You can get detailed information about OSDs, their status, read and write bandwidth, read and write ops. As well as information about pools, cephFS and etc...

ceph-dashdoard-osd

Prometheus

Prometheus is a open-source software ecosystem for monitoring and alerting, with focus on reliability and simplicity. Using prometheus you will monitor metrics over period of time. What is very convenient. In our case, I will be collecting ceph metrics.
To get metrics from ceph we need to use ceph exporter, which takes metrics from ceph and presents them to prometheus.
Ceph has ready to use prometheus module in the ceph-mgr package. By default the prometheus module accepts HTTP requests on port 9283. You just need to enable it.

To enable prometheus module, execute the following:

> ceph mgr module enable prometheus

or use ceph.conf

[mon]
        mgr initial modules = prometheus

or kubernetes helm chart, set next values to values.yaml

ceph_mgr_enabled_modules:
  - prometheus

Configure prometheus to receive ceph metrics start with adding following configuration:

    scrape_configs:
    
      - job_name: ceph-built-in-exporter
        static_configs:
          - targets:
            - < active mgr host >:9283

Prometheus config example:

 prometheus.yml:
   global:
     scrape_interval: 10s
     scrape_timeout: 10s
     evaluation_interval: 10s

   rule_files:
     - /etc/config/rules
     - /etc/config/alerts

   scrape_configs:

     - job_name: ceph-built-in-exporter
       static_configs:
         - targets:
           - < active mgr host >:9283

We got ceph-built-in-exporter connected and prometheus is getting metrics from ceph.
Screen-Shot-2018-06-26-at-14.49.42

To see which metrics ceph sends to prometheus and get explanation about each metric. Browse to http://< active mgr host >:9283/metrics

Example output:

# HELP ceph_osd_op_out_bytes Client operations total read size
# TYPE ceph_osd_op_out_bytes counter
ceph_osd_op_out_bytes{ceph_daemon="osd.1"} 643856.0
ceph_osd_op_out_bytes{ceph_daemon="osd.0"} 532622.0
# HELP ceph_pg_incomplete PG incomplete
# TYPE ceph_pg_incomplete gauge
ceph_pg_incomplete 0.0
# HELP ceph_paxos_commit Commits
# TYPE ceph_paxos_commit counter
ceph_paxos_commit{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 10735.0
ceph_paxos_commit{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 12578.0
ceph_paxos_commit{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 12607.0
# HELP ceph_paxos_collect_uncommitted Uncommitted values in started and handled collects
# TYPE ceph_paxos_collect_uncommitted counter
ceph_paxos_collect_uncommitted{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 0.0
ceph_paxos_collect_uncommitted{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 0.0
ceph_paxos_collect_uncommitted{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 0.0
# HELP ceph_paxos_store_state_keys Keys in transaction in stored state
# TYPE ceph_paxos_store_state_keys counter
ceph_paxos_store_state_keys{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 0.0
ceph_paxos_store_state_keys{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 96332.0
ceph_paxos_store_state_keys{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 97015.0
# HELP ceph_osd_in OSD status in
# TYPE ceph_osd_in untyped
ceph_osd_in{ceph_daemon="osd.1"} 1.0
ceph_osd_in{ceph_daemon="osd.2"} 0.0
ceph_osd_in{ceph_daemon="osd.0"} 1.0
# HELP ceph_paxos_accept_timeout Accept timeouts
# TYPE ceph_paxos_accept_timeout counter
ceph_paxos_accept_timeout{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 0.0
ceph_paxos_accept_timeout{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 0.0
ceph_paxos_accept_timeout{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 0.0
# HELP ceph_paxos_refresh Refreshes
# TYPE ceph_paxos_refresh counter
ceph_paxos_refresh{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 10736.0
ceph_paxos_refresh{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 12579.0
ceph_paxos_refresh{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 12607.0
# HELP ceph_pg_degraded PG degraded
# TYPE ceph_pg_degraded gauge
ceph_pg_degraded 0.0
# HELP ceph_mon_num_sessions Open sessions
# TYPE ceph_mon_num_sessions gauge
ceph_mon_num_sessions{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 3.0
ceph_mon_num_sessions{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 3.0
ceph_mon_num_sessions{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 9.0
# HELP ceph_mon_num_elections Elections participated in
# TYPE ceph_mon_num_elections counter
ceph_mon_num_elections{ceph_daemon="mon.ip-10-0-0-226.eu-west-1.compute.internal"} 6.0
ceph_mon_num_elections{ceph_daemon="mon.ip-10-0-1-53.eu-west-1.compute.internal"} 14.0
ceph_mon_num_elections{ceph_daemon="mon.ip-10-0-1-120.eu-west-1.compute.internal"} 20.0
...

We can visualise our metrics and event trigger alert when we need it to. For instance: graf shows us cluster health status ceph_health_status.

Screen-Shot-2018-06-26-at-14.58.22

For graf to show latency of OSD's. We use metric ceph_osd_apply_latency_ms and ceph_osd_commit_latency_ms

Screen-Shot-2018-06-26-at-15.25.36

Prometheus has other exporters for ceph. With it's own set of metrics.
For example: I use https://hub.docker.com/r/primeroz/ceph_exporter/ this exporter seems to work ceph Luminous with out any bugs. Digitalocean suggests another good exporter: https://github.com/digitalocean/ceph_exporter/
The whole list of prometheus exporters: https://prometheus.io/docs/instrumenting/exporters/

External exporters give us different set of metrics. For example: with external exporters we can monitor disk io, ceph_client_io_read_ops and ceph_client_io_write_ops
Screen-Shot-2018-06-26-at-15.24.23

ELK

ELK is a set of open source projects: Elasticsearch, Logstash, and Kibana.
I recommend you to use ELK to store your ceph cluster logs. ELK helps you to store, analyze and show logs in convenient WebUI. ELK is very flexible and has a lot of filters. You can store ceph logs during some period of time and then analyze them. It gives you full picture of what is going on in a ceph cluster.
To get more information about ELK visit official website: https://www.elastic.co/elk-stack