cmaven

Worknotes

:bulb: How to get health, OSD, and capacity metrics from a Ceph cluster built on bare metal outside Kubernetes into an already-running Prometheus as time series. Identifying details are replaced with documentation ranges and pseudonyms — storage nodes node-s1/s2/s3 = 192.0.2.11/12/13, Ceph public network 192.0.2.0/24, OSD replication network 198.51.100.0/24, admin node node-a1, SSH account user.
Audience: operators who attach Ceph to Kubernetes in Rook external mode and already run a monitoring stack

There are two paths. One registers the endpoint exposed by the Ceph manager as a static Prometheus target. The other tells Rook the external manager addresses and lets it create a ServiceMonitor. Exposing the metrics is identical in both; only the way Prometheus connects to them differs.

graph LR
    subgraph BM["Bare-metal Ceph (outside Kubernetes)"]
        MGR["active mgr
prometheus module
:9283/metrics"]
        SB["standby mgr
:9283 → 503"]
    end
    subgraph K8S["Kubernetes (monitoring)"]
        A["Method A
additionalScrapeConfigs
static_configs"]
        B["Method B
rook-ceph-mgr-external
Service + ServiceMonitor"]
        P["Prometheus"]
    end
    MGR --> A
    SB --> A
    MGR --> B
    SB --> B
    A --> P
    B --> P

    style MGR fill:#e8f5e9,stroke:#2e7d32
    style SB fill:#eceff1,stroke:#607d8b
    style A fill:#fff3e0,stroke:#e65100
    style B fill:#e3f2fd,stroke:#1565c0
    style P fill:#fce4ec,stroke:#c62828

Nothing here is hard to undo. It is only a manager module activation plus a scrape config, and [08] lists the rollback commands. That said, it does open port 9283 on the storage network, so check the firewall policy first.

[00] Three Terms

[00-1] mgr (Ceph Manager)

The management daemon of a Ceph cluster. If mon handles consensus on cluster state, mgr collects that state and presents it outward. The dashboard, the orchestrator, and metrics exposure all run as modules on top of mgr. It is made redundant with one active and one or more standby instances.

[00-2] prometheus module

A module bundled with mgr. Once enabled, the active mgr serves Prometheus-format metrics at :9283/metrics. No separate exporter is needed, and mon, OSD, pool, and PG metrics all come from a single endpoint.

[00-3] Rook external mode

A mode where the Rook operator does not own Ceph and attaches only as a client. It creates no OSDs; it takes the credentials and mon addresses of an existing Ceph cluster and provides dynamic provisioning through the CSI driver. In this mode Ceph daemons are not Kubernetes pods, so there is no target for Prometheus Operator to discover in the first place.

[01] Current State of the System

Ceph runs on three dedicated bare-metal nodes deployed with cephadm: Quincy 17.2.8, a 3-mon quorum, one OSD per node. The client network 192.0.2.0/24 and the OSD replication network 198.51.100.0/24 are separated. The pools in use are general (general-purpose RBD), vms, volumes, backups, and general-multi-attach-{data,metadata} for CephFS.

Here is the ceph -s output recorded at build time.



1
2
3
4
5
6
7
8
9
10
11
12



  cluster:
    id:     9b3c1d7a-...
    health: HEALTH_OK
  services:
    mon: 3 daemons, quorum node-s1,node-s2,node-s3
    mgr: node-s1.xxxx(active), standbys: 1
    mds: 1/1 daemons up, 2 standby
    osd: 3 osds: 3 up, 3 in
  data:
    pools:   7 pools, 193 pgs
    usage:   ... 49 TiB avail
    pgs:     193 active+clean

Kubernetes is attached to this cluster in Rook external mode. A CephCluster sits in the rook-ceph-external namespace in Connected state, StorageClass general (RBD) is the default, and general-multi-attach (CephFS) serves RWX.

The monitoring stack is already running. kube-prometheus-stack is deployed on the admin node node-a1, and Prometheus uses a 100Gi PVC. Grafana was installed the operator way with six dashboards provisioned. Exporters include node-exporter and kube-state-metrics, plus OpenStack, MySQL, memcached, libvirtd, and blackbox.

[01-1] What Is Missing

Ceph. None of the six dashboards has a Ceph panel, and Ceph is absent from the Prometheus target list. There are three ways to look at Ceph state, and all three require a person to look at that exact moment.

Path Command or address Limitation
CLI on a storage node ssh user@192.0.2.11 'sudo ceph -s' A value appears only when someone types it
Ceph Dashboard https://192.0.2.11:8443 Short retention, and alert integration is separate
CR lookup from Kubernetes kubectl -n rook-ceph-external get cephcluster Just the one-line summary the operator refreshes

Without history you cannot answer “since when,” and without alerts a missing OSD goes unnoticed until the next manual check.

[01-2] The Dependency Direction

The 100Gi PVC of Prometheus lives on StorageClass general — that is, on this very Ceph cluster. When Ceph wobbles, the Prometheus that is supposed to record it is affected first.

Adding metric collection does not remove this circular dependency. What it does give you is a record up to the moment of failure, and if the alert delivery path is routed outside Alertmanager (mail, messenger), post-mortem tracing stays possible.

[02] Work Required

Three steps.

  1. Expose: have Ceph serve metrics over HTTP. Enabling the mgr prometheus module is all it takes. This is the shared prerequisite for both methods.
  2. Connect: make Prometheus treat that endpoint as a scrape target. This is where Method A and Method B diverge.
  3. Use: add a Grafana dashboard and PrometheusRule alerts.

The connect step diverges because of what an external Ceph is. Prometheus Operator discovers targets based on Kubernetes Services. Ceph daemons are outside Kubernetes, so there is no Service to point at. Method A writes the static addresses directly without any Service; Method B has Rook create a Service that points at the external addresses.

Network reachability is another thing to verify up front. The node where the Prometheus pod runs must be able to reach 9283 on 192.0.2.0/24. Do not skip this check in environments where the storage-network route differs per node. Most cases of “targets registered but all down” turn out to be routing.

[03] Shared Preparation — the mgr prometheus Module

[03-1] Enabling the Module

It does not have to be the node holding the active mgr; running it on any node applies to the whole cluster.



1
2



ssh user@192.0.2.11 'sudo ceph mgr module enable prometheus'
ssh user@192.0.2.11 'sudo ceph mgr services'

Expected output:



1
2
3
4



{
    "dashboard": "https://192.0.2.11:8443/",
    "prometheus": "http://192.0.2.11:9283/"
}

A prometheus entry means the active mgr has opened the endpoint. Note that this output shows only the active mgr’s address.

[03-2] Pinning Behavior — What to Do About the standby mgr



1
2
3
4
5



ssh user@192.0.2.11 'sudo ceph config set mgr mgr/prometheus/server_addr 0.0.0.0
sudo ceph config set mgr mgr/prometheus/server_port 9283
sudo ceph config set mgr mgr/prometheus/scrape_interval 30
sudo ceph config set mgr mgr/prometheus/standby_behaviour error
sudo ceph config set mgr mgr/prometheus/standby_error_status_code 503'

The first three lines mostly make defaults explicit. The last two actually change behavior, so here is why.

By default standby_behaviour makes a standby mgr redirect incoming requests to the active mgr. Prometheus follows redirects, so if standby addresses are in the target list you receive the active mgr’s metrics a second time. The same time series arrives twice, differing only in the instance label, and sum()-style aggregations inflate accordingly.

Setting it to error makes the standby return 503 instead of redirecting, and that target stays down in Prometheus. Between inflated aggregation and a down marker, the latter is preferable. When the mgr fails over, the up target moves automatically.

[03-3] Verifying Exposure



1



ssh user@192.0.2.11 'curl -s http://192.0.2.11:9283/metrics | head -6; echo ---; curl -s http://192.0.2.11:9283/metrics | grep -c "^ceph_"'

Expected output:



1
2
3
4
5
6
7
8



# HELP ceph_health_status Cluster health status
# TYPE ceph_health_status untyped
ceph_health_status 0.0
# HELP ceph_mon_quorum_status Monitors in quorum
# TYPE ceph_mon_quorum_status gauge
ceph_mon_quorum_status{ceph_daemon="mon.node-s1"} 1.0
---
1842

For ceph_health_status, 0 is HEALTH_OK, 1 is HEALTH_WARN, 2 is HEALTH_ERR. The metric count scales with the number of pools and OSDs, so it varies per environment.

[03-4] Reachability from the Kubernetes Side

What matters is whether traffic leaves the node where Prometheus runs, so check from a pod.



1
2
3
4



kubectl -n monitoring run nettest --rm -i --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"role":"admin-node"}}}' \
  --image=curlimages/curl -- \
  curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 http://192.0.2.11:9283/metrics

Expected output:



1
2



200
pod "nettest" deleted

A 000 means routing or firewall. Resolve that before adding any scrape config.

[04] Method A — Registering Static Targets

Put a job in raw Prometheus format into the additionalScrapeConfigs of kube-prometheus-stack. It does not go through Rook, so it works with nothing but Ceph and Prometheus.

[04-1] Writing the Scrape Config



1
2
3
4
5
6
7
8
9
10



# ceph-scrape.yaml
- job_name: ceph-mgr
  scrape_interval: 30s
  static_configs:
    - targets:
        - 192.0.2.11:9283
        - 192.0.2.12:9283
        - 192.0.2.13:9283
      labels:
        cluster: onprem

All three nodes are listed because of mgr failover: whichever node the active mgr moves to, that target becomes up. Match the cluster label value to your existing dashboard variable. If it differs, Ceph metrics show up under a separate entry in the dashboard’s cluster selector.

[04-2] Creating the Secret and Applying via Helm



1
2



kubectl -n monitoring create secret generic additional-scrape-configs \
  --from-file=ceph-scrape.yaml --dry-run=client -o yaml | kubectl apply -f -





1
2
3
4
5
6



# addition to kube-prometheus-stack values
prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      name: additional-scrape-configs
      key: ceph-scrape.yaml





1
2



helm -n monitoring upgrade kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack --reuse-values -f values-ceph.yaml

The operator regenerates the Prometheus configuration and the pod restarts. Propagation takes tens of seconds.

[04-3] Verification

Use the Prometheus targets API. The NodePort is 31010, the one the existing stack already uses.



1
2



curl -s http://192.0.2.20:31010/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job=="ceph-mgr") | "\(.labels.instance)\t\(.health)\t\(.lastError)"'

Expected output:



1
2
3



192.0.2.11:9283	up
192.0.2.12:9283	down	server returned HTTP status 503 Service Unavailable
192.0.2.13:9283	down	server returned HTTP status 503 Service Unavailable
Result Reading
One up + two down Normal — the result of the standby behavior set in [03-2]
All three down Reachability problem; go back to [03-4]
All three up standby_behaviour did not take effect and you are double scraping

[04-4] Trade-offs

Pros: No dependency on the Rook version or the CephCluster CR schema. It works as-is in environments where the only Ceph consumers are outside Kubernetes (for example OpenStack using RBD directly).

Cons: Addresses are hardcoded inside a Secret. Adding storage nodes or changing IPs means editing this file by hand. Unless the Secret creation step is written into your rebuild procedure, it is easy to forget.

[05] Method B — Letting Rook Own the ServiceMonitor

Tell the CephCluster CR the external mgr addresses, and the Rook operator creates a Service and Endpoints pointing at them, along with a ServiceMonitor. From there Prometheus Operator discovers the target the way it always does.

[05-1] Checking the Current Setting and Operator Version

The external monitoring fields are handled by the Rook operator, so the first step is checking whether the version in use handles them.



1
2



kubectl -n rook-ceph get deploy rook-ceph-operator -o jsonpath='{..image}{"\n"}'
kubectl -n rook-ceph-external get cephcluster rook-ceph-external -o jsonpath='{.spec.monitoring}{"\n"}'

Expected output:



1
2



rook/ceph:v1.13.7
{"enabled":false}

[05-2] Editing the Rook values

The CephCluster is managed by a Helm release, so put this in values. Patching the CR directly gets reverted on the next helm upgrade.



1
2
3
4
5
6
7
8
9



# addition to rook-values-external.yaml
cephClusterSpec:
  monitoring:
    enabled: true
    externalMgrEndpoints:
      - ip: 192.0.2.11
      - ip: 192.0.2.12
      - ip: 192.0.2.13
    externalMgrPrometheusPort: 9283





1
2



helm -n rook-ceph-external upgrade rook-ceph-cluster rook-release/rook-ceph-cluster \
  --set operatorNamespace=rook-ceph -f rook-values-external.yaml

[05-3] Loosening the ServiceMonitor Selector

By default kube-prometheus-stack picks up only ServiceMonitors carrying its own release label. The ServiceMonitor Rook creates does not have that label, so without loosening the selector you end up in the state where the resources exist but nothing is scraped.



1
2
3
4
5



# addition to kube-prometheus-stack values
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorNamespaceSelector: {}

[05-4] Verification



1



kubectl -n rook-ceph-external get svc,endpoints,servicemonitor | grep mgr-external

Expected output:



1
2
3



service/rook-ceph-mgr-external     ClusterIP   <ClusterIP>   <none>   9283/TCP   45s
endpoints/rook-ceph-mgr-external               192.0.2.11:9283,192.0.2.12:9283,192.0.2.13:9283   45s
servicemonitor.monitoring.coreos.com/rook-ceph-mgr-external                                      45s

If all three addresses are in Endpoints, Rook applied the values as written. Next, check whether Prometheus actually picked up that ServiceMonitor.



1
2



curl -s http://192.0.2.20:31010/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job|test("mgr-external")) | "\(.labels.instance)\t\(.health)"'

Expected output:



1
2
3



192.0.2.11:9283	up
192.0.2.12:9283	down
192.0.2.13:9283	down

Empty output means the ServiceMonitor selector is still too narrow. Recheck the setting in [05-3] and whether the Prometheus pod restarted.

[05-5] Trade-offs

Pros: The addresses live in one place, the Rook values. Since the Rook external connection is already managed by Helm, keeping that single values file during a rebuild restores metric collection along with it. Labeling also follows the normal Prometheus Operator path.

Cons: It depends on the Rook operator version. The script that extracts Ceph credentials in external mode also takes a --monitoring-endpoint argument, so if that value and the CR’s externalMgrEndpoints disagree, it is hard to tell which one is actually in effect. It is worth confirming whether the value passed at initial connection was a mon address or a mgr address. Even on the same node, the two mean different things.

[06] How to Choose

The two methods are not mutually exclusive, but enabling both means receiving the same metrics twice. Pick one.

Situation Pick
Ceph consumers are not only Kubernetes (OpenStack etc. using RBD directly) A
The Rook external connection is already config-managed through Helm values B
The Rook operator version must stay pinned for a while A
Storage node expansion is planned B
You want the monitoring stack and the storage integration to change independently A

When it is a toss-up, start with A. It is one Secret and four lines of values, so it is easy to revert, and when you later move to B the metric names and dashboards carry over unchanged. The only thing that changes is the job label.

[07] Dashboards and Alerts

[07-1] Metrics Worth Watching

Metric Meaning
ceph_health_status 0=OK, 1=WARN, 2=ERR
ceph_mon_quorum_status Per-mon quorum participation
ceph_osd_up, ceph_osd_in Per-OSD running / data-accepting state
ceph_cluster_total_bytes, ceph_cluster_total_used_raw_bytes Total capacity and raw usage
ceph_osd_apply_latency_ms, ceph_osd_commit_latency_ms OSD latency

Query one to confirm.



1
2



curl -sG http://192.0.2.20:31010/api/v1/query \
  --data-urlencode 'query=ceph_health_status' | jq -r '.data.result[] | "\(.metric.instance)\t\(.value[1])"'

Expected output:



1



192.0.2.11:9283	0

[07-2] PrometheusRule Alerts

For the same reason as the ServiceMonitor, the release label is required. Without it the rule file is never loaded into Prometheus.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26



apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ceph-external
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: ceph
      rules:
        - alert: CephHealthError
          expr: ceph_health_status == 2
          for: 5m
        - alert: CephHealthWarn
          expr: ceph_health_status == 1
          for: 30m
        - alert: CephOsdDown
          expr: ceph_osd_up == 0
          for: 5m
        - alert: CephClusterNearFull
          expr: ceph_cluster_total_used_raw_bytes / ceph_cluster_total_bytes > 0.85
          for: 30m
        - alert: CephMetricsMissing
          expr: absent(ceph_health_status)
          for: 10m

The last rule is the one that matters in practice. Because standby_behaviour: error leaves some targets permanently down, the common up == 0 condition fires constantly in this setup. To catch metrics not arriving at all, check for the existence of the metric itself with absent().

CephHealthWarn uses a 30-minute for because harmless HEALTH_WARN states, such as pool autoscale warnings, can flicker briefly. Start generous, watch for a week which warnings actually appear, then tighten.

[07-3] Grafana Dashboard

Register it the same way as the existing dashboards: build a ConfigMap from the JSON and reference it from a GrafanaDashboard CR. Ceph community dashboards tend to have many panels and therefore large JSON, so for files exceeding the 256KB last-applied annotation limit of kubectl apply, register them with kubectl create configmap instead.

[08] Rollback

Method A:



1
2



kubectl -n monitoring delete secret additional-scrape-configs
# remove the additionalScrapeConfigs block from values, then helm upgrade

Method B:



1
2
3



cephClusterSpec:
  monitoring:
    enabled: false

Shared:



1



ssh user@192.0.2.11 'sudo ceph mgr module disable prometheus'

Disabling the module closes the 9283 endpoint. Cluster operation and data are unaffected.

[09] Summary

  • Exposing metrics is done entirely by the mgr prometheus module. No separate exporter to install.
  • The part that diverges is how Prometheus connects to that endpoint. A static scrape config stands on its own regardless of Rook; Rook external monitoring gathers the configuration into one Helm values file.
  • Decide how to handle the standby mgr first. Leaving the redirect in place means metrics arrive twice; switching to error means some targets are permanently down. Match your alert conditions to that choice.
  • Prometheus Operator’s selectors bite once for the ServiceMonitor and once for the PrometheusRule. When the resources exist but nothing happens, start with labels and selectors.
  • If the monitoring stack’s own storage sits on Ceph, that dependency remains even after adding metrics. At minimum, route the alert delivery path outside Ceph.

References

Environment: Ceph Quincy 17.2.8 (cephadm, 3 bare-metal nodes) / Rook v1.13.x external mode / kube-prometheus-stack. Written as of 2026-08-20; the schema of Rook’s monitoring field varies by operator version.

You may also enjoy