208 lines
10 KiB
Markdown
208 lines
10 KiB
Markdown
[](https://circleci.com/gh/prometheus-community/smartctl_exporter)
|
||
[](https://quay.io/repository/prometheuscommunity/smartctl-exporter)
|
||
|
||
# smartctl_exporter
|
||
Export smartctl statistics to prometheus
|
||
|
||
Example output you can show in [EXAMPLE.md](EXAMPLE.md)
|
||
|
||
## Need more?
|
||
**If you need additional metrics - contact me :)**
|
||
|
||
**Create a feature request, describe the metric that you would like to have and
|
||
attach exported from smartctl json file**
|
||
|
||
# Requirements
|
||
`smartmontools` >= 7.0, because export to json [released in 7.0](https://www.smartmontools.org/browser/tags/RELEASE_7_0/smartmontools/NEWS#L11)
|
||
|
||
# Configuration
|
||
## Command line options
|
||
|
||
The exporter will scan the system for available devices if no `--smartctl.device`
|
||
flags are used.
|
||
|
||
```
|
||
usage: smartctl_exporter [<flags>]
|
||
|
||
Flags:
|
||
-h, --help Show context-sensitive help (also try --help-long and --help-man).
|
||
--smartctl.path="/usr/sbin/smartctl"
|
||
The path to the smartctl binary
|
||
--smartctl.interval=60s The interval between smartctl polls
|
||
--smartctl.rescan=10m The interval between rescanning for new/disappeared devices. If the interval is smaller than 1s no
|
||
rescanning takes place. If any devices are configured with smartctl.device also no rescanning takes
|
||
place.
|
||
--smartctl.device=SMARTCTL.DEVICE ...
|
||
The device to monitor (repeatable)
|
||
--smartctl.device-exclude=""
|
||
Regexp of devices to exclude from automatic scanning. (mutually exclusive to
|
||
device-include)
|
||
--smartctl.device-include=""
|
||
Regexp of devices to include in automatic scanning. (mutually exclusive to
|
||
device-exclude)
|
||
--web.telemetry-path="/metrics"
|
||
Path under which to expose metrics
|
||
--web.systemd-socket Use systemd socket activation listeners instead of port listeners (Linux only).
|
||
--web.listen-address=:9633 ...
|
||
Addresses on which to expose metrics and web interface. Repeatable for multiple
|
||
addresses.
|
||
--web.config.file="" [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.
|
||
--log.level=info Only log messages with the given severity or above. One of: [debug, info, warn,
|
||
error]
|
||
--log.format=logfmt Output format of log messages. One of: [logfmt, json]
|
||
--version Show application version.
|
||
```
|
||
|
||
## TLS and basic authentication
|
||
|
||
This exporter supports TLS and basic authentication.
|
||
|
||
To use TLS and/or basic authentication, you need to pass a configuration file
|
||
using the `--web.config.file` parameter. The format of the file is described
|
||
[in the exporter-toolkit repository](https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md).
|
||
|
||
## Example of running in Docker
|
||
|
||
Minimal functional `docker-compose.yml`:
|
||
```yaml
|
||
version: "3"
|
||
|
||
services:
|
||
smartctl-exporter:
|
||
image: prometheuscommunity/smartctl-exporter
|
||
privileged: true
|
||
user: root
|
||
ports:
|
||
- "9633:9633"
|
||
```
|
||
|
||
# Troubleshooting
|
||
## Troubleshooting data inconsistencies
|
||
`smartmon_exporter` uses the JSON output from `smartctl` to provide the data to
|
||
Prometheus. If the data is incorrect, look at the data from `smartctl` to
|
||
determine if the issue should be reported upstream to smartmontools or to this
|
||
repo. In general, the `smartctl_exporter` should not modify the data in flight.
|
||
If the data is missing from `smartctl`, it should not be in `smartctl_exporter`.
|
||
If the data from `smartctl` is incorrect, it should be reported upstream.
|
||
Requests for `smartctl_exporter` to "fix" incorrect data where `smartctl` is
|
||
reporting incorrect data will be closed. The grey area is when invalid or
|
||
missing data from smartctl is causing multiple invalid or incorrect data
|
||
in `smartctl_exporter`. This could happen if the data is used in a calculation
|
||
for other data. This will need to be researched on a case by case basis.
|
||
|
||
| - | smartctl valid | smartctl missing | smartctl invalid/incorrect |
|
||
|---------------------------|-----------------------------|-------------------------------------------------|----------------------------------|
|
||
| smartctl_exporter valid | all good | N/A | N/A |
|
||
| smartctl_exporter missing | issue for smartctl_exporter | report upstream to smartmontools | report upstream to smartmontools |
|
||
| smartctl_exporter invalid | issue for smartctl_exporter | issue for smartctl_exporter and report upstream | report upstream to smartmontools |
|
||
|
||
### smartctl output vs smartctl_exporter output
|
||
|
||
The S.M.A.R.T. attributes are mapped in
|
||
[smartctl.go](https://github.com/prometheus-community/smartctl_exporter/blob/master/smartctl.go).
|
||
Each function has a `prometheus.MustNewConstMetric` or similar function with the
|
||
first parameter being the metric name. Find the metric name in
|
||
[metrics.go](https://github.com/prometheus-community/smartctl_exporter/blob/master/metrics.go)
|
||
to see how the exporter displays the information. This may sound technical, but
|
||
it's crucial for understanding how data flows from `smartctl` to
|
||
`smartctl_exporter` to Prometheus.
|
||
|
||
If the data looks incorrect, check the
|
||
[Smartmontools Frequently Asked Questions (FAQ)](https://www.smartmontools.org/wiki/FAQ).
|
||
It's likely your question may already have an answer. If you still have
|
||
questions, open an [issue]().
|
||
|
||
## Gathering smartctl data
|
||
Follow these steps to gather smartctl data for troubleshooting purposes. If you
|
||
have unique drives/data/edge cases and would like to "donate" the data, open a
|
||
PR with the redacted JSON files.
|
||
|
||
1. Run `scripts/collect-smartctl-json.sh` to export all drives to a
|
||
`smartctl-data` directory (created in the current directory).
|
||
2. Run `scripts/redact_fake_json.py` to redact sensitive data.
|
||
3. Provide the JSON file for the drive in question.
|
||
|
||
```bash
|
||
cd scripts
|
||
./collect-smartctl-json.sh
|
||
./redact-fake-json.py smartctl-data/*.json
|
||
```
|
||
|
||
## Run smartctl_exporter using JSON data
|
||
The `smartctl_exporter` can be run using local JSON data. The device names are
|
||
pulled from actual devices in the machine while the data is redirected to the
|
||
`debug` directory. Save the JSON data in the `debug` directory using the actual
|
||
device names using a 1:1 ratio. If you have 3 devices, `sda`, `sdb` and `sdc`,
|
||
the `smartctl_exporter` will expect 3 files: `debug/sda.json`, `debug/sdb.json`
|
||
and `debug/sdc.json`.
|
||
|
||
Once the "fake devices" (JSON files) are in place, run the exporter passing the
|
||
hidden `--smartctl.fake-data` switch on the command line. The port is specified
|
||
to prevent conflicts with an existing `smartctl_exporter` on the default port.
|
||
|
||
```bash
|
||
smartctl_exporter --web.listen-address 127.0.0.1:19633 --smartctl.fake-data
|
||
```
|
||
|
||
# FAQ
|
||
## How do I run `smartctl_exporter` against a JSON file?
|
||
|
||
If you're helping someone else, request the output of the `smartctl` command
|
||
above. Feed this into the `smartctl_exporter` using the
|
||
hidden `--smartctl.fake-data` flag. If a `smartctl_exporter` is already running,
|
||
use a different port; in this case, it's `19633`. Run `collect_fake_json.sh`
|
||
first to collect the JSON files for **your** devices. Copy the requested JSON
|
||
file into one of the fake files. After starting the exporter, you can query it
|
||
to see the data generated.
|
||
|
||
```bash
|
||
# Dump the JSON files for your devices into debug/
|
||
./collect_fake_json.sh
|
||
|
||
# copy the test JSON into one of the files in debug/
|
||
cp extracted-from-above-sda.json debug/sda.json
|
||
|
||
# Make sure you have the latest version
|
||
go build
|
||
# Use a different port in case smartctl_exporter is already running
|
||
sudo ./smartctl_exporter --web.listen-address=127.0.0.1:19633 --log.level=debug --smartctl.fake-data
|
||
|
||
# Use curl with grep
|
||
curl --silent 127.0.0.1:19633/metrics | grep -i nvme
|
||
# Or xh with ripgrep
|
||
xh --body :19633/metrics | rg nvme
|
||
```
|
||
|
||
## Why is root required? Can't I add a user to the "disk" group?
|
||
|
||
A blogger had the same question and opened a ticket on smartmontools. This is
|
||
their response. `smartctl` needs to be run as root.
|
||
|
||
[RFE: add O_RDRW mode for sat/scsi/ata devices](https://www.smartmontools.org/ticket/1064)
|
||
|
||
> According to function `blk_verify_command()` from current kernel sources
|
||
> (see [block/scsi_ioctl.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/scsi_ioctl.c)),
|
||
> O_RDONLY or O_RDWR make no difference if device was opened as root (or with
|
||
> CAP_SYS_RAWIO).
|
||
>
|
||
> The SCSI commands listed in function `blk_set_cmd_filter_defaults()` show
|
||
> that some of the `smartctl -d scsi` functionality might work with O_RDONLY
|
||
> for non-root users. Some more might work with O_RDWR.
|
||
>
|
||
> But `smartctl -d sat` (to access SATA devices) won't work at all because the
|
||
> SCSI commands ATA_12 and ATA_16
|
||
> (see [scsi_proto.h](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/scsi/scsi_proto.h))
|
||
> are **always blocked for non-root users**.
|
||
|
||
## What about my NVMe drive?
|
||
From the smartmontools FAQ: [My NVMe drive is not in the smartctl/smartd database](https://www.smartmontools.org/wiki/FAQ#MyNVMedriveisnotinthesmartctlsmartddatabase)
|
||
> SCSI/SAS and NVMe drives do not provide ATA/SATA-like SMART Attributes.
|
||
> Therefore the drive database does not contain any entries for these drives.
|
||
> This may change in the future as some drives provide similar info via vendor
|
||
> specific commands (see ticket #870).
|
||
|
||
smartmontools also has a [wiki page for NVMe](https://www.smartmontools.org/wiki/NVMe_Support) devices.
|
||
|
||
## How do I report upstream to smartmontools?
|
||
Check their FAQ: [How to create a bug report](https://www.smartmontools.org/wiki/FAQ#Howtocreateabugreport).
|