smartctl_exporter/README.md

208 lines
10 KiB
Markdown
Raw Permalink Normal View History

[![CircleCI](https://circleci.com/gh/prometheus-community/smartctl_exporter.svg?style=svg)](https://circleci.com/gh/prometheus-community/smartctl_exporter)
[![Container Repository on Quay](https://quay.io/repository/prometheuscommunity/smartctl-exporter/status "Container Repository on Quay")](https://quay.io/repository/prometheuscommunity/smartctl-exporter)
2019-08-16 10:05:30 +02:00
2019-08-14 22:28:21 +02:00
# smartctl_exporter
Export smartctl statistics to prometheus
2019-08-14 22:34:49 +02:00
2019-08-17 12:24:28 +02:00
Example output you can show in [EXAMPLE.md](EXAMPLE.md)
2019-08-14 22:34:49 +02:00
## Need more?
**If you need additional metrics - contact me :)**
**Create a feature request, describe the metric that you would like to have and
attach exported from smartctl json file**
2019-08-14 22:34:49 +02:00
2019-08-26 10:27:28 +02:00
# Requirements
`smartmontools` >= 7.0, because export to json [released in 7.0](https://www.smartmontools.org/browser/tags/RELEASE_7_0/smartmontools/NEWS#L11)
2019-08-26 10:27:28 +02:00
2019-08-14 22:34:49 +02:00
# Configuration
## Command line options
The exporter will scan the system for available devices if no `--smartctl.device`
flags are used.
2019-08-14 22:34:49 +02:00
```
usage: smartctl_exporter [<flags>]
Flags:
-h, --help Show context-sensitive help (also try --help-long and --help-man).
--smartctl.path="/usr/sbin/smartctl"
The path to the smartctl binary
--smartctl.interval=60s The interval between smartctl polls
--smartctl.rescan=10m The interval between rescanning for new/disappeared devices. If the interval is smaller than 1s no
rescanning takes place. If any devices are configured with smartctl.device also no rescanning takes
place.
--smartctl.device=SMARTCTL.DEVICE ...
The device to monitor (repeatable)
--smartctl.device-exclude=""
Regexp of devices to exclude from automatic scanning. (mutually exclusive to
device-include)
--smartctl.device-include=""
Regexp of devices to include in automatic scanning. (mutually exclusive to
device-exclude)
--web.telemetry-path="/metrics"
Path under which to expose metrics
--web.systemd-socket Use systemd socket activation listeners instead of port listeners (Linux only).
--web.listen-address=:9633 ...
Addresses on which to expose metrics and web interface. Repeatable for multiple
addresses.
--web.config.file="" [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.
--log.level=info Only log messages with the given severity or above. One of: [debug, info, warn,
error]
--log.format=logfmt Output format of log messages. One of: [logfmt, json]
--version Show application version.
2019-08-14 22:34:49 +02:00
```
## TLS and basic authentication
This exporter supports TLS and basic authentication.
To use TLS and/or basic authentication, you need to pass a configuration file
using the `--web.config.file` parameter. The format of the file is described
[in the exporter-toolkit repository](https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md).
## Example of running in Docker
Minimal functional `docker-compose.yml`:
```yaml
version: "3"
services:
smartctl-exporter:
image: prometheuscommunity/smartctl-exporter
privileged: true
user: root
ports:
- "9633:9633"
```
# Troubleshooting
## Troubleshooting data inconsistencies
`smartmon_exporter` uses the JSON output from `smartctl` to provide the data to
Prometheus. If the data is incorrect, look at the data from `smartctl` to
determine if the issue should be reported upstream to smartmontools or to this
repo. In general, the `smartctl_exporter` should not modify the data in flight.
If the data is missing from `smartctl`, it should not be in `smartctl_exporter`.
If the data from `smartctl` is incorrect, it should be reported upstream.
Requests for `smartctl_exporter` to "fix" incorrect data where `smartctl` is
reporting incorrect data will be closed. The grey area is when invalid or
missing data from smartctl is causing multiple invalid or incorrect data
in `smartctl_exporter`. This could happen if the data is used in a calculation
for other data. This will need to be researched on a case by case basis.
| - | smartctl valid | smartctl missing | smartctl invalid/incorrect |
|---------------------------|-----------------------------|-------------------------------------------------|----------------------------------|
| smartctl_exporter valid | all good | N/A | N/A |
| smartctl_exporter missing | issue for smartctl_exporter | report upstream to smartmontools | report upstream to smartmontools |
| smartctl_exporter invalid | issue for smartctl_exporter | issue for smartctl_exporter and report upstream | report upstream to smartmontools |
### smartctl output vs smartctl_exporter output
The S.M.A.R.T. attributes are mapped in
[smartctl.go](https://github.com/prometheus-community/smartctl_exporter/blob/master/smartctl.go).
Each function has a `prometheus.MustNewConstMetric` or similar function with the
first parameter being the metric name. Find the metric name in
[metrics.go](https://github.com/prometheus-community/smartctl_exporter/blob/master/metrics.go)
to see how the exporter displays the information. This may sound technical, but
it's crucial for understanding how data flows from `smartctl` to
`smartctl_exporter` to Prometheus.
If the data looks incorrect, check the
[Smartmontools Frequently Asked Questions (FAQ)](https://www.smartmontools.org/wiki/FAQ).
It's likely your question may already have an answer. If you still have
questions, open an [issue]().
## Gathering smartctl data
Follow these steps to gather smartctl data for troubleshooting purposes. If you
have unique drives/data/edge cases and would like to "donate" the data, open a
PR with the redacted JSON files.
1. Run `scripts/collect-smartctl-json.sh` to export all drives to a
`smartctl-data` directory (created in the current directory).
2. Run `scripts/redact_fake_json.py` to redact sensitive data.
3. Provide the JSON file for the drive in question.
```bash
cd scripts
./collect-smartctl-json.sh
./redact-fake-json.py smartctl-data/*.json
```
## Run smartctl_exporter using JSON data
The `smartctl_exporter` can be run using local JSON data. The device names are
pulled from actual devices in the machine while the data is redirected to the
`debug` directory. Save the JSON data in the `debug` directory using the actual
device names using a 1:1 ratio. If you have 3 devices, `sda`, `sdb` and `sdc`,
the `smartctl_exporter` will expect 3 files: `debug/sda.json`, `debug/sdb.json`
and `debug/sdc.json`.
Once the "fake devices" (JSON files) are in place, run the exporter passing the
hidden `--smartctl.fake-data` switch on the command line. The port is specified
to prevent conflicts with an existing `smartctl_exporter` on the default port.
```bash
smartctl_exporter --web.listen-address 127.0.0.1:19633 --smartctl.fake-data
```
# FAQ
## How do I run `smartctl_exporter` against a JSON file?
If you're helping someone else, request the output of the `smartctl` command
above. Feed this into the `smartctl_exporter` using the
hidden `--smartctl.fake-data` flag. If a `smartctl_exporter` is already running,
use a different port; in this case, it's `19633`. Run `collect_fake_json.sh`
first to collect the JSON files for **your** devices. Copy the requested JSON
file into one of the fake files. After starting the exporter, you can query it
to see the data generated.
```bash
# Dump the JSON files for your devices into debug/
./collect_fake_json.sh
# copy the test JSON into one of the files in debug/
cp extracted-from-above-sda.json debug/sda.json
# Make sure you have the latest version
go build
# Use a different port in case smartctl_exporter is already running
sudo ./smartctl_exporter --web.listen-address=127.0.0.1:19633 --log.level=debug --smartctl.fake-data
# Use curl with grep
curl --silent 127.0.0.1:19633/metrics | grep -i nvme
# Or xh with ripgrep
xh --body :19633/metrics | rg nvme
```
## Why is root required? Can't I add a user to the "disk" group?
A blogger had the same question and opened a ticket on smartmontools. This is
their response. `smartctl` needs to be run as root.
[RFE: add O_RDRW mode for sat/scsi/ata devices](https://www.smartmontools.org/ticket/1064)
> According to function `blk_verify_command()` from current kernel sources
> (see [block/scsi_ioctl.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/scsi_ioctl.c)),
> O_RDONLY or O_RDWR make no difference if device was opened as root (or with
> CAP_SYS_RAWIO).
>
> The SCSI commands listed in function `blk_set_cmd_filter_defaults()` show
> that some of the `smartctl -d scsi` functionality might work with O_RDONLY
> for non-root users. Some more might work with O_RDWR.
>
> But `smartctl -d sat` (to access SATA devices) won't work at all because the
> SCSI commands ATA_12 and ATA_16
> (see [scsi_proto.h](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/scsi/scsi_proto.h))
> are **always blocked for non-root users**.
## What about my NVMe drive?
From the smartmontools FAQ: [My NVMe drive is not in the smartctl/smartd database](https://www.smartmontools.org/wiki/FAQ#MyNVMedriveisnotinthesmartctlsmartddatabase)
> SCSI/SAS and NVMe drives do not provide ATA/SATA-like SMART Attributes.
> Therefore the drive database does not contain any entries for these drives.
> This may change in the future as some drives provide similar info via vendor
> specific commands (see ticket #870).
smartmontools also has a [wiki page for NVMe](https://www.smartmontools.org/wiki/NVMe_Support) devices.
## How do I report upstream to smartmontools?
Check their FAQ: [How to create a bug report](https://www.smartmontools.org/wiki/FAQ#Howtocreateabugreport).