4 minute read

Telegraf for heavy lifting?

As I'm still thinking how to move ahead with SFC, I've been checking out Telegraf again.

I thought maybe I should just use Telegraf instead, so today I've been revisiting Telegraf again.

This post shows how to use SNMP as a Telegraf input that collects SolidFire metrics, but SolidFir's SNMP output is quite basic, so that approach couldn't work well.

A better - but also time-consuming - approach is to use another input and gater SolidFire metrics from the API.

PoC with PowerShell

For this PoC I used PowerShell to collect volume metrics. I probably shouldn't have (more on that later).

Thanks to a PowerShell bug I ended up collecting metrics indirectly.

That is, instead of using inputs.exec with a PowerShell script that sends output to InfluxDB, I had to run the script independently of Telegraf, spit data out to a place where Telegraf's input.file can fetch it and submit to InfluxDB.

Telegraf configuration file:

[agent]
  interval = "60s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "60s"
  flush_jitter = "5s"
  precision = "60s"
  omit_hostname = true
[[inputs.file]]
  data_format = "influx"
  files = ["/etc/telegraf/telegraf01.out", "/etc/telegraf/telegraf02.out"]
[[outputs.influxdb]]
  urls = ["http://192.168.50.184:32290"]
  database = "solidgraf"
  skip_database_creation = false
  timeout = "5s"
  content_encoding = "gzip"

I know that location shouldn't be /etc/telegraf/, but that's Telegraf's home directory and I ran out of patience. Anyways, this is how my config looks like after volume properties (Get-SFVolume) and stats (Get-SFVolumeStats) are saved to two files in the InfluxDB format.

I run a PowerShell script from crontab and overwrite those two files every minute.

Telegraf agent picks them up and sends them to InfluxDB version 1 (in my case - see output.influxdb above) and that gives me these metrics in the solidgraf DB:

> select * from volumeStats
name: volumeStats
time                 accountID actualIOPS_final averageIOpSize_final host         latencyUSec_final readLatencyUSec_final throttle_final volUtilPct_final volumeId volumeName                               volumeUtil_final writeLatencyUSec_final zeroBlocks_final
----                 --------- ---------------- -------------------- ----         ----------------- --------------------- -------------- ---------------- -------- ----------                               ---------------- ---------------------- ----------------
2024-05-20T07:11:00Z 1         0                1920                 scaleoutsean 0                 0                     0              0                1        data                                     0                0                      486871
2024-05-20T07:11:00Z 1         0                1964                 scaleoutsean 0                 0                     0              0                2        log                                      0                0                      260851
2024-05-20T07:11:00Z 10        0                0                    scaleoutsean 0                 0                     0              0                108      win1101                                  0                0                      1220864
2024-05-20T07:11:00Z 11        0                0                    scaleoutsean 0                 0                     0              0                111      pvc-8d31e43b-f942-4cf8-94db-a08762c745ee 0                0                      524288
2024-05-20T07:11:00Z 11        0                0                    scaleoutsean 0                 0                     0              0                112      pvc-14a51322-16c8-4b95-a7e4-28d9963450b3 0                0                      524288

> select * from volumeProps
name: volumeProps
time                 accountID fifoSize_final host         minFifoSize_final qosPolicyId_final scsiNaaDeviceID                  volumeBlockSize_final volumeId volumeName                               volumeSize_final
----                 --------- -------------- ----         ----------------- ----------------- ---------------                  --------------------- -------- ----------                               ----------------
2024-05-20T07:11:00Z 1         5              scaleoutsean 0                 2                 6f47acc1000000007763776200000001 4096                  1        data                                     2000683008
2024-05-20T07:11:00Z 1         5              scaleoutsean 0                 1                 6f47acc1000000007763776200000002 4096                  2        log                                      1073741824
2024-05-20T07:11:00Z 10        5              scaleoutsean 0                 2                 6f47acc100000000776377620000006c 4096                  108      win1101                                  5000658944
2024-05-20T07:11:00Z 11        5              scaleoutsean 0                 0                 6f47acc100000000776377620000006f 4096                  111      pvc-8d31e43b-f942-4cf8-94db-a08762c745ee 2147483648
2024-05-20T07:11:00Z 11        5              scaleoutsean 0                 0                 6f47acc1000000007763776200000070 4096                  112      pvc-14a51322-16c8-4b95-a7e4-28d9963450b3 2147483648
2024-05-20T07:11:00Z 11        5              scaleoutsean 0                 0                 6f47acc1000000007763776200000071 4096                  113      pvc-4fb5d7f5-4d98-429e-ab36-1b2118b7e55c 2147483648

Known BS in this release:

  • host name of the system where Telegraf agent is running is sent to InfluxDB although agent configuration says not to
  • _final appears in some field names because I mistakenly used the same value for both key and field in the same measurement; avoid that and you won't have this problem

You can find the PowerShell script in the scripts directory of Awesome SolidFire. In the case you wonder why the metrics-related code looks so laborious, it's because it is - it's all manually crafted to minimize code complexity at the expense of coding time. SolidFire.Core is the only external PowerShell module I used.

Also, not all fields from Get-SFVolume and Get-SFVolumeStats responses are sent to InfluxDB - only about 50% of the important stuff is. Feel free to adjust if necessary.

Conclusion

Volume stats and properties are probably two most intensive parts of metrics collection in SFC as these are usually collected most frequently and take up most space which is one of the reasons why the script doesn't send all output to InfluxDB.

I suspect I would have been better off if I used Python, but it's too late now.

Meanwhile I rewrote the script in Python (notes below), which behaves a bit better on Linux. With it I also took a similarly minimalist approach.

After evaluating Telegraf with both Python and PowerShell, I've concluded it's easier to just use Python (or PowerShell) and keep things simple. SolidFire Collector (SFC) is a simple program and it's easier to add a scheduler module and keep everything within one script than use Telegraf for scheduling and InfluxDB communication.

Telegraf is good, but SFC doesn't do much and adding a 250 MB container to do schedule Web POST events is hard to justify for this use case.

Appendix A - Python version

Rewritten for Python, it works slightly less badly. Telegraf is weird. Working around its quirks, I did it like this:

  • Python service that collects data every 60 seconds and outputs to /etc/telegraf/out/telegraf-solidfire.out. Python service runs as another user who's member of the telegraf group
  • /etc/telegraf/out is owned by this user, let's say sean:telegraf, so that he can write to it. /etc/telegraf/out/telegraf-solidfire.out is made accessible world with 755
  • [[inputs.file]] plugin collects /etc/telegraf/out/telegraf-solidfire.out and sends to InfluxDB v1
[[inputs.file]]
  files = ["/etc/telegraf/out/solidfire-telegraf.out"]

Compared to known BS from PowerShell code, there are no such problems in the Python script.

You may also enjoy