Date: Mon, 12 Dec 2016 06:16:26 -0800 From: Larry Maloney <larry.maloney@hackerdojo.com> To: Ed Schouten <ed@nuxi.nl> Cc: hackers@freebsd.org Subject: Re: Sysctl as a Service, or: making sysctl(3) more friendly for monitoring systems Message-ID: <7E6EAE02-F059-4C88-8D43-21F0811BC07B@hackerdojo.com> In-Reply-To: <CABh_MKk87hJTsu1ETX8Ffq9E8gqRPELeSEKzf1jKk_wwUROgAw@mail.gmail.com> References: <CABh_MKk87hJTsu1ETX8Ffq9E8gqRPELeSEKzf1jKk_wwUROgAw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Cool! I like it Sent from my iPhone > On Dec 11, 2016, at 11:35 AM, Ed Schouten <ed@nuxi.nl> wrote: >=20 > Hi there, >=20 > The last couple of months I've been playing around with a monitoring > system called Prometheus (https://prometheus.io/). In short, > Prometheus works like this: >=20 > ----- If you already know Prometheus, skip this ----- >=20 > 1. For the thing you want to monitor, you either integrate the > Prometheus client library into your codebase or you write a separate > exporter process. The client library or the exporter process then > exposes key metrics of your application over HTTP. Simplified example: >=20 > $ curl http://localhost:12345/metrics > # HELP open_file_descriptors The number of files opened by the process > # TYPE open_file_descriptors gauge > open_file_descriptors 12 > # HELP http_requests The number of HTTP requests received. > # TYPE http_requests counter > http_requests{result=3D"2xx"} 100 > http_requests{result=3D"4xx"} 14 > http_requests{result=3D"5xx"} 0 >=20 > 2. You fire op Prometheus and configure it to scrape and store all of > the things you want to monitor. Prometheus can then add more labels to > the metrics it scrapes. So the example above may get transformed by > Prometheus to look like this: >=20 > open_file_descriptors{job=3D"nginx",instance=3D"web1.mycompany.com"} 12 > http_requests{job=3D"nginx",instance=3D"web1.mycompany.com",result=3D"2xx"= } 100 > http_requests{job=3D"nginx",instance=3D"web1.mycompany.com",result=3D"4xx"= } 14 > http_requests{job=3D"nginx",instance=3D"web1.mycompany.com",result=3D"5xx"= } 0 >=20 > Fun fact: Prometheus can also scrape Prometheus, so if you operate > multiple datacenters, you can let a global instance scrape a per-DC > instance and add a dc=3D"..." label to all metrics. >=20 > 3. After scraping data for some time, you can do fancy queries like these:= >=20 > - Compute the 5-minute rate of HTTP requests per server and per HTTP error= code: > rate(http_requests[5m]) >=20 > - Compute the 5-minute rate of all HTTP requests on the entire cluster: > sum(rate(http_requests[5m])) >=20 > - Same as the above, but aggregate by HTTP error code: > sum(rate(http_requests[5m])) by (result) >=20 > Prometheus can do alerting as well by using these expressions as matchers.= >=20 > 4. Set up Grafana and voila: you can create fancy dashboards! >=20 > ----- If you skipped the introduction, start reading here ----- >=20 > The Prometheus folks have developed a tool called the node_exporter > (https://github.com/prometheus/node_exporter). Basically it extracts a > whole bunch of interesting system-related metrics (disk usage, network > I/O, etc) through sysctl(3), invoking ioctl(2), parsing /proc files, > etc. and exposes that information using Prometheus' syntax. >=20 > The other day I was thinking: in a certain way, the node exporter is a > bit of a redundant tool on the BSDs. Instead of needing to write > custom collectors for every kernel subsystem, we could write a generic > exporter for converting the entire sysctl(3) tree to Prometheus > metrics, which is exactly what I'm experimenting with here: >=20 > https://github.com/EdSchouten/prometheus_sysctl_exporter >=20 > An example of what this tool's output looks like: >=20 > $ ./prometheus_sysctl_exporter > ... > # HELP kern_maxfiles Maximum number of files > sysctl_kern_maxfiles 1043382 > # HELP kern_openfiles System-wide number of open files > sysctl_kern_openfiles 316 > ... >=20 > You could use this to write alerting rules like this: >=20 > ALERT FileDescriptorUsageHigh > IF sysctl_kern_openfiles / sysctl_kern_maxfiles > 0.5 > FOR 15m > ANNOTATIONS { > description =3D "More than half of all FDs are in use!", > } >=20 > There you go. Access to a very large number of metrics without too much ef= fort. >=20 > My main question here is: are there any people in here interested in > seeing something like this being developed into something usable? If > so, let me know and I'll pursue this further. >=20 > I also have a couple of technical questions related to sysctl(3)'s > in-kernel design: >=20 > - Prometheus differentiates between gauges (memory usage), counters > (number of HTTP requests), histograms (per-RPC latency stats), etc., > while sysctl(3) does not. It would be nice if we could have that info > on a per-sysctl basis. Mind if I add a CTLFLAG_GAUGE, CTLFLAG_COUNTER, > etc? >=20 > - Semantically sysctl(3) and Prometheus are slightly different. > Consider this sysctl: >=20 > hw.acpi.thermal.tz0.temperature: 27.8C >=20 > My tool currently converts this metric's name to > sysctl_hw_acpi_thermal_tz0_temperature. This is suboptimal, as it > would ideally be called > sysctl_hw_acpi_thermal_temperature{sensor=3D"tz0"}. Otherwise you > wouldn't be able to write generic alerting rules, use aggregation in > queries, etc. >=20 > I was thinking: we could quite easily do such a translation by > attaching labels to SYSCTL_NODE objects. As in, the hw.acpi.thermal > node would get a label "sensor". Any OID placed underneath this node > will not become a midfix of the sysctl name, but the value of that > label instead. Thoughts? >=20 > A final remark I want to make: a concern might be that changes like > these would not be generic, but only apply to Prometheus. I tend to > disagree. First of all, an advantage of Prometheus is that the > coupling is very loose: it's just a GET request with key-value pairs. > Anyone is free to add his/her own implementation. >=20 > Second, emaste@ also pointed me to another monitoring framework being > developed by Intel right now: >=20 > https://github.com/intelsdi-x/snap >=20 > The changes I'm proposing would also seem to make exporting sysctl > data to that system easier. >=20 > Anyway, thanks for reading this huge wall of text. >=20 > Best regards, > --=20 > Ed Schouten <ed@nuxi.nl> > Nuxi, 's-Hertogenbosch, the Netherlands > KvK-nr.: 62051717 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7E6EAE02-F059-4C88-8D43-21F0811BC07B>