From owner-freebsd-stable@freebsd.org  Tue May  8 09:00:44 2018
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 827FAFAE3CB
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Tue,  8 May 2018 09:00:44 +0000 (UTC)
 (envelope-from SRS0=6gX9=H3=quip.cz=000.fbsd@elsa.codelab.cz)
Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 25207750F2
 for <freebsd-stable@freebsd.org>; Tue,  8 May 2018 09:00:43 +0000 (UTC)
 (envelope-from SRS0=6gX9=H3=quip.cz=000.fbsd@elsa.codelab.cz)
Received: from elsa.codelab.cz (localhost [127.0.0.1])
 by elsa.codelab.cz (Postfix) with ESMTP id 13CAE28416;
 Tue,  8 May 2018 11:00:36 +0200 (CEST)
Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz
 [86.49.16.209])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (No client certificate requested)
 by elsa.codelab.cz (Postfix) with ESMTPSA id 25A3128412;
 Tue,  8 May 2018 11:00:35 +0200 (CEST)
Subject: Re: Notification for gmirror failures
To: Andrea Brancatelli <abrancatelli@schema31.it>,
 freebsd-stable <freebsd-stable@freebsd.org>
References: <e792884d579cd1f2c8d05ff4002ad1a3@schema31.it>
From: Miroslav Lachman <000.fbsd@quip.cz>
Message-ID: <f271c8ef-36a7-0a93-9e0b-4bf01ac5552a@quip.cz>
Date: Tue, 8 May 2018 11:00:34 +0200
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101
 Firefox/52.0 SeaMonkey/2.49.1
MIME-Version: 1.0
In-Reply-To: <e792884d579cd1f2c8d05ff4002ad1a3@schema31.it>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.25
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 08 May 2018 09:00:44 -0000

Andrea Brancatelli wrote on 2018/05/08 09:28:
> Hi,
> 
> out of curiosity, does any kind of GMirror-failure notification tools
> exist?
> 
> Either native... SNMP... external ports... whatever?

I don't know any daemon like solution so I wrote some simple shell 
script. Its used on all our machines. It should be run be cron in some 
sane interval (like each 5 or 10 minutes)

#!/bin/sh

## $Id: check_gmirror.sh 8c1f418ba7f5 2017-06-15 12:36 +0200 lachman $

export PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"

mailto="you@example.com"
subject_pref="$(hostname) gmirror"
notify_state="/var/run/${0##*/}.notified"

cmd_out=$(gmirror status -s)

components="$(echo "$cmd_out" | grep -F -v COMPLETE)"

if [ -n "$components" ]; then
     if [ ! -f "$notify_state" ]; then
         subject="$subject_pref FAILED - $(date '+%Y-%m-%d %H:%M:%S')"
         echo "$cmd_out" | sed -E 's/(SYNCHRONIZING,) [0-9]+%/\1 x%/' > 
"$notify_state"
         echo "$cmd_out" | mail -s "$subject" "$mailto"
         echo "$subject" | logger
     else
         subject="$subject_pref FAILED - $(date '+%Y-%m-%d %H:%M:%S')"
         echo "$cmd_out" | sed -E 's/(SYNCHRONIZING,) [0-9]+%/\1 x%/' > 
"$notify_state.now"
         diff -u -b -B "$notify_state" "$notify_state.now" > 
"$notify_state.diff"

         if [ -s "$notify_state.diff" ]; then    ## if diff is not empty
             cat "$notify_state.diff" | mail -s "$subject" "$mailto"
             echo "$subject additional failure occurred" | logger
             cp "$notify_state.now" "$notify_state"
         fi
         rm "$notify_state.now" "$notify_state.diff"
     fi
else
     if [ -f "$notify_state" ]; then
         rm "$notify_state"
         subject="$subject_pref now OK - $(date '+%Y-%m-%d %H:%M:%S')"
         echo "$cmd_out" | mail -s "$subject" "$mailto"
         echo "$subject" | logger
     fi
fi


We have similar script for checking ZFS too.

Miroslav Lachman