From owner-freebsd-geom@freebsd.org Mon Nov 27 13:08:45 2017 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24E2FE4FCD2 for ; Mon, 27 Nov 2017 13:08:45 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (unknown [IPv6:2a02:b90:3002:411::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA97870619 for ; Mon, 27 Nov 2017 13:08:44 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from dilbert.ingresso.co.uk ([2a02:b90:3002:411::6]) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89 (FreeBSD)) (envelope-from ) id 1eJJ9M-0006ch-Km for freebsd-geom@freebsd.org; Mon, 27 Nov 2017 13:08:40 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.89 (FreeBSD)) (envelope-from ) id 1eJJ9M-000FSw-GZ for freebsd-geom@freebsd.org; Mon, 27 Nov 2017 13:08:40 +0000 To: freebsd-geom@freebsd.org Subject: GELI strangeness with gstat Message-Id: From: Pete French Date: Mon, 27 Nov 2017 13:08:40 +0000 X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 13:08:45 -0000 So, I have a set of machines running rul disc encryption with GELI. The output from gstat on an example one looks something like this: 0 27 3 100 0.4 16 100 0.1 1.1| ada0p4 0 27 3 100 0.9 16 100 0.2 1.3| ada0p4.eli I uapgraded a couple of thme to much faster CPUs - the output then started looking like this: 0 146 0 0 0.0 125 604 0.1 5.7| ada0p4 2 146 0 0 0.0 125 604 0.1 104.3| ada0p4.eli ...so the .eli device is now running at 100% despite the underlying disc only being about 6% busy. This was software encryption - my assumption was that the faster COU's were now enabling me to overload the encryption somehow, so I enabled AES-NI on the COU. Now I have hardware encryption. But the output from gstat still looks the same. Whats going on here ? Its very ouzzlking. What is even odder is that these machines are ina HAST pair, and the secondary side looks fine - i.e. only a few percent busy on the disc and the encrypted device. If I sap roles then the efect persists - the HAST primary has a massively busy ELI device. I realise the oprimary will be doing reads as well as writes, but as you can see from the snapshot above, its not singificany compares to the writes, and the effect is also ther when the load is dominated by writes. I am teoprted to think that gstat is being screwy here, but it bothers me not knowing (especially as I am trying to tarck diwn bottlenecks in the system), Anyone got any opinions on what might be showing up here ? -pete.